## 1. Project Overview & Quickstart (pipecat-ai/pipecat)
## File: README.md
[](https://pypi.org/project/pipecat-ai) [](https://codecov.io/gh/pipecat-ai/pipecat) [](https://docs.pipecat.ai) [](https://discord.gg/pipecat) [](https://deepwiki.com/pipecat-ai/pipecat)
# ποΈ Pipecat: Real-Time Voice & Multimodal AI Agents
**Pipecat** is an open-source Python framework for building real-time voice and multimodal conversational agents. Build a single voice agent or a full multi-agent system where specialists hand off, fan out in parallel, and coordinate over a shared bus, locally or distributed across processes and machines. Orchestrate audio and video, AI services, transports, and conversation pipelines effortlessly, so you can focus on what makes your agents unique.
> Want to dive right in? Run `pipecat init quickstart` or follow the [quickstart guide](https://docs.pipecat.ai/getting-started/quickstart).
## π What you can build
- **Voice Assistants** β natural, streaming conversations with AI
- **Multi-Agent Systems** β specialists that hand off, fan out in parallel, or run as sidecars over a shared bus
- **AI Companions** β coaches, meeting assistants, characters
- **Multimodal Interfaces** β voice, video, images, and more
- **Interactive Storytelling** β creative tools with generative media
- **Business Agents** β customer intake, support bots, guided flows
- **Complex Dialog Systems** β design logic with structured conversations
## π§ Why Pipecat?
- **Voice-first**: Integrates speech recognition, text-to-speech, and conversation handling
- **Pluggable**: Supports many AI services and tools
- **Composable Pipelines**: Build complex behavior from modular components
- **Multi-Agent Ready**: Each pipeline is an agent. Compose them with handoff, parallel fan-out, sidecar workers, or distributed deployments
- **Real-Time**: Ultra-low latency interaction with different transports (e.g. WebSockets or WebRTC)
## π Pipecat ecosystem
### π± Client SDKs
Building client applications? You can connect to Pipecat from any platform using our official SDKs:
[JavaScript](https://docs.pipecat.ai/client/js/introduction) | [React](https://docs.pipecat.ai/client/react/introduction) | [React Native](https://docs.pipecat.ai/client/react-native/introduction) |
[Swift](https://docs.pipecat.ai/client/ios/introduction) | [Kotlin](https://docs.pipecat.ai/client/android/introduction) | [C++](https://docs.pipecat.ai/client/c++/introduction) | [ESP32](https://github.com/pipecat-ai/pipecat-esp32)
### π§ Structured conversations
Need predefined or dynamic conversation paths with state management? [Pipecat Flows](https://docs.pipecat.ai/guides/features/pipecat-flows) is built into Pipecat. Browse the [examples](https://github.com/pipecat-ai/pipecat/tree/main/examples/flows) to see it in action.
### πͺ Beautiful UIs
Want to build beautiful and engaging experiences? Checkout the [Voice UI Kit](https://github.com/pipecat-ai/voice-ui-kit), a collection of components, hooks and templates for building voice AI applications quickly.
### π οΈ Create and deploy projects
The [Pipecat CLI](https://docs.pipecat.ai/api-reference/cli/overview) ships with `pipecat-ai` β install it with `uv tool install "pipecat-ai[cli]"`. Run `pipecat init` to start a project: it sets you up so an AI coding assistant (Claude Code, Codex) builds it for you, and can scaffold a runnable bot in under a minute. Then use the CLI to monitor and deploy your agent to production.
### π Debugging
Looking for help debugging your pipeline and processors? Check out [Whisker](https://github.com/pipecat-ai/whisker), a real-time Pipecat debugger.
### π₯οΈ Terminal
Love terminal applications? Check out [Tail](https://github.com/pipecat-ai/tail), a terminal dashboard for Pipecat.
### π€ Claude Code skills
Use [Pipecat Skills](https://github.com/pipecat-ai/skills) with [Claude Code](https://claude.ai/code) to scaffold projects, deploy to Pipecat Cloud, and more. Install the marketplace with:
```
claude plugin marketplace add pipecat-ai/skills
```
and install any of the available plugins.
### 𧩠Community integrations
Build and share your own Pipecat service integrations! Browse existing [community integrations](https://docs.pipecat.ai/api-reference/server/services/supported-services) or check out our [guide](COMMUNITY_INTEGRATIONS.md) to create your own.
### πΊοΈ Pipecat TV channel
Catch new features, interviews, and how-tos on our [Pipecat TV](https://www.youtube.com/playlist?list=PLzU2zoMTQIHjqC3v4q2XVSR3hGSzwKFwH) channel.
## π¬ See it in action
[](https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot)
[](https://github.com/pipecat-ai/pipecat-examples/tree/main/storytelling-chatbot)
[](https://github.com/pipecat-ai/pipecat-examples/tree/main/daily-multi-translation)
[](https://github.com/pipecat-ai/pipecat/blob/main/examples/vision/vision-moondream.py)
## 𧩠Available services
| Category | Services |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Speech-to-Text | [AssemblyAI](https://docs.pipecat.ai/api-reference/server/services/stt/assemblyai), [AWS](https://docs.pipecat.ai/api-reference/server/services/stt/aws), [Azure](https://docs.pipecat.ai/api-reference/server/services/stt/azure), [Cartesia](https://docs.pipecat.ai/api-reference/server/services/stt/cartesia), [Deepgram](https://docs.pipecat.ai/api-reference/server/services/stt/deepgram), [ElevenLabs](https://docs.pipecat.ai/api-reference/server/services/stt/elevenlabs), [Fal Wizper](https://docs.pipecat.ai/api-reference/server/services/stt/fal), [FunASR](https://docs.pipecat.ai/api-reference/server/services/stt/funasr), [Gladia](https://docs.pipecat.ai/api-reference/server/services/stt/gladia), [Google](https://docs.pipecat.ai/api-reference/server/services/stt/google), [Gradium](https://docs.pipecat.ai/api-reference/server/services/stt/gradium), [Groq (Whisper)](https://docs.pipecat.ai/api-reference/server/services/stt/groq), [Mistral](https://docs.pipecat.ai/api-reference/server/services/stt/mistral), [Moonshine](https://docs.pipecat.ai/api-reference/server/services/stt/moonshine), [NVIDIA](https://docs.pipecat.ai/api-reference/server/services/stt/nvidia), [OpenAI (Whisper)](https://docs.pipecat.ai/api-reference/server/services/stt/openai), [Sarvam](https://docs.pipecat.ai/api-reference/server/services/stt/sarvam), [Soniox](https://docs.pipecat.ai/api-reference/server/services/stt/soniox), [Speechmatics](https://docs.pipecat.ai/api-reference/server/services/stt/speechmatics), [Together](https://docs.pipecat.ai/api-reference/server/services/stt/together), [Whisper](https://docs.pipecat.ai/api-reference/server/services/stt/whisper), [xAI](https://docs.pipecat.ai/api-reference/server/services/stt/xai) |
| LLMs | [Anthropic](https://docs.pipecat.ai/api-reference/server/services/llm/anthropic), [AWS](https://docs.pipecat.ai/api-reference/server/services/llm/aws), [Azure](https://docs.pipecat.ai/api-reference/server/services/llm/azure), [Baseten](https://docs.pipecat.ai/api-reference/server/services/llm/baseten), [Cerebras](https://docs.pipecat.ai/api-reference/server/services/llm/cerebras), [Crusoe](https://docs.pipecat.ai/api-reference/server/services/llm/crusoe), [DeepSeek](https://docs.pipecat.ai/api-reference/server/services/llm/deepseek), [Fireworks AI](https://docs.pipecat.ai/api-reference/server/services/llm/fireworks), [Gemini](https://docs.pipecat.ai/api-reference/server/services/llm/gemini), [Grok](https://docs.pipecat.ai/api-reference/server/services/llm/grok), [Groq](https://docs.pipecat.ai/api-reference/server/services/llm/groq), [Inception](https://docs.pipecat.ai/api-reference/server/services/llm/inception), [Mistral](https://docs.pipecat.ai/api-reference/server/services/llm/mistral), [Nebius](https://docs.pipecat.ai/api-reference/server/services/llm/nebius), [Novita](https://docs.pipecat.ai/api-reference/server/services/llm/novita), [NVIDIA NIM](https://docs.pipecat.ai/api-reference/server/services/llm/nvidia), [Ollama](https://docs.pipecat.ai/api-reference/server/services/llm/ollama), [OpenAI](https://docs.pipecat.ai/api-reference/server/services/llm/openai), [OpenAI Responses](https://docs.pipecat.ai/api-reference/server/services/llm/openai-responses), [OpenRouter](https://docs.pipecat.ai/api-reference/server/services/llm/openrouter), [Perplexity](https://docs.pipecat.ai/api-reference/server/services/llm/perplexity), [Qwen](https://docs.pipecat.ai/api-reference/server/services/llm/qwen), [SambaNova](https://docs.pipecat.ai/api-reference/server/services/llm/sambanova), [Sarvam](https://docs.pipecat.ai/api-reference/server/services/llm/sarvam), [Together AI](https://docs.pipecat.ai/api-reference/server/services/llm/together) |
| Text-to-Speech | [Async](https://docs.pipecat.ai/api-reference/server/services/tts/asyncai), [AWS](https://docs.pipecat.ai/api-reference/server/services/tts/aws), [Azure](https://docs.pipecat.ai/api-reference/server/services/tts/azure), [Bland](https://docs.pipecat.ai/api-reference/server/services/tts/bland), [Camb AI](https://docs.pipecat.ai/api-reference/server/services/tts/camb), [Cartesia](https://docs.pipecat.ai/api-reference/server/services/tts/cartesia), [Deepgram](https://docs.pipecat.ai/api-reference/server/services/tts/deepgram), [ElevenLabs](https://docs.pipecat.ai/api-reference/server/services/tts/elevenlabs), [Fish](https://docs.pipecat.ai/api-reference/server/services/tts/fish), [Google](https://docs.pipecat.ai/api-reference/server/services/tts/google), [Gradium](https://docs.pipecat.ai/api-reference/server/services/tts/gradium), [Groq](https://docs.pipecat.ai/api-reference/server/services/tts/groq), [Hume](https://docs.pipecat.ai/api-reference/server/services/tts/hume), [Inworld](https://docs.pipecat.ai/api-reference/server/services/tts/inworld), [Kokoro](https://docs.pipecat.ai/api-reference/server/services/tts/kokoro), [LMNT](https://docs.pipecat.ai/api-reference/server/services/tts/lmnt), [MiniMax](https://docs.pipecat.ai/api-reference/server/services/tts/minimax), [Mistral](https://docs.pipecat.ai/api-reference/server/services/tts/mistral), [Neuphonic](https://docs.pipecat.ai/api-reference/server/services/tts/neuphonic), [NVIDIA](https://docs.pipecat.ai/api-reference/server/services/tts/nvidia), [OpenAI](https://docs.pipecat.ai/api-reference/server/services/tts/openai), [Piper](https://docs.pipecat.ai/api-reference/server/services/tts/piper), [Resemble](https://docs.pipecat.ai/api-reference/server/services/tts/resemble), [Rime](https://docs.pipecat.ai/api-reference/server/services/tts/rime), [Sarvam](https://docs.pipecat.ai/api-reference/server/services/tts/sarvam), [Smallest](https://docs.pipecat.ai/api-reference/server/services/tts/smallest), [Soniox](https://docs.pipecat.ai/api-reference/server/services/tts/soniox), [Speechify](https://docs.pipecat.ai/api-reference/server/services/tts/speechify), [Speechmatics](https://docs.pipecat.ai/api-reference/server/services/tts/speechmatics), [Together](https://docs.pipecat.ai/api-reference/server/services/tts/together), [xAI](https://docs.pipecat.ai/api-reference/server/services/tts/xai), [XTTS](https://docs.pipecat.ai/api-reference/server/services/tts/xtts) |
| Speech-to-Speech | [AWS Nova Sonic](https://docs.pipecat.ai/api-reference/server/services/s2s/aws), [Gemini Multimodal Live](https://docs.pipecat.ai/api-reference/server/services/s2s/gemini), [Grok Voice Agent](https://docs.pipecat.ai/api-reference/server/services/s2s/grok), [OpenAI Realtime](https://docs.pipecat.ai/api-reference/server/services/s2s/openai), [Ultravox](https://docs.pipecat.ai/api-reference/server/services/s2s/ultravox), |
| Transport | [Daily (WebRTC)](https://docs.pipecat.ai/api-reference/server/services/transport/daily), [FastAPI Websocket](https://docs.pipecat.ai/api-reference/server/services/transport/fastapi-websocket), [LiveKit (WebRTC)](https://docs.pipecat.ai/api-reference/server/services/transport/livekit), [SmallWebRTCTransport](https://docs.pipecat.ai/api-reference/server/services/transport/small-webrtc), [Vonage (WebRTC)](https://docs.pipecat.ai/api-reference/server/services/transport/vonage), [WebSocket Server](https://docs.pipecat.ai/api-reference/server/services/transport/websocket-server), [WhatsApp](https://docs.pipecat.ai/api-reference/server/services/transport/whatsapp), Local |
| Serializers | [Exotel](https://docs.pipecat.ai/api-reference/server/services/serializers/exotel), [Genesys](https://docs.pipecat.ai/api-reference/server/services/serializers/genesys), [Plivo](https://docs.pipecat.ai/api-reference/server/services/serializers/plivo), [Twilio](https://docs.pipecat.ai/api-reference/server/services/serializers/twilio), [Telnyx](https://docs.pipecat.ai/api-reference/server/services/serializers/telnyx), [Vonage](https://docs.pipecat.ai/api-reference/server/services/serializers/vonage) |
| Video | [HeyGen](https://docs.pipecat.ai/api-reference/server/services/video/heygen), [LemonSlice](https://docs.pipecat.ai/api-reference/server/services/transport/lemonslice), [Tavus](https://docs.pipecat.ai/api-reference/server/services/video/tavus), [Simli](https://docs.pipecat.ai/api-reference/server/services/video/simli) |
| Memory | [mem0](https://docs.pipecat.ai/api-reference/server/services/memory/mem0) |
| Vision & Image | [fal](https://docs.pipecat.ai/api-reference/server/services/image-generation/fal), [Google Imagen](https://docs.pipecat.ai/api-reference/server/services/image-generation/google-imagen), [Moondream](https://docs.pipecat.ai/api-reference/server/services/vision/moondream) |
| Audio Processing | [Silero VAD](https://docs.pipecat.ai/api-reference/server/utilities/audio/silero-vad-analyzer), [Krisp Viva](https://docs.pipecat.ai/guides/features/krisp-viva), [Koala](https://docs.pipecat.ai/api-reference/server/utilities/audio/koala-filter), [ai-coustics](https://docs.pipecat.ai/api-reference/server/utilities/audio/aic-filter), [RNNoise](https://docs.pipecat.ai/api-reference/server/utilities/audio/rnnoise-filter) |
| Analytics & Metrics | [OpenTelemetry](https://docs.pipecat.ai/api-reference/server/utilities/opentelemetry), [Sentry](https://docs.pipecat.ai/api-reference/server/services/analytics/sentry) |
| Community | [Browse community integrations β](https://docs.pipecat.ai/api-reference/server/services/supported-services) |
π [View full services documentation β](https://docs.pipecat.ai/api-reference/server/services/supported-services)
## β‘ Getting started
Run Pipecat on your local machine, then move your agent processes to the cloud when you're ready. Either way, you'll need [uv](https://docs.astral.sh/uv/getting-started/installation/) β install it first:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```
### Start a new project with the CLI
The quickest path: install the Pipecat CLI and scaffold a new phone or web/mobile bot interactively.
```bash
uv tool install "pipecat-ai[cli]"
pipecat init
```
Follow the [quickstart guide](https://docs.pipecat.ai/getting-started/quickstart) to get your very first bot running, or dive a little deeper into [bootstrapping a project](https://docs.pipecat.ai/pipecat/get-started/build-your-next-bot).
### Manual installation
Prefer to wire things up yourself?
1. Install the module
```bash
# For new projects
uv init my-pipecat-app
cd my-pipecat-app
uv add pipecat-ai
# Or for existing projects
uv add pipecat-ai
```
2. Set up your environment
```bash
cp env.example .env
```
3. To keep things lightweight, only the core framework is included by default. If you need support for third-party AI services, you can add the necessary dependencies with:
```bash
uv add "pipecat-ai[option,...]"
```
> **Using pip?** You can still use `pip install pipecat-ai` and `pip install "pipecat-ai[option,...]"` to get set up.
From here, the code examples below are the best way to learn β agents you can run, read, and adapt.
## π§ͺ Code examples
- [Focused examples](https://github.com/pipecat-ai/pipecat/tree/main/examples) β small agents that each illustrate one or two specific services or concepts
- [Example apps](https://github.com/pipecat-ai/pipecat-examples) β complete applications you can use as starting points for development
## π οΈ Developing Pipecat
### Prerequisites
**Minimum Python Version:** 3.11
**Recommended Python Version:** >= 3.12
### Setup steps
1. Clone the repository and navigate to it:
```bash
git clone https://github.com/pipecat-ai/pipecat.git
cd pipecat
```
2. Install development and testing dependencies:
```bash
uv sync --group dev --all-extras \
--no-extra gstreamer \
--no-extra local \
```
3. Install the git pre-commit hooks:
```bash
uv run pre-commit install
```
> **Note**: Some extras (local, gstreamer) require system dependencies. See documentation if you encounter build errors.
### Claude Code skills
Install development workflow skills for contributing to Pipecat with [Claude Code](https://claude.ai/code):
```
claude plugin marketplace add pipecat-ai/pipecat
claude plugin install pipecat-dev@pipecat-dev-skills
```
### Running tests
To run all tests, from the root directory:
```bash
uv run pytest
```
Run a specific test suite:
```bash
uv run pytest tests/test_name.py
```
## π€ Contributing
We welcome contributions from the community! Whether you're fixing bugs, improving documentation, or adding new features, here's how you can help:
- **Found a bug?** Open an [issue](https://github.com/pipecat-ai/pipecat/issues)
- **Have a feature idea?** Start a [discussion](https://discord.gg/pipecat)
- **Want to contribute code?** Check our [CONTRIBUTING.md](CONTRIBUTING.md) guide
- **Documentation improvements?** [Docs](https://github.com/pipecat-ai/docs) PRs are always welcome
Before submitting a pull request, please check existing issues and PRs to avoid duplicates.
We aim to review all contributions promptly and provide constructive feedback to help get your changes merged.
## π Getting help
β‘οΈ [Join our Discord](https://discord.gg/pipecat)
β‘οΈ [Read the docs](https://docs.pipecat.ai)
β‘οΈ [Reach us on X](https://x.com/pipecat_ai)
β‘οΈ [Enterprise support](mailto:help@daily.co) for consulting and support agreements
---
## File: docs/api/index.rst
Pipecat API Reference
=====================
Welcome to the Pipecat API reference.
Use the navigation on the left to browse modules, or search using the search box.
**New to Pipecat?** Check out the `main documentation `_ for tutorials, guides, and client SDK information.
Quick Links
-----------
* `GitHub Repository `_
* `Join our Community `_
.. toctree::
:maxdepth: 2
:caption: API Reference
:hidden:
Adapters
Audio
Clocks
Extensions
Flows
Frames
Metrics
Observers
Pipeline
Processors
Runner
Serializers
Services
Transcriptions
Transports
Turns
Utils
---
## File: docs/api/README.md
# Pipecat API Documentation
This directory contains the source files for auto-generating Pipecat's API reference documentation.
## Building Documentation
From this directory:
```bash
# Build docs (warnings shown but don't fail the build)
cd docs/api && uv run ./build-docs.sh
# Build with strict mode (warnings treated as errors)
cd docs/api && uv run ./build-docs.sh --strict
```
The build script will:
1. Install documentation dependencies via `uv sync --group docs`
2. Clean previous build output
3. Run `sphinx-build` to generate HTML documentation
4. Open the result in your browser (macOS)
## Directory Structure
```
.
βββ api/ # Auto-generated API documentation (created during build)
βββ _build/ # Built documentation output
βββ conf.py # Sphinx configuration (mock imports, extensions, etc.)
βββ index.rst # Main documentation entry point
βββ build-docs.sh # Local build script
βββ rtd-test.sh # ReadTheDocs test build script (uses pip, not uv)
```
## How It Works
- `conf.py` runs `sphinx-apidoc` during Sphinx's `setup()` phase to generate `.rst` files from Python source
- Sphinx autodoc imports each module to extract docstrings
- Modules with unavailable dependencies are listed in `autodoc_mock_imports` in `conf.py`
- Napoleon extension converts Google-style docstrings to reStructuredText
## Troubleshooting
**Module not appearing in docs:**
1. Check the build output for `autodoc: failed to import` warnings
2. If the module has an unresolvable import dependency, add it to `autodoc_mock_imports` in `conf.py`
3. Verify the module is importable: `uv run python -c "import pipecat.module.name"`
**Duplicate object warnings:**
These come from re-export modules or Sphinx discovering the same class through multiple import paths. Usually cosmetic.
**Docstring formatting warnings:**
Docstrings use reStructuredText, not Markdown. Common issues:
- Use `Example::` with indented code blocks, not `` ```python ``
- Ensure blank lines between directive content and subsequent sections
- Use `Parameters:` (not `Attributes:`) for dataclass field documentation to avoid duplicate entries
---
## File: examples/README.md
# Pipecat Examples
This directory contains examples showing how to build voice and multimodal agents with Pipecat.
## Setup
1. Follow the [README](https://github.com/pipecat-ai/pipecat/blob/main/README.md#%EF%B8%8F-developing-pipecat) steps to get your local environment configured.
> **Run from root directory**: Make sure you are running the steps from the root directory.
> **Using local audio?**: The `LocalAudioTransport` requires a system dependency for `portaudio`. Install the dependency to use the transport.
2. Copy the [`env.example`](../env.example) file and add API keys for services you plan to use:
```bash
cp env.example .env
# Edit .env with your API keys
```
3. Run any example:
```bash
uv run python getting-started/01-say-one-thing.py
```
4. Open the web interface at http://localhost:7860/client/ and click "Connect"
## Running examples with other transports
Most examples support running with other transports, like Twilio or Daily.
### Daily
You need to create a Daily account at https://dashboard.daily.co/u/signup. Once signed up, you can create your own room from the dashboard and set the environment variables `DAILY_ROOM_URL` and `DAILY_API_KEY`. Alternatively, you can let the example create a room for you (still needs `DAILY_API_KEY` environment variable). Then, start any example with `-t daily`:
```bash
uv run getting-started/06-voice-agent.py -t daily
```
### Twilio
It is also possible to run the example through a Twilio phone number. You will need to setup a few things:
1. Install and run [ngrok](https://ngrok.com/download).
```bash
ngrok http 7860
```
2. Configure your Twilio phone number. One way is to setup a TwiML app and set the request URL to the ngrok URL from step (1). Then, set your phone number to use the new TwiML app.
Then, run the example with:
```bash
uv run getting-started/06-voice-agent.py -t twilio -x NGROK_HOST_NAME
```
## Directory Structure
### [`getting-started/`](./getting-started/)
Progressive introduction to Pipecat, from minimal TTS to a full voice agent with function calling.
### [`flows/`](./flows/)
Structured conversations with [Pipecat Flows](../src/pipecat/flows): predefined and dynamic conversation paths with state management, across multiple LLM providers.
### [`voice/`](./voice/)
Full STT + LLM + TTS voice agent pipelines showcasing different speech service providers (Deepgram, ElevenLabs, Cartesia, etc.)
### [`function-calling/`](./function-calling/)
Function calling with different LLM providers (OpenAI, Anthropic, Google, etc.)
### [`transcription/`](./transcription/)
Speech-to-text examples with various STT providers.
### [`vision/`](./vision/)
Image description and vision capabilities with different multimodal LLMs.
### [`realtime/`](./realtime/)
Realtime and multimodal live APIs (OpenAI Realtime, Gemini Live, AWS Nova Sonic, Ultravox, Grok).
### [`persistent-context/`](./persistent-context/)
Maintaining conversation context across sessions with different providers.
### [`context-summarization/`](./context-summarization/)
Summarizing conversation context to manage token limits.
### [`update-settings/`](./update-settings/)
Changing service settings at runtime, organized by service type:
- **[`stt/`](./update-settings/stt/)** β Speech-to-text settings
- **[`tts/`](./update-settings/tts/)** β Text-to-speech settings
- **[`llm/`](./update-settings/llm/)** β LLM settings
### [`turn-management/`](./turn-management/)
Turn detection, interruption handling, and user input management.
### [`thinking-and-mcp/`](./thinking-and-mcp/)
LLM thinking/reasoning modes and MCP (Model Context Protocol) tool server integration.
### [`transports/`](./transports/)
Transport layer examples (WebRTC, Daily, LiveKit).
### [`video-avatar/`](./video-avatar/)
Video avatar integrations (Tavus, HeyGen, Simli, LemonSlice).
### [`video-processing/`](./video-processing/)
Video processing, mirroring, GStreamer, and custom video tracks.
### [`audio/`](./audio/)
Audio recording, background sounds, and sound effects.
### [`observability/`](./observability/)
Pipeline monitoring: observers, heartbeats, and Sentry metrics.
### [`rag/`](./rag/)
Retrieval-augmented generation, grounding, and long-term memory (Mem0, Gemini).
### [`features/`](./features/)
Miscellaneous features: wake phrases, live translation, service switching, voice switching, DTMF keypad menus, and more.
## Advanced Usage
### Customizing Network Settings
```bash
uv run python --host 0.0.0.0 --port 8080
```
### Troubleshooting
- **No audio/video**: Check browser permissions for microphone and camera
- **Connection errors**: Verify API keys in `.env` file
- **Port conflicts**: Use `--port` to change the port
For more examples, visit the [pipecat-examples repository](https://github.com/pipecat-ai/pipecat-examples).
## 2. Official Technical Reference & Guides (pipecat-ai/docs)
# Pipecat
> Pipecat is an open source ecosystem for building voice and multimodal AI
> agents. Build with the Python framework, connect users with the client SDKs,
> structure conversations with Pipecat Flows, and deploy to production on
> Pipecat Cloud.
## About Pipecat
Pipecat is an open source framework and hosted platform for real-time voice
and multimodal AI agents. Its primary components:
- **Pipecat framework**: An open source Python framework that orchestrates AI
services (STT, LLM, TTS, and more) into real-time pipelines. Start with the
[quickstart](https://docs.pipecat.ai/pipecat/get-started/quickstart.md).
- **Pipecat client SDKs**: JavaScript, React, React Native, iOS, Android, and
C++ SDKs that connect users to agents over WebRTC or WebSockets. See the
[client introduction](https://docs.pipecat.ai/client/introduction.md).
- **Pipecat Flows**: A framework for structured conversations - define
conversation paths as nodes with functions and actions. See the
[Flows introduction](https://docs.pipecat.ai/pipecat-flows/introduction.md).
- **Pipecat Cloud**: Managed infrastructure for deploying and scaling agents,
run by the Pipecat team. See the
[Cloud introduction](https://docs.pipecat.ai/pipecat-cloud/introduction.md).
- **Pipecat CLI**: Scaffold projects, run evals, and deploy from the terminal.
See the [CLI overview](https://docs.pipecat.ai/api-reference/cli/overview.md).
Server API reference lives under `api-reference/server/`: services (STT, TTS,
LLM, transports, serializers), pipelines, frames, workers, and utilities.
# Pipecat Documentation
Source: https://docs.pipecat.ai/overview/introduction.md
Documentation for the Pipecat ecosystem: the open source framework, client SDKs, Pipecat Flows, and Pipecat Cloud hosting.
Pipecat is an open source ecosystem for building voice and multimodal AI agents. It provides everything you need to create, deploy, and scale real-time AI applications that can see, hear, and speak. The framework is free to use under the BSD-2 license, and works with any AI provider and any hosting environment β you pay only for the services you choose.
## The Pipecat Ecosystem
Open source Python framework for building voice and multimodal AI pipelines.
Orchestrate 100+ AI services with ultra-low latency.
Client SDKs for JavaScript, React, React Native, iOS, Android, and C++.
Connect users to your agents via web and mobile.
Build structured conversations with defined paths and state management.
Break complex tasks into focused steps for better LLM accuracy.
Managed hosting platform for deploying and scaling Pipecat agents in
production with built-in infrastructure.
## How It All Fits Together
A typical Pipecat application has a **client** and a **server**. The client connects users via browser, mobile app, or phone. The server runs a Pipecat pipeline that processes audio, runs LLMs, and generates speech in real-time. Your hosting provider β Pipecat Cloud or self-hosted β manages deployment and scales instances to handle concurrent sessions.
## Getting Started
Follow the [Quickstart](/pipecat/get-started/quickstart) to create a voice
AI bot in 5 minutes.
Work through the [Learning Pipecat](/pipecat/learn/overview) guide to
understand pipelines, processors, and transports.
Connect users to your agent with a [Client SDK](/client/introduction) for
web or mobile.
Ship to production with [Pipecat Cloud](/pipecat-cloud/introduction) or
[self-host](/pipecat/deployment/overview) on your own infrastructure.
## Community
Connect with other developers, share projects, and get support.
Explore the source code, open issues, and contribute.
## Enterprise Support
Pipecat is built and maintained by Daily. If your team is taking voice agents to production, we're here to help.
[Learn more about enterprise support](/enterprise-support)
# Pipecat Open Source Framework
Source: https://docs.pipecat.ai/overview/pipecat.md
Pipecat is an open source Python framework for voice and multimodal AI agents, orchestrating AI services and transports.
Pipecat is an open source Python framework for building voice and multimodal AI agents. It orchestrates AI services, network transports, and audio processing to enable ultra-low latency conversations that feel natural and responsive.
Want to dive right in? Build and run your first Pipecat application
## What You Can Build
Natural, real-time conversations with AI using speech recognition and
synthesis
Connect to your agent via phone for support, intake, and customer service
interactions
Applications that combine voice, video, images, and text for rich interactions
Storytelling experiences and social companions that engage users
Voice-controlled games and interactive experiences with real-time AI responses
Build structured conversations with Pipecat Flows to complete tasks and improve LLM accuracy
## How It Works
Pipecat orchestrates AI services in a **pipeline**, which is a series of processors that handle real-time audio, text, and video frames with ultra-low latency.
Here's what happens in a typical voice conversation:
1. **Transport** receives audio from the user (browser, phone, etc.)
2. **Speech Recognition** converts speech to text in real-time
3. **LLM** generates intelligent responses based on context
4. **Speech Synthesis** converts responses back to natural speech
5. **Transport** streams audio back to the user
In most cases, the entire round-trip interaction happens between 500-800ms, creating a natural conversation experience for the user.
## A Multi-Agent System
A typical bot is a single agent, but Pipecat is a multi-agent system: you can coordinate many agents that talk to each other over a shared message **bus**. The same `WorkerRunner` you use for one bot coordinates many agents, so any Pipecat app is multi-agent ready.
- The **`WorkerRunner`** owns the bus and manages every agent's lifecycle and discovery.
- Each agent is a **worker**. `BaseWorker` is the foundation and coordinates purely over the bus with no pipeline of its own; `PipelineWorker` and `LLMWorker` add a Pipecat pipeline.
- Any agent can **start other agents** at runtime: a worker adds child workers (or the runner does), and they immediately join the shared bus.
- Agents coordinate in more than one way. **Handoff** transfers control of the conversation to another agent, while **jobs** and **job groups** dispatch work to one or many agents in parallel and collect their results.
Agents can run together in one process over an in-process bus, or across separate processes and machines over a Redis or Postgres bus, without changing your agent code.
Learn how agents, the runner, and the bus work together
## Ready to Build?
Build and run your first Pipecat application
Learn about pipelines, processors, transports, and context management
Browse the complete list of 100+ AI service integrations
Deploy to Pipecat Cloud or self-host on your own infrastructure
# Pipecat Client SDKs
Source: https://docs.pipecat.ai/overview/clients.md
The Pipecat client SDK family connects users to your agents from web and mobile apps, handling the real-time media layer.
Pipecat Clients are a family of SDKs that connect users to your Pipecat agents through web and mobile applications. They handle real-time audio/video transport, session management, and provide messaging and events for building responsive voice AI interfaces.
## Supported Platforms
Web applications with vanilla JavaScript
React applications with hooks and components
Cross-platform mobile apps
Native iOS applications
Native Android applications
High-performance native applications
## What the SDKs Provide
All Pipecat Client SDKs include:
- **Transport management** β WebRTC and WebSocket connections to your Pipecat pipeline
- **Media handling** β Microphone, camera, and speaker device management
- **Session lifecycle** β Connect, disconnect, reconnect, and error handling
- **Messaging** β Send custom messages to your bot and receive responses
- **Events** β Callbacks for bot state changes, transcriptions, and more
## Next Steps
Introduction to client-side development with Pipecat.
Pre-built React components for voice AI interfaces.
# Pipecat Flows
Source: https://docs.pipecat.ai/overview/flows.md
Pipecat Flows adds structured conversations to your voice agents: define conversation paths as nodes with functions and actions.
Pipecat Flows is a framework for building structured conversations in your AI applications. It lets you define conversation paths as a graph of nodes, where each node focuses the LLM on a single task with only the tools it needs.
This approach solves a common problem: monolithic prompts with many tools lead to hallucinations and lower accuracy. Pipecat Flows breaks complex tasks into focused steps with clear, specific instructions.
## When to Use Pipecat Flows
Pipecat Flows is best suited for use cases where:
- **You need precise control** over how a conversation progresses through specific steps
- **Your bot handles complex tasks** that can be broken down into smaller, manageable pieces
- **You want to improve LLM accuracy** by focusing the model on one specific task at a time instead of managing multiple responsibilities simultaneously
## How Pipecat Flows Builds on the Pipeline
A Pipecat **pipeline** provides your bot's core mechanics β receiving audio, transcribing input, running LLM completions, converting responses to audio, and sending audio back to the user.
**Pipecat Flows** builds on that pipeline to structure the conversation, managing context and tools as it moves from one state to the next. This keeps your conversation logic cleanly separated from the pipeline mechanics.
## Ready to Build?
Build your first conversation flow in minutes
Complete reference docs and technical details
Explore real-world examples and use cases
Source code, issues, and contributions
# Pipecat Cloud
Source: https://docs.pipecat.ai/overview/cloud.md
Pipecat Cloud is the managed hosting platform for deploying and scaling Pipecat agents, from the team behind the framework.
[Pipecat Cloud](https://pipecat.daily.co) is a managed platform for deploying and scaling Pipecat agents in production. It handles infrastructure, scaling, and operations so you can focus on building your agent.
## Key Capabilities
- **One-command deploy**: Package and deploy agents with `pipecat cloud deploy`
- **Auto-scaling**: Scale from zero to thousands of concurrent sessions
- **Built-in WebRTC**: Daily WebRTC transport included, no separate infrastructure needed
- **Secrets management**: Securely store and inject API keys and credentials
- **Session management**: Start, stop, and monitor agent sessions via REST API or SDK
- **Logging & monitoring**: Built-in logging with Datadog integration support
- **Global regions**: Deploy close to your users for lowest latency
## How It Works
Write a Pipecat pipeline as you normally would, using any supported services.
Use the CLI to build and deploy your agent image to Pipecat Cloud.
```bash
pipecat cloud deploy
```
Use the REST API or Python SDK to start agent sessions on demand.
```bash
curl --request POST \
--url https://api.pipecat.daily.co/v1/public/{agentName}/start \
--header 'Authorization: Bearer ' \
--header 'Content-Type: application/json' \
```
## Next Steps
Set up your account and deploy your first agent.
Learn about accounts, agent images, secrets, and scaling.
# Enterprise Support
Source: https://docs.pipecat.ai/enterprise-support/index.md
Enterprise support for Pipecat and Pipecat Cloud from Daily: architecture guidance, production readiness, and compliance.
Pipecat is open source and free to use under the BSD-2 license. For teams running voice agents in production, [Daily](https://www.daily.co) β the team behind Pipecat and [Pipecat Cloud](/pipecat-cloud/introduction) β offers enterprise support.
## What enterprise support covers
- **Architecture guidance** β designing pipelines, reviewing benchmarks, choosing services, and tuning latency for your use case.
- **Production readiness** β [capacity planning](/pipecat-cloud/guides/capacity-planning), observability, and evals.
- **Custom Pipecat deployments** β support for devops teams for scaling, compliance, and multi-region deployments.
- **Forward Deployed Engineering** β work directly with senior Pipecat engineers in FDE engagements.
## Get in touch
Talk to Daily about enterprise support, limits, and pricing.
Reach Daily directly at help@daily.co.
Not an enterprise customer? Community support is available in the [Pipecat
Discord](https://discord.gg/pipecat), and issues can be filed on
[GitHub](https://github.com/pipecat-ai/pipecat/issues).
# Introduction to Pipecat
Source: https://docs.pipecat.ai/pipecat/get-started/introduction.md
Start here with the Pipecat framework: what it is, how pipelines orchestrate AI services, and where to go next.
Pipecat is an open source Python framework for building voice and multimodal AI agents. It orchestrates AI services, network transports, and audio processing to enable ultra-low latency conversations that feel natural and responsive. The framework is free to use under the BSD-2 license, and you choose your own stack: any speech-to-text, LLM, and text-to-speech provider, hosted on your own infrastructure or [Pipecat Cloud](https://www.daily.co/pricing/pipecat-cloud) β you pay only for the providers you choose.
Want to dive right in? Build and run your first Pipecat application
## What You Can Build
Natural, real-time conversations with AI using speech recognition and
synthesis
Connect to your agent via phone for support, intake, and customer service
interactions
Applications that combine voice, video, images, and text for rich interactions
Storytelling experiences and social companions that engage users
Voice-controlled games and interactive experiences with real-time AI responses
Build structured conversations with Pipecat Flows to complete tasks and improve LLM accuracy
## How It Works
Pipecat orchestrates AI services in a **pipeline**, which is a series of processors that handle real-time audio, text, and video frames with ultra-low latency.
Here's what happens in a typical voice conversation:
1. **Transport** receives audio from the user (browser, phone, etc.)
2. **Speech Recognition** converts speech to text in real-time
3. **LLM** generates intelligent responses based on context
4. **Speech Synthesis** converts responses back to natural speech
5. **Transport** streams audio back to the user
In most cases, the entire round-trip interaction happens between 500-800ms, creating a natural conversation experience for the user.
## A Multi-Agent System
The bot above is a single agent. Pipecat is also a multi-agent system: the same `WorkerRunner` can coordinate many agents that communicate over a shared message **bus**. Most agents run their own pipeline (`PipelineWorker`, `LLMWorker`), while a coordinator can be a plain `BaseWorker` with no pipeline at all. A normal bot is just the one-agent case, so every Pipecat app is multi-agent ready.
Specialized agents that transfer control seamlessly during a conversation.
Dispatch work to multiple agents in parallel and collect their results.
Run agents across separate processes or machines on a shared bus.
## Installation
Pipecat requires Python 3.11 or later and is published on PyPI as [`pipecat-ai`](https://pypi.org/project/pipecat-ai/). Install the base package with optional **extras** for each transport, AI service, and utility you need:
```bash
uv add "pipecat-ai[daily,deepgram,openai,cartesia,silero]"
```
Each [service page](/api-reference/server/services/supported-services) lists the exact extra to install. To upgrade to the latest version:
```bash
uv lock --upgrade-package pipecat-ai
uv sync
```
## Ready to Build?
Build and run your first Pipecat application
Learn about pipelines, processors, transports, and context management
Browse the complete list of 100+ AI service integrations
Deploy to Pipecat Cloud or self-host on your own infrastructure
# Pipecat Quickstart
Source: https://docs.pipecat.ai/pipecat/get-started/quickstart.md
Build and run your first Pipecat voice AI bot in under 5 minutes: a simple conversational agent you can talk to in a browser.
This quickstart guide will help you build and deploy your first Pipecat voice AI bot. You'll create a simple conversational agent that you can talk to in real-time, then deploy it to production on Pipecat Cloud.
**Two steps**: Local Development (5 min) β Production Deployment (5 min)
## Step 1: Local Development
### Prerequisites
**Environment**
- Python 3.11 or later
- [uv](https://docs.astral.sh/uv/getting-started/installation/) package manager installed
Adding Pipecat to an existing project instead of scaffolding a new one? See
[Installation](/pipecat/get-started/introduction#installation) for manual
setup.
**AI Service API Keys**
This quickstart uses three AI services working together in a pipeline. You'll need API keys from each service:
Create an account and generate your API key for real-time
speech recognition.
Create an account and generate an API key for intelligent conversation
responses.
Sign up and generate your API key for natural voice
synthesis.
Have these API keys ready. You'll add them to your environment file in the next section.
### Setup
1. Install the Pipecat CLI and scaffold the quickstart project
```bash
# Install the Pipecat CLI
uv tool install "pipecat-ai[cli]"
# Scaffold the quickstart project (also writes AGENTS.md + CLAUDE.md)
pipecat init quickstart
# Change to the project directory
cd pipecat-quickstart
```
Planning to build with a coding agent? Set up the [Pipecat Context
Hub](/api-reference/context-hub) with `pipecat context-hub install`, so your
agent queries current Pipecat APIs instead of its training data. It ships with
the CLI, so there's nothing extra to install β but `pipecat init quickstart`
deliberately skips the setup to stay a short path to a running bot, so run
`install` yourself. Building its index takes a few minutes and about 900 MB.
2. Configure your API keys
Create your environment file:
```bash
cp .env.example .env
```
Open the `.env` file in your text editor and add your API keys:
```ini
DEEPGRAM_API_KEY=your_deepgram_api_key
OPENAI_API_KEY=your_openai_api_key
CARTESIA_API_KEY=your_cartesia_api_key
```
3. Set up virtual environment and install dependencies
```bash
uv sync
```
### Run your bot locally
Now you're ready to run your bot! Start it with
```bash
uv run bot.py
```
You should see output similar to this:
```
π WebRTC server starting at http://localhost:7860/client
Open this URL in your browser to connect!
```
Open http://localhost:7860/client in your browser and click **Connect** to start talking to your bot.
**First run note**: The initial startup may take ~20 seconds as Pipecat
downloads required models and imports. Subsequent runs will be much faster.
π **Success!** Your bot is running locally. Now let's deploy it to production so others can use it.
---
## Step 2: Deploy to Production
Transform your local bot into a production-ready service. Pipecat Cloud handles scaling, monitoring, and global deployment.
### Prerequisites
1. Sign up for Pipecat Cloud
[Create your Pipecat Cloud account](https://pipecat.daily.co/sign-up) to deploy and manage your bots.
2. Pipecat CLI
Log in to Pipecat Cloud using the CLI to get started:
```bash
pipecat cloud auth login
```
Select `Allow` to authenticate your CLI session in the browser page that opens.
### Your deployment configuration
The `pcc-deploy.toml` file tells Pipecat Cloud how to build and deploy your bot:
```toml
agent_name = "pipecat-quickstart"
secret_set = "pipecat-quickstart-secrets"
[scaling]
min_agents = 1
```
**Understanding the configuration:**
- `agent_name`: Your bot's name in Pipecat Cloud
- `secret_set`: Where your API keys are stored securely
- `min_agents`: Number of bot instances to keep ready (1 = instant start)
### Configure secrets
Upload your API keys to Pipecat Cloud's secure storage:
```bash
pipecat cloud secrets set pipecat-quickstart-secrets --file .env
```
This creates a secret set called `pipecat-quickstart-secrets` (matching your TOML file) and uploads all your API keys from `.env`.
### Build and Deploy
Build and deploy your bot to Pipecat Cloud:
```bash
pipecat cloud deploy
```
The CLI automatically builds your image using the pcc-deploy.toml file and Dockerfile in your project and deploys it to Pipecat Cloud β no container registry needed.
Want to use your own container registry? See the [container registries
guide](/pipecat-cloud/guides/container-registries/overview) for advanced
deployment options.
### Connect to your agent
1. Open your [Pipecat Cloud dashboard](https://pipecat.daily.co/)
2. Select your `pipecat-quickstart` agent β **Sandbox**
3. Allow microphone access and click **Connect**
π **Your bot is now live in production!**
Explore advanced Pipecat Cloud features like scaling, monitoring, secrets
management, and production best practices.
---
## Understanding the Quickstart Bot
Let's walk through the generated `bot.py` to understand what's happening. When you speak to your bot, here's the real-time pipeline that processes your conversation:
1. **Audio Capture**: Your browser captures microphone audio and sends it via WebRTC
2. **Voice Activity Detection**: Silero VAD detects when you start and stop speaking
3. **Speech Recognition**: Deepgram converts your speech to text in real-time
4. **Language Processing**: OpenAI's GPT model generates an intelligent response
5. **Speech Synthesis**: Cartesia converts the response text back to natural speech
6. **Audio Playback**: The generated audio streams back to your browser
Each step happens with minimal latency, typically completing the full round-trip in under one second.
### AI Services
Your bot uses three AI services, each configured with API keys from your `.env` file:
```python
# Create AI Services
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
settings=CartesiaTTSService.Settings(
voice=os.getenv("CARTESIA_VOICE_ID", "71a7ad14-091c-4e8e-a314-022ece01c121"),
),
)
llm = OpenAIResponsesLLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAIResponsesLLMService.Settings(
model=os.getenv("OPENAI_MODEL", "gpt-4.1"),
system_instruction="You are a helpful assistant in a voice conversation. Your responses will be spoken aloud, so avoid emojis, bullet points, or other formatting that can't be spoken. Respond to what the user said in a creative, helpful, and brief way.",
),
)
```
Pipecat supports many different AI services. You can swap out Deepgram for Soniox, OpenAI for Anthropic, or Cartesia for ElevenLabs without changing the rest of your code.
See the [supported services documentation](/api-reference/server/services/supported-services) for all available options.
### Context and Messages
Your bot maintains conversation history using a context object, enabling multi-turn interactions where the bot remembers what was said earlier.
The context is initialized and used to store the conversation history by the context aggregators:
```python
context = LLMContext()
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
vad_analyzer=SileroVADAnalyzer(),
),
)
```
### RTVI Protocol
When building web or mobile clients, you can use [Pipecat's client SDKs](/client/introduction) that communicate with your bot via the [RTVI (Real-Time Voice Interaction) protocol](/client/rtvi-standard). Pipecat comes with RTVI enabled by default, allowing your the Pipecat server and client to exchange messages and events in real-time.
### Pipeline Configuration
The core of your bot is a Pipeline that processes data through a series of processors:
```python
# Create the pipeline with the processors
pipeline = Pipeline([
transport.input(), # Receive audio from browser
stt, # Speech-to-text (Deepgram)
user_aggregator, # Add user message to context
llm, # Language model (OpenAI)
tts, # Text-to-speech (Cartesia)
transport.output(), # Send audio back to browser
assistant_aggregator, # Add bot response to context
])
```
Data flows through the pipeline as "frames", objects containing audio, text, or other data types. The ordering is crucial: audio must be transcribed before it can be processed by the LLM, and text must be synthesized before it can be played back.
The pipeline is managed by a PipelineWorker:
```python
# Create a PipelineWorker to manage the pipeline execution
worker = PipelineWorker(
pipeline,
params=PipelineParams(
enable_metrics=True,
enable_usage_metrics=True,
),
)
```
The worker handles pipeline execution, collects metrics, and manages RTVI events through [observers](/api-reference/server/utilities/observers/observer-pattern).
### Event Handlers
Event handlers manage the bot's lifecycle and user interactions:
```python
# Kick off the conversation when the client signals it's ready
@worker.rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
context.add_message({"role": "developer", "content": "Start by concisely introducing yourself."})
await worker.queue_frames([LLMRunFrame()])
# Event handler for when a client connects
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
logger.info("Client connected")
# Event handler for when a client disconnects
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
logger.info("Client disconnected")
# Cancel the worker when the client disconnects
# This stops the pipeline and all processors, cleaning up resources
await worker.cancel()
```
When the client signals it's ready over the RTVI protocol, the bot adds a greeting instruction and queues an `LLMRunFrame` to start the conversation. When the client disconnects, the worker is cancelled to clean up resources.
### Running the Pipeline
Finally, the pipeline is executed by a WorkerRunner:
```python
# Create a WorkerRunner to run the worker
runner = WorkerRunner(handle_sigint=False)
# Finally, add the worker and run it
# This will start the pipeline and begin processing frames
await runner.add_workers(worker)
await runner.run()
```
The runner manages the pipeline's execution lifecycle. Note that `handle_sigint=False` because the main runner handles system signals.
### Bot Entry Point
The quickstart uses Pipecat's runner system:
```python
async def bot(runner_args: RunnerArguments):
"""Main bot entry point."""
# Configure transport parameters for different environments
transport_params = {
"daily": lambda: DailyParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
"webrtc": lambda: TransportParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
}
transport = await create_transport(runner_args, transport_params)
await run_bot(transport, runner_args)
if __name__ == "__main__":
from pipecat.runner.run import main
main()
```
This runner automatically handles WebRTC connection setup and management, making it easy to get started with minimal configuration. The same code works for both local development and production deployment.
**Production ready**: This bot pattern is fully compatible with Pipecat Cloud,
meaning you can deploy your bot without any code changes.
## Troubleshooting
- **Browser permissions**: Make sure to allow microphone access when prompted by your browser.
- **Connection issues**: If the WebRTC connection fails, first try a different browser. If that fails, make sure you don't have a VPN or firewall rules blocking traffic. WebRTC uses UDP to communicate.
- **Audio issues**: Check that your microphone and speakers are working and not muted.
## Next Steps
Congratulations! You've built and deployed your first Pipecat bot. Here's what to do next based on your goals:
### π Ready to Build Your Own Application?
The quickstart gave you a working example, but the Pipecat CLI helps you scaffold production-ready projects with your choice of platform (phone vs web/mobile), transport providers, and AI servicesβall tailored to your specific use case.
Use the CLI to scaffold phone or web/mobile projects customized for your needs
### π§ Want to Understand How It Works?
Dive deeper into Pipecat's architecture and learn how to build custom solutions.
Master pipelines, processors, transports, and context management
30+ production-ready examples for inspiration
# Build Your Next Bot
Source: https://docs.pipecat.ai/pipecat/get-started/build-your-next-bot.md
Scaffold a new Pipecat project and build it with a coding agent or by hand
Ready to build your own bot? The Pipecat CLI takes you from zero to a runnable project, whether you build with an AI coding agent or scaffold it by hand.
**New to Pipecat?** We recommend completing the
[Quickstart](/pipecat/get-started/quickstart) first to understand Pipecat
basics before scaffolding your own project.
## Install the Pipecat CLI
Install the CLI globally with [uv](https://docs.astral.sh/uv/):
```bash
uv tool install "pipecat-ai[cli]"
```
Verify installation:
```bash
pipecat --version
```
## Start with `pipecat init`
`pipecat init` is the starting point. It writes the Pipecat coding-agent guide (`AGENTS.md` + `CLAUDE.md`) so your coding agent works well with Pipecat, then asks how you want to build:
```bash
pipecat init
```
- **Build with a coding agent (recommended).** Let an AI coding assistant (Claude Code, Codex, β¦) do the building. Continue with [Build with a coding agent](#build-with-a-coding-agent) below.
- **Scaffold a runnable bot now.** Prefer to drive the CLI yourself? Skip to [Scaffold it yourself](#scaffold-it-yourself).
## Build with a coding agent
AI coding tools like Claude Code and Codex write your agent code. The agent follows the `AGENTS.md` guide that `init` wrote, so it uses Pipecat conventions, scaffolds the app for you, and verifies its own work. This path also writes `GETTING_STARTED.md`, a short guide to driving the agent well.
### The Pipecat Context Hub
The Context Hub indexes Pipecat docs, examples, and API source into a local database, so your agent queries live context instead of stale training data. It ships with the CLI, and `init` sets it up for you: it registers the MCP server with the coding agents it finds, then offers to build the index.
The build takes upwards of three minutes and about 900 MB of disk, so `init` asks rather than assuming. If you skipped it, run:
```bash
pipecat context-hub refresh
```
If `init` found no coding agent CLI to register with β the case for editors configured by hand, like Cursor, VS Code, and Zed β run `pipecat context-hub install` instead, which prints the config block to paste and builds the index.
Do it before opening your coding session β MCP servers load at session start, and the server won't start against an empty index.
Full setup, the tool list, CLI lookups, and MCP config for Cursor and VS Code
### Start a coding session
Open Claude Code or Codex in your project and prompt it to build. The agent follows `AGENTS.md`, queries the Context Hub for accurate APIs, and scaffolds and iterates on your bot.
Prefer to feed context by hand? Pipecat docs support the [llms.txt
standard](https://llmstxt.org/):
[`llms.txt`](https://docs.pipecat.ai/llms.txt) is a structured index of all
pages, and [`llms-full.txt`](https://docs.pipecat.ai/llms-full.txt) is the
full documentation in a single file, useful for tools that ingest docs in
bulk.
## Scaffold it yourself
Prefer to drive the CLI directly? When you pick **Scaffold a runnable bot now**, `init` runs an interactive setup wizard that walks you through your platform (phone or web/mobile), transport provider, AI services (STT, LLM, TTS), and deployment target, then scaffolds the project in place alongside `AGENTS.md` + `CLAUDE.md`. The generated README includes Pipecat Context Hub setup, so the project is ready for a coding agent too.
Once scaffolding completes, the CLI provides specific next steps for your generated project:
- How to configure your API keys
- Installing dependencies
- Running your bot locally
- Customizing the bot logic and behavior
## Deploy to Production
Ready to deploy your bot? Choose between managed cloud hosting or self-hosted infrastructure.
Deploy and manage your agents with the CLI - purpose-built for Pipecat
Deploy to Fly.io, Modal, AWS, or your own infrastructure
## Learn More
Understand pipelines, processors, and transports
30+ production-ready examples for inspiration
# Continue Learning
Source: https://docs.pipecat.ai/pipecat/get-started/next-steps.md
Where to go after the Pipecat quickstart: learning paths, examples, and guides for building production voice agents.
Now that you've run your first Pipecat bot, here's how to continue learning and build your own applications.
## Choose Your Path
Understand pipelines, processors, transports, and how to build custom AI
applications from the ground up.
Dive into complete examples including multimodal bots, creative
applications, and enterprise integrations.
## What's Your Goal?
Choose your path based on what you want to accomplish:
### π Build a Production Application
Ready to create your own bot? Use the CLI to scaffold a project tailored to your needs.
Scaffold phone or web/mobile projects with the Pipecat CLI
### π§ Understand How Pipecat Works
Master the fundamentals to build custom solutions and debug effectively.
Learn about pipelines, processors, transports, and context management
Function calling, audio recording, transcripts, and custom processors
### π‘ Get Inspired
Explore real-world examples and recipes for common use cases.
30+ production-ready examples including multimodal bots and games
Common patterns and solutions for specific use cases
### π Deploy to Production
Scale your bot with managed hosting or self-hosted infrastructure.
Managed hosting with auto-scaling and built-in WebRTC
Deploy on Fly.io, Modal, Cerebrium, or your own infrastructure
## Need Help?
Get support and share projects with other developers
Report bugs or request features
# Migrating to Pipecat 1.0
Source: https://docs.pipecat.ai/pipecat/migration/migration-1.0.md
Upgrade a Pipecat application from 0.0.x to 1.0: removed deprecated APIs, their replacements, and the migration path.
Pipecat 1.0 removes many deprecated APIs from the 0.0.x series. If you were already using the non-deprecated replacements, most of these changes won't affect you.
Before upgrading, search your codebase for the deprecated imports and patterns listed below to identify what needs to change.
Pipecat 1.0 requires **Python 3.11 or later**. Support for Python 3.10 has
been dropped. Python 3.11 through 3.14 are supported.
**Removed parameters are silently ignored, not errored.** `TransportParams`
and `PipelineParams` are Pydantic models that drop unknown fields, so leftover
0.0.x params like `vad_analyzer`, `turn_analyzer`, and `allow_interruptions`
are accepted with no error or warning. Your app can start cleanly and look
upgraded while turn detection is actually broken (missed user speech, no bot
responses). Don't rely on the app starting: delete these params from your
transport/pipeline config and move them to `LLMUserAggregatorParams` as shown
below.
## 1. Universal LLMContext
`LLMContext` is a universal context that works with all LLM providers. It allows you to dynamically switch `LLMService` at runtime, passing a compatible context between services. Pipecat maintains adapters for each LLM service, which are applied automatically at the time of inference.
Use the standard messages format along with [`FunctionSchema` and `ToolsSchema`](/pipecat/learn/function-calling) to define tools in a provider-agnostic way.
### Before (1.0)
```python
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
# Service-specific context
context = OpenAILLMContext(messages, tools)
# Service-specific aggregator factory
context_aggregator = llm.create_context_aggregator(context)
pipeline = Pipeline([
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
])
```
### After (1.0)
```python
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
# Universal context β works with any LLM provider
tools = ToolsSchema(standard_tools=[
FunctionSchema(
name="get_weather",
description="Get current weather",
properties={"location": {"type": "string", "description": "City name"}},
required=["location"],
)
])
context = LLMContext(messages=messages, tools=tools)
# Universal aggregator
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
pipeline = Pipeline([
transport.input(),
stt,
user_aggregator,
llm,
tts,
transport.output(),
assistant_aggregator,
])
```
### What was removed
| Removed | Replacement |
| ---------------------------------------- | -------------------------------------- |
| `OpenAILLMContext` | `LLMContext` |
| `OpenAILLMContextFrame` | `LLMContextFrame` |
| `AnthropicLLMContext` | `LLMContext` |
| `AWSBedrockLLMContext` | `LLMContext` |
| `GatedOpenAILLMContextAggregator` | `GatedLLMContextAggregator` |
| `llm.create_context_aggregator(context)` | `LLMContextAggregatorPair(context)` |
| `VisionImageFrame` | `LLMContext.add_image_frame_message()` |
## 2. Turn Management
Turn management was unified into the user context aggregator configuration. Strategies for interruptions, muting, and idle detection are now configured via `LLMUserAggregatorParams`.
### Before (1.0)
```python
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.user_idle_processor import UserIdleProcessor
# Interruptions configured on the pipeline task
task = PipelineTask(
pipeline,
params=PipelineParams(allow_interruptions=True),
)
# Separate idle processor
idle = UserIdleProcessor(timeout=5.0, callback=handle_idle)
# STTMuteFilter
stt_mute_filter = STTMuteFilter(
config=STTMuteConfig(strategies={STTMuteStrategy.FIRST_SPEECH})
)
pipeline = Pipeline([
transport.input(),
stt,
stt_mute_filter,
idle,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
])
```
### After (1.0)
```python
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_start import (
TranscriptionUserTurnStartStrategy,
VADUserTurnStartStrategy,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies
# Everything configured on the aggregator
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
user_turn_strategies=UserTurnStrategies(
start=[
VADUserTurnStartStrategy(),
TranscriptionUserTurnStartStrategy(),
],
stop=[
TurnAnalyzerUserTurnStopStrategy(
turn_analyzer=LocalSmartTurnAnalyzerV3()
),
],
),
user_mute_strategies=[
# Mute strategies
],
user_turn_stop_timeout=5.0,
user_idle_timeout=8.0,
vad_analyzer=SileroVADAnalyzer(),
),
)
```
**Where did `allow_interruptions` go?** The `allow_interruptions` parameter
was a legacy parameter used for non-voice AI applications. If you're looking
to prevent an interruption from occurring, consider using a user mute strategy
instead. See the [User Mute Strategies
reference](/api-reference/server/utilities/turn-management/user-mute-strategies)
for more information.
### What was removed
| Removed | Replacement |
| ---------------------------------------- | ------------------------------------------------------------ |
| `PipelineParams.allow_interruptions` | `user_turn_strategies` on `LLMUserAggregatorParams` |
| `PipelineParams.interruption_strategies` | `user_turn_strategies` on `LLMUserAggregatorParams` |
| `UserResponseAggregator` | `LLMUserAggregator` (created via `LLMContextAggregatorPair`) |
| `UserIdleProcessor` | `user_idle_timeout` on `LLMUserAggregatorParams` |
| `STTMuteFilter` | `user_mute_strategies` on `LLMUserAggregatorParams` |
| `MinWordsInterruptionStrategy` | `MinWordsUserTurnStartStrategy` |
| `TranscriptionUserTurnStopStrategy` | `SpeechTimeoutUserTurnStopStrategy` |
## 3. VAD & Turn Analyzer Configuration
VAD and turn detection configuration moved from transport params to the user aggregator. Additionally, the `VADParams` default `stops_secs` changed from 0.8 to 0.2. This ensures that STT services create transcripts with optimal latency and sufficient silence padding.
The `UserTurnStrategies` (`stop`) now control how long to wait before the user's turn has stopped:
- `SpeechTimeoutUserTurnStopStrategy` has a `user_speech_timeout` parameter that you can configure to control how long to wait before the user's turn has stopped.
- `TurnAnalyzerUserTurnStopStrategy` uses the `smart-turn` model to dynamically modify the wait time. No configuration is required.
### Before (1.0)
```python
from pipecat.transports.services.daily import DailyParams, DailyTransport
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
transport = DailyTransport(
room_url,
token,
"Bot",
DailyParams(
vad_enabled=True,
vad_audio_passthrough=True,
vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
turn_analyzer=LocalSmartTurnAnalyzerV3(params=SmartTurnParams()),
),
)
```
### After (1.0)
By default, the `TurnAnalyzerUserTurnStopStrategy` stop strategy is used, which includes the `LocalSmartTurnAnalyzerV3` turn analyzer.
```python
from pipecat.transports.daily.transport import DailyTransport, DailyParams
from pipecat.audio.vad.silero import SileroVADAnalyzer
transport = DailyTransport(
room_url,
token,
"Bot",
DailyParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
)
# VAD is now configured on the aggregator
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
vad_analyzer=SileroVADAnalyzer(),
),
)
```
### What was removed
| Removed | Replacement |
| --------------------------------------- | --------------------------------------------------------------------------------- |
| `TransportParams.vad_analyzer` | `LLMUserAggregatorParams.vad_analyzer` |
| `TransportParams.vad_enabled` | Removed (VAD is active when analyzer is set) |
| `TransportParams.vad_audio_passthrough` | Removed |
| `TransportParams.turn_analyzer` | `UserTurnStrategies(stop=[TurnAnalyzerUserTurnStopStrategy()])` (now the default) |
| `VADParams.stop_secs` default (0.8) | Now defaults to 0.2; turn timing is controlled by stop strategies |
## 4. Service Import Paths
All flat service imports have been removed. Services now use submodule imports.
### Pattern change
```python
# Before (1.0)
from pipecat.services.openai import OpenAILLMService
# After (1.0)
from pipecat.services.openai.llm import OpenAILLMService
```
### Renamed/moved service modules
| Old module | New module |
| ----------------------------------------- | ------------------------------------- |
| `pipecat.services.gemini_multimodal_live` | `pipecat.services.google.gemini_live` |
| `pipecat.services.aws_nova_sonic` | `pipecat.services.aws.nova_sonic` |
| `pipecat.services.openai_realtime` | `pipecat.services.openai.realtime` |
| `pipecat.services.riva` (STT) | `pipecat.services.nvidia.stt` |
| `pipecat.services.riva` (TTS) | `pipecat.services.nvidia.tts` |
| `pipecat.services.nim` | `pipecat.services.nvidia.llm` |
### Removed service classes
These service have been removed.
| Removed class | Replacement class |
| ------------------------------ | -------------------------- |
| `PollyTTSService` | `AWSPollyTTSService` |
| `OpenAIRealtimeBetaLLMService` | `OpenAIRealtimeLLMService` |
| `GoogleLLMOpenAIBetaService` | `GoogleLLMService` |
## 5. Transport Import Paths
Transport imports moved from `services` subdirectory to transport-specific subdirectories.
| Old import | New import |
| ---------------------------------------------- | -------------------------------------- |
| `pipecat.transports.services.daily` | `pipecat.transports.daily.transport` |
| `pipecat.transports.services.livekit` | `pipecat.transports.livekit.transport` |
| `pipecat.transports.network.websocket_server` | `pipecat.transports.websocket.server` |
| `pipecat.transports.network.websocket_client` | `pipecat.transports.websocket.client` |
| `pipecat.transports.network.fastapi_websocket` | `pipecat.transports.websocket.fastapi` |
| `pipecat.sync` | `pipecat.utils.sync` |
## 6. Frames
Several frame classes were renamed or removed. Update any direct references in your code.
| Removed | Replacement |
| ------------------------------------ | --------------------------------------------------------------- |
| `TransportMessageFrame` | `OutputTransportMessageFrame` |
| `TransportMessageUrgentFrame` | `OutputTransportMessageUrgentFrame` |
| `InputTransportMessageUrgentFrame` | `InputTransportMessageFrame` |
| `KeypadEntryFrame` | `DTMFFrame` |
| `StartInterruptionFrame` | `InterruptionFrame` |
| `BotInterruptionFrame` | `InterruptionWorkerFrame` |
| `TranscriptionMessage` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` |
| `TranscriptionUpdateFrame` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` |
| `DailyTransportMessageFrame` | `DailyOutputTransportMessageFrame` |
| `DailyTransportMessageUrgentFrame` | `DailyOutputTransportMessageUrgentFrame` |
| `LiveKitTransportMessageFrame` | `LiveKitOutputTransportMessageFrame` |
| `LiveKitTransportMessageUrgentFrame` | `LiveKitOutputTransportMessageUrgentFrame` |
## 7. Service-Specific Parameter Changes
Individual services renamed or removed configuration parameters. Find your service below to see what changed.
| Service | Removed param | Replacement |
| -------------------------- | ---------------------------- | -------------------------------------- |
| `DeepgramSTTService` | `url` | `base_url` |
| `DeepgramSTTService` | `vad_events` | Removed |
| `FishAudioTTSService` | `model` | `reference_id` |
| `GladiaSTTService` | `InputParams.language` | Removed |
| `GladiaSTTService` | `InputParams.confidence` | Removed |
| `GeminiTTSService` | `api_key` | Removed |
| `GeminiLiveLLMService` | `base_url` | `http_options` |
| `GoogleVertexLLMService` | `InputParams` | Removed (`project_id` now required) |
| `MiniMaxHttpTTSService` | `english_normalization` | `text_normalization` |
| `SimliVideoService` | `simli_config` | `api_key` and `face_id` (required str) |
| `AnthropicLLMService` | `enable_prompt_caching_beta` | `enable_prompt_caching` |
| `AWSNovaSonicLLMService` | `send_transcription_frames` | Removed |
| `OpenAIRealtimeLLMService` | `send_transcription_frames` | Removed |
## 8. Transport Parameters
All `camera_*` transport parameters were renamed to `video_*`. VAD-related transport params were removed β VAD is now configured on the user aggregator (see [section 3](#3-vad-&-turn-analyzer-configuration)).
| Removed | Replacement |
| ---------------------------------------- | -------------------------------------- |
| `camera_in_enabled` | `video_in_enabled` |
| `camera_in_is_live` | `video_in_is_live` |
| `camera_in_width` / `camera_in_height` | `video_in_width` / `video_in_height` |
| `camera_out_enabled` | `video_out_enabled` |
| `camera_out_is_live` | `video_out_is_live` |
| `camera_out_width` / `camera_out_height` | `video_out_width` / `video_out_height` |
| `camera_out_bitrate` | `video_out_bitrate` |
| `camera_out_framerate` | `video_out_framerate` |
| `camera_out_codec` | `video_out_codec` |
| `vad_enabled` | Removed |
| `vad_audio_passthrough` | Removed |
## 9. Other API Changes
Smaller breaking changes across the rest of the API.
### Pipeline
| Removed | Replacement |
| ------------------------------------ | ---------------------------------------------- |
| `PipelineParams.observers` | Pass `observers` to `PipelineTask` constructor |
| `PipelineTask.on_pipeline_ended` | `on_pipeline_finished` |
| `PipelineTask.on_pipeline_cancelled` | `on_pipeline_finished` |
| `PipelineTask.on_pipeline_stopped` | `on_pipeline_finished` |
| `DailyRunner.configure_with_args()` | Use `WorkerRunner` with `RunnerArguments` |
### TTS service
| Removed | Replacement |
| ---------------------------- | ---------------------- |
| `TTSService.say()` | Push a `TTSSpeakFrame` |
| `text_aggregator` init param | Use `LLMTextProcessor` |
| `text_filter` init param | `text_filters` |
### Processors
| Removed | Replacement |
| --------------------------------------------- | --------------------------------------------------------------- |
| `AudioBufferProcessor.user_continuous_stream` | `user_audio_passthrough` |
| `UserBotLatencyLogObserver` | `UserBotLatencyObserver` (use `on_latency_measured` event) |
| `TranscriptProcessor` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` |
| `add_pattern_pair()` | `add_pattern()` |
### RTVI
- Deprecated RTVI models (`RTVIConfig`, `RTVIServiceConfig`) removed
- `RTVIActionFrame` removed
- `RTVIProcessor.handle_function_call` and `handle_function_call_start` removed
### Other
| Removed | Replacement |
| ----------------------------------------------------------- | -------------------------------------------- |
| `expect_stripped_words` from `LLMAssistantAggregatorParams` | Removed |
| `context` field from `UserImageRequestFrame` | Removed |
| Old multi-parameter `on_push_frame` observer signature | Use new signature |
| `pipecat.utils.tracing.class_decorators` | Removed |
| `NoisereduceFilter` | Use `KrispVivaFilter` or other audio filters |
| `KrispFilter` | Use `KrispVivaFilter` |
| `pipecat.turns.mute` | `pipecat.turns.user_mute` |
# Overview of Pipecat
Source: https://docs.pipecat.ai/pipecat/learn/overview.md
Learn the foundational concepts of Pipecat's architecture for building voice AI agents
## What You'll Learn
This comprehensive guide will teach you how to build real-time voice AI agents with Pipecat. By the end, you'll be equipped with the knowledge to create custom applicationsβfrom simple voice assistants to complex multimodal bots that can see, hear, and speak.
**Prerequisites**: Basic Python knowledge is recommended. The guide takes
approximately 45-60 minutes to complete, with hands-on examples throughout.
## Why Voice AI is Challenging
Building responsive voice AI applications involves coordinating multiple AI services in real-time:
- **Speech recognition** must transcribe audio as users speak
- **Language models** need to process context and generate responses
- **Speech synthesis** has to convert text back to natural audio
- **Network transports** must handle streaming audio with minimal delay
Doing this manually means managing complex timing, buffering, error handling, and service coordination. Most developers end up rebuilding the same orchestration logic repeatedly.
## Pipecat's Solution
Pipecat solves this orchestration problem with a **pipeline architecture** that handles the complexity for you. Instead of managing individual API calls and timing, you define a flow of processing steps that work together automatically.
Here's what makes Pipecat different:
Typical voice interactions complete in 500-800ms for natural conversations
Swap AI providers, add features, or customize behavior without rewriting
code
Stream processing eliminates waiting for complete responses at each step
Built-in error handling, logging, and scaling considerations
## Core Architecture Concepts
Before diving into how voice AI works, let's understand Pipecat's four foundational concepts:
### Frames
Think of frames as **data packages** moving through your application. Each frame contains a specific type of information:
- Audio data from a microphone
- Transcribed text from speech recognition
- Generated responses from an LLM
- Synthesized audio for playback
### Frame Processors
Frame processors are **specialized building blocks** that handle specific tasks:
- A speech-to-text processor converts audio frames into text frames
- An LLM processor takes text frames and produces response frames
- A text-to-speech processor converts response frames into audio frames
### Pipelines
Pipelines **connect processors together**, creating a path for frames to flow through your application. They handle the orchestration automatically.
### Workers
A **worker** runs a pipeline. A worker that owns a pipeline is an **agent** in your application -- a standalone voice bot is a single worker (a `PipelineWorker`). Pipecat is a multi-agent system, so you can run several workers that coordinate over a shared bus. The `WorkerRunner` starts your workers and manages their lifecycle.
## Voice AI Processing Flow
Now let's see how these concepts work together in a typical voice AI interaction:
User speaks β Transport receives streaming audio β Creates audio frames
STT processor receives audio frames β Transcribes speech in real-time β
Outputs text frames
Context processor aggregates text frames with conversation history β Creates
formatted input for LLM
LLM processor receives context β Generates streaming response β Outputs text
frames
TTS processor receives text frames β Converts to speech β Outputs audio
frames
Transport receives audio frames β Streams to user's device β User hears
response
The key insight: **everything happens in parallel**. While the LLM is generating later parts of a response, earlier parts are already being converted to speech and played back to the user.
## Pipeline Architecture
Here's how this flow translates into a Pipecat pipeline:
Each processor in the pipeline:
1. Receives specific frame types as input
2. Performs its specialized task (transcription, language processing, etc.)
3. Outputs new frames for the next processor
4. Passes through frames it doesn't handle
While frames can flow upstream or downstream, most data flows downstream as
shown above. We'll discuss pushing frames in later sections.
## What's Next
In the following sections, we'll build a complete agent and explore each component in detail:
- Building and running your first agent
- How to initialize sessions and connect users
- Configuring different transport options (Daily, WebRTC, Twilio, etc.)
- Setting up speech recognition and synthesis services
- Managing conversation context and LLM integration
- Handling the complete pipeline lifecycle
- Coordinating multiple agents that share a message bus
Each section includes practical examples and configuration options to help you build production-ready voice AI applications.
Let's build and run your first agent
# Your First Agent
Source: https://docs.pipecat.ai/pipecat/learn/your-first-agent.md
Build and run a single agent: a PipelineWorker, the WorkerRunner, and the pipeline lifecycle.
In Pipecat, an **agent** is a worker that runs a pipeline. A standalone voice bot is a single agent. Pipecat is a multi-agent system, so once you have one agent you can add more that coordinate over a shared bus -- but let's start with one. This page shows the whole shape; the sections that follow break down each piece (transports, speech-to-text, the LLM, text-to-speech, and more).
## Agent types
Pipecat provides built-in worker types, each building on the previous:
| Type | Purpose |
| ------------------ | --------------------------------------------------------------------------------------------------- |
| `BaseWorker` | Foundation for all agents. Connects to the bus, manages lifecycle, coordinates jobs. |
| `PipelineWorker` | Extends `BaseWorker` to run a Pipecat pipeline. A standalone agent is a single `PipelineWorker`. |
| `LLMWorker` | Extends `PipelineWorker` with an LLM pipeline and automatic `@tool` registration. |
| `LLMContextWorker` | Extends `LLMWorker` with a built-in `LLMContext` and aggregator pair (its own or a shared context). |
| `UIWorker` | Extends `LLMContextWorker` to read and drive a client GUI over the RTVI UI channel. |
If you've built a Pipecat bot before, you've already used these.
`PipelineTask` is a deprecated alias for `PipelineWorker`, and
`PipelineRunner` for `WorkerRunner`, so existing `PipelineRunner().run(task)`
code keeps working unchanged. New code should use `PipelineWorker` and
`WorkerRunner` directly.
`LLMWorker` gives an agent its own LLM and tools. It's covered in [Multiple
LLM Agents](/pipecat/learn/multiple-llm-agents); this page focuses on building
and running a single agent with a `PipelineWorker`.
## The WorkerRunner
The `WorkerRunner` is the entry point. It:
- Creates and manages the message bus
- Starts agents and manages their lifecycle
- Tracks agent readiness through a registry
- Coordinates graceful shutdown
When you don't provide a bus, the runner creates an `AsyncQueueBus` automatically -- an in-process bus backed by asyncio queues. For distributed setups, you can pass a network bus such as `RedisBus` or `PgmqBus` instead.
Agents get their bus from the runner. You never pass a `bus=` argument to a worker constructor; you register agents with `runner.add_workers(...)`, which attaches them to the runner's bus and registry. Inside a running agent you can read `self.bus`.
## Building the agent
A single agent is one `PipelineWorker` wrapping a complete pipeline -- the same `transport -> STT -> LLM -> TTS -> transport` flow you'd build for a standalone bot. The LLM runs inline in the pipeline.
```
/* Detailed source-code truncated for AI context efficiency. */
```
That's a complete agent: one worker running one pipeline.
## Running the agent
`runner.add_workers(...)` attaches each agent to the runner's bus and registry and starts it. `runner.run()` then blocks until one of these happens:
- The session ends normally (the agent calls `end()`)
- A signal is received (`SIGINT`/`SIGTERM`, when `handle_sigint=True`)
- You call `runner.cancel()`
- An unhandled error occurs
In the example above, the runner is cancelled when the client disconnects.
## Configuring execution
`PipelineParams` controls how the agent runs -- audio sample rates, metrics, and more:
```python
params = PipelineParams(
audio_in_sample_rate=16000,
audio_out_sample_rate=24000,
enable_metrics=True,
enable_usage_metrics=True,
)
```
The full list of execution parameters
## Adding more agents
This is a single agent. Because Pipecat is a multi-agent system, you can run several agents that each handle part of a conversation and coordinate over the shared bus. Later in this guide you'll see how: giving an agent its own LLM and tools, transferring control between agents with handoff, and dispatching work as jobs.
## What's next
Now you've seen the shape of an agent. Next, let's connect users to it, starting with session initialization.
Connect users and set up a session
# Session Initialization
Source: https://docs.pipecat.ai/pipecat/learn/session-initialization.md
Learn how to set up connections between users and your Pipecat voice AI bot
Before your voice AI bot can start processing audio and generating responses, you need to establish a connection between the user and your bot. This process is called **session initialization** - it's how users and bots find each other and establish a communication channel for real-time audio exchange.
## Understanding the Architecture
Session initialization involves multiple components working together:
- **Runner**: A FastAPI server that handles incoming connection requests and manages session setup
- **Pipecat Bot**: Your voice AI application running as a separate server-side service
- **Client Application**: The user-facing app (web browser, mobile app, etc.)
The runner acts as the coordinator, setting up the necessary resources and starting bot instances, while the Pipecat bot handles the actual voice AI processing.
## Development Runner
For most development and many production use cases, Pipecat provides a **development runner** that handles all the session initialization complexity for you. Instead of building FastAPI servers and managing WebRTC connections yourself, you focus on your bot logic while the runner handles the infrastructure.
### Using the Development Runner
Your bot needs a single entry point function that the runner will call:
```python
from pipecat.runner.types import RunnerArguments
async def bot(runner_args: RunnerArguments):
"""Main bot entry point called by the development runner."""
# Create your transport based on the runner arguments
transport = SmallWebRTCTransport(
params=TransportParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
webrtc_connection=runner_args.webrtc_connection,
)
# Run your bot logic
await run_bot(transport)
if __name__ == "__main__":
from pipecat.runner.run import main
main()
```
Then start your bot with different connection types:
```bash
# P2P WebRTC (opens browser interface)
python bot.py -t webrtc
# Daily room-based WebRTC
python bot.py -t daily
# Telephony (requires ngrok or similar)
python bot.py -t twilio -x your_domain.ngrok.io
```
where `-t` specifies the transport type (e.g., `webrtc`, `daily`, `twilio`) and `-x` is the optional proxy domain for telephony. `-t` is optional β omit it to serve all transports from one server and let the client choose via the `/start` request; pass it to restrict the runner to a single transport.
The development runner automatically:
- Creates the FastAPI server
- Sets up the appropriate endpoints
- Handles connection management
- Starts your bot instances
- Provides a web interface (for WebRTC)
Learn more about building with the development runner in the [runner
guide](/api-reference/server/utilities/runner/guide).
## Connection Types Under the Hood
While the development runner handles the complexity, understanding the three connection patterns helps you choose the right approach and debug issues:
### 1. P2P WebRTC Connections
**What happens:**
1. Runner serves a web interface at `http://localhost:7860/client`
2. When you open the page and connect, browser creates a WebRTC offer
3. Runner receives the offer, establishes connection, starts your bot
4. Browser and bot communicate directly via WebRTC
**When to use:** Direct client connections, embedded applications, local development
### 2. Room-Based WebRTC (Daily)
**What happens:**
User visits the client application and clicks to start a session
Runner calls Daily's API to create a room and tokens using
`pipecat.runner.daily.configure()`
Both user's browser and your bot join the same Daily room
Once media streams are established, browser sends `client_ready` message
Your bot receives the event and starts the conversation
**When to use:** Video calls, group sessions, production deployments
Room-based WebRTC can also be used for SIP or PSTN connections, which require
different connection patterns. Refer to the [telephony
guide](/pipecat/telephony/overview) for details.
### 3. WebSocket Connections (Telephony)
**What happens:**
1. Telephony provider (Twilio, etc.) receives a phone call
2. Provider connects to your runner's webhook endpoint
3. Runner accepts WebSocket connection and parses telephony-specific messages
4. Your bot starts immediately with the parsed connection data
**When to use:** Phone bots, telephony integrations
## Starting Conversations
How and when your bot begins talking depends on the connection type:
### Immediate Start (P2P WebRTC, WebSocket)
These connections are ready immediately, so you can start talking right after connection:
```python
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
logger.info("Client connected - starting conversation")
messages.append({
"role": "developer",
"content": "Say hello and introduce yourself."
})
await worker.queue_frames([LLMRunFrame()])
```
### Handshake Required (Client/Server Room-based WebRTC)
For client/server applications using room-based WebRTC, a handshake ensures both sides are ready and the client won't miss the opening message:
```python
@rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
await rtvi.set_bot_ready() # Confirm readiness to client
# Start the conversation
await worker.queue_frames([LLMRunFrame()])
```
For client/server room-based connections, waiting for `on_client_ready` is
crucial - starting too early can cause the client to miss part of the initial
message.
## Process Isolation
Each session runs its own dedicated bot instance for:
- **Resource Management**: Dedicated CPU and memory per session
- **Error Isolation**: One session crash doesn't affect others
- **Clean Cleanup**: Resources automatically freed when sessions end
The development runner handles this process management automatically.
## Custom Runners: When You Need More Control
The development runner works for most cases, but sometimes you need custom behavior - specific authentication, custom endpoints, or integration with existing systems. For these cases, you can create your own FastAPI runner.
The development runner source code ([available on GitHub](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/runner/run.py)) provides excellent examples for:
**Daily Integration Example:**
```python
from pipecat.runner.daily import configure
@app.post("/start")
async def start_bot(background_tasks: BackgroundTasks):
async with aiohttp.ClientSession() as session:
room_url, token = await configure(session)
# Start bot instance
background_tasks.add_task(run_bot, room_url, token)
return {"room_url": room_url, "token": token}
```
**WebSocket Telephony Example:**
```python
from pipecat.runner.utils import parse_telephony_websocket
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
# Parse provider-specific messages
transport_type, call_data = await parse_telephony_websocket(websocket)
# Start bot with parsed data
await run_telephony_bot(websocket, transport_type, call_data)
```
Refer to the development runner source code to understand these patterns
before building custom runners. It handles many edge cases and provides
battle-tested implementations.
## Key Takeaways
- **Start with the development runner** for fastest development and learning
- **Understand connection types** to choose the right approach for your use case
- **Handle startup timing correctly** - immediate start vs. handshake patterns matter
- **Plan for process isolation** - one bot instance per session is the recommended pattern
- **Reference the source code** when building custom runners for production
## What's Next
Now that you understand session initialization, let's explore the different transport options and how to configure them for your specific needs.
Learn how Pipecat's pipeline architecture orchestrates frame processing for
voice AI applications
# Pipeline & Frame Processing
Source: https://docs.pipecat.ai/pipecat/learn/pipeline.md
Learn how Pipecat's pipeline architecture orchestrates frame processing for voice AI applications
The **Pipeline** is the core orchestration component in Pipecat that connects frame processors together, creating a structured path for data to flow through your voice AI application.
## Basic Pipeline Structure
A Pipeline takes a list of frame processors and connects them in sequence. Here's a simple voice AI pipeline that matches the voice AI agent architecture we discussed earlier:
```python
pipeline = Pipeline([
transport.input(), # Receives user audio
stt, # Speech-to-text conversion
context_aggregator.user(), # Collect user responses
llm, # Language model processing
tts, # Text-to-speech conversion
transport.output(), # Sends audio to user
context_aggregator.assistant(), # Collect assistant responses
])
```
## Understanding Frames and Frame Processing
Before diving deeper into pipelines, it's important to understand how data moves through them using **frames** and **frame processors**.
### What Are Frames?
**Frames** are data containers that carry information through your pipeline. Think of them as packages on a conveyor belt. Each frame contains a specific type of data that processors can examine and act upon.
Frames automatically receive unique identifiers and names (like `TranscriptionFrame#1`) that help with debugging and tracking data flow through your pipeline.
Common frame types include:
- **Audio frames**: Raw audio data from users or generated by TTS
- **Text frames**: Transcriptions, LLM responses, or other text content
- **Image frames**: Visual data for multimodal applications
- **System frames**: Control signals for pipeline management
- **Context frames**: Conversation history and state information
Frames flow through your pipeline from processor to processor, carrying the data your voice AI application needs to operate.
### What Are Frame Processors?
**Frame Processors** are the workers in your pipeline. Each processor has a specific job - like converting speech to text, generating responses, or playing audio. They:
- **Receive frames** from the previous processor in the pipeline
- **Process the data** (transcribe audio, generate text, etc.)
- **Create new frames** with their output
- **Pass frames along** to the next processor
Frame processors are modular and reusable. You can swap out different STT services or LLM providers without changing the rest of your pipeline.
### Frame Types
Frames in Pipecat have different base classes that determine how they're processed:
```python
@dataclass
class SystemFrame(Frame):
"""System frames are queued with high priority."""
pass
@dataclass
class DataFrame(Frame):
"""Data frames are queued and processed in order."""
pass
@dataclass
class ControlFrame(Frame):
"""Control frames are queued and processed in order."""
pass
```
**Key differences:**
- **SystemFrames**: High-priority and ordered with other SystemFrames; interruptions do not discard them (interruptions, pipeline control, user input)
- **DataFrames & ControlFrames**: Queued and processed in order (audio output, text, images)
**Examples by type:**
```python
# SystemFrames (high-priority and ordered with other SystemFrames)
InputAudioRawFrame # User audio input
UserStartedSpeakingFrame # Speech detection events
InterruptionFrame # Interruption control
ErrorFrame # Error notifications
# DataFrames (queued and ordered)
OutputAudioRawFrame # Audio for playback
TextFrame # Text content
TranscriptionFrame # Speech-to-text results
LLMTextFrame # LLM responses
AggregatedTextFrame # Text aggregated into a describable unit
TTSTextFrame # Text-to-speech text output
# ControlFrames (queued and ordered)
EndFrame # Pipeline shutdown
TTSStartedFrame # TTS response boundaries
LLMFullResponseStartFrame # LLM response boundaries
```
### Frame Processing Order
**Frames are processed in guaranteed order within their processing lane**, even across ParallelPipelines. SystemFrames are queued in a high-priority lane and ordered with other SystemFrames; DataFrames and ControlFrames are queued together in the non-system lane and ordered with each other. This enables reliable sequencing. For example, you can push two non-system frames in order and the order will be respected. Additionally, the corresponding processing will finish before allowing the next non-system frame to be processed. Let's look at an example where we push two framesβ`TTSSpeakFrame` and `EndFrame`βin order to say goodbye then end the pipeline:
```python
from pipecat.frames.frames import EndFrame, TTSSpeakFrame
# These will execute in order: speak first, then end pipeline
await worker.queue_frames([
TTSSpeakFrame("Goodbye!"),
EndFrame()
])
```
### How Frame Processors Work
Every frame processor follows the same pattern with two key methods:
```python
class TranscriptionLogger(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection):
# Always call parent first
await super().process_frame(frame, direction)
# Handle specific frame types
if isinstance(frame, TranscriptionFrame):
print(f"Transcription: {frame.text}")
# Push frame to next processor
await self.push_frame(frame, direction)
```
**Key methods:**
- **`process_frame()`**: Inspect and handle incoming frames
- **`push_frame()`**: Send frames upstream or downstream
Learn how to build your own frame processors
## How Data Flows Through Pipelines
Understanding data flow is crucial for building effective pipelines:
### Frame Processing Order
**Order matters**: Processors must be arranged so that each receives the frame types it needs:
1. `transport.input()` creates `InputAudioRawFrame`s from user audio
2. `stt` receives audio frames and outputs `TranscriptionFrame`s
3. `llm` processes text and generates `LLMTextFrame`s
4. `tts` converts text frames to `TTSAudioRawFrame`s, `AggregatedTextFrame`s, and `TTSTextFrame`s
5. `transport.output()` creates `OutputAudioRawFrame`s and sends audio back to user
- Note: An `LLMTextProcessor` can sit between the `llm` and `tts` to pre-aggregate `LLMTextFrame`s into `AggregatedTextFrame`s. This simply moves the aggregation step
out of the TTS.
### Frame Propagation
**Processors always push frames**: Processors don't consume frames, they pass them along:
```python
# This pipeline allows multiple processors to use the same audio
pipeline = Pipeline([
transport.input(), # Creates InputAudioRawFrame
stt, # Uses audio β creates TranscriptionFrame
# ... # Other processors can use the same audio
tts, # Uses various text frames β creates TTSAudioRawFrame
transport.output(), # Uses TTSAudioRawFrame β sends audio to user
audio_buffer_processor, # Also uses the same user and assistant audio for recording
])
```
This design allows multiple processors to operate on the same data stream without interfering with each other.
### Parallel Processing Patterns
Use `ParallelPipeline` to create branches where each branch receives all upstream frames. Frames are collected and pushed individually from each branch:
```python
pipeline = Pipeline([
transport.input(),
stt,
context_aggregator.user(),
llm,
ParallelPipeline([
# English branch
[FunctionFilter(english_filter), english_tts],
# Spanish branch
[FunctionFilter(spanish_filter), spanish_tts],
]),
transport.output(),
context_aggregator.assistant(),
])
```
In this example:
- Both TTS branches receive all LLM output
- Each branch can filter and process frames independently
- Results from both branches flow to `transport.output()`
ParallelPipelines are traditionally paired with filters or gates to control which frames go where, allowing for complex conditional logic.
### Frame Queuing and Processing
Frame processors have internal queues that ensure ordered processing:
- **SystemFrames use a high-priority input queue** and are processed in order with other SystemFrames (interruptions, errors, input audio)
- **DataFrames and ControlFrames use the non-system process queue** and are processed in order with each other
- **Queuing is managed automatically** by the pipeline infrastructure
- **Order is guaranteed within each lane** even across complex pipeline structures
Learn more about frame flow patterns in the [Custom Frame Processor
Guide](/pipecat/fundamentals/custom-frame-processor).
## Key Takeaways
- **Order matters** - arrange processors so each gets the frames it needs
- **Processors push frames** - processors pass frames downstream, not consume them
- **Frame types determine processing** - SystemFrames use the high-priority input queue, while DataFrames and ControlFrames use the non-system process queue
- **Queuing ensures reliability** - frames are processed in guaranteed order within their processing lane
- **Parallel processing** enables conditional logic and multi-modal handling
## What's Next
Now that you understand how pipelines orchestrate processing, let's explore the different transport options that connect your pipeline to users.
Learn about the different ways users can connect to your voice AI pipeline
# Transports
Source: https://docs.pipecat.ai/pipecat/learn/transports.md
Learn about the different ways users can connect to your Pipecat voice AI bot
**Transports** are the communication layer between users and your Pipecat bot. They handle receiving and sending audio, video, and data, serving as the media interface that enables real-time interaction.
## Available Transport Types
Pipecat supports multiple transport types to fit different use cases and deployment scenarios:
WebRTC-based transport using Daily's infrastructure for video calls and
conferencing
WebSocket transport for telephony providers and custom WebSocket connections
Specialized transport for HeyGen LiveAvatar video generation and streaming
WebRTC transport using LiveKit's real-time communication platform
Direct peer-to-peer WebRTC connections without cloud infrastructure
Specialized transport for Tavus video generation and streaming
General-purpose WebSocket transport for custom implementations
## Pipeline Integration
Transports provide two key components for your pipeline: `input()` and `output()` methods. These methods define how the transport interacts with the pipeline:
### Transport Input and Output
```python
pipeline = Pipeline([
transport.input(), # Receives user audio/video
stt,
context_aggregator.user(),
llm,
tts,
transport.output(), # Sends bot audio/video
context_aggregator.assistant(), # Processes after output
])
```
**Key points about transport placement:**
- **`transport.input()`** typically goes first in the pipeline to receive user input
- **`transport.output()`** doesn't always go last - you may want processors after it
- **Post-output processing** enables synchronized actions like:
- Recording with word-level accuracy
- Displaying subtitles synchronized to audio
- Capturing context information precisely timed to output
## Transport Modularity
Transports are modular components in your Pipeline, allowing you to flexibly change how users connect to your bot depending on the context. This modularity enables you to:
- **Switch environments easily**: Use P2P WebRTC for development, Daily for production
- **Support multiple connection types**: Same bot logic works across different transports
- **Optimize for use case**: Choose the best transport for your specific requirements
## Transport Configuration
All transports are configured using `TransportParams`, which provides common settings across transport types:
```python
from pipecat.transports.base_transport import TransportParams
params = TransportParams(
# Audio settings
audio_in_enabled=True,
audio_out_enabled=True,
# Video settings
video_in_enabled=False,
video_out_enabled=False,
# Video stream configuration
video_out_width=1024,
video_out_height=576,
video_out_bitrate=800000,
video_out_framerate=30,
)
```
Each transport may have its own specialized parameters class that extends
TransportParams with transport-specific options. Check the individual
transport documentation for details.
For advanced turn detection (like Smart Turn), configure [User Turn
Strategies](/api-reference/server/utilities/turn-management/user-turn-strategies)
on the context aggregator instead of using the transport's turn_analyzer
parameter.
Complete reference for all transport configuration options
## Telephony Integration
Telephony services (phone calls) use WebSocket connections with specialized serialization:
### Supported Telephony Providers
Media Streams over WebSocket with TwilioFrameSerializer
Real-time media streaming with TelnyxFrameSerializer
Voice streaming API with PlivoFrameSerializer
Voice streaming integration with ExotelFrameSerializer
### Telephony Transport Setup
Telephony requires a `FrameSerializer` to handle provider-specific message formats:
```python
# Create provider-specific serializer
serializer = TwilioFrameSerializer(
stream_sid=stream_sid,
call_sid=call_sid,
account_sid=os.getenv("TWILIO_ACCOUNT_SID", ""),
auth_token=os.getenv("TWILIO_AUTH_TOKEN", ""),
)
# Configure transport with serializer
transport = FastAPIWebsocketTransport(
websocket=websocket_client,
params=FastAPIWebsocketParams(
audio_in_enabled=True,
audio_out_enabled=True,
add_wav_header=False,
serializer=serializer, # Provider-specific serialization
),
)
```
The development runner automatically detects and configures the appropriate
serializer when using `parse_telephony_websocket()`.
## Conditional Transport Selection
The development runner provides a pattern for conditionally selecting transports based on the environment:
```python
async def bot(runner_args: RunnerArguments):
"""Main bot entry point compatible with Pipecat Cloud."""
transport = None
if isinstance(runner_args, DailyRunnerArguments):
from pipecat.transports.daily.transport import DailyParams, DailyTransport
transport = DailyTransport(
runner_args.room_url,
runner_args.token,
"Pipecat Bot",
params=DailyParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
)
elif isinstance(runner_args, SmallWebRTCRunnerArguments):
from pipecat.transports.base_transport import TransportParams
from pipecat.transports.network.small_webrtc import SmallWebRTCTransport
transport = SmallWebRTCTransport(
params=TransportParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
webrtc_connection=runner_args.webrtc_connection,
)
else:
logger.error(f"Unsupported runner arguments type: {type(runner_args)}")
return
if transport is None:
logger.error("Failed to create transport")
return
await run_bot(transport)
```
This pattern allows you to run the same bot code across different environments with different connection types.
## WebRTC vs WebSocket Considerations
Understanding when to use each connection type is crucial for building effective voice AI applications:
### WebRTC (Recommended for Client Applications)
**Best for:** Browser apps, mobile apps, real-time conversations
**Advantages:**
- **Low latency**: Optimized for real-time media with minimal delay
- **Built-in resilience**: Handles packet loss and network variations
- **Advanced audio processing**: Echo cancellation, noise reduction, automatic gain control
- **Quality monitoring**: Detailed performance and media quality statistics
- **Automatic timestamping**: Simplifies interruption and playout logic
- **Robust reconnection**: Built-in connection management
**Use WebRTC when:**
- Building client-facing applications (web, mobile)
- Conversational latency is critical
- Users are on potentially unreliable networks
- You need built-in audio processing features
### WebSocket (Good for Server-to-Server)
**Best for:** Telephony integration, server-to-server communication, prototyping
**Limitations for real-time media:**
- **TCP-based**: Subject to head-of-line blocking
- **Network sensitivity**: Less resilient to packet loss and jitter
- **Manual implementation**: Requires custom logic for reconnection, timestamping
- **Limited observability**: Harder to monitor connection quality
**Use WebSocket when:**
- Integrating with telephony providers (Twilio, Telnyx, etc.)
- Building server-to-server connections
- Prototyping or latency isn't critical
- Working within existing WebSocket infrastructure
## Key Takeaways
- **Transports are modular** - swap them without changing bot logic
- **Choose based on use case** - WebRTC for clients, WebSocket for telephony
- **Configuration is standardized** - TransportParams work across transport types
- **Pipeline placement matters** - consider what processing happens after output
- **Development runner helps** - provides patterns for multi-transport bots
## What's Next
Now that you understand how transports connect users to your bot, let's explore how to configure speech recognition to convert user audio into text.
Learn how to configure speech recognition in your voice AI pipeline
# Speech Input & Turn Detection
Source: https://docs.pipecat.ai/pipecat/learn/speech-input.md
Learn how Pipecat detects user turns using VAD, transcriptions, and turn detection models
A key to natural conversations is properly detecting when the user starts and stops speaking. This is more nuanced than simply detecting audio; a brief pause doesn't always mean the user is done talking.
## Overview
Pipecat uses [user turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) to determine when user turns start and end. These strategies can use different techniques:
**For detecting turn start:**
- Voice Activity Detection (VAD): triggers when speech is detected
- Transcription-based (fallback): triggers when transcription is received but VAD didn't detect speech
- Minimum words: waits for a minimum number of spoken words before triggering
**For detecting turn end (default: Smart Turn):**
- Turn detection model (default): uses AI to understand if the user has finished their thought
- Speech timeout: waits for silence after transcription to determine when the user is done
Custom strategies can also be implemented for specific use cases. By combining these techniques, you can create responsive yet natural conversations that don't interrupt users mid-sentence or wait too long after they've finished.
## Voice Activity Detection (VAD)
### What VAD Does
VAD is responsible for detecting when a user starts and stops speaking. Pipecat includes [Silero VAD](https://github.com/snakers4/silero-vad), an open-source model that runs locally on CPU with minimal overhead. [Krisp VIVA VAD](/api-reference/server/services/vad/krisp-viva-vad-analyzer) is also available for applications requiring support for higher sample rates.
**Silero VAD performance characteristics:**
- Processes 30+ms audio chunks in less than 1ms
- Runs on a single CPU thread
- Minimal system resource impact
### VAD Configuration
VAD is configured through `VADParams` in the `LLMContextAggregatorPair`:
```python
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
vad_analyzer = SileroVADAnalyzer(
params=VADParams(
confidence=0.7, # Minimum confidence for voice detection
start_secs=0.2, # Time to wait before confirming speech start
stop_secs=0.2, # Time to wait before confirming speech stop
min_volume=0.6, # Minimum volume threshold
)
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(vad_analyzer=vad_analyzer),
)
```
VAD is configured on the user aggregator because its speech start/stop signals feed the [user turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) that decide when a turn begins and ends. In the vast majority of cases, the default values will work well. Only adjust these parameters if you have specific audio conditions that require it.
### Key Parameters
**`start_secs` (default: 0.2)**
- How long a user must speak before VAD confirms speech has started
- Lower values = more responsive, but may trigger on brief sounds
- Higher values = less sensitive, but may miss quick utterances, like "yes", "no", or "ok"
**`stop_secs` (default: 0.2)**
- How much silence must be detected before confirming speech has stopped
- Critical for turn-taking behavior
- A short value (0.2s) allows STT services to finalize sooner, improving transcription speed
- **Important**: Built-in STT P99 latency values are measured with `stop_secs=0.2`. If you change this value, re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your settings and pass the measured latency to your STT service via `ttfs_p99_latency`
**`confidence` and `min_volume`**
- Generally work well with defaults
- Only adjust after extensive testing with your specific audio conditions
Changing confidence and min_volume requires careful profiling to ensure
optimal performance across different audio environments and use cases.
`confidence` and `min_volume` only raise the bar for what counts as speech β
blunt instruments for noisy audio. If your bot reacts to background voices or
your STT transcribes noise, it's usually better to remove the noise upstream
with an input audio filter β [Krisp VIVA](/pipecat/features/krisp-viva),
[ai-coustic](/api-reference/server/services/audio-filters/aic-filter), or
[RNNoise](/api-reference/server/services/audio-filters/rnnoise-filter) β
before the audio reaches VAD and STT.
## User Turn Detection
While VAD detects speech vs. silence, it can't understand linguistic context. A pause doesn't mean the user is done. User turn strategies interpret VAD signals and transcriptions to determine actual turn boundaries.
### How It Works
1. **Turn Start**: When VAD detects speech (or transcription arrives), the start strategy emits `UserStartedSpeakingFrame` and optionally triggers an interruption
2. **Turn End**: When the stop strategy determines the user is done, it emits `UserStoppedSpeakingFrame`
VAD also emits its own frames (`VADUserStartedSpeakingFrame`,
`VADUserStoppedSpeakingFrame`) which indicate raw speech/silence detection.
These are inputs to the turn strategies, not the final turn decisions.
### Detecting Turn End
Turn end detection determines when the user has finished speaking and expects a response:
**Smart Turn Model (Default)**: Uses an AI model to analyze audio and determine if the user has finished their thought. This is the default turn stop strategy.
```python
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
stop_strategy = TurnAnalyzerUserTurnStopStrategy(
turn_analyzer=LocalSmartTurnAnalyzerV3()
)
```
**Speech Timeout**: Waits for a configurable timeout after VAD detects silence and a transcript is received. Useful as a simpler alternative to Smart Turn.
```python
from pipecat.turns.user_stop import SpeechTimeoutUserTurnStopStrategy
stop_strategy = SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=0.6)
```
Complete reference for start and stop strategies
Smart Turn model implementation guide
### Interruptions
When a user turn starts with interruptions enabled (the default, via the `enable_interruptions` parameter on start strategies), the bot immediately stops speaking, in-flight work is cancelled, and the pipeline is ready for the new user input.
Keep interruptions enabled for natural conversations. For how the cancellation works, what ends up in the LLM context, and how to tune, disable, or trigger interruptions yourself, see [Interruptions](/pipecat/fundamentals/interruptions).
## Best Practices
### Optimal Configuration
**For most voice AI use cases, the defaults work well:**
```python
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
transport = YourTransport(
params=TransportParams(),
)
# Default configuration: Smart Turn detection + VAD with stop_secs=0.2
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)
worker = PipelineWorker(pipeline)
```
The defaults use `TurnAnalyzerUserTurnStopStrategy` with `LocalSmartTurnAnalyzerV3` for turn detection and `SileroVADAnalyzer` with `stop_secs=0.2` for voice activity detection.
### Performance Considerations
- **Use local VAD**: 150-200ms faster than remote VAD services
- **Tune for your use case**: Test with real audio conditions
- **Monitor CPU usage**: VAD adds minimal overhead but monitor in production
- **Consider turn detection**: Improves conversation quality but adds complexity
## Key Takeaways
- **VAD detects speech activity** but turn detection understands conversation context
- **Configuration affects user experience** - tune parameters for your specific use case
- **System frames coordinate behavior** - enable interruptions and natural turn-taking
- **Local processing is faster** - Silero VAD provides low-latency speech detection
- **Turn detection improves quality** - but requires careful VAD configuration
## What's Next
Now that you understand how speech input is detected and processed, let's explore how that audio gets converted to text through speech recognition.
Learn how to configure speech recognition in your voice AI pipeline
# Speech to Text
Source: https://docs.pipecat.ai/pipecat/learn/speech-to-text.md
Learn how to configure speech recognition to convert user audio into text in your Pipecat pipeline
**Speech to Text (STT)** services are responsible for converting user audio into text transcriptions. They receive audio input from users and provide real-time transcriptions that your bot can process and respond to.
## Pipeline Placement
STT processors must be positioned correctly in your pipeline to receive and process audio frames:
```python
pipeline = Pipeline([
transport.input(), # Creates InputAudioRawFrames
stt, # Processes audio β creates TranscriptionFrames
context_aggregator.user(), # Uses transcriptions for context
llm,
tts,
transport.output(),
])
```
**Placement requirements:**
- **After `transport.input()`**: STT needs `InputAudioRawFrame`s from the transport
- **Before context processing**: Transcriptions must be available for context aggregation
- **Before LLM processing**: Text must be ready for language model input
## STT Service Types
Pipecat provides two types of STT services based on how they process audio:
### 1. STTService (Streaming)
**How it works:**
- Establishes a WebSocket connection to the STT provider
- Continuously streams audio for real-time transcription
- Lower latency due to persistent connection
### 2. SegmentedSTTService (HTTP-based)
**How it works:**
- Uses local VAD (Voice Activity Detection) to chunk speech
- Sends audio segments to STT service as wav files
- Higher latency due to segmentation and HTTP POST requests
STT services are modular and can be swapped out with no additional overhead.
You can easily switch between streaming and segmented services based on your
needs.
## Supported STT Services
Pipecat supports a wide range of STT providers to fit different needs and budgets:
View the complete list of supported speech-to-text providers
Popular options include:
Fast, accurate streaming STT with excellent real-time performance
Real-time streaming STT with strong multilingual support and language hints
AI-powered transcription with speaker diarization and sentiment analysis
Advanced speech recognition with strong accent and dialect handling
Low-latency real-time transcription from the Cartesia voice platform
Real-time streaming STT with a low-latency transcription API
## STT Configuration
### Service-Specific Configuration
Each STT service has its own customization options. Refer to specific service documentation for details:
Explore configuration options for each supported STT provider
For example, let's look at configuring the **DeepgramSTTService** using the `LiveOptions` class:
```python
from deepgram import LiveOptions
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.transcriptions.language import Language
# Configure using LiveOptions for full control
live_options = LiveOptions(
model="nova-2",
language=Language.EN_US,
interim_results=True, # Enable interim transcripts
punctuate=True, # Add punctuation
profanity_filter=True, # Filter profanity
vad_events=False, # Use pipeline VAD instead
)
stt = DeepgramSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
live_options=live_options,
)
```
### STTService Base Class Configuration
All STT services inherit from the STTService base class. The base class has base configuration options which are set with smart defaults:
```python
stt = YourSTTService(
# Service-specific options...
audio_passthrough=True, # Pass audio frames downstream (recommended)
sample_rate=16000, # Audio sample rate (better set in PipelineParams)
)
```
**Key options:**
- **`audio_passthrough=True`**: Allows audio frames to continue downstream to other processors (like audio recording)
- **`sample_rate`**: Audio sampling rate - best practice is to **set the `audio_in_sample_rate` in `PipelineParams` for consistency**
Setting `audio_passthrough=False` will stop audio frames from being passed
downstream, which may break audio recording or other audio-dependent
processors.
### Pipeline-Level Audio Configuration
Instead of setting sample rates on individual services, configure them pipeline-wide:
```python
worker = PipelineWorker(
pipeline,
params=PipelineParams(
audio_in_sample_rate=16000, # All input processors use this rate
audio_out_sample_rate=24000, # All output processors use this rate
),
)
```
This ensures all audio processors use consistent sample rates without manual configuration.
Always set audio sample rates in `PipelineParams` to avoid mismatches between
different audio processors. This simplifies configuration and ensures
consistent audio quality across your pipeline.
## Multilingual Transcription
Many STT services in Pipecat default to `Language.EN` (English). If you need to transcribe speech in other languages or let the model auto-detect the spoken language, you can enable multilingual support. However, providers implement this differently:
**`language=None`** β Whisper-based services (Groq, OpenAI, local Whisper) and ElevenLabs support automatic language detection when no language is specified:
```python
from pipecat.services.groq.stt import GroqSTTService
stt = GroqSTTService(
api_key=os.getenv("GROQ_API_KEY"),
settings=GroqSTTService.Settings(
language=None, # Auto-detect language
),
)
```
**`language="multi"`** β Deepgram uses a special `"multi"` language code to enable multilingual transcription:
```python
from pipecat.services.deepgram.stt import DeepgramSTTService
stt = DeepgramSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
settings=DeepgramSTTService.Settings(
language="multi", # Enable multilingual mode
),
)
```
**Language array** β Google Cloud STT accepts a list of languages for multi-language recognition. See the [Google STT docs](/api-reference/server/services/stt/google) for details.
Some services have additional multilingual features. For example, Soniox
supports language hints, AssemblyAI offers a dedicated multilingual model, and
Speechmatics supports bilingual transcription. See individual service docs for
details.
## Best Practices
### Enable Interim Results
When available, enable interim transcripts for better user experience:
```python
stt = DeepgramSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
live_options=LiveOptions(
interim_results=True,
)
)
```
**Benefits:**
- Notifies context aggregation that more text is coming
- Prevents premature LLM completions
- Enables interruption detection
- Improves conversation flow
### Enable Punctuation and Formatting
Use punctuation when available for better LLM comprehension:
```python
stt = DeepgramSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
live_options=LiveOptions(
punctuate=True, # Adds punctuation
profanity_filter=True, # Optional content filtering
)
)
```
**Benefits:**
- Professional-looking transcripts
- Better LLM comprehension
- Eliminates post-processing needs
- Improved context understanding
### Use Local VAD
While many STT services provide Voice Activity Detection, use Pipecat's local Silero VAD for better performance:
```python
from pipecat.audio.vad.silero import SileroVADAnalyzer
# Configure in context aggregator
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
vad_analyzer=SileroVADAnalyzer(),
),
)
```
**Advantages:**
- **150-200ms faster** speech detection (no network round trip)
- More responsive conversation flow
- Better interruption handling
- Reduced latency overall
### Tune STT Latency
Each STT service has a measured P99 latency for delivering final transcripts after the user stops speaking. This value is used by turn stop strategies to decide how long to wait before ending the user's turn. If you notice the bot responding too early (cutting off the user) or too late (long pauses), tuning this value can help.
Learn about TTFS latency, see default values for every STT service, and how to
measure and override for your deployment
## Key Takeaways
- **Pipeline placement matters** - STT must come after transport input, before context processing
- **Service types differ** - streaming services have lower latency than segmented
- **Services are modular** - easily swap providers without code changes
- **Best practices improve performance** - use interim results, formatting, and local VAD
- **Configuration affects quality** - proper setup significantly impacts transcription accuracy
## What's Next
Now that you understand speech recognition, let's explore how to manage conversation context and memory in your voice AI bot.
Learn how to handle conversation history and context in your pipeline
# Context Management
Source: https://docs.pipecat.ai/pipecat/learn/context-management.md
Work with Pipecat's LLM context and context aggregators: how conversation history is built and shared in a pipeline.
## What is Context in Pipecat?
In Pipecat, **context** refers to the conversation history that the LLM uses to generate responses. The context consists of user/assistant messages representing the conversation history, and can also include **developer messages** for task-specific instructions to the LLM.
```python
# Example context structure
messages = [
{"role": "developer", "content": "Keep responses under two sentences."},
{"role": "user", "content": "Hello!"},
{"role": "assistant", "content": "Hi there! How can I help you?"},
# Context aggregators automatically add new messages here
]
```
The system prompt is typically set via `system_instruction` in the LLM service's [Settings](/pipecat/fundamentals/service-settings), not as a message in the context. See [System Instruction, Developer Messages, and System Messages](#system-instruction-developer-messages-and-system-messages) below for details.
Since Pipecat is a real-time voice AI framework, context management happens automatically as the conversation flows, but you can also control it manually when needed.
## How Context Updates During Conversations
Context updates happen automatically as frames flow through your pipeline:
**User Messages:**
1. User speaks β `InputAudioRawFrame` β STT Service β `TranscriptionFrame`
2. `context_aggregator.user()` receives `TranscriptionFrame` and adds user message to context
**Assistant Messages:**
1. LLM generates response β `LLMTextFrame` β TTS Service β `TTSTextFrame`
2. `context_aggregator.assistant()` receives `TTSTextFrame` and adds assistant message to context
**Frame types that update context:**
- **`TranscriptionFrame`**: Contains user speech converted to text by STT service
- **`LLMTextFrame`**: Contains LLM-generated responses
- **`TTSTextFrame`**: Contains bot responses converted to text by TTS service (represents what was actually spoken)
The TTS service processes `LLMTextFrame`s but outputs `TTSTextFrame`s, which
represent the actual spoken text returned by the TTS provider. This ensures
context matches what users actually hear.
## Setting Up Context Management
Pipecat includes a context aggregator that creates and manages context for both user and assistant messages:
### 1. Create the Context and Context Aggregator
```python
# Create LLM service with system instruction
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAILLMService.Settings(
model="gpt-4o",
system_instruction="You are a helpful voice assistant.",
),
)
# Create context (no system message needed β system_instruction handles it)
context = LLMContext()
# Create context aggregator instance
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```
The context aggregator also supports configuring [user turn
strategies](/api-reference/server/utilities/turn-management/user-turn-strategies)
and [user mute
strategies](/api-reference/server/utilities/turn-management/user-mute-strategies)
via `LLMUserAggregatorParams`.
**About LLMContext:**
`LLMContext` is Pipecat's universal context container that stores conversation messages, tool definitions, and tool choice settings. It uses an OpenAI-compatible format that works across all LLM services through automatic adapter translation.
Key properties:
- **`messages`**: List of conversation messages (user, assistant, developer, tool)
- **`tools`**: Optional available functions β a list of direct functions and/or `FunctionSchema` objects (or a `ToolsSchema`)
- **`tool_choice`**: Optional strategy for tool selection
### 2. Context with Function Calling
Context can also include [tools](/pipecat/learn/function-calling#1-define-a-tool) (function definitions) that the LLM can call during conversations:
```python
from pipecat.services.llm_service import FunctionCallParams
# A direct function: schema is auto-derived from the signature and docstring
async def get_current_weather(params: FunctionCallParams, location: str, format: str):
"""Get the current weather.
Args:
location: The city and state, e.g. "San Francisco, CA".
format: The temperature unit to use. Must be either "celsius" or "fahrenheit".
"""
await params.result_callback({"conditions": "sunny", "temperature": "75"})
# Create context with both messages and tools
context = LLMContext(messages, tools=[get_current_weather])
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```
Function call results are also automatically stored in the context, maintaining a complete conversation history including tool interactions.
We'll cover function calling in detail in an upcoming section. The context
aggregator handles function call storage automatically.
### 3. Add Context Aggregators to Your Pipeline
```python
pipeline = Pipeline([
transport.input(),
stt,
context_aggregator.user(), # User context aggregator
llm,
tts,
transport.output(),
context_aggregator.assistant(), # Assistant context aggregator
])
```
## System Instruction, Developer Messages, and System Messages
Pipecat provides three ways to give instructions to your LLM, each suited for a different purpose.
### Using `system_instruction` (recommended)
Set the bot's personality and core behavior via the LLM service's Settings. The service automatically prepends it to the context messages on each request:
```python
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAILLMService.Settings(
model="gpt-4o",
system_instruction="You are a helpful voice assistant.",
),
)
# Context only needs conversation messages
context = LLMContext()
```
This approach is recommended because:
- **Survives context updates**: The system prompt is always prepended, even after `LLMMessagesUpdateFrame` replaces the context or context summarization compresses older messages.
- **Shared context**: Multiple LLM services can share a single `LLMContext` while each provides its own system instruction.
- **Runtime updates**: You can change the system prompt mid-conversation via `LLMUpdateSettingsFrame`.
### Using developer messages in context
**Developer messages** (`"role": "developer"`) are task-specific instructions placed directly in the context. Use them for supplementary guidance that applies to a particular phase of the conversation rather than defining the bot's overall personality.
```python
messages = [
{"role": "developer", "content": "Keep responses under two sentences. Use metric units."},
]
context = LLMContext(messages)
```
Key characteristics:
- **Task-specific**: Use for instructions like response format constraints, domain rules, or workflow steps. The bot's personality belongs in `system_instruction`.
- **Part of normal context flow**: Developer messages participate in context like any other message. They are included in the summarization range and may be compressed or dropped during [context summarization](/pipecat/fundamentals/context-summarization).
- **Cross-provider translation**: Pipecat's adapters automatically convert developer messages for providers that don't support the role natively (for example, Anthropic receives them as user messages).
### Using a context system message (legacy)
You can also include a system message directly in the context messages:
```python
messages = [
{"role": "system", "content": "You are a helpful voice assistant."},
]
context = LLMContext(messages)
```
This works but has limitations: the system message can be lost during full context replacement, and it cannot be shared across multiple LLM services with different system prompts. Context summarization does preserve `messages[0]` when it's a system message, but `system_instruction` is still more reliable because it sits outside the context entirely.
For new projects, prefer `system_instruction` for personality and developer messages for task-specific instructions.
If both `system_instruction` and a system message in the context are set,
`system_instruction` takes precedence and a warning is logged. Avoid using
both at the same time.
## Context Aggregator Placement
The placement of context aggregator instances in your pipeline is **crucial** for proper operation:
### User Context Aggregator
Place the user context aggregator **downstream from the STT service**. Since the user's speech results in `TranscriptionFrame` objects pushed by the STT service, the user aggregator needs to be positioned to collect these frames.
### Assistant Context Aggregator
Place the assistant context aggregator **after `transport.output()`**. This positioning is important because:
- The TTS service outputs `TTSTextFrame`s in addition to audio
- The assistant aggregator must be downstream to collect those frames
- It ensures context updates happen word-by-word for specific services (e.g. Cartesia, ElevenLabs, and Rime)
- Your context stays updated at the word level in case an interruption occurs
Always place the assistant context aggregator **after** `transport.output()`
to ensure proper word-level context updates during interruptions. See
[Interruptions](/pipecat/fundamentals/interruptions#what-ends-up-in-the-context)
for what the context contains when the bot is cut off mid-sentence.
## Manual Context Control
You can programmatically add new messages to the context by pushing or queueing specific frames:
### Adding Messages
- **`LLMMessagesAppendFrame`**: Appends a new message to the existing context
- **`LLMMessagesUpdateFrame`**: Completely replaces the existing context with new messages
- **`LLMMessagesTransformFrame`**: Edits the existing context in place using a transform function
```python
# Add a new user message to context and trigger a response
new_message = {"role": "user", "content": "Tell me about your capabilities."}
await worker.queue_frames([
LLMMessagesAppendFrame([new_message], run_llm=True), # Optionally trigger bot response, too
])
```
#### Adding a message silently
All three frames take a `run_llm` argument that controls whether the change also prompts a bot response. Pass `run_llm=True` to respond; the default (`None`, which behaves like `False`) updates the context silently. This is useful when you collect information in the background and don't want the bot to react every time:
```python
# Add a message to context without triggering a bot response
note = {"role": "user", "content": "Caller's name is Maria. Account verified."}
await worker.queue_frames([
LLMMessagesAppendFrame([note], run_llm=False), # Silent: no response
])
```
#### Editing or removing specific messages
To surgically edit or remove individual messages without rebuilding the whole list, push an `LLMMessagesTransformFrame`. It takes a function that receives the current list of messages and returns a modified list. Use it to drop stale instructions, remove an offensive turn, or rewrite content:
```python
from pipecat.frames.frames import LLMMessagesTransformFrame
def remove_language_instructions(messages):
# Drop any message that contains a per-turn language instruction
return [m for m in messages if "LANGUAGE INSTRUCTION" not in str(m.get("content", ""))]
await worker.queue_frames([
LLMMessagesTransformFrame(transform=remove_language_instructions, run_llm=False),
])
```
To make the bot **speak** specific text and have it recorded in context (for
example, a greeting the LLM did not generate), use [`TTSSpeakFrame` with
`append_to_context=True`](/pipecat/learn/text-to-speech). That path is for
driving speech output; the frames above are for editing the context directly.
### Retrieving Current Context
The context aggregator provides a `context` property for getting the current context:
```python
context = context_aggregator.user().context
```
## Context Summarization
In long-running conversations, context grows with every exchange, increasing token usage and potentially hitting context window limits. Pipecat includes built-in context summarization that automatically compresses older conversation history while preserving recent messages.
Enable it by setting `enable_auto_context_summarization=True` when creating your context aggregators (default: `False`):
```python
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
assistant_params=LLMAssistantAggregatorParams(
enable_auto_context_summarization=True,
),
)
```
Learn how to configure summarization triggers, customize behavior, and control
what gets preserved.
## Triggering Bot Responses
You may want to manually trigger the bot to speak in two scenarios:
1. **Starting a pipeline** where the bot should speak first
2. **After editing the context** using `LLMMessagesAppendFrame` or `LLMMessagesUpdateFrame`
```python
# Example: Bot speaks first when pipeline starts
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
# Trigger a response
await worker.queue_frames([LLMRunFrame()])
```
```python
# Example: Bot speaks after context is edited
new_message = {"role": "user", "content": "Tell me a fun fact."}
await worker.queue_frames([
LLMMessagesAppendFrame([new_message], run_llm=True), # Trigger bot response
])
```
This gives you fine-grained control over when and how the bot responds during the conversation flow.
## Key Takeaways
- **Context is conversation history** - automatically maintained as users and bots exchange messages
- **Use `system_instruction` for system prompts** - set it in LLM Settings rather than as a context message for reliability across context updates and summarization
- **Use developer messages for task instructions** - place task-specific guidance in context with `"role": "developer"` rather than overloading the system prompt
- **Frame types matter** - `TranscriptionFrame` for users, `TTSTextFrame` for assistants
- **Placement matters** - user aggregator after STT, assistant aggregator after transport output
- **Tools are included** - function definitions and results are stored in context
- **Manual control available** - use frames to append messages or trigger responses when needed
- **Word-level precision** - proper placement ensures context accuracy during interruptions
- **Automatic summarization** - enable context summarization to manage long conversations efficiently and reduce token costs
## What's Next
Now that you understand context management, let's explore how to configure the LLM services that process this context to generate intelligent responses.
Learn how to configure language models in your voice AI pipeline
# LLM Inference
Source: https://docs.pipecat.ai/pipecat/learn/llm.md
Learn how to configure language models to generate intelligent responses in your voice AI pipeline
**LLM services** are responsible for chat completions and tool calling based on the provided context (conversation history). The LLM responds by streaming tokens via `LLMTextFrame`s, which are used by subsequent processors to create audio output for the bot.
## Pipeline Placement
The LLM instance should be placed after the user context aggregator and before any downstream services that depend on the LLM's output stream:
```python
pipeline = Pipeline([
transport.input(),
stt, # Creates TranscriptionFrames
context_aggregator.user(), # Processes user context β creates LLMContextFrame
llm, # Processes context β streams LLMTextFrames
tts, # Processes LLMTextFrames β creates TTSAudioRawFrames
transport.output(),
context_aggregator.assistant(),
])
```
**Frame flow:**
- **Input**: Receives `LLMContextFrame` containing conversation history
- **Processing**:
- Analyzes context and generates streaming response
- Handles function calls if tools are available
- Tracks token usage for metrics
- **Output**:
- Denotes the start of the streaming response by pushing an `LLMFullResponseStartFrame`
- Streams `LLMTextFrame`s containing response tokens to downstream processors (enables real-time TTS processing)
- Ends with an `LLMFullResponseEndFrame` to mark the completion of the response
- Output frames can be configured to [skip TTS](/pipecat/learn/text-to-speech#skipping-tts-output) via `LLMConfigureOutputFrame(skip_tts=True)`, allowing text to flow through the pipeline without being spoken
- **Function calls:**
- `FunctionCallsStartedFrame`: Indicates function execution beginning
- `FunctionCallInProgressFrame`: Indicates a function is currently executing
- `FunctionCallResultFrame`: Contains results from executed functions
## Supported LLM Services
Pipecat supports a wide range of LLM providers to fit different needs, performance requirements, and budgets:
### Text-Based LLMs
Most LLM services are built on the OpenAI chat completion specification for compatibility:
GPT models with the original chat completion API
Claude models with advanced reasoning capabilities
Multimodal capabilities with competitive performance
Enterprise-grade hosting for various foundation models
**Compatible APIs**: Any OpenAI-spec compatible service can be used via the `base_url` parameter.
### Speech-to-Speech Models
For lower latency, some providers offer direct speech-to-speech models:
- **OpenAI Realtime**: Direct speech input/output with GPT models
- **Gemini Live**: Real-time speech conversations with Gemini
- **AWS Nova Sonic**: Speech-optimized models on Bedrock
For a complete list of supported LLM services, see the Supported Services page:
View the complete list of supported language model providers
## LLM Service Architecture
### BaseOpenAILLMService
Many LLM services use the OpenAI chat completion specification. Pipecat provides a `BaseOpenAILLMService` that most providers extend, enabling easy switching between compatible services:
```python
from pipecat.services.openai.llm import OpenAILLMService
# Native OpenAI
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
# OpenAI-compatible service via base_url
llm = OpenAILLMService(
api_key=os.getenv("OTHER_API_KEY"),
base_url="https://api.other-provider.com/v1" # Custom endpoint
)
```
This architecture allows you to quickly plug in different LLM services without changing your pipeline code.
## LLM Configuration
### Service-Specific Configuration
Each LLM service has its own configuration options. For example, configuring OpenAI with various parameters:
```python
from pipecat.services.openai.llm import OpenAILLMService
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAILLMService.Settings(
model="gpt-4.1",
system_instruction="You are a helpful voice assistant.",
temperature=0.7, # Response creativity (0.0-2.0)
max_completion_tokens=150, # Maximum response length
frequency_penalty=0.5, # Reduce repetition (0.0-2.0)
presence_penalty=0.5, # Encourage topic diversity (0.0-2.0)
),
)
```
`system_instruction` defines the bot's personality and core behavior. For task-specific instructions (response format constraints, domain rules, workflow steps), use developer messages in context instead. See [Context Management](/pipecat/learn/context-management#system-instruction-developer-messages-and-system-messages) for how these interact.
For detailed configuration options specific to each provider:
Explore configuration options for each supported LLM provider
### Base Class Configuration
All LLM services inherit from the `LLMService` base class with shared configuration options:
```python
llm = YourLLMService(
# Service-specific options...
run_in_parallel=True, # Whether function calls run in parallel (default: True)
)
```
**Key options:**
- **`run_in_parallel`**: Controls whether function calls execute simultaneously or sequentially
- `True` (default): Faster execution when multiple functions are called
- `False`: Sequential execution for dependent function calls
## Event Handlers
LLM services provide event handlers for monitoring completion lifecycle:
```python
@llm.event_handler("on_completion_timeout")
async def on_completion_timeout(service):
logger.warning("LLM completion timed out")
# Handle timeout (retry, fallback, etc.)
@llm.event_handler("on_function_calls_started")
async def on_function_calls_started(service, function_calls):
logger.info(f"Starting {len(function_calls)} function calls")
# Optionally notify user that bot is "thinking"
await tts.queue_frame(TTSSpeakFrame("Let me check on that."))
```
**Available events:**
- **`on_completion_timeout`**: Triggered when LLM requests timeout
- **`on_function_calls_started`**: Triggered when function calls are initiated
These handlers enable you to provide user feedback and implement error recovery strategies.
## Function Calling
LLMs can call external functions to access real-time data and perform actions beyond their training data. This enables capabilities like checking weather, querying databases, or controlling external APIs.
Function calls and their results are automatically stored in the conversation context by the context aggregator.
Learn how to enable LLMs to interact with external services and APIs
## Key Takeaways
- **Pipeline placement matters** - LLM goes after user context, before TTS
- **Token streaming enables real-time responses** - no waiting for complete generation
- **OpenAI compatibility** enables easy provider switching
- **Function calling extends capabilities** beyond training data
- **Configuration affects behavior** - tune temperature, penalties, and limits
- **Services are modular** - swap providers without changing pipeline code
## What's Next
Now that you understand LLM configuration, let's explore how function calling enables your bot to interact with external services and real-time data.
Learn how to enable LLMs to interact with external services and APIs
# Function Calling
Source: https://docs.pipecat.ai/pipecat/learn/function-calling.md
Enable LLMs to interact with external services and APIs in your voice AI pipeline
**Function calling** (also known as tool calling) allows LLMs to request information from external services and APIs during conversations. This extends your voice AI bot's capabilities beyond its training data to access real-time information and perform actions.
## Pipeline Integration
Function calling works seamlessly within your existing pipeline structure. The LLM service handles function calls automatically when they're needed:
```python
pipeline = Pipeline([
transport.input(),
stt,
context_aggregator.user(), # Collects user transcriptions
llm, # Processes context, calls functions when needed
tts,
transport.output(),
context_aggregator.assistant(), # Collects function results and responses
])
```
**Function call flow:**
1. User asks a question requiring external data
2. LLM recognizes the need and calls appropriate function
3. Your function handler executes and returns results
4. LLM incorporates results into its response
5. Response flows to TTS and user as normal
**Context integration:** Function calls and their results are automatically stored in conversation context by the context aggregators, maintaining complete conversation history.
## Understanding Function Calling
Function calling allows your bot to access real-time data and perform actions that aren't part of its training data.
For example, you could give your bot the ability to:
- Check current weather conditions
- Look up stock prices
- Query a database
- Control smart home devices
- Schedule appointments
Here's how it works:
1. You define functions the LLM can use and make them available to the LLM service used in your pipeline
2. When needed, the LLM requests a function call
3. Your application executes any corresponding functions
4. The result is sent back to the LLM
5. The LLM uses this information in its response
## Implementation
### 1. Define a tool
A tool needs two things: a handler β the code to run when the LLM calls the tool β and a schema that describes the tool to the LLM (its name, what it does, and its parameters) so the model knows it exists and how to call it.
The preferred way to define a tool is with a **direct function**: a single async function that is _both_ the handler and the schema. Pipecat auto-derives the tool's metadata β name, description, parameter properties (with their descriptions), and which parameters are required β from the function's signature and docstring.
The first parameter is always `params` (a `FunctionCallParams`); the tool's own arguments follow. Document each argument in a Google-style docstring.
```python
from pipecat.services.llm_service import FunctionCallParams
async def get_current_weather(params: FunctionCallParams, location: str, format: str):
"""Get the current weather.
Args:
location: The city and state, e.g. "San Francisco, CA".
format: The temperature unit to use. Must be either "celsius" or "fahrenheit". Infer this from the user's location.
"""
weather_data = {"conditions": "sunny", "temperature": "75"}
await params.result_callback(weather_data)
```
The direct-function schema generator doesn't yet map `Literal` types to a
JSON-schema `enum`. Express enum-like constraints in the docstring prose
instead (e.g. _'Must be either "celsius" or "fahrenheit"'_), as shown above.
If you need a strict `enum` in the schema, use the [verbose
`FunctionSchema`](#advanced-defining-tools-with-functionschema) pattern.
### 2. Add the tool to the context
List your direct functions in `LLMContext(tools=[...])`:
```python
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
context = LLMContext(tools=[get_current_weather, get_restaurant_recommendation])
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```
The bot's personality (e.g. "You are a helpful assistant") is set via [`system_instruction`](/pipecat/learn/context-management#using-system_instruction-recommended) in the LLM service's Settings, not as a context message. Tools are automatically converted to the correct format for your LLM provider through adapters.
### 3. Create the pipeline
Include your LLM service in the pipeline:
```python
# Create the pipeline
pipeline = Pipeline([
transport.input(), # Input from the transport
stt, # STT processing
user_aggregator, # User context aggregation
llm, # LLM processing
tts, # TTS processing
transport.output(), # Output to the transport
assistant_aggregator, # Assistant context aggregation
])
```
## Per-Tool Options with `@tool_options`
By default, a direct function is cancelled if the user interrupts, and it uses the LLM service's global timeout. To override either, decorate the function with `@tool_options`. The decorator only attaches call options β the schema is still auto-derived β so decorated functions can stay at module level.
```python
import asyncio
from pipecat.adapters.schemas.direct_function import tool_options
from pipecat.services.llm_service import FunctionCallParams
@tool_options(cancel_on_interruption=False, timeout_secs=30)
async def get_current_weather(params: FunctionCallParams, location: str, format: str):
"""Get the current weather.
Args:
location: The city and state, e.g. "San Francisco, CA".
format: The temperature unit to use. Must be either "celsius" or "fahrenheit". Infer this from the user's location.
"""
# Simulate a long-running API call.
await asyncio.sleep(20)
await params.result_callback({"conditions": "nice", "temperature": "75"})
```
**Options:**
- **`cancel_on_interruption`** (default `True`): When `True`, the call is cancelled if the user interrupts. When `False`, the call is treated as **asynchronous** β see below.
- **`timeout_secs`** (default `None`): Per-tool timeout in seconds. Overrides the global `function_call_timeout_secs` for this function. Use a longer timeout for slow operations (e.g. database queries) or a shorter one for quick lookups.
`@tool_options` also sets call options on the handler of a
[`FunctionSchema`](#advanced-defining-tools-with-functionschema) tool, not
just a direct function.
On an [`LLMWorker`](/api-reference/server/workers/llm-context-worker), mark
tool methods with `@tool` instead. It applies the same options _and_ marks the
method for automatic collection as one of the worker's own tools.
### Synchronous vs. asynchronous calls
With `cancel_on_interruption=True` (the default), the call is **synchronous**: the LLM waits for the result before generating its next response. This ensures the LLM has complete information before responding.
With `cancel_on_interruption=False`, the call is **asynchronous**: the LLM continues the conversation immediately without waiting. Once the result returns, it's injected back into the context as a developer message, triggering a new LLM inference at that point. This enables truly non-blocking calls where the conversation proceeds while the function runs in the background. Async calls can also send [intermediate updates](#intermediate-results-for-async-functions) before their final result.
#### Async function call cancellation
For async functions (`cancel_on_interruption=False`), you can also enable model-directed cancellation:
```python
llm = OpenAILLMService(
api_key="your-api-key",
enable_async_tool_cancellation=True,
)
```
When `enable_async_tool_cancellation=True` and at least one async function is available, Pipecat automatically adds the built-in `cancel_async_tool_call` tool and supporting system instructions. The LLM can call that tool to cancel a stale in-progress async function call β for example, when the user changes their request before a long-running lookup completes.
## Parallel and Multiple Tool Calls
When the LLM calls more than one tool in a single turn, two `LLMService` constructor options control how those calls run and when the LLM responds:
```python
llm = OpenAILLMService(
api_key="your-api-key",
run_in_parallel=True, # default
group_parallel_tools=True, # default
)
```
Whether multiple tool calls in one turn run in parallel or one after another.
When `True`, all tool calls in a batch are grouped so the LLM is triggered
**exactly once** after every call in the batch completes. When `False`, each
function call result triggers the LLM independently as it arrives.
If multiple tools firing in one turn produce **duplicate or repeated
responses** (the bot answers once per tool instead of once total), check that
`group_parallel_tools` is `True`. With it disabled, each result re-triggers
the LLM, so the model responds once for every tool call in the batch.
## Changing Tools Mid-Conversation
To change the set of tools the LLM can use during a session, push an `LLMSetToolsFrame`. Its `tools` field takes the same things as `LLMContext(tools=[...])` β a list of direct functions and/or `FunctionSchema` objects. Whatever you pass becomes the LLM's new tool set.
```python
from pipecat.frames.frames import LLMSetToolsFrame
from pipecat.processors.aggregators.llm_context import NOT_GIVEN
# Make get_current_weather the only tool the LLM can call
await worker.queue_frame(LLMSetToolsFrame(tools=[get_current_weather]))
# Clear all tools
await worker.queue_frame(LLMSetToolsFrame(tools=NOT_GIVEN))
```
## Tools Across Service Switches
When you use an [`LLMSwitcher`](/api-reference/server/utilities/service-switchers/llm-switcher) to swap LLM providers mid-session, the tools you list in `LLMContext(tools=[...])` are available on whichever provider is active. You define them once for the whole switcher.
```python
from pipecat.pipeline.llm_switcher import LLMSwitcher
llm_switcher = LLMSwitcher(llms=[llm_openai, llm_google])
# Tools in the context are available on whichever provider is active
context = LLMContext(tools=[get_current_weather, get_restaurant_recommendation])
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```
## Advanced: Defining Tools with `FunctionSchema`
Direct functions cover most cases. Reach for the verbose `FunctionSchema` pattern when you need explicit control over the schema β for example, a strict `enum` constraint (which the direct-function generator doesn't yet emit) β or when the tool's handler isn't shaped like a direct function.
A `FunctionSchema` spells out the tool's name, description, and parameters by hand. Pass the `handler` that runs when the LLM calls the tool as the schema's `handler`, then list the schema in `LLMContext(tools=[...])` β exactly as you would a direct function.
```python
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.services.llm_service import FunctionCallParams
async def fetch_weather_from_api(params: FunctionCallParams):
weather_data = {"conditions": "sunny", "temperature": "75"}
await params.result_callback(weather_data)
weather_function = FunctionSchema(
name="get_current_weather",
description="Get the current weather in a location",
properties={
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"format": {
"type": "string",
# A strict enum β the kind of explicit control a direct function can't yet express.
"enum": ["celsius", "fahrenheit"],
"description": "The temperature unit to use.",
},
},
required=["location", "format"],
handler=fetch_weather_from_api, # bundle the handler on the schema
)
# List the schema in the context, just like a direct function.
context = LLMContext(tools=[weather_function])
```
To override the handler's default call options, decorate it with [`@tool_options`](#per-tool-options-with-@tool_options) β the same decorator direct functions use, with the same [synchronous vs. asynchronous](#synchronous-vs-asynchronous-calls) semantics:
```python
from pipecat.adapters.schemas.direct_function import tool_options
@tool_options(cancel_on_interruption=False, timeout_secs=30)
async def fetch_weather_from_api(params: FunctionCallParams):
...
```
These schemas behave just like direct functions everywhere else in this guide β swap them mid-conversation with an `LLMSetToolsFrame`, and they keep working across an `LLMSwitcher`'s providers.
### Registering a handler manually
Bundling the handler on the schema (above) is the recommended approach. If you'd rather keep the handler separate, list a handler-free `FunctionSchema` in the context as usual and register its handler by name:
```python
# weather_function here is the same schema, just defined without handler=.
context = LLMContext(tools=[weather_function])
llm.register_function("get_current_weather", fetch_weather_from_api)
```
If the handler carries [`@tool_options`](#per-tool-options-with-@tool_options), `register_function` honors it the same way bundling does β or pass `cancel_on_interruption` / `timeout_secs` to `register_function` directly to override.
To remove the tool, un-advertise it with an [`LLMSetToolsFrame`](#changing-tools-mid-conversation); call `llm.unregister_function(...)` only afterward, since unregistering a still-advertised tool leaves the LLM able to call a handler that's no longer there.
This is uncommon β bundling keeps a tool and its handler together β but the option is there when you need to manage registration directly.
### Provider-Specific Custom Tools
For normal function calling, prefer `standard_tools` with `FunctionSchema` or direct functions so Pipecat can convert them to each provider's native format. When a provider has tools that don't fit Pipecat's standard function schema, add those provider-native definitions through `ToolsSchema.custom_tools`. These custom tools are passed only to the matching adapter and are appended to the converted standard tools.
```python OpenAI-family adapter
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema
# Standard function converted by Pipecat
weather_function = FunctionSchema(
name="get_current_weather",
description="Get the current weather",
properties={"location": {"type": "string"}},
required=["location"],
)
# Provider-native tool appended only for OpenAI-family adapters.
# This object must match the target OpenAI API you are using.
provider_tool = {"type": "tool_search"}
tools = ToolsSchema(
standard_tools=[weather_function],
custom_tools={AdapterType.OPENAI: [provider_tool]},
)
```
```python Gemini
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema
# Standard function converted by Pipecat
weather_function = FunctionSchema(
name="get_current_weather",
description="Get the current weather",
properties={"location": {"type": "string"}},
required=["location"],
)
# Provider-native tool appended only for Gemini adapters.
# This object must match the Gemini API you are using.
gemini_search_tool = {"google_search": {}}
tools = ToolsSchema(
standard_tools=[weather_function],
custom_tools={AdapterType.GEMINI: [gemini_search_tool]},
)
```
Raw provider-native tool lists are not the normal `LLMContext` path. Some
lower-level adapter code still preserves non-`ToolsSchema` tools for legacy or
direct provider-specific paths, but `LLMContext(tools=...)` validates tools as
a `ToolsSchema`. Use `custom_tools` as the provider-specific escape hatch
while staying in the universal context flow.
For normal callable functions, use direct functions or `FunctionSchema`
instead of provider-native function definitions. Today, `custom_tools` is
supported for OpenAI-family adapters and Gemini. Anthropic standard functions
should be represented with `FunctionSchema`.
## Function Handler Details
### FunctionCallParams
Every function handler receives a `FunctionCallParams` object containing all the information needed for execution:
```python
@dataclass
class FunctionCallParams:
function_name: str # Name of the called function
tool_call_id: str # Unique identifier for this call
arguments: Mapping[str, Any] # Arguments from the LLM
llm: LLMService # Reference to the LLM service
context: LLMContext # Current conversation context
result_callback: FunctionCallResultCallback # Return results here
app_resources: Any # Application-defined resources shared across tool calls
```
**Using the parameters:**
```python
async def example_function_handler(params: FunctionCallParams):
# Access function details
print(f"Called function: {params.function_name}")
print(f"Call ID: {params.tool_call_id}")
# Extract arguments
location = params.arguments["location"]
# Access LLM context for conversation history
messages = params.context.messages
# Access shared resources (database, API clients, etc.)
if params.app_resources:
db = params.app_resources.database
user_id = params.app_resources.current_user_id
# Use LLM service for additional operations
await params.llm.push_frame(TTSSpeakFrame("Looking up weather data..."))
# Return results
await params.result_callback({"conditions": "nice", "temperature": "75"})
```
See the [API reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.services.llm_service.html#pipecat.services.llm_service.FunctionCallParams) for complete details.
`params.tool_resources` is a deprecated alias for `params.app_resources`. Use
`app_resources` in new code.
### Handler Structure
Your function handler should:
1. Receive necessary arguments, either:
- From `params.arguments`
- Directly from function arguments, if using [direct functions](#1-define-a-tool)
2. Process data or call external services
3. Return results via `params.result_callback(result)`
```python Non-Direct Function
async def fetch_weather_from_api(params: FunctionCallParams):
try:
# Extract arguments
location = params.arguments.get("location")
format_type = params.arguments.get("format", "celsius")
# Call external API
api_result = await weather_api.get_weather(location, format_type)
# Return formatted result
await params.result_callback({
"location": location,
"temperature": api_result["temp"],
"conditions": api_result["conditions"],
"unit": format_type
})
except Exception as e:
# Handle errors
await params.result_callback({
"error": f"Failed to get weather: {str(e)}"
})
```
```python Direct Function
async def get_current_weather(params: FunctionCallParams, location: str, format: str):
"""Get the current weather.
Args:
location: The city and state, e.g. "San Francisco, CA".
format: The temperature unit to use. Must be either "celsius" or "fahrenheit".
"""
try:
# Call external API
api_result = await weather_api.get_weather(location, format)
# Return formatted result
await params.result_callback({
"location": location,
"temperature": api_result["temp"],
"conditions": api_result["conditions"],
"unit": format
})
except Exception as e:
# Handle errors
await params.result_callback({
"error": f"Failed to get weather: {str(e)}"
})
```
### Sharing Resources with app_resources
When function handlers need access to shared resources like database connections, API clients, or application state, you can pass them via `app_resources` when creating the `PipelineWorker`. These resources are then accessible in every function handler via `params.app_resources`.
```python
from dataclasses import dataclass
from pipecat.pipeline.worker import PipelineWorker
from pipecat.services.llm_service import FunctionCallParams
# Define your application resources
@dataclass
class AppResources:
database: DatabaseConnection
api_client: WeatherAPIClient
user_id: str
# Create your resources
resources = AppResources(
database=db_connection,
api_client=weather_client,
user_id="user-123"
)
# Pass resources to the pipeline worker
worker = PipelineWorker(
pipeline,
app_resources=resources
)
# Access resources in function handlers
async def query_user_preferences(params: FunctionCallParams):
# Access shared resources
db = params.app_resources.database
user_id = params.app_resources.user_id
# Query database with shared connection
prefs = await db.query("SELECT * FROM preferences WHERE user_id = ?", user_id)
await params.result_callback(prefs)
```
**Key points:**
- Resources are **passed by reference** β the caller retains their handle and can read mutations after the task finishes
- The framework **never copies or clears** the `app_resources` object
- All function handlers in the pipeline share the same `app_resources` instance
- Useful for database connections, API clients, caches, or any shared state
`PipelineWorker(tool_resources=...)` and `FunctionCallParams.tool_resources`
are deprecated aliases retained for compatibility. Prefer
`PipelineWorker(app_resources=...)` and `params.app_resources`.
## Advanced: Controlling Function Call Behavior
When returning results from a function handler, you can control how the LLM processes those results using a `FunctionCallResultProperties` object passed to the result callback.
### Properties
`FunctionCallResultProperties` provides fine-grained control over LLM execution:
```python
@dataclass
class FunctionCallResultProperties:
run_llm: bool | None = None # Whether to run LLM after this result
on_context_updated: Callable | None = None # Callback when context is updated
is_final: bool = True # Whether this is the final result
```
**Property options:**
- **`run_llm=True`**: Run LLM after function call (default behavior)
- **`run_llm=False`**: Don't run LLM after function call (useful for chained calls)
- **`on_context_updated`**: Async callback executed after the function result is added to context
- **`is_final=False`**: Treat this as an intermediate result for an async function call. Only use this for async functions (`cancel_on_interruption=False`)
Skip LLM execution (`run_llm=False`) when you have back-to-back function
calls. If you skip a completion, you must manually trigger one from the
context aggregator.
See the [API reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.frames.frames.html#pipecat.frames.frames.FunctionCallResultProperties) for complete details.
### Example Usage
```python
from pipecat.frames.frames import FunctionCallResultProperties
from pipecat.services.llm_service import FunctionCallParams
async def fetch_weather_from_api(params: FunctionCallParams):
# Fetch weather data
weather_data = {"conditions": "sunny", "temperature": "75"}
# Don't run LLM after this function call
properties = FunctionCallResultProperties(run_llm=False)
await params.result_callback(weather_data, properties=properties)
async def query_database(params: FunctionCallParams):
# Query database
results = await db.query(params.arguments["query"])
async def on_update():
await notify_system("Database query complete")
# Run LLM after function call and notify when context is updated
properties = FunctionCallResultProperties(
run_llm=True,
on_context_updated=on_update
)
await params.result_callback(results, properties=properties)
```
### Intermediate Results for Async Functions
Async function calls can send progress updates before their final result. Make the function async with `@tool_options(cancel_on_interruption=False)`, then call `params.result_callback(..., properties=FunctionCallResultProperties(is_final=False))` for each intermediate update. Finish with a normal `params.result_callback(...)`.
```python
from pipecat.adapters.schemas.direct_function import tool_options
from pipecat.frames.frames import FunctionCallResultProperties
from pipecat.services.llm_service import FunctionCallParams
@tool_options(cancel_on_interruption=False)
async def track_delivery(params: FunctionCallParams):
"""Track a delivery, reporting each status update until it arrives."""
await params.result_callback(
{"status": "picked_up"},
properties=FunctionCallResultProperties(is_final=False),
)
await params.result_callback(
{"status": "nearby"},
properties=FunctionCallResultProperties(is_final=False),
)
await params.result_callback({"status": "delivered"})
```
Intermediate results are injected into the LLM context as async-tool developer messages. They do not close the function call; the call remains in progress until the final result is sent.
## Key Takeaways
- **Function calling extends LLM capabilities** beyond training data to real-time information
- **Context integration is automatic** - function calls and results are stored in conversation history
- **Direct functions are the preferred approach** - one async function is both schema and handler; list it in `LLMContext(tools=[...])` or add it via `LLMSetToolsFrame` to make it available. When you need explicit schema control, use a `FunctionSchema` with its `handler` bundled in
- **Async function calls are opt-in** - set `cancel_on_interruption=False` for deferred results, intermediate updates, and optional async-tool cancellation
- **Pipeline integration is seamless** - functions work within your existing voice AI architecture
- **Advanced control available** - fine-tune LLM execution and monitor function call lifecycle
## What's Next
Now that you understand function calling, let's explore how to configure text-to-speech services to convert your LLM's responses (including function call results) into natural-sounding speech.
Learn how to configure speech synthesis in your voice AI pipeline
# Text to Speech
Source: https://docs.pipecat.ai/pipecat/learn/text-to-speech.md
Learn how to configure speech synthesis to convert text into natural-sounding audio in your voice AI pipeline
**Text to Speech (TTS)** services are responsible for converting text into natural-sounding speech audio. They receive text input from LLMs and other sources, then generate audio output that users can hear through their connected devices.
## Pipeline Placement
TTS processors must be positioned correctly in your pipeline to receive text and generate audio frames:
```python
pipeline = Pipeline([
transport.input(),
stt,
context_aggregator.user(),
llm, # Generates LLMTextFrames
tts, # Processes text β creates TTSAudioRawFrames
transport.output(), # Sends audio to user
context_aggregator.assistant(), # Processes TTSTextFrames for context
])
```
**Placement requirements:**
- **After LLM processing**: TTS needs `LLMTextFrame`s from language model responses
- **Before transport output**: Audio must be generated before sending to user
- **Before assistant context aggregator**: Ensures spoken text is captured in conversation history
### Frame Processing Flow
**TTS generates speech through two primary mechanisms:**
1. **Streamed LLM tokens** via `LLMTextFrame`s:
- By default, TTS aggregates streaming tokens into complete sentences before synthesis (`TextAggregationMode.SENTENCE`)
- Set `text_aggregation_mode=TextAggregationMode.TOKEN` to stream tokens directly for lower latency
- Audio bytes stream back and play immediately
- End-to-end latency often under 200ms
2. **Direct speech requests** via `TTSSpeakFrame`s:
- Bypasses LLM for immediate audio generation
- Optionally appends text to conversation context via `append_to_context` parameter
- Useful for developer messages, greetings, or injected speech
**Frame output:**
- `TTSAudioRawFrame`s: Raw audio data for playback
- `TTSTextFrame`s: Text that was actually spoken (for context updates)
- `TTSStartedFrame`/`TTSStoppedFrame`: Speech boundary markers
## Supported TTS Services
Pipecat supports a wide range of TTS providers with different capabilities and performance characteristics:
View the complete list of supported text-to-speech providers
### Service Categories
**WebSocket-Based Services (Recommended):**
- **Cartesia**: Ultra-low latency with word timestamps
- **ElevenLabs**: High-quality voices with emotion control
- **Rime**: Ultra-realistic voices with advanced features
**HTTP-Based Services:**
- **OpenAI TTS**: High-quality synthesis with multiple voices
- **Azure Speech**: Enterprise-grade with extensive language support
- **Google Text-to-Speech**: Reliable with WaveNet voices
**Advanced Features:**
- **Word timestamps**: Enable word-level accuracy for context and subtitles
- **Voice cloning**: Custom voice creation from samples
- **Emotion control**: Dynamic emotional expression
- **SSML support**: Fine-grained pronunciation control
WebSocket services typically provide the lowest latency, while HTTP services
may have intermittent higher latency due to their request/response nature.
## TTS Configuration
### Service-Specific Configuration
Each TTS service has its own configuration options. Here's an example with Cartesia:
```python
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.transcriptions.language import Language
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
settings=CartesiaTTSService.Settings(
model="sonic-3.5",
voice="voice-id-here",
language=Language.EN, # Speech language
),
# Word timestamps automatically enabled for precise context updates
)
```
**Word timestamps:** Services like Cartesia, ElevenLabs, and Rime provide word-level timestamps that enable precise context updates during interruptions and better synchronization with other pipeline components. For example, if an interruption occurs while the bot is speaking, the word timestamps allow you to accurately capture which words were spoken up to that point, enabling better context management and user experience. Additionally, transcription events streamed from server to client can be done in sync with the audio output, allowing for real-time subtitles or captions.
Explore configuration options for each supported TTS provider
### Pipeline-Level Audio Configuration
Set consistent audio settings across your entire pipeline:
```python
worker = PipelineWorker(
pipeline,
params=PipelineParams(
audio_in_sample_rate=16000, # Input audio quality
audio_out_sample_rate=24000, # Output audio quality (TTS)
),
)
```
Set the `audio_out_sample_rate` to match your TTS service's requirements for
optimal quality. This is preferred to setting the sample_rate directly in the
TTS service as the PipelineParam ensures that all output sample_rates match.
## Text Processing and Filtering
### Custom Text Aggregation
By default, TTS services have a built-in text aggregator that collects streaming text into sentences before passing them to the underlying service. However, you can customize this behavior by inserting an [`LLMTextProcessor`](/api-reference/server/utilities/frame/llm-text-processor) with a different text aggregator before the TTS in your pipeline. This allows the ability to categorize and structure text into logical units beyond simple sentences, such as code blocks, URLs, or custom tags. You can then configure the TTS to handle these different text types appropriately, such as skipping code blocks or transforming them in a just-in-time manner before speaking.
### Skipping Text Aggregations
To skip certain text aggregations (e.g., code snippets or URLs) and keep them from being spoken, use a custom text aggregator like [`PatternPairAggregator`](/api-reference/server/utilities/text/pattern-pair-aggregator) within an [`LLMTextProcessor`](/api-reference/server/utilities/frame/llm-text-processor), and configure it to identify and handle specific patterns in the text stream. With this, you can then pass any aggregated types you want to skip (like "code") to the TTS service's `skip_aggregator_types` parameter.
```python
# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()
# Add pattern for JSON data
pattern_aggregator.add_pattern(
type="code",
start_pattern="",
end_pattern="",
action=MatchAction.AGGREGATE
)
# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)
# Initialize TTS service, and don't speak JSON data
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
skip_aggregator_types=["code"], # The strings here should match the types defined in the PatternPairAggregator
)
# add the llm_text_processor to your pipeline after the llm and before the tts
# llm -> llm_text_processor -> tts
```
### Text Transforms
For TTS-specific text preprocessing, you can provide custom text transforms that modify text in a just-in-time manner before sending the text off to the TTS service. This is useful for handling special text segments that need to be altered for better pronunciation or clarity, such as spelling out phone numbers, removing URLs, or expanding abbreviations. These text transforms can be mapped to a specific text aggregation type, like with `skip_aggregator_types`, or applied globally to all text using `'*'` as the type.
Text transforms are registered directly on the TTS service instance via the `add_text_transformer()` method or during initialization using the `text_transforms` parameter.
The intentions of text transforms are meant to be TTS-specific modifications
that do not affect the underlying LLM text or context. That said, since the
context aggregator attempts to base its context on what was actually spoken,
for services that support word timestamps, like Cartesia, ElevenLabs, and
Rime,these transforms will modify the context as they modify what is spoken.
```python
# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()
# Add patterns for different parts of an explanation
pattern_aggregator.add_pattern(
type="phone_number",
start_pattern="",
end_pattern="",
action=MatchAction.AGGREGATE
)
# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)
# Text-to-Speech service
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
)
# Text transformers for TTS
# This will insert Cartesia's spell tags around the provided text.
async def spell_out_text(text: str, type: str) -> str:
# CartesiaTTSService provides a helper for this along with other common transforms
return CartesiaTTSService.SPELL(text)
async def replace_acronyms(text: str, type: str) -> str:
# Replace "SEC" with "Southeastern Conference"
return text.replace(" SEC ", " Southeastern Conference ")
# Setup the text transformers in TTS to spell out phone numbers and replace
# acronyms. The string below matches the type defined in the PatternPairAggregator
# above so that whenever those segments are encountered, this transform
# is applied
tts.add_text_transformer(spell_out_text, "phone_number")
tts.add_text_transformer(replace_acronyms, "*") # Apply to all text
# add the llm_text_processor to your pipeline after the llm and before the tts
# llm -> llm_text_processor -> tts
```
### Text Filters
Text filters are no longer the preferred method for text preprocessing and
will be deprecated in future releases. Instead, you should use one of the
methods described above.
Apply preprocessing to text before synthesis:
```python
from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter
tts = YourTTSService(
# ... other options
text_filters=[
MarkdownTextFilter(), # Remove markdown formatting
CustomTextFilter(), # Your custom processing
],
)
```
**Common filters:**
- **MarkdownTextFilter**: Strips markdown formatting from LLM responses
- **Custom filters**: Implement your own text preprocessing logic
## Skipping TTS Output
Sometimes you want text from the LLM to flow through the pipelineβupdating the conversation context, reaching observers, or being processed by custom frame processorsβwithout being spoken by the TTS service. Pipecat provides a `skip_tts` attribute on text and response frames for this purpose.
When `skip_tts` is `True` on a frame, the TTS service passes it through without generating audio, but the text still reaches downstream processors like the assistant context aggregator.
### Configuring All LLM Output
Use `LLMConfigureOutputFrame` to tell the LLM service to mark **all** subsequent output frames (`LLMTextFrame`, `LLMFullResponseStartFrame`, `LLMFullResponseEndFrame`) with `skip_tts`:
```python
from pipecat.frames.frames import LLMConfigureOutputFrame
# Tell the LLM to skip TTS for all output
await worker.queue_frame(LLMConfigureOutputFrame(skip_tts=True))
# ... LLM responses will not be spoken ...
# Re-enable TTS
await worker.queue_frame(LLMConfigureOutputFrame(skip_tts=False))
```
This is useful when you want to toggle TTS on or off for an entire stretch of conversation, such as switching between voice and text input modes.
### Setting skip_tts on Individual Frames
For more granular control, set `skip_tts=True` directly on individual text frames. This is useful when building custom frame processors that selectively silence certain parts of the LLM output:
```python
from pipecat.frames.frames import LLMTextFrame
# In a custom frame processor
frame = LLMTextFrame(text)
frame.skip_tts = True
await self.push_frame(frame)
```
The `skip_tts` attribute is available on `TextFrame` and all its subclasses
(`LLMTextFrame`, `AggregatedTextFrame`, `TTSTextFrame`, etc.), as well as
`LLMFullResponseStartFrame` and `LLMFullResponseEndFrame`.
### Common Use Cases
**Encoding structured output from the LLM.** You can instruct the LLM to include markers or metadata in its response that should be processed by pipeline logic but not spoken. For example, Pipecat's [turn completion detection](/api-reference/server/utilities/turn-management/filter-incomplete-turns) uses this approach β the LLM outputs completion markers (`β`, `β`, `β`) that are pushed with `skip_tts=True` so they update the context but aren't spoken.
**Switching between voice and text input.** When a client sends text input instead of speech, you may want the bot to respond with text only. The client SDKs support this via `sendText()` with `audio_response: false`, which uses `LLMConfigureOutputFrame` internally to temporarily disable TTS for that response.
**Testing without audio.** When building test pipelines, you can use `LLMConfigureOutputFrame(skip_tts=True)` to bypass audio generation entirely while still exercising the rest of the pipeline.
## Advanced TTS Features
### Direct Speech Commands
Use `TTSSpeakFrame` for immediate speech synthesis:
```python
from pipecat.frames.frames import TTSSpeakFrame
# Make bot speak directly (added to context by default)
await tts.queue_frame(TTSSpeakFrame("Hello, how can I help you?"))
# Explicitly append spoken text to conversation context
await tts.queue_frame(
TTSSpeakFrame("Welcome! Let's begin.", append_to_context=True)
)
# Speak without adding to context
await tts.queue_frame(
TTSSpeakFrame("Processing...", append_to_context=False)
)
```
The `append_to_context` parameter controls whether the spoken text is added to the conversation history. When `append_to_context=True`, the text is automatically committed to the context after being spoken, making it useful for bot greetings and injected speech that should be part of the conversation flow.
As of Pipecat v1.4.0, `append_to_context` defaults to `True`. A plain
`TTSSpeakFrame("...")` **is** added to the conversation context after it is
spoken; pass `append_to_context=False` to speak without recording it. (`None`
was the previous default and is no longer supported.)
### Dynamic Settings Updates
Update TTS settings during conversation using typed settings objects:
```python
from pipecat.frames.frames import TTSUpdateSettingsFrame
from pipecat.services.cartesia.tts import CartesiaTTSSettings
# Change voice speed during conversation
await worker.queue_frames([
TTSUpdateSettingsFrame(delta=CartesiaTTSSettings(speed="fast")),
TTSSpeakFrame("I'm speaking faster now!")
])
```
## Key Takeaways
- **Pipeline placement matters** - TTS must come after LLM, before transport output
- **Service types differ** - WebSocket services provide lower latency than HTTP
- **Text processing affects quality** - use aggregation and filters for better results
- **Word timestamps enable precision** - better interruption handling and context accuracy
- **Configuration impacts performance** - balance quality, latency, and bandwidth needs
- **Services are modular** - easily swap providers without changing pipeline code
## What's Next
You've now learned how to build a complete voice AI pipeline! Let's explore some additional topics to enhance your implementation.
Learn how to terminate your voice AI pipeline at the end of a conversation
# Pipeline Termination
Source: https://docs.pipecat.ai/pipecat/learn/pipeline-termination.md
Learn how to properly terminate Pipecat pipelines for clean shutdown and resource management
**Pipeline termination** ensures your voice AI applications shut down cleanly without resource leaks or hanging processes. Understanding the different termination methods helps you handle various scenarios from natural conversation endings to unexpected disconnections.
## Pipeline Integration
Pipeline termination works through the same frame-based system as other pipeline operations:
```python
pipeline = Pipeline([
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
])
# EndFrame or CancelFrame flows through entire pipeline for shutdown
```
**Termination frames:**
- **`EndFrame`**: A queued ControlFrame that triggers graceful shutdown after processing pending frames
- **`CancelFrame`**: A SystemFrame that triggers immediate shutdown, discarding pending frames
Both frames flow downstream through the pipeline, allowing each processor to clean up resources appropriately.
### Termination frames at a glance
| Frame | Job | Push from / direction |
| ------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `EndFrame` | Graceful shutdown (drains pending frames) | From outside the pipeline via `worker.queue_frame(EndFrame())` |
| `CancelFrame` | Immediate shutdown (discards pending frames) | Via `worker.cancel()` |
| `EndWorkerFrame` | Graceful shutdown signal from inside the pipeline | `push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM)`; the source converts it to an `EndFrame` |
| `CancelWorkerFrame` | Immediate shutdown signal from inside the pipeline | `push_frame(CancelWorkerFrame(), FrameDirection.DOWNSTREAM)`; the source converts it to a `CancelFrame` |
If you see `EndTaskFrame`, `CancelTaskFrame`, `PipelineTask`, or
`task.cancel()` in older code or examples, those are **deprecated aliases**
(since 1.3.0/1.4.0) of `EndWorkerFrame`, `CancelWorkerFrame`,
`PipelineWorker`, and `worker.cancel()`. They still work but will be removed
in 2.0.0. Use the `Worker` names in new code.
## Termination Methods
Pipecat provides two primary approaches for pipeline termination, each designed for different scenarios:
### 1. Graceful Termination
Graceful termination allows the bot to complete its current processing before shutting down. This is ideal when you want the bot to properly end a conversation. For example, after completing a specific task or reaching a natural conclusion.
**When to use:**
- Natural conversation endings
- Task completion scenarios
- When the bot should say goodbye
**Implementation options:**
Push an `EndFrame` from outside your pipeline:
```python
# From outside the pipeline
from pipecat.frames.frames import EndFrame
await worker.queue_frame(EndFrame())
```
Push an `EndWorkerFrame` downstream from inside your pipeline. Here `end_conversation` is a **direct function** β an `async` function whose first parameter is `params: FunctionCallParams`, with a Google-style docstring that becomes the tool description. Register it by passing the function itself in your tools list:
```python
from pipecat.frames.frames import EndWorkerFrame, TTSSpeakFrame
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.llm_service import FunctionCallParams
async def end_conversation(params: FunctionCallParams):
"""End the conversation and shut down the bot.
Call this when the user says goodbye or the task is complete.
"""
await params.llm.push_frame(TTSSpeakFrame("Have a nice day!"))
# Resolve the function call so the LLM call doesn't hang
await params.result_callback({"status": "ended"})
# Signal that the worker should end after processing this frame
await params.llm.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM)
# Pass the function directly in the tools list; it's registered automatically
context = LLMContext(tools=[end_conversation])
```
Always call `params.result_callback(...)` in your handler before pushing the
end frame. Skipping it can leave the LLM function call unresolved. If you
don't want the LLM to respond, you can provide `None` as the result.
**How graceful termination works:**
1. `EndFrame` is queued and processes after any pending frames (like goodbye messages)
2. All processors shutdown when they receive the `EndFrame`
3. Once the `EndFrame` reaches the end of the pipeline, shutdown is complete
4. Resources are cleaned up and the process terminates
Graceful termination allows your bot to say goodbye and complete any final
actions before terminating.
### 2. Immediate Termination
Immediate termination cancels the pipeline without waiting for pending frames to complete. This is appropriate when the user is no longer active in the conversation.
**When to use:**
- User disconnections (browser closed, call ended)
- Error conditions requiring immediate shutdown
- When completing the conversation is no longer necessary
**Implementation:**
Use event handlers to detect disconnections and trigger cancellation:
```python
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
logger.info("Client disconnected - terminating pipeline")
await worker.cancel()
```
**How immediate termination works:**
1. An event triggers the cancellation (like client disconnection)
2. `worker.cancel()` pushes a `CancelFrame` downstream from the PipelineWorker
3. `CancelFrame`s are `SystemFrame`s, so they use the high-priority input queue and are processed before queued non-system frames
4. Processors handle the `CancelFrame` and shut down without waiting for pending non-system work to drain
5. Any pending frames are discarded during shutdown
Immediate termination will discard any pending frames in the pipeline. Use
this approach when completing the conversation is no longer necessary.
## Automatic Termination
### Pipeline Idle Detection
Pipecat includes automatic idle detection to prevent hanging pipelines. This feature monitors activity and can automatically cancel tasks when no meaningful bot interactions occur for an extended period.
**How it works:**
- Monitors pipeline activity for meaningful bot interactions
- Automatically triggers termination after configured idle timeout
- Serves as a safety net for anomalous behavior or forgotten sessions
**Configuration:**
```python
worker = PipelineWorker(
pipeline,
# Configure idle detection timeout
cancel_on_idle_timeout=True, # Default is True
idle_timeout_secs=600, # Default is 300 seconds
idle_timeout_frames=(BotSpeakingFrame,), # Default is (BotSpeakingFrame, UserSpeakingFrame)
)
```
You can further configure the idle detection behavior. To learn more, refer to the Pipeline Idle Detection documentation:
Learn how to configure and customize idle detection for your use case
Pipeline Idle Detection is enabled by default and helps prevent resources from
being wasted on inactive conversations.
### Maximum Call Duration
Idle detection ends a call that goes quiet, but it does not cap the total length of an active call. To enforce a maximum call duration, run an `asyncio` timer that speaks a goodbye and then queues an `EndFrame` so the bot can sign off gracefully before shutdown:
```python
import asyncio
from pipecat.frames.frames import EndFrame, TTSSpeakFrame
async def end_after(worker, timeout_secs: float):
await asyncio.sleep(timeout_secs)
await worker.queue_frame(TTSSpeakFrame("We've reached our time limit. Goodbye!"))
await worker.queue_frame(EndFrame()) # Graceful: plays the goodbye, then shuts down
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
asyncio.create_task(end_after(worker, timeout_secs=300)) # 5-minute cap
```
On Pipecat Cloud, there is also a platform-level hard cap via
[`maxSessionDuration`](/pipecat-cloud/fundamentals/active-sessions#session-duration-limits)
(default 7200s). That cap cancels the bot and shuts the pipeline down with no
goodbye, so use the bot-level timer above when you want the bot to speak
before the call ends.
## Implementation Patterns
### Event-Driven Termination
Connect termination to transport events for automatic cleanup:
```python
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
logger.info("Client connected - starting conversation")
await worker.queue_frames([LLMRunFrame()])
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
logger.info("Client disconnected - immediate termination")
await worker.cancel()
# Run the pipeline
runner = WorkerRunner(handle_sigint=False)
await runner.run(worker)
```
### Conditional Termination
Use function calling or other logic to determine when conversations should end:
```python
async def check_conversation_complete(params: FunctionCallParams):
# Your logic to determine if conversation should end
conversation_complete = await evaluate_completion_criteria()
if conversation_complete:
await params.llm.push_frame(TTSSpeakFrame("Thank you for using our service!"))
await params.llm.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM)
await params.result_callback({"status": "complete" if conversation_complete else "continuing"})
```
### Error Handling
Ensure pipelines can terminate properly even when exceptions occur:
```python
try:
runner = WorkerRunner(handle_sigint=False)
await runner.run(worker)
except Exception as e:
logger.error(f"Pipeline error: {e}")
# Ensure cleanup happens even on errors
await worker.cancel()
```
### Running Cleanup Code on Shutdown
To run cleanup or persist data when a call ends, use the `on_pipeline_finished` event handler. It fires after the pipeline reaches any terminal state, so it runs for **both** graceful (`EndFrame`) and cancelled (`CancelFrame`) shutdowns. This makes it the single write point for end-of-call work like saving a transcript or recording:
```python
@worker.event_handler("on_pipeline_finished")
async def on_pipeline_finished(worker, frame):
# Runs for both graceful and cancelled shutdown
await save_transcript_to_db()
```
`on_client_disconnected`, by contrast, fires only when the client disconnects. Use it to _tag the reason_ for the shutdown (for example, "user hung up"), and do the actual persistence in `on_pipeline_finished` so you write data exactly once regardless of how the call ended. See the [`PipelineWorker` events](/api-reference/server/pipeline/pipeline-worker) reference for the full event signature.
## Troubleshooting
If your pipeline isn't shutting down properly, check these common issues:
### Custom Processors Not Propagating Frames
**Problem:** Custom processors that don't call `push_frame()` can block termination frames from reaching the end of the pipeline.
**Solution:** Ensure your custom processors propagate all frames downstream, including `EndFrame` and `CancelFrame`:
```python
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
# Your custom processing logic here
# Always push frames downstream (including termination frames)
await self.push_frame(frame, direction)
```
### Incorrect Termination Frame Direction
**Problem:** Pushing `EndFrame` or `CancelFrame` from the middle of the pipeline may not reach the pipeline source properly.
**Solution:** Use the appropriate frame type and direction:
```python
await self.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM)
await self.push_frame(CancelWorkerFrame(), FrameDirection.DOWNSTREAM)
# The pipeline source will then convert these to proper termination frames
# and push them downstream through the entire pipeline
```
The pipeline source automatically converts `EndWorkerFrame` to `EndFrame` and
`CancelWorkerFrame` to `CancelFrame` when pushing downstream, ensuring proper
termination handling throughout the pipeline.
### "dangling tasks detected" Warning
**Problem:** On shutdown you see a log warning like `PipelineWorker#0 dangling tasks detected: [...]`.
**Solution:** This means `asyncio` tasks created during the session were never cancelled or awaited before the pipeline shut down. The usual causes are the two above: a custom processor that doesn't propagate termination frames, or a background task started inside a processor (for example, a timer or long-running coroutine) that isn't cleaned up. Create background tasks through the pipeline task manager so they are tracked and cancelled on shutdown, and make sure your processors push `EndFrame`/`CancelFrame` downstream.
## Key Takeaways
- **Frame-based termination** - shutdown uses the same frame system as processing
- **Choose the right method** - graceful for natural endings, immediate for disconnections
- **Event handlers enable automatic termination** - respond to user disconnections cleanly
- **Idle detection provides safety net** - prevents hanging processes and resource waste
- **SystemFrames have priority** - CancelFrames are processed before queued non-system frames for fast shutdown
- **Resource cleanup is automatic** - proper termination ensures clean resource disposal
## What's Next
You now understand how to build, run, and properly terminate voice AI pipelines! With the single-agent basics covered, let's see how Pipecat coordinates multiple agents, starting with giving an agent its own LLM and tools.
Give an agent its own LLM and register tools with the @tool decorator
# Multiple LLM Agents
Source: https://docs.pipecat.ai/pipecat/learn/multiple-llm-agents.md
Run separate agents that each own their own LLM, tools, and conversation context.
So far you've built a single agent: one LLM, one context, one set of tools. But many problems are better served by **several agents working together**, each owning its own LLM. A greeter hands off to a support agent. A researcher runs in the background while the main agent keeps talking. A screen-driving agent acts on the UI while a voice agent converses.
Giving each agent its own LLM keeps every context small and focused. Instead of one model juggling every instruction and tool, each agent reasons over just its own job. That's cheaper, faster, and less prone to the model getting lost.
## LLMWorker overview
Multi-agent systems often run several LLM-backed agents -- a greeter, a support agent, a researcher -- each with its own instructions, tools, and (optionally) its own conversation context. `LLMWorker` is the building block for each one. It extends `PipelineWorker` with everything you need to run an LLM-powered agent:
- A pipeline with your LLM service, automatically built
- Tool registration via the `@tool` decorator
- Activation handling that injects messages and runs the LLM
To create an LLM agent, subclass `LLMWorker` so you can host `@tool` methods, then instantiate it with its own LLM service. Pass `bridged=()` so the agent receives frames from the bus:
```python
import os
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.workers.llm import LLMWorker
class MyAgent(LLMWorker):
"""An LLM agent. ``@tool`` methods go here."""
def build_agent() -> MyAgent:
llm = OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
settings=OpenAILLMService.Settings(
system_instruction="You are a helpful assistant.",
),
)
return MyAgent("assistant", llm=llm, bridged=())
```
You never pass a `bus=` argument to the constructor -- the worker gets its bus when you register it with `runner.add_workers(...)`. The default pipeline is `Pipeline([llm])`, with tools from `build_tools()` automatically registered. When `bridged=()` is set, the framework wraps this pipeline with edge processors that connect it to the bus.
## The @tool decorator
The `@tool` decorator marks a method as an LLM-callable tool. The framework automatically collects all `@tool`-decorated methods and registers them with the LLM service.
```python
from pipecat.services.llm_service import FunctionCallParams
from pipecat.workers.llm import tool
class MyAgent(LLMWorker):
@tool
async def get_weather(self, params: FunctionCallParams, city: str):
"""Get the current weather for a city.
Args:
city (str): The city name (e.g. 'San Francisco').
"""
weather = await fetch_weather(city)
await params.result_callback(weather)
```
The tool's name comes from the method name. The docstring becomes the tool description. Parameter types and descriptions are extracted from the type annotations and the `Args` section in the docstring.
### Tool options
The `@tool` decorator accepts options:
```python
@tool(cancel_on_interruption=False, timeout=60)
async def long_running_tool(self, params: FunctionCallParams, query: str):
"""A tool that takes a while.
Args:
query (str): The search query.
"""
result = await expensive_search(query)
await params.result_callback(result)
```
| Option | Default | Description |
| ------------------------ | ------- | -------------------------------------- |
| `cancel_on_interruption` | `True` | Cancel the tool if the user interrupts |
| `timeout` | `None` | Maximum execution time in seconds |
### Tool parameters
Every tool method receives `self` and `params: FunctionCallParams` as the first two arguments. Additional arguments are the tool's parameters that the LLM fills in.
The `params` object gives you access to:
- `params.result_callback(result)` -- return the result to the LLM
- `params.llm` -- the LLM service instance, useful for queuing frames
### Returning results
Always call `params.result_callback()` to return the tool result to the LLM:
```python
@tool
async def lookup(self, params: FunctionCallParams, item: str):
"""Look up an item.
Args:
item (str): The item to look up.
"""
data = await database.get(item)
await params.result_callback({"found": True, "data": data})
```
## Activation with messages
When an `LLMWorker` is activated, you can inject messages into its context. Pass an `LLMWorkerActivationArgs` via the `args` parameter:
```python
from pipecat.workers.llm import LLMWorkerActivationArgs
await self.activate_worker(
"support",
args=LLMWorkerActivationArgs(
messages=[{"role": "developer", "content": "The user asked about pricing."}],
run_llm=True, # Run the LLM immediately after injection
),
)
```
The default `on_activated()` implementation:
1. Sets the tools from `build_tools()`
2. Injects the provided messages into the LLM context
3. Runs the LLM if `run_llm` is `True` (the default when `messages` is set)
## Managing context with LLMContextWorker
A plain `LLMWorker` runs an LLM but doesn't manage conversation context on its own -- it relies on context coming from elsewhere (for example, the main agent's aggregators bridged in). When an agent needs to keep its **own** history, use `LLMContextWorker`. It extends `LLMWorker` with a built-in `LLMContext` and the user/assistant aggregator pair, building the pipeline as `[user_aggregator, llm, assistant_aggregator]` for you.
```python
from pipecat.workers.llm import LLMContextWorker
class AssistantAgent(LLMContextWorker):
"""An LLM agent that keeps its own conversation context."""
agent = AssistantAgent("assistant", llm=llm) # gets its own context
```
Each `LLMContextWorker` gets its own context by default, so agents don't see each other's history. To give several agents a **shared** conversation, pass the same `context=` to each:
```python
from pipecat.processors.aggregators.llm_context import LLMContext
shared = LLMContext()
agent_a = AssistantAgent("agent_a", llm=llm_a, context=shared)
agent_b = AssistantAgent("agent_b", llm=llm_b, context=shared)
```
Access the managed aggregators via `self.user_aggregator` and `self.assistant_aggregator`.
## Custom pipelines
If you need more control, you can pass a custom `pipeline=` to the `LLMWorker` constructor. For example, to add TTS to the agent's own pipeline:
```python
from pipecat.pipeline.pipeline import Pipeline
from pipecat.services.cartesia.tts import CartesiaTTSService
class AgentWithTTS(LLMWorker):
def __init__(self, name: str, *, llm, voice_id: str):
tts = CartesiaTTSService(
api_key=os.environ["CARTESIA_API_KEY"],
settings=CartesiaTTSService.Settings(voice=voice_id),
)
super().__init__(name, llm=llm, pipeline=Pipeline([llm, tts]), bridged=())
```
This is how you give each agent its own voice -- each agent adds its own TTS after the LLM, so a handoff sounds like a real transfer between distinct speakers.
## What's next
Now that your agents can run LLMs and call tools, here's a powerful one: an agent that sees and drives the user's screen.
A UIWorker that reads the screen and acts on it over a two-way RTVI interface
# Controlling the UI
Source: https://docs.pipecat.ai/pipecat/learn/ui-worker.md
Bridge a voice agent and a client GUI with a UIWorker over a two-way RTVI interface.
## What is a UIWorker?
When you put a voice agent in front of an app, talking isn't enough β the agent needs to _see what the user sees_ and _act on the screen_: read the page, point at things, fill in fields, click buttons. A `UIWorker` is the server-side agent that makes this possible. It voice-enables a client UI by connecting an LLM to whatever the user is looking at.
The connection is **two-way**, over the RTVI UI channel:
- **Client β server.** The client streams the screen to the worker as accessibility snapshots, and forwards the user's UI interactions as events.
- **Server β client.** The worker drives the page back β scrolling, highlighting, selecting text, filling inputs, clicking, or running app-defined commands β and can surface long-running work as progress cards.
A `UIWorker` is the screen half of a voice/UI split: a **voice agent** owns the conversation, and the `UIWorker` owns the screen. Each is a separate LLM with its own focused context. The worker auto-injects the latest screen state into _its_ context before every turn, so the conversational voice LLM never has to carry a giant accessibility tree:
- The **voice agent** converses and decides what's worth saying.
- The **UIWorker** reasons over the current page and acts on it.
The result is two small, fast contexts instead of one bloated one β cheaper, and less prone to the model getting lost.
The two directions map to RTVI UI messages: the client sends `ui-snapshot` and
`ui-event`; the worker sends `ui-command` and `ui-job-group`. You rarely touch
these directly β `PipelineWorker` wires the channel up automatically when RTVI
is enabled (the default). See [The RTVI
Standard](/client/rtvi-standard#user-interface) for the wire protocol and the
[UIWorker API reference](/api-reference/server/workers/ui-worker) for the
class.
## The two-way interface
A `UIWorker` gives an LLM a handful of capabilities, split across the two directions of the interface. Everything below works out of the box on any `UIWorker` subclass.
### What the worker sees (client β server)
**The screen, as a snapshot.** The client sends an accessibility snapshot of the page whenever it changes. The worker renders the latest one as a `` block and β with `auto_inject_ui_state` on (the default) β injects it into the LLM context before every turn, so the model always reasons over what's currently on screen. Each element carries a stable `ref` the worker uses to act on it:
```
- heading "Shopping list" [level=1] [ref=e3]
- list:
- checkbox "milk" [checked] [ref=e5]
- checkbox "eggs" [ref=e6]
```
When the user has text selected, the snapshot includes a `` block so the LLM can resolve deictic references like "this paragraph" or "what I selected".
**User interactions, as events.** The client dispatches app-defined events (a button click, a custom gesture) with `sendUIEvent(name, payload)`. Route them to handlers with `@ui_event(name)`; each runs in its own task:
```python
from pipecat.workers.ui import UIWorker, ui_event
class MyUIWorker(UIWorker):
@ui_event("note_click")
async def on_note_click(self, message):
ref = (message.payload or {}).get("ref")
await self.scroll_to(ref)
await self.select_text(ref)
```
### What the worker does (server β client)
**Drives the page.** The worker acts on the screen by sending UI commands. The built-in helpers cover the common actions, and `send_command(name, payload)` sends any app-defined command:
| Helper | Effect |
| ----------------------------- | -------------------------------------------- |
| `scroll_to(ref)` | Bring an element into view |
| `highlight(ref)` | Briefly flash an element |
| `select_text(ref)` | Select an element's text (pointing / deixis) |
| `click(ref)` | Click a checkbox, radio, or button |
| `set_input_value(ref, value)` | Fill a text input or textarea |
| `send_command(name, payload)` | Any app-defined command (e.g. `"add_note"`) |
The standard client handlers ship in `@pipecat-ai/client-react`; apps can override them or define their own command names.
**Answers back.** A `UIWorker` answers via a built-in single-flight `respond` job: a requester dispatches `job("ui", name="respond", payload={"query": ...})`, the worker runs one screen-grounded LLM turn, and a `@tool` ends it by calling `respond_to_job()`. That call chooses how the answer reaches the user:
- `respond_to_job(text, tts_speak=True)` β speak `text` verbatim through the requester's TTS.
- `respond_to_job(text)` β return `{"answer": text}` for the requester's voice LLM to phrase.
- `respond_to_job()` β complete the turn silently (the worker acted, but said nothing).
**Surfaces long work.** When a turn kicks off background work, `ui_job_group` / `start_ui_job_group` fan it out to peer workers _and_ surface it to the client as a cancellable progress card, streaming each worker's updates as they arrive:
```python
await self.start_ui_job_group(
"wikipedia", "news", "scholar",
payload={"query": research_query},
label=f"Research: {research_query}",
)
```
By default a `UIWorker` is stateless: it clears its context at the start of each
`respond` job, so every turn sees only the current `` and query. Set
`keep_history=True` to accumulate history across turns β useful for multi-turn
references like "can we add a note for that?" β at the cost of more tokens.
## Hello world
The smallest `UIWorker` ties the interface together: a delegate that answers questions about the page. The voice agent forwards screen-relevant utterances to it; the worker reads the screen (client β server) and speaks the reply (server β client).
The worker needs only an LLM and one `@tool` that ends the turn with `respond_to_job()`:
```python
from pipecat.workers.ui import UIWorker
from pipecat.workers.llm import tool
class HelloWorker(UIWorker):
@tool
async def answer(self, params, text: str):
"""Speak `text` back to the user."""
await self.respond_to_job(text, tts_speak=True)
await params.result_callback(None)
```
The voice agent exposes a tool that dispatches a `respond` job to the worker and speaks back whatever it returns:
```python
async def answer_about_screen(params, query: str):
"""Ask the screen-aware UI layer to answer about the current page."""
async with params.pipeline_worker.job(
"hello", name="respond", payload={"query": query}, timeout=30
) as t:
pass
await params.result_callback(t.response)
```
Register both with the runner β the `UIWorker` comes online to receive snapshots and jobs as soon as its pipeline starts:
```python
await runner.add_workers(HelloWorker(), worker)
```
Here's the full round trip for one utterance:
The client streams the current screen as a `ui-snapshot`. `PipelineWorker`
broadcasts it on the bus; the `UIWorker` stores the latest one.
The user speaks. The voice LLM calls `answer_about_screen`, which dispatches
a `respond` job to the `UIWorker`.
The worker's `respond` job runs one LLM turn with the latest ``
auto-injected, so its answer is grounded in what's on screen.
The worker's `answer` tool calls `respond_to_job(text, tts_speak=True)`. The
voice agent speaks the reply verbatim.
## Patterns
How the worker gets triggered β and who speaks β falls into two patterns.
### Delegation
In the hello-world example, the voice agent is the gatekeeper: it decides which turns involve the screen and routes those to the `UIWorker`, then voices the result. This is the **delegation** pattern, and it's the common one. The voice LLM stays small and screen-unaware; the worker owns all screen reasoning.
Most apps don't need a custom tool per action. `ReplyToolMixin` provides a single bundled `reply` tool β a required spoken `answer` plus optional `scroll_to`, `highlight`, `select_text`, `fills`, and `click` β covering pointing, reading, and form apps:
```python
from pipecat.workers.ui import ReplyToolMixin, UIWorker
class FormWorker(ReplyToolMixin, UIWorker):
def __init__(self):
super().__init__("ui", llm=OpenAILLMService(api_key="..."))
```
The LLM uses whichever fields fit the turn β `select_text` to point at "this paragraph", `fills` + `click` to complete a form β and unused fields stay `null`.
Delegation also scales to background work. A `@tool` can fan out to peer workers with `start_ui_job_group`, which surfaces a cancellable progress card on the client and returns immediately so the voice agent isn't blocked:
```python
class ResearchWorker(UIWorker):
@tool
async def reply(self, params, answer: str, research_query: str | None = None):
if research_query:
await self.start_ui_job_group(
"wikipedia", "news", "scholar",
payload={"query": research_query},
label=f"Research: {research_query}",
)
await self.respond_to_job(answer)
await params.result_callback(None)
```
### Parallel handling
Sometimes the screen should react to _every_ user turn, not only the ones the voice agent chooses to delegate. In the **parallel handling** pattern, both agents receive each turn and act in parallel: the voice agent converses while the `UIWorker` updates the screen, independently.
The key difference: there's no tool call routing work to the worker. Instead, the voice pipeline's user aggregator fires `on_user_turn_stopped` once per turn, and that handler dispatches the transcript to the worker as a `respond` job. Because it runs in its own task, the voice LLM (running from the same turn) and the worker act concurrently:
```python
@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message):
transcript = (message.content or "").strip()
if not transcript:
return
async with worker.job("ui", name="respond", payload={"query": transcript}, timeout=15):
pass # fire-and-forget; the worker acts on its own
```
The worker acts **silently** β its `@tool` completes the job with `respond_to_job()` and no answer, so nothing it does reaches TTS. The separate voice layer owns speech:
```python
class ListWorker(UIWorker):
@tool
async def update_list(self, params, add=None, check=None, remove=None):
for text in add or []:
await self.send_command("add_item", {"text": text})
for ref in check or []:
await self.send_command("set_checked", {"ref": ref, "checked": True})
for ref in remove or []:
await self.send_command("remove_item", {"ref": ref})
await self.respond_to_job() # no answer β acts silently
await params.result_callback(None)
```
Here the **snapshot is the shared source of truth**. The worker acts on it, and the voice agent reads it through a read-only tool β so the voice agent can answer "what's left on my list?" from what's actually on screen (including items the user checked off by hand), not from conversation memory:
```python
async def check_list(params):
"""Look up what's currently on the list."""
await params.result_callback(list_worker.list_summary()) # reads the live snapshot
```
### Choosing a pattern
| | Delegation | Parallel handling |
| ------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| **How the worker is triggered** | The voice LLM calls a tool | The voice pipeline's `on_user_turn_stopped` event, every turn |
| **Who speaks** | Often the worker (`tts_speak=True`), or the voice LLM phrases the worker's result | The voice agent; the worker acts silently |
| **Voice LLM's role** | Gatekeeper β decides what's screen-relevant | Converses; reads shared state but never mutates the UI |
| **Best when** | The page matters only for some turns | Every turn should drive the UI and speech is incidental |
Both keep the voice LLM's context small. Delegation gives the voice agent control over when the screen is involved; parallel handling makes the screen a first-class output of every turn.
## What's next
You've built agents that converse, call tools, and drive the screen. Next, learn how to transfer control between them.
Activation, deactivation, and seamless control transfer
Full reference for the `UIWorker` class, UI commands, job groups, and
`ReplyToolMixin`.
# Agent Handoff
Source: https://docs.pipecat.ai/pipecat/learn/agent-handoff.md
Transfer control between agents with activation, deactivation, and handoff.
## The activation model
In a multi-agent system, only one agent is **active** at a time (per bridge). The active agent receives frames from the bus. Inactive agents exist but don't process frames.
Every worker has an `active` property. `LLMWorker` defaults to `active=False`, so in a handoff setup the LLM agents start inactive and the main agent activates the first one explicitly.
You control which agent is active with two methods:
| Method | What it does |
| --------------------------------------------- | ------------------------------------------ |
| `activate_worker(name, args=...)` | Activate another agent |
| `deactivate_worker(name)` | Deactivate another agent |
| `activate_worker(name, deactivate_self=True)` | Hand off: deactivate self, activate target |
Passing `deactivate_self=True` to `activate_worker()` is the most common form -- it's a single call that transfers control from the current agent to another.
## Building a handoff system
Let's walk through how the two-agent handoff works. You need three pieces.
### 1. A main agent with a bus bridge
The main agent owns the transport (audio I/O) and places a `BusBridgeProcessor` in its pipeline instead of an LLM. The bridge routes frames to whichever LLM agent is active. The main agent is a `PipelineWorker` wrapping that pipeline:
```python
from pipecat.bus import BusBridgeProcessor
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
MAIN_NAME = "main"
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = CartesiaTTSService(api_key=os.getenv("CARTESIA_API_KEY"))
context = LLMContext()
aggregators = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)
bridge = BusBridgeProcessor(bus=runner.bus, worker_name=MAIN_NAME)
pipeline = Pipeline([
transport.input(),
stt,
aggregators.user(),
bridge, # Where the LLM would go
tts,
transport.output(),
aggregators.assistant(),
])
main = PipelineWorker(pipeline, name=MAIN_NAME, params=PipelineParams())
```
The `BusBridgeProcessor` is what lets a transport-owning agent delegate its
LLM turn to other agents over the bus. For how it routes frames and how to
filter by bridge name, see [Understanding the Bus
Bridge](/pipecat/fundamentals/understanding-the-bus-bridge).
### 2. LLM agents with bridged=()
Each LLM agent is an `LLMWorker` created with `bridged=()` so it receives frames from the bus. Subclass `LLMWorker` to add tools, then instantiate it with its own LLM:
```python
from pipecat.workers.llm import LLMWorker, tool
class AcmeAgent(LLMWorker):
# tools defined below
def build_greeter() -> AcmeAgent:
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAILLMService.Settings(
system_instruction="You are a friendly greeter. Route product questions to support.",
),
)
return AcmeAgent("greeter", llm=llm, bridged=())
```
`bridged=()` means the agent receives frames from all bridges. You can filter
by bridge name with `bridged=("voice",)` if you have multiple bridges.
### 3. Handoff via tools
The LLM decides when to transfer by calling a tool. The tool calls `activate_worker()` with `deactivate_self=True`:
```python
from pipecat.workers.llm import LLMWorker, LLMWorkerActivationArgs, tool
class AcmeAgent(LLMWorker):
@tool(cancel_on_interruption=False)
async def transfer_to_agent(self, params: FunctionCallParams, agent: str, reason: str):
"""Transfer the user to another agent.
Args:
agent (str): The agent to transfer to (e.g. 'support').
reason (str): Why the user is being transferred.
"""
await self.activate_worker(
agent,
args=LLMWorkerActivationArgs(
messages=[{"role": "developer", "content": reason}],
),
deactivate_self=True,
result_callback=params.result_callback,
)
```
For `LLMWorker`, `activate_worker()` also accepts a `messages` parameter. These messages are injected and spoken by the current agent _before_ the transfer happens -- useful for announcing the handoff:
```python
await self.activate_worker(
agent,
messages=[{"role": "developer", "content": f"Tell the user about the transfer ({reason})."}],
args=LLMWorkerActivationArgs(
messages=[{"role": "developer", "content": reason}],
),
deactivate_self=True,
result_callback=params.result_callback,
)
```
When the greeter calls `activate_worker("support", deactivate_self=True, ...)`:
1. The greeter is deactivated -- it stops receiving frames from the bus
2. The support agent is activated with the provided arguments
3. The support agent's `on_activated()` fires, injecting the reason message into its LLM context
4. The support agent starts responding to the user
The transition is seamless -- the user experiences it as a natural conversation flow.
## Activation arguments
When activating an `LLMWorker`, pass `LLMWorkerActivationArgs` via `args=` to give the target agent context about why it was activated:
```python
from pipecat.workers.llm import LLMWorkerActivationArgs
await self.activate_worker(
"support",
args=LLMWorkerActivationArgs(
messages=[{"role": "developer", "content": "The user asked about Rocket Boots."}],
),
)
```
The target agent receives these messages in its LLM context and immediately runs the LLM to respond.
## Activating the first agent
Before you can activate an agent, it needs to be ready (its pipeline must be started). The simplest place to kick off the conversation is the transport's `on_client_connected` handler, where you activate the first agent:
```python
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
await main.activate_worker(
"greeter",
args=LLMWorkerActivationArgs(
messages=[{"role": "developer", "content": "Welcome the user."}],
),
)
```
If you need to react to a specific agent registering (for example in a distributed setup), use the `@worker_ready` decorator instead. See [Agent Registry and Discovery](/pipecat/fundamentals/agent-registry-and-discovery).
## Putting it all together
Here's the full flow:
`WorkerRunner` creates the bus. You add every agent (the main agent and
the LLM agents) with `runner.add_workers(...)`.
The transport's `on_client_connected` handler activates the greeter.
Audio flows: transport -> STT -> BusBridge -> bus -> greeter's LLM -> bus ->
BusBridge -> TTS -> transport.
The greeter's LLM calls `transfer_to_agent`. The tool calls
`activate_worker("support", deactivate_self=True, ...)`.
Greeter deactivates, support activates with context. Audio now flows through
the support agent.
## What's next
Handoff transfers a conversation between agents. But sometimes you need agents to do work in parallel. Next, let's look at job coordination.
Dispatch work to multiple agents in parallel
# Job Coordination
Source: https://docs.pipecat.ai/pipecat/learn/job-coordination.md
Dispatch work from one Pipecat agent to others and collect results with job coordination in the multi-agent framework.
## What are jobs?
Jobs let one agent dispatch work to other agents. While handoff transfers the conversation, jobs are for background work -- an agent asks other agents to do something and collects their results.
Use cases for jobs:
- Researching a topic from multiple perspectives in parallel
- Running a code analysis while talking to the user
- Fetching data from multiple sources simultaneously
## Single job
The simplest form is `self.job()`, which sends a job to one agent and waits for the response:
```python
async with self.job("worker", payload={"question": "What is Pipecat?"}, timeout=30) as j:
pass
print(j.response)
```
`self.job()` is a context manager that handles the full lifecycle: it waits for the agent to be ready, sends the request, and collects the response. Under the hood, this is equivalent to calling `request_job()` (fire-and-forget) and then handling `on_job_response()` and `on_job_completed()` yourself.
## Job group
When you need to dispatch work to multiple agents in parallel, use `self.job_group()`:
```python
async with self.job_group("worker1", "worker2", "worker3", payload={"topic": "AI safety"}, timeout=30) as jg:
pass
for worker_name, response in jg.responses.items():
print(f"{worker_name}: {response}")
```
`self.job_group()` works the same way but for multiple agents: it waits for all of them to be ready, sends a request to each, and collects all responses. Under the hood, this is equivalent to calling `request_job_group()` and then handling `on_job_response()` for each agent and a final `on_job_completed()` yourself.
The same payload is sent to every agent. If you need different arguments per agent, you can structure the payload so each one reads its own key:
```python
async with self.job_group("researcher", "fact_checker", payload={
"researcher": {"topic": "AI safety", "depth": "detailed"},
"fact_checker": {"claims": ["AI can self-improve", "AGI is near"]},
}, timeout=30) as jg:
pass
```
## Handling job requests
Agents handle incoming jobs in two ways.
### The @job decorator
The `@job` decorator marks a method as a job handler. The framework automatically dispatches matching requests to it. The decorator requires a `name` argument. The handler responds with `message.job_id`:
```python
from pipecat.pipeline.job_decorator import job
class MyWorker(BaseWorker):
@job(name="process")
async def on_process(self, message: BusJobRequestMessage):
result = await self._do_work(message.payload)
await self.send_job_response(message.job_id, result)
```
You can use different job names to route work to different handlers:
```python
class MyWorker(BaseWorker):
@job(name="research")
async def on_research(self, message: BusJobRequestMessage):
await self.send_job_response(message.job_id, {"answer": "..."})
@job(name="summarize")
async def on_summarize(self, message: BusJobRequestMessage):
await self.send_job_response(message.job_id, {"summary": "..."})
```
The requester specifies the job name when dispatching:
```python
async with self.job("worker", name="research", payload={"topic": "AI"}) as j:
pass
```
Each request runs in its own asyncio task, so multiple requests to the same handler are executed concurrently without blocking the bus message loop.
### Overriding on_job_request
Alternatively, you can override `on_job_request()` directly without the `@job` decorator:
```python
class MyWorker(BaseWorker):
async def on_job_request(self, message: BusJobRequestMessage) -> None:
await super().on_job_request(message)
result = await self._do_work(message.payload)
await self.send_job_response(message.job_id, result)
```
This is useful when you need custom routing logic or want to integrate with an existing pipeline, as shown in the example below.
`send_job_response()`, `send_job_update()`, and `send_job_stream_*()` all
require an explicit `job_id`. This lets an agent handle multiple concurrent
jobs and respond to each one correctly. For simple handlers, pass
`message.job_id` from the request. For asynchronous responses (see the example
below), track the `job_id` yourself until you're ready to respond.
## Building a job system
Let's build a debate system where a moderator dispatches a topic to three agents, each arguing from a different perspective.
### Worker agents
Each debate agent runs its own LLM pipeline. The LLM response arrives asynchronously through an event handler, so the agent tracks the current job ID until it has something to respond with:
```python
from pipecat.workers.llm import LLMContextWorker
from pipecat.bus.messages import BusJobRequestMessage
from pipecat.frames.frames import LLMMessagesAppendFrame
class DebateWorker(LLMContextWorker):
def __init__(self, role: str):
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAILLMService.Settings(system_instruction=f"You argue as a {role}."),
)
super().__init__(role, llm=llm)
self._role = role
self._current_job_id: str | None = None
@self.assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message):
if self._current_job_id:
job_id = self._current_job_id
self._current_job_id = None
await self.send_job_response(job_id, {"role": self._role, "text": message.content})
async def on_job_request(self, message: BusJobRequestMessage) -> None:
await super().on_job_request(message)
self._current_job_id = message.job_id
await self.queue_frame(
LLMMessagesAppendFrame(
messages=[{"role": "developer", "content": f"Topic: {message.payload['topic']}"}],
run_llm=True,
)
)
```
`LLMContextWorker` extends `LLMWorker` with a built-in `LLMContext` and
aggregator pair. It builds the pipeline as `[user_aggregator, llm,
assistant_aggregator]` automatically, so you don't need to wire the context
plumbing yourself. Access the aggregators via `self.user_aggregator` and
`self.assistant_aggregator`.
The agent:
1. Receives a job request and stores the `job_id`
2. Injects the topic into its LLM context and runs the LLM
3. When the LLM finishes its turn, the event handler sends the response with the stored `job_id`
### Coordinator agent
The moderator triggers jobs via a tool and synthesizes the results:
```python
from pipecat.workers.llm import LLMWorker, tool
class ModeratorAgent(LLMWorker):
@tool(cancel_on_interruption=False)
async def debate(self, params: FunctionCallParams, topic: str):
"""Analyze a topic from multiple perspectives.
Args:
topic (str): The topic to debate.
"""
async with self.job_group("advocate", "critic", "analyst", payload={"topic": topic}, timeout=30) as jg:
pass
result = "\n\n".join(
f"{r['role'].upper()}: {r['text']}" for r in jg.responses.values()
)
await params.result_callback(result)
```
Create the debate agents and the moderator, then register them all with `runner.add_workers(ModeratorAgent("moderator", llm=...), DebateWorker("advocate"), DebateWorker("critic"), DebateWorker("analyst"))`.
When the user says "debate whether AI should be regulated," the moderator:
1. The LLM calls the `debate` tool
2. `job_group()` sends the topic to all three agents in parallel
3. Each agent runs its own LLM and responds with its perspective
4. The moderator collects all responses and returns them to the LLM
5. The LLM synthesizes a balanced summary and speaks it to the user
## Job lifecycle
The requester calls `job()` or `job_group()`. The framework waits for agents
to be ready, then sends a job request to each.
Agents receive the request in `on_job_request()` and do their work.
Agents call `send_job_response(job_id, ...)` with their results.
When all agents respond (or timeout occurs), the context manager exits and
results are available.
The framework also supports fire-and-forget jobs, progress updates, and streaming.
## Job cancellation
Jobs can be cancelled in several ways. The framework handles cleanup automatically and notifies agents so they can stop in-progress work.
### Automatic cancellation with context managers
When using `job()` or `job_group()`, cancellation happens automatically if the context block raises an exception. This includes tool interruptions (`CancelledError`) when using `cancel_on_interruption=True`:
```python
# If an exception or CancelledError is raised inside the block,
# all agents are cancelled automatically
async with self.job_group("w1", "w2", payload=data) as jg:
# ... if this raises, the agents get cancelled
pass
```
### Worker errors
By default (`cancel_on_error=True`), if any agent responds with an error status, the remaining agents in the group are cancelled and `JobGroupError` is raised:
```python
try:
async with self.job_group("w1", "w2", payload=data) as jg:
pass
except JobGroupError as e:
# An agent errored -- remaining agents were cancelled
pass
```
### Manual cancellation
For fire-and-forget jobs (using `request_job()`), you manage cancellation yourself by tracking job IDs:
```python
job_ids = []
try:
job_ids.append(await self.request_job("w1", payload={"job": 1}))
job_ids.append(await self.request_job("w2", payload={"job": 2}))
# ...
except asyncio.CancelledError:
for jid in job_ids:
await self.cancel_job_group(jid, reason="tool cancelled")
```
### Handling cancellation on the agent side
When a job is cancelled, the agent's `on_job_cancelled` hook fires. The framework automatically sends a `CANCELLED` response back to the requester, so you only need to override this hook if you have resources to clean up:
```python
class MyWorker(BaseWorker):
async def on_job_cancelled(self, message: BusJobCancelMessage) -> None:
# Optional: clean up resources, stop in-progress work
logger.info(f"Job {message.job_id} cancelled: {message.reason}")
```
### Agent shutdown
When an agent stops with jobs still in flight, it automatically sends a `CANCELLED` response for each active job so requesters aren't left waiting on a timeout.
## What's next
You can now build agents, hand off between them, and dispatch jobs. So far everything has run in a single process. Next, let's scale across processes and machines.
Run agents across processes and machines on a shared bus
# Distributed Agents
Source: https://docs.pipecat.ai/pipecat/learn/distributed-agents.md
Run Pipecat agents across processes and machines connected to the same bus for distributed multi-agent deployments.
## Overview
Distributed agents are agents connected to the **same bus** but running in different processes or on different machines. By default, all agents run in a single process using a local bus. For distributed setups, swap to a network bus -- all agents share the same channel and discover each other automatically. Your agent code stays the same.
Distributed agents share a bus. If you need to connect agents on **different
buses** (separate networks, third-party services), see [Proxy
Agents](/pipecat/learn/proxy-agents) instead.
## Setting up a distributed bus
Pipecat provides two distributed bus implementations: RedisBus and PgmqBus. Choose based on your infrastructure.
### RedisBus
Each process creates its own `WorkerRunner` with a `RedisBus` connected to the same Redis channel:
```python
from redis.asyncio import Redis
from pipecat.bus.network.redis import RedisBus
from pipecat.workers.runner import WorkerRunner
redis = Redis.from_url("redis://localhost:6379")
bus = RedisBus(redis=redis, channel="pipecat:my-app")
runner = WorkerRunner(bus=bus, handle_sigint=True)
```
All runners sharing the same `channel` can exchange messages. Agents discover each other automatically through registry snapshots.
Install the Redis extra: `uv add "pipecat-ai[redis]"`
### PgmqBus
Alternatively, use `PgmqBus` backed by PostgreSQL Message Queue:
```python
from pgmq.async_queue import PGMQueue
from pipecat.bus.network.pgmq import PgmqBus
from pipecat.workers.runner import WorkerRunner
pgmq = PGMQueue(
host="localhost",
port="5432",
database="postgres",
username="postgres",
password="...",
pool_size=4,
)
await pgmq.init()
bus = PgmqBus(pgmq=pgmq, channel="pipecat:my-app")
runner = WorkerRunner(bus=bus, handle_sigint=True)
```
Install the PGMQ extra: `uv add "pipecat-ai[pgmq]"`
## Example: distributed handoff
This example splits the two-agent handoff across separate processes. The main agent handles transport on one machine, and LLM agents run independently on other machines.
### Process 1: Main transport agent
The main agent owns the transport and bridges frames to the bus. It has no LLM -- it waits for the remote greeter agent to register, then activates it.
```
/* Detailed source-code truncated for AI context efficiency. */
```
```bash
python main.py --redis-url redis://localhost:6379
```
### Process 2 & 3: LLM agents
Each LLM agent runs as a standalone process. It connects to the same Redis channel and waits for activation. Subclass `LLMWorker` to host `@tool` methods and pass the LLM service in the constructor:
```
/* Detailed source-code truncated for AI context efficiency. */
```
```bash
# Run on Machine B
python llm.py greeter --redis-url redis://your-redis-host:6379
# Run on Machine C
python llm.py support --redis-url redis://your-redis-host:6379
```
Each LLM agent watches its sibling so that, when the user is transferred, the target is already known to be available. You can do the same with the `@worker_ready` decorator on a worker subclass to react automatically when a specific agent registers.
## How discovery works
Runners exchange registry information automatically over the shared bus. To get notified when an agent is ready, watch it with `runner.registry.watch(...)`, call `watch_workers()` from inside a worker, or use the `@worker_ready` decorator -- they all work the same way locally and distributed.
## Considerations
- **Latency**: Network buses add overhead. For latency-sensitive voice applications, keep the main transport agent and its active LLM agent geographically close to each other and the bus server (Redis or PostgreSQL).
- **Serialization**: Both `RedisBus` and `PgmqBus` serialize messages to JSON. Custom frame types need to be registered with the serializer.
- **Single channel**: All agents on the same channel see all messages. Use different channels for different sessions or applications.
## What's next
Distributed agents share one bus. When you need to connect agents across separate buses or networks, bridge them with a proxy.
Bridge agents on different buses point-to-point over WebSocket
# Proxy Agents
Source: https://docs.pipecat.ai/pipecat/learn/proxy-agents.md
Connect Pipecat agents running on different buses with proxy agents, extending multi-agent pipelines across processes.
## Overview
Proxy agents connect agents running on **different buses**. Unlike [distributed agents](/pipecat/learn/distributed-agents) (which all share the same bus), proxy agents bridge two isolated bus instances point-to-point. This is useful when:
- You want to run an LLM agent on a separate server without shared infrastructure
- You need fine-grained control over which messages cross the network
- You're connecting to a third-party service that hosts agents
## Architecture
Each side has its own `WorkerRunner` and bus. The proxy agents relay messages between the two buses over a WebSocket connection. Like any worker, a proxy gets its bus from the runner when you register it -- you never pass `bus=` to its constructor.
## Client side: WebSocketProxyClient
The client connects to a remote server and forwards specific message types:
```python
from pipecat.bus import BusFrameMessage
from pipecat.workers.proxy.websocket import WebSocketProxyClient
proxy = WebSocketProxyClient(
"proxy",
url="ws://remote-server:8765/ws",
local_worker_name="acme", # Agent on this bus
remote_worker_name="assistant", # Agent on remote bus
forward_messages=(BusFrameMessage,),
)
await runner.add_workers(proxy)
```
| Parameter | Description |
| -------------------- | ------------------------------------------------------------- |
| `url` | WebSocket URL of the remote server |
| `local_worker_name` | Name of the local agent that exchanges frames with the remote |
| `remote_worker_name` | Name of the remote agent to communicate with |
| `forward_messages` | Tuple of message types to forward across the connection |
| `headers` | Optional HTTP headers (e.g. authentication tokens) |
The proxy connects when activated. Activate it when the client connects, by calling `activate_worker` on the main agent:
```python
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
await main.activate_worker("proxy")
```
## Server side: WebSocketProxyServer
The server accepts WebSocket connections and creates a proxy for each session:
```python
from fastapi import FastAPI, WebSocket
from pipecat.bus import BusFrameMessage
from pipecat.workers.runner import WorkerRunner
from pipecat.workers.proxy.websocket import WebSocketProxyServer
app = FastAPI()
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
runner = WorkerRunner(handle_sigint=False)
proxy = WebSocketProxyServer(
"gateway",
websocket=websocket,
worker_name="assistant", # Agent on this bus
remote_worker_name="acme", # Agent on remote bus
forward_messages=(BusFrameMessage,),
)
assistant = build_assistant() # an LLMWorker named "assistant"
await runner.add_workers(proxy, assistant)
await runner.run()
```
Each WebSocket connection gets its own `WorkerRunner`, bus, and set of agents. This isolates sessions from each other.
## Message filtering
Proxy agents provide security through message filtering:
- Only messages targeted at the configured agent names cross the connection
- Broadcast messages (no target) are **not** forwarded
- Local-only messages (`BusLocalMessage`) never cross
- `forward_messages` controls which message types are allowed
This means internal bus traffic stays local. Only the specific message types you opt into are relayed.
## Full example
### Client (main.py)
```python
async def run_bot(transport, runner_args):
runner = WorkerRunner(handle_sigint=runner_args.handle_sigint)
# ... build the main transport pipeline with a BusBridgeProcessor ...
main = PipelineWorker(pipeline, name="acme", params=PipelineParams(...))
proxy = WebSocketProxyClient(
"proxy",
url=runner_args.cli_args.remote_url,
local_worker_name="acme",
remote_worker_name="assistant",
forward_messages=(BusFrameMessage,),
)
async def on_assistant_ready(_data):
await main.activate_worker(
"assistant",
args=LLMWorkerActivationArgs(
messages=[{"role": "developer", "content": "Welcome the user."}],
),
)
await runner.registry.watch("assistant", on_assistant_ready)
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
await main.activate_worker("proxy")
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
await runner.cancel()
await runner.add_workers(proxy, main)
await runner.run()
```
### Server (assistant.py)
```python
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
runner = WorkerRunner(handle_sigint=False)
proxy = WebSocketProxyServer(
"gateway",
websocket=websocket,
worker_name="assistant",
remote_worker_name="acme",
forward_messages=(BusFrameMessage,),
)
@proxy.event_handler("on_client_connected")
async def on_client_connected(proxy, client):
logger.info("WebSocket client connected")
@proxy.event_handler("on_client_disconnected")
async def on_client_disconnected(proxy, client):
await runner.cancel()
assistant = AcmeAssistant() # an LLMWorker named "assistant"
await runner.add_workers(proxy, assistant)
await runner.run()
```
### Running
```bash
# Terminal 1: Start the LLM server
python assistant.py --port 8765
# Terminal 2: Start the transport client
python main.py --remote-url ws://localhost:8765/ws
```
Install the WebSocket extra: `uv add "pipecat-ai[websocket]"`
## Proxy agents vs distributed agents
| | Proxy Agents (different buses) | Distributed (same bus) |
| --------------------- | ------------------------------------- | ------------------------------ |
| **Topology** | Point-to-point | Many-to-many |
| **Session isolation** | Each connection is isolated | All agents share a channel |
| **Use case** | Separate networks, third-party agents | Scaling agents across machines |
## What's next
You've built a multi-agent system and scaled it across processes, machines, and networks. Here's where to go from here.
Explore Fundamentals and advanced patterns
# What's Next
Source: https://docs.pipecat.ai/pipecat/learn/whats-next.md
Continue your Pipecat journey with advanced features, examples, and production deployment
## You've Mastered Voice AI Pipelines! π
Congratulations! You've learned how to build complete voice AI applications with Pipecat. You're now well on your way to understanding pipelines, processors, transports, and all the components needed to create sophisticated conversational AI.
## Choose Your Path
**Production-ready applications** - Dive into 30+ complete examples including
multimodal bots, creative applications, and enterprise integrations.
**Master advanced features** - Explore specialized topics like telephony,
deployment, custom processors, and production optimization.
## π¬ Join the Community
Get involved with the Pipecat community to share your projects, get support, and contribute to the ecosystem:
- **[Discord Community](https://discord.gg/pipecat)**: Get support, share projects, and connect with other developers
- **[GitHub](https://github.com/pipecat-ai/pipecat)**: Contribute to the project and report issues
- **[Examples Repository](https://github.com/pipecat-ai/pipecat-examples)**: Community-contributed applications
# Service Settings
Source: https://docs.pipecat.ai/pipecat/fundamentals/service-settings.md
Learn how to configure and update AI service settings at initialization and runtime.
Settings are the runtime-configurable properties of AI services, including things like the model, voice, language, temperature, and other provider-specific options. Every service exposes a `Settings` class that you can use in two ways:
1. **Configure at initialization**: pass a `settings=` argument when constructing a service.
2. **Update at runtime**: push an `*UpdateSettingsFrame` through the pipeline to change settings mid-conversation.
## Configuring settings at initialization
Pass a `settings=` argument to any service constructor using that service's `Settings` class:
```python
from pipecat.services.openai.llm import OpenAILLMService
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAILLMService.Settings(
model="gpt-4o",
temperature=0.7,
system_instruction="You are a helpful assistant.",
),
)
```
```python
from pipecat.services.cartesia.tts import CartesiaTTSService
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
settings=CartesiaTTSService.Settings(
voice="71a7ad14-091c-4e8e-a314-022ece01c121",
),
)
```
```python
from pipecat.services.deepgram.stt import DeepgramSTTService
stt = DeepgramSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
settings=DeepgramSTTService.Settings(
model="nova-3",
language="en",
smart_format=True,
),
)
```
You only need to specify the settings you want to override. Everything else uses the service's defaults.
## Common settings by service type
Each service type has a base set of settings. Individual services may extend these with provider-specific fields.
### LLM settings
| Setting | Type | Description |
| -------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `model` | `str` | Model identifier (e.g. `"gpt-4o"`, `"claude-sonnet-4-5-20250929"`) |
| `system_instruction` | `str` | System prompt for the model. See [Context Management](/pipecat/learn/context-management#system-instruction-developer-messages-and-system-messages) for how this interacts with developer messages and context system messages. |
| `temperature` | `float` | Sampling temperature |
| `max_tokens` | `int` | Maximum tokens to generate |
| `top_p` | `float` | Nucleus sampling probability |
| `top_k` | `int` | Top-k sampling parameter |
| `frequency_penalty` | `float` | Frequency penalty |
| `presence_penalty` | `float` | Presence penalty |
| `seed` | `int` | Random seed for reproducibility |
### TTS settings
| Setting | Type | Description |
| ---------- | ----------------- | ----------------------------- |
| `model` | `str` | TTS model identifier |
| `voice` | `str` | Voice identifier or name |
| `language` | `Language \| str` | Language for speech synthesis |
### STT settings
| Setting | Type | Description |
| ---------- | ----------------- | ------------------------------- |
| `model` | `str` | STT model identifier |
| `language` | `Language \| str` | Language for speech recognition |
Individual services extend these base settings with provider-specific fields.
For example, Deepgram STT adds `endpointing`, `smart_format`, `diarize`, and
more. See the individual [service
documentation](/api-reference/server/services/supported-services) for the full
list of available settings.
## Updating settings at runtime
You can change service settings while the pipeline is running by pushing an update settings frame. This is useful for scenarios like switching languages, changing voices, or adjusting LLM parameters mid-conversation.
Use these frame types:
| Frame | Target |
| ------------------------ | ------------ |
| `LLMUpdateSettingsFrame` | LLM services |
| `TTSUpdateSettingsFrame` | TTS services |
| `STTUpdateSettingsFrame` | STT services |
Only include the fields you want to change. Unspecified fields are left as-is.
```python
from pipecat.frames.frames import TTSUpdateSettingsFrame
# Change TTS voice mid-conversation
await worker.queue_frame(
TTSUpdateSettingsFrame(
delta=CartesiaTTSService.Settings(voice="new-voice-id")
)
)
```
```python
from pipecat.frames.frames import LLMUpdateSettingsFrame
# Lower the temperature for more deterministic responses
await worker.queue_frame(
LLMUpdateSettingsFrame(
delta=OpenAILLMService.Settings(temperature=0.2)
)
)
```
```python
from pipecat.frames.frames import STTUpdateSettingsFrame
from pipecat.transcriptions.language import Language
# Switch STT language to Spanish
await worker.queue_frame(
STTUpdateSettingsFrame(
delta=DeepgramSTTService.Settings(language=Language.ES)
)
)
```
Update settings frames are uninterruptible. They will always be processed even
if a user interruption occurs.
## Service-specific settings
Services can extend the base settings with provider-specific fields. For example:
- **Cartesia TTS** adds `generation_config` (volume, speed, emotion) and `pronunciation_dict_id`
- **OpenAI TTS** adds `instructions` and `speed`
- **Deepgram STT** adds `endpointing`, `smart_format`, `diarize`, `punctuate`, and many more
- **OpenAI LLM** adds `max_completion_tokens`
These service-specific fields work the same way β set them at initialization or update them at runtime:
```python
from pipecat.services.cartesia.tts import CartesiaTTSService, GenerationConfig
tts = CartesiaTTSService(
api_key=os.getenv("CARTESIA_API_KEY"),
settings=CartesiaTTSService.Settings(
voice="71a7ad14-091c-4e8e-a314-022ece01c121",
generation_config=GenerationConfig(speed=1.2, emotion="excited"),
),
)
```
See the individual service documentation for a complete list of available settings:
## Passing extra parameters
Every Settings class includes an `extra` dict for passing provider-specific parameters that Pipecat doesn't have a dedicated field for. This is useful when a provider supports options that haven't been explicitly added to the Settings dataclass yet:
```python
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
settings=OpenAILLMService.Settings(
model="gpt-4o",
extra={"logprobs": True, "top_logprobs": 5},
),
)
```
Values in `extra` are passed through to the underlying API call. You can also update `extra` at runtime:
```python
await worker.queue_frame(
LLMUpdateSettingsFrame(
delta=OpenAILLMService.Settings(
extra={"logprobs": False},
)
)
)
```
The `extra` dict is merged on updates β new keys are added and existing keys
are overwritten, but keys not present in the delta are left unchanged.
## Migration from InputParams
The `InputParams` / `params=` pattern is deprecated as of v0.0.105. Use `Settings` / `settings=` instead.
**Before:**
```python
llm = OpenAILLMService(
model="gpt-4o",
params=OpenAILLMService.InputParams(
temperature=0.7,
),
)
```
**After:**
```python
llm = OpenAILLMService(
settings=OpenAILLMService.Settings(
model="gpt-4o",
temperature=0.7,
),
)
```
Note that `model` has moved from a top-level constructor argument into `Settings`. Both the old and new patterns still work during the deprecation period, but `settings` values take precedence when both are provided.
# Interruptions
Source: https://docs.pipecat.ai/pipecat/fundamentals/interruptions.md
How Pipecat stops the bot when the user speaks, what happens to in-flight LLM and TTS output, and how to control it.
Interruptions (also called barge-in) let the user talk over the bot. When the user starts speaking while the bot is talking, the bot stops immediately, in-flight work is cancelled, and the pipeline is ready for the new user input.
Interruptions are **enabled by default**. This page explains how they work under the hood, what ends up in the conversation context, and how to configure or trigger them yourself.
Looking for how Pipecat decides *when* a user turn starts and ends? See
[Speech Input & Turn Detection](/pipecat/learn/speech-input) and [User Turn
Strategies](/api-reference/server/utilities/turn-management/user-turn-strategies).
This page covers what happens *after* an interruption is triggered.
## What happens when the user interrupts
When a [user turn start strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#start-strategies) triggers with `enable_interruptions=True` (the default), the user aggregator broadcasts an `InterruptionFrame` both upstream and downstream through the pipeline. From there:
`InterruptionFrame` is a
[SystemFrame](/api-reference/server/frames/system-frames), so each processor
handles it immediately instead of waiting behind queued frames. Each
processor cancels its processing task and discards queued `DataFrame`s and
`ControlFrame`s. Frames marked uninterruptible (like
`FunctionCallResultFrame` and `EndFrame`) are preserved and still processed.
The in-flight LLM completion is cancelled mid-stream. Any registered
function calls with `cancel_on_interruption=True` are cancelled and emit a
`FunctionCallCancelFrame`. See [Function
Calling](/pipecat/learn/function-calling) for details.
The TTS service stops synthesizing, clears its text aggregation and word
timestamps, and drops pending output.
The output transport drains its audio queue, discarding audio that was
generated but not yet played. If a background audio mixer is active, the
transport drains only the bot's speech so the background audio keeps playing
without a gap.
The result: the bot goes silent within roughly one audio write, and the pipeline is clean and ready for the user's new turn.
## What ends up in the context
A common question: if the bot is cut off mid-sentence, what does the LLM context contain?
**Only the words that were actually spoken.** As the bot speaks, the output transport pushes `TTSTextFrame`s downstream in sync with audio playback. Text that never played never reaches the assistant context aggregator. On interruption, the aggregator commits the partial, spoken-so-far text to the context as the assistant message.
This means the LLM's next completion sees an accurate transcript of the conversation: the bot's message ends where the user cut it off, not where the LLM's generation ended.
You can observe this with the `on_assistant_turn_stopped` event, which reports the committed text and whether the turn was interrupted:
```python
@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message):
if message.interrupted:
print(f"Bot was cut off after saying: {message.content}")
```
## Controlling interruptions
**Let the bot finish speaking** with [user input muting](/pipecat/fundamentals/user-input-muting). Mute strategies block user audio, transcriptions, and interruption signals while active, so speech over the bot is discarded rather than answered later. Use `AlwaysUserMuteStrategy` to mute whenever the bot is speaking, or `FirstSpeechUserMuteStrategy` to protect just the introduction:
```python
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.turns.user_mute import AlwaysUserMuteStrategy
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
user_mute_strategies=[AlwaysUserMuteStrategy()],
),
)
```
**Require a minimum number of words** so short utterances like "okay" or "yeah" don't interrupt the bot. This is configured on the [user turn start strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#start-strategies):
```python
from pipecat.turns.user_start import MinWordsUserTurnStartStrategy
start_strategy = MinWordsUserTurnStartStrategy(min_words=3)
```
**Filter out backchannels with a model.** The [Krisp VIVA Interruption Prediction strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#krispvivaipuserturnstartstrategy) distinguishes genuine interruptions from acknowledgments like "uh-huh".
**Disable interruptions** so in-flight work is never cancelled:
```python
from pipecat.turns.user_start import VADUserTurnStartStrategy
start_strategy = VADUserTurnStartStrategy(enable_interruptions=False)
```
When an STT service with built-in turn detection drives turns, it routes its own `should_interrupt` setting to the strategies it requests, so pass `should_interrupt=False` to the service instead. See [External Turn Management](/api-reference/server/utilities/turn-management/external-turn-management#external-services).
Disabling interruptions does not ignore the user. Speech over the bot is still
transcribed and processed as a normal user turn β the bot's reply is queued
and plays as soon as the current speech finishes. If you want the bot to
finish speaking *and* discard what the user said over it, use mute strategies
instead.
## Triggering an interruption yourself
Sometimes the bot should stop itself: a timeout fires, an external event arrives, or your own logic decides the current response is no longer relevant.
From inside a custom `FrameProcessor`, broadcast the interruption directly:
```python
await self.broadcast_interruption()
```
From code that has a reference to a processor or the worker, you can also push an `InterruptionWorkerFrame`. The pipeline worker converts it into an `InterruptionFrame` and sends it through the whole pipeline:
```python
from pipecat.frames.frames import InterruptionWorkerFrame
await worker.queue_frame(InterruptionWorkerFrame())
```
Both approaches run the same interruption flow described above: the bot stops speaking, in-flight work is cancelled, and the spoken-so-far text is committed to context.
## Speech-to-speech services
Realtime speech-to-speech services (like Gemini Live and OpenAI Realtime) handle interruption detection on the provider side. In these pipelines, the service emits the speaking events and Pipecat's aggregators follow along using [external turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies#externaluserturnstartstrategy). The provider decides when the user barged in; Pipecat still flushes local audio output so the bot goes silent right away.
## Related
- [Speech Input & Turn Detection](/pipecat/learn/speech-input) - how user turns are detected
- [User Turn Strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) - full strategy reference
- [User Input Muting](/pipecat/fundamentals/user-input-muting) - suppress user input while the bot speaks
- [System Frames](/api-reference/server/frames/system-frames) - `InterruptionFrame` reference
# User Input Muting
Source: https://docs.pipecat.ai/pipecat/fundamentals/user-input-muting.md
Learn how to control when user speech is processed in your conversational bot
## Overview
In conversational applications, there are moments when you don't want to process user speech, such as during bot introductions or while executing function calls. Pipecat's user mute strategies let you selectively "mute" user input based on different conversation states.
## When to Use Mute Strategies
Common scenarios for muting user input include:
- **During introductions**: Prevent the bot from being interrupted during its initial greeting
- **While processing functions**: Block input while the bot is retrieving external data
- **During bot speech**: Reduce false transcriptions while the bot is speaking
- **For guided conversations**: Create more structured interactions with clear turn-taking
## How It Works
User mute strategies work by blocking specific user-related frames from flowing through your pipeline. When muted, the following frames are filtered:
- Voice activity detection (VAD) events
- Interruption signals
- Raw audio input frames
- Transcription frames (both interim and final)
This prevents user speech from being processed during muted periods.
Mute strategies are configured on the `LLMUserAggregator` via the
`user_mute_strategies` parameter.
## Mute Strategies
Pipecat provides several built-in strategies for determining when to mute user input:
Mute only during the bot's first speech utterance. Useful for introductions
when you want the bot to complete its greeting before the user can speak.
Start muted and remain muted until the first bot utterance completes.
Ensures the bot's initial instructions are fully delivered.
Mute during function calls. Prevents users from speaking while the bot is
processing external data requests.
Mute whenever the bot is speaking. Creates a strict turn-taking conversation
pattern.
The `FirstSpeechUserMuteStrategy` and
`MuteUntilFirstBotCompleteUserMuteStrategy` strategies should not be used
together as they handle the first bot speech differently.
## Basic Implementation
Import and configure the mute strategies you need:
```python
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.turns.user_mute import AlwaysUserMuteStrategy
# Configure with one or more strategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
user_mute_strategies=[AlwaysUserMuteStrategy()],
),
)
```
## Combining Multiple Strategies
Multiple strategies can be combined. They use OR logicβif **any** strategy indicates the user should be muted, input is suppressed:
```python
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.turns.user_mute import (
MuteUntilFirstBotCompleteUserMuteStrategy,
FunctionCallUserMuteStrategy,
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
user_mute_strategies=[
MuteUntilFirstBotCompleteUserMuteStrategy(), # Mute until first response
FunctionCallUserMuteStrategy(), # Mute during function calls
],
),
)
```
## Building Custom Strategies
Subclass `BaseUserMuteStrategy` (in `pipecat.turns.user_mute`) when none of the built-in strategies fit. A strategy only needs to answer one question per frame: should the user be muted right now? Override `process_frame(self, frame: Frame) -> bool` to update internal state and return the current mute decision.
### Which frames reach a strategy
Each strategy's `process_frame` is called for every frame that passes through the user aggregator, **except** `StartFrame`, `EndFrame`, and `CancelFrame`. This includes:
- User-direction frames from the input transport and STT: `TranscriptionFrame`, `InterimTranscriptionFrame`, `UserStartedSpeakingFrame`, `UserStoppedSpeakingFrame`, `ProposedUserStartedSpeakingFrame`, `ProposedUserStoppedSpeakingFrame`, `VADUserStartedSpeakingFrame`, `VADUserStoppedSpeakingFrame`, `InputAudioRawFrame`, `InterruptionFrame`
- Bot and function-calling lifecycle frames from elsewhere in the pipeline: `BotStartedSpeakingFrame`, `BotStoppedSpeakingFrame`, `FunctionCallsStartedFrame`, `FunctionCallResultFrame`, `FunctionCallCancelFrame`
Frames that don't naturally reach the user aggregator (for example
`LLMTextFrame` or `TTSTextFrame`, which flow downstream from the LLM or TTS)
won't be seen by a strategy directly. To react to those signals, place a
companion `FrameProcessor` where the frames do flow and have it toggle state
on your strategy. See [Toggling a strategy at
runtime](#toggling-a-strategy-at-runtime) below.
### Which frames get suppressed when muted
Returning `True` from your strategy sets the aggregator's mute state. While muted, only these frame types are actually dropped:
- `InterruptionFrame`
- `VADUserStartedSpeakingFrame`, `VADUserStoppedSpeakingFrame`
- `UserStartedSpeakingFrame`, `UserStoppedSpeakingFrame`
- `ProposedUserStartedSpeakingFrame`, `ProposedUserStoppedSpeakingFrame`
- `InputAudioRawFrame`
- `InterimTranscriptionFrame`, `TranscriptionFrame`
All other frames continue to flow so the rest of the pipeline keeps functioning.
### Toggling a strategy at runtime
Strategies are plain Python objects. Anything that holds a reference to one can flip its state between frames, which means a companion processor placed elsewhere in the pipeline can drive the mute decision based on signals the strategy can't observe directly (LLM text, tool results, external events).
This example strategy adds its own `enable`/`disable` methods (not part of the base contract) and returns their state from `process_frame`:
```python
from pipecat.frames.frames import Frame
from pipecat.turns.user_mute import BaseUserMuteStrategy
class ToggleableUserMuteStrategy(BaseUserMuteStrategy):
def __init__(self):
super().__init__()
self._muted = False
def enable(self):
self._muted = True
def disable(self):
self._muted = False
async def process_frame(self, frame: Frame) -> bool:
await super().process_frame(frame)
return self._muted
```
A companion processor watches for the trigger and toggles the strategy:
```python
from pipecat.frames.frames import (
BotStartedSpeakingFrame,
BotStoppedSpeakingFrame,
Frame,
LLMTextFrame,
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
class DisclaimerGuardProcessor(FrameProcessor):
def __init__(self, strategy: ToggleableUserMuteStrategy, trigger_phrase: str, **kwargs):
super().__init__(**kwargs)
self._strategy = strategy
self._trigger = trigger_phrase
# Keep a small sliding window so cross-frame matches work without
# the buffer growing unbounded if the trigger never appears.
self._max_buffer = max(len(trigger_phrase) * 4, 512)
self._buffer = ""
self._active = False
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, BotStartedSpeakingFrame):
# Start each bot turn with a fresh buffer.
self._buffer = ""
elif isinstance(frame, LLMTextFrame) and direction == FrameDirection.DOWNSTREAM:
self._buffer = (self._buffer + frame.text)[-self._max_buffer :]
if not self._active and self._trigger in self._buffer:
self._active = True
self._strategy.enable()
elif isinstance(frame, BotStoppedSpeakingFrame) and self._active:
self._active = False
self._buffer = ""
self._strategy.disable()
await self.push_frame(frame, direction)
```
Wire them together by passing the same strategy instance to both the aggregator and the processor:
```python
mute_strategy = ToggleableUserMuteStrategy()
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(user_mute_strategies=[mute_strategy]),
)
disclaimer_guard = DisclaimerGuardProcessor(
strategy=mute_strategy,
trigger_phrase="Please read the following disclosure",
)
pipeline = Pipeline([
transport.input(),
stt,
user_aggregator,
llm,
disclaimer_guard, # positioned where LLMTextFrame flows downstream
tts,
transport.output(),
assistant_aggregator,
])
```
## Responding to Mute Events
You can register event handlers to be notified when muting starts or stops. This is particularly useful for providing visual feedback to users:
```python
@user_aggregator.event_handler("on_user_mute_started")
async def on_user_mute_started(aggregator):
logger.info("User mute started")
# Send a visual indicator to your client
# e.g., show a "Bot is speaking" indicator
@user_aggregator.event_handler("on_user_mute_stopped")
async def on_user_mute_stopped(aggregator):
logger.info("User mute stopped")
# Update your client UI
# e.g., show a "You can speak now" indicator
```
These events fire whenever the mute state changes, allowing you to keep your UI synchronized with the bot's state.
### RTVI Events
When mute strategies activate or deactivate, the server automatically sends RTVI messages (`user-mute-started` and `user-mute-stopped`) to the client. You can listen for these in the JavaScript client to update your UI:
The client should continue sending audio normally during mute. These events
are purely informational β muting happens server-side.
```typescript JavaScript
import { PipecatClient, RTVIEvent } from "@pipecat-ai/client-js";
const pcClient = new PipecatClient({
callbacks: {
onUserMuteStarted: () => {
// Show a visual indicator that the bot is not listening
// e.g., disable a microphone button or show "Bot is speaking..."
},
onUserMuteStopped: () => {
// Remove the indicator, show the user they can speak
},
},
});
// Or using event listeners
pcClient.on(RTVIEvent.UserMuteStarted, () => {
console.log("Server is ignoring user audio");
});
pcClient.on(RTVIEvent.UserMuteStopped, () => {
console.log("Server is listening to user audio again");
});
```
## Best Practices
- **Choose strategies wisely**: Select the minimal set of strategies needed for your use case
- **Test user experience**: Excessive muting can frustrate users; balance control with usability
- **Provide feedback**: Use the mute event handlers to show visual cues when the user is muted to improve the experience
## Next Steps
Read the complete API reference documentation for all available mute
strategies and their behavior.
Learn how to configure turn detection behavior for more control over
conversation flow.
Experiment with different muting strategies to find the right balance for your application.
# Detecting Idle Users
Source: https://docs.pipecat.ai/pipecat/fundamentals/detecting-user-idle.md
Learn how to detect and respond when users are inactive in conversations
## Overview
In conversational applications, it's important to handle situations where users go silent or inactive. Pipecat provides built-in idle detection through `LLMUserAggregator` and `UserTurnProcessor`, allowing your bot to respond appropriately when users haven't spoken for a defined period.
## How It Works
Idle detection monitors user activity and:
1. Starts a timer when the bot finishes speaking (`BotStoppedSpeakingFrame`)
2. Cancels the timer when the user or bot starts speaking
3. Suppresses the timer during function calls and active user turns (to avoid false triggers during interruptions)
4. Emits an `on_user_turn_idle` event when the timer expires
5. Allows you to implement escalating responses or gracefully end the conversation in your application code
## Basic Implementation
### Step 1: Enable Idle Detection
Enable idle detection by setting the `user_idle_timeout` parameter when creating your aggregator:
```python
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
user_idle_timeout=5.0, # Detect idle after 5 seconds
),
)
```
### Step 2: Handle Idle Events
Create an event handler to respond when the user becomes idle:
```python
@user_aggregator.event_handler("on_user_turn_idle")
async def on_user_turn_idle(aggregator):
# Send a reminder to the user
message = {
"role": "developer",
"content": "The user has been quiet. Politely ask if they're still there.",
}
await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
```
### Step 3: Implement Retry Logic (Optional)
For escalating responses, track retry count in your application:
```python
class IdleHandler:
def __init__(self):
self._retry_count = 0
def reset(self):
self._retry_count = 0
async def handle_idle(self, aggregator):
self._retry_count += 1
if self._retry_count == 1:
# First attempt - gentle reminder
message = {
"role": "developer",
"content": "The user has been quiet. Politely ask if they're still there.",
}
await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
elif self._retry_count == 2:
# Second attempt - more direct
message = {
"role": "developer",
"content": "The user is still inactive. Ask if they'd like to continue.",
}
await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
else:
# Third attempt - end conversation
await aggregator.push_frame(
TTSSpeakFrame("It seems like you're busy. Have a nice day!")
)
await aggregator.push_frame(EndWorkerFrame(), FrameDirection.UPSTREAM)
# Use the handler
idle_handler = IdleHandler()
@user_aggregator.event_handler("on_user_turn_idle")
async def on_user_turn_idle(aggregator):
await idle_handler.handle_idle(aggregator)
@user_aggregator.event_handler("on_user_turn_started")
async def on_user_turn_started(aggregator, strategy):
idle_handler.reset() # Reset retry count when user speaks
```
## Updating Timeout at Runtime
You can enable, disable, or change the idle timeout at runtime by pushing a `UserIdleTimeoutUpdateFrame`:
```python
from pipecat.frames.frames import UserIdleTimeoutUpdateFrame
# Enable idle detection (or change timeout)
await worker.queue_frame(UserIdleTimeoutUpdateFrame(timeout=10.0))
# Disable idle detection
await worker.queue_frame(UserIdleTimeoutUpdateFrame(timeout=0))
```
**Immediate application**: Timeout updates take effect immediately. If an idle timer is currently running, it restarts with the new duration. If the bot is waiting for the user to speak and you enable a positive timeout (e.g., upgrading from `0` to `10.0`), the timer arms right away without waiting for the next bot turn.
This is useful when you want to enable idle detection only at certain points in the conversation, or adjust the timeout based on context.
## Best Practices
- **Set appropriate timeouts**: Shorter timeouts (5-10 seconds) work well for voice conversations
- **Use escalating responses**: Start with gentle reminders and gradually become more direct
- **Limit retry attempts**: After 2-3 unsuccessful attempts, consider [ending the conversation](/pipecat/learn/pipeline-termination) gracefully by pushing an `EndWorkerFrame`
- **Reset on user activity**: Use the `on_user_turn_started` event to reset your retry counter when the user speaks
- **Let the LLM respond naturally**: Use developer messages to prompt the LLM rather than hardcoded TTS responses for more natural interactions
## Next Steps
Explore a complete working example that demonstrates how to detect and
respond to user inactivity in Pipecat.
Learn about all available turn events and their parameters.
Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate.
# STT Latency Tuning
Source: https://docs.pipecat.ai/pipecat/fundamentals/stt-latency-tuning.md
Measure and tune STT latency in Pipecat to improve turn detection timing and end-of-turn responsiveness.
## What is TTFS?
**Time To Final Segment (TTFS)** measures how long it takes from the moment a user stops speaking until the STT service delivers the final transcript. This latency directly affects how long your bot waits before it starts responding.
```
User stops β [TTFS latency] β Final transcript arrives β Bot starts
```
Every STT service has a different TTFS profile based on its architecture, model complexity, and infrastructure. Pipecat ships with measured P99 latency values for each supported service so that turn detection can account for this delay automatically. Measured values were benchmarked using the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) tool.
## Why TTFS matters
TTFS feeds directly into [turn stop strategies](/api-reference/server/utilities/turn-management/user-turn-strategies), which decide when the user has finished speaking and the bot should respond.
- **Value too low**: The turn stop strategy gives up waiting before the final transcript arrives. The bot responds based on incomplete text, or misses the user's input entirely.
- **Value too high**: The bot waits longer than necessary after the user stops speaking, creating awkward pauses in the conversation.
- **Value just right**: The bot waits long enough for the transcript to arrive, then responds immediately.
Getting TTFS right is one of the most impactful tuning knobs for perceived conversation responsiveness.
TTFS is a configuration value the turn stop strategy uses, not a metric that
is logged at runtime. To observe live latency in your running bot (time to
first byte, user-to-bot latency), see the [Metrics
guide](/pipecat/fundamentals/metrics).
## Default P99 latency values
Pipecat includes measured P99 TTFS values for every supported STT service. These are used automatically when you create a service β no configuration required.
For the current P99 TTFS value for each service, see [`stt_latency.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/stt_latency.py). These values are refined and added to often, so the source file is the source of truth.
These built-in values were all measured with `VADParams.stop_secs=0.2`, the
recommended default. If you change `stop_secs`, the built-in value no longer
matches your setup and Pipecat logs a warning. Re-run the benchmark with your
VAD settings and pass the measured value to your STT service constructor. See
the [Stop Strategies
section](/api-reference/server/utilities/turn-management/user-turn-strategies)
for the full explanation of how this interacts with turn detection.
Local services (NVIDIA, Whisper) default to 1.0s since actual latency depends
entirely on your hardware. Always measure and override for local deployments.
Turn-based STT services (for example `CartesiaTurnsSTTService` and
`DeepgramFluxSTTService`) have no meaningful TTFS value. The server defines
the turn boundary directly, so there is no separate "speech end to final
transcript" interval to measure.
## Measuring latency for your deployment
The default values are measured under standard conditions, but your actual latency depends on:
- **Network distance** to the STT provider
- **Region** where the service is hosted
- **Service configuration** (model size, language, features enabled)
- **Audio quality** and encoding settings
Use the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) tool to measure TTFS for your specific setup. The tool sends standardized audio samples to your STT service and reports P50, P90, and P99 latency values.
To run it, clone the repo and install with [`uv`](https://docs.astral.sh/uv/):
```bash
git clone https://github.com/pipecat-ai/stt-benchmark
cd stt-benchmark
uv sync
# Add your provider API keys
cp env.example .env
# Download standardized audio samples
uv run stt-benchmark download --num-samples 100
# Run the benchmark for one or more services
uv run stt-benchmark run --services deepgram,openai
# View the P50/P90/P99 report
uv run stt-benchmark report --service deepgram
```
If you run your bot with a non-default VAD setting, match the benchmark to it with `--vad-stop-secs` so the measured value reflects your configuration:
```bash
uv run stt-benchmark run --services deepgram --vad-stop-secs 0.3
```
See the [stt-benchmark README](https://github.com/pipecat-ai/stt-benchmark) for the full command reference.
## Overriding the default value
Pass the `ttfs_p99_latency` parameter to any STT service constructor to override the built-in default:
```python
from pipecat.services.deepgram.stt import DeepgramSTTService
# Use a measured value from your deployment
stt = DeepgramSTTService(
api_key=os.getenv("DEEPGRAM_API_KEY"),
ttfs_p99_latency=0.45, # Override with your measured P99
)
```
This value is broadcast to the pipeline via an `STTMetadataFrame` at startup, so turn stop strategies automatically adjust their timing.
If you're deploying to a specific region or using a self-hosted STT service,
always measure and override the default TTFS value. Even small differences
(e.g., 0.35s vs 0.55s) can noticeably affect conversation responsiveness.
# Context Summarization
Source: https://docs.pipecat.ai/pipecat/fundamentals/context-summarization.md
Automatically compress older conversation history in long-running Pipecat conversations to manage LLM token usage and cost.
## Overview
In long-running voice AI conversations, context grows with every exchange. This increases token usage, raises costs, and can eventually hit context window limits. Pipecat includes built-in context summarization that automatically compresses older conversation history while preserving recent messages and important context.
## How It Works
Context summarization automatically triggers when **either** condition is met:
- **Token limit reached**: Context size exceeds `max_context_tokens` (estimated using ~4 characters per token)
- **Message count reached**: Number of new messages exceeds `max_unsummarized_messages`
You can disable either threshold by setting it to `None`, but at least one must remain active. Summarization always generates a summary and cannot be reduced to pure truncation.
When triggered, the system:
1. Sends a `LLMContextSummaryRequestFrame` to the LLM service
2. The LLM generates a concise summary of older messages
3. Context is reconstructed as: `[system_message (if present)] + [summary] + [recent_messages]`
4. Incomplete function call sequences and recent messages are preserved
Context summarization is asynchronous and happens in the background without
blocking the pipeline. The system uses request IDs to match summary requests
with results and handles interruptions gracefully.
## Enabling Context Summarization
Enable summarization by setting `enable_auto_context_summarization=True` in `LLMAssistantAggregatorParams`:
```python
from pipecat.processors.aggregators.llm_response_universal import (
LLMAssistantAggregatorParams,
LLMContextAggregatorPair,
)
# Create aggregators with summarization enabled
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
assistant_params=LLMAssistantAggregatorParams(
enable_auto_context_summarization=True,
),
)
```
Automatic summarization is **disabled by default** (`enable_auto_context_summarization=False`). When enabled with the default configuration, summarization triggers at 8000 estimated tokens or after 20 new messages, whichever comes first.
## Customizing Behavior
Use `LLMAutoContextSummarizationConfig` and `LLMContextSummaryConfig` to tune the summarization triggers and output:
```python
from pipecat.utils.context.llm_context_summarization import (
LLMAutoContextSummarizationConfig,
LLMContextSummaryConfig,
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
assistant_params=LLMAssistantAggregatorParams(
enable_auto_context_summarization=True,
auto_context_summarization_config=LLMAutoContextSummarizationConfig(
max_context_tokens=4000, # Trigger at 4000 tokens
max_unsummarized_messages=10, # Or trigger after 10 new messages
summary_config=LLMContextSummaryConfig(
target_context_tokens=3000, # Target summary size
min_messages_after_summary=2, # Keep last 2 messages uncompressed
),
),
),
)
```
See the [reference page](/api-reference/server/utilities/context-summarization) for all available configuration parameters.
## What Gets Preserved
Context summarization intelligently preserves:
- **System messages**: If the first message (`messages[0]`) is a system message, it is preserved as the initial system prompt. Mid-conversation system messages (e.g., idle notifications or context injections) are treated as regular messages and included in the summarization range. When using [`system_instruction`](/pipecat/learn/context-management#using-system_instruction-recommended) in LLM Settings instead, the system prompt is not part of the context messages and is automatically prepended by the service on each request, so there is nothing to preserve in the context.
- **Recent messages**: The last N messages stay uncompressed (configured by `min_messages_after_summary`)
- **Function call sequences**: Incomplete function call/result pairs are not split during summarization
- **Developer messages are NOT preserved**: Developer messages (`"role": "developer"`) are included in the summarization range like any other message and may be compressed or dropped. If instructions need to survive summarization, use [`system_instruction`](/pipecat/learn/context-management#using-system_instruction-recommended) instead.
## Custom Summarization Prompts
You can override the default summarization prompt to control how the LLM generates summaries:
```python
custom_prompt = """Summarize this conversation concisely.
Focus on: key decisions, user preferences, and action items.
Keep the summary under {target_tokens} tokens."""
config = LLMAutoContextSummarizationConfig(
summary_config=LLMContextSummaryConfig(
summarization_prompt=custom_prompt,
),
)
```
When no custom prompt is provided, Pipecat uses a built-in prompt that instructs the LLM to create a concise summary preserving key information, user preferences, and conversation flow.
## Dedicated Summarization LLM
By default, summarization uses the same LLM service that handles conversation. You can route summarization to a separate, cheaper model by setting the `llm` field:
```python
from pipecat.services.google.llm import GoogleLLMService
# Use a fast/cheap model for summarization
summarization_llm = GoogleLLMService(
api_key=os.getenv("GOOGLE_API_KEY"),
model="gemini-2.5-flash",
)
config = LLMAutoContextSummarizationConfig(
summary_config=LLMContextSummaryConfig(
llm=summarization_llm,
),
)
```
When a dedicated LLM is configured, summarization requests bypass the pipeline entirely and call the dedicated service directly, so the primary conversation LLM is never interrupted.
## On-Demand Summarization
In addition to automatic summarization, you can trigger context summarization on demand by pushing an `LLMSummarizeContextFrame` into the pipeline. This is useful when you want to give users explicit control over when summarization happens β for example, via a function call tool.
```python
from pipecat.frames.frames import LLMSummarizeContextFrame
from pipecat.services.llm_service import FunctionCallParams
async def summarize_conversation(params: FunctionCallParams):
"""Summarize and compress the conversation history. Call this when the user asks you to summarize the conversation or when you want to free up context space."""
await params.result_callback({"status": "summarization_requested"})
await params.llm.queue_frame(LLMSummarizeContextFrame())
```
Above, `summarize_conversation` is a [direct function](/pipecat/learn/function-calling#1-define-a-tool). List it in the context's `tools` so the LLM can invoke it when the user asks to summarize:
```python
context = LLMContext(messages, tools=[summarize_conversation])
```
On-demand summarization works even when `enable_auto_context_summarization` is `False` β the summarizer is always created internally to handle manually pushed frames. You can also pass a per-request `LLMContextSummaryConfig` to override the default settings:
```python
from pipecat.utils.context.llm_context_summarization import LLMContextSummaryConfig
await llm.queue_frame(
LLMSummarizeContextFrame(
config=LLMContextSummaryConfig(
target_context_tokens=2000,
min_messages_after_summary=2,
)
)
)
```
See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/context-summarization/context-summarization-manual-openai.py) for a full working implementation.
## Observability
The summarizer emits an `on_summary_applied` event after each successful summarization, providing message count metrics:
```python
from pipecat.processors.aggregators.llm_context_summarizer import SummaryAppliedEvent
summarizer = assistant_aggregator._summarizer
if summarizer:
@summarizer.event_handler("on_summary_applied")
async def on_summary_applied(summarizer, event: SummaryAppliedEvent):
logger.info(
f"Context summarized: {event.original_message_count} messages -> "
f"{event.new_message_count} messages "
f"({event.summarized_message_count} summarized, "
f"{event.preserved_message_count} preserved)"
)
```
## Next Steps
Full reference for configuration parameters, events, and classes.
Learn how Pipecat manages conversation context in pipelines.
# Saving Conversation Transcripts
Source: https://docs.pipecat.ai/pipecat/fundamentals/saving-transcripts.md
Learn how to collect and save conversation transcripts between users and your bot
## Overview
Recording transcripts of conversations between users and your bot is useful for debugging, analysis, and creating a record of interactions. Pipecat's turn events make it easy to collect both user and assistant messages as they occur.
## How It Works
Transcripts are collected using turn events on the context aggregators:
1. Capturing what the user says via `on_user_turn_stopped`
2. Capturing what the assistant says via `on_assistant_turn_stopped`
3. Each event provides the complete transcript for that turn
4. Allowing you to handle these events with custom logic
Turn events are emitted by the context aggregators (`LLMUserAggregator` and
`LLMAssistantAggregator`), which are created as part of the
`LLMContextAggregatorPair`.
## Basic Implementation
### Step 1: Create Context Aggregators
First, create the context aggregator pair and get references to both aggregators:
```python
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
UserTurnStoppedMessage,
AssistantTurnStoppedMessage,
)
# Create context aggregator pair
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```
### Step 2: Add to Your Pipeline
Include the aggregators in your pipeline:
```python
pipeline = Pipeline(
[
transport.input(),
stt, # Speech-to-text
user_aggregator,
llm,
tts, # Text-to-speech
transport.output(),
assistant_aggregator,
]
)
```
### Step 3: Handle Turn Events
Register event handlers to capture transcripts when turns complete:
```python
@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
print(f"[{message.timestamp}] user: {message.content}")
@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
if message.content:
print(f"[{message.timestamp}] assistant: {message.content}")
if message.interrupted:
print(f"[{message.timestamp}] (assistant was interrupted)")
```
In addition to console logging, you can save transcripts to a database or file
for later analysis.
With a realtime (speech-to-speech) service and
`LLMContextAggregatorPair(context, realtime_service_mode=True)`, capture the
user transcript from
[`on_user_turn_message_added`](/api-reference/server/utilities/turn-management/turn-events#on_user_turn_message_added)
instead of `on_user_turn_stopped`, whose `message.content` is `None` in that
mode.
## Next Steps
Learn about all available turn events and their parameters.
See more examples for collecting and processing transcriptions.
Consider implementing transcript recording in your application for debugging during development and preserving important conversations in production. The transcript data can also be useful for analyzing conversation patterns and improving your bot's responses over time.
# Recording Conversation Audio
Source: https://docs.pipecat.ai/pipecat/fundamentals/recording-audio.md
Learn how to record and save audio from conversations between users and your bot
## Overview
Recording audio from conversations provides valuable data for analysis, debugging, and quality control. You have two options for how to record with Pipecat:
### Option 1: Record using your transport service provider
Record without writing custom code by using your [transport](/api-reference/server/services/supported-services#transports) provider's recording capabilities. In addition to saving you development time, some providers offer unique recording capabilities.
Refer to your service provider's documentation to learn more.
For example, see [Recording](/api-reference/server/services/transport/daily#recording) in the `DailyTransport` reference.
### Option 2: Create your own recording pipeline
Pipecat's `AudioBufferProcessor` makes it easy to capture high-quality audio recordings of both the user and bot during interactions. Opt for this approach if you want more control over your recording.
The `AudioBufferProcessor.start_recording()` / `stop_recording()` methods
below control this in-pipeline buffer. They are separate from a transport
provider's own recording.
This guide focuses on how to recording using the `AudioBufferProcessor`, including high-level guidance for how to set up post-processing jobs for longer recordings.
## How the AudioBufferProcessor Works
The `AudioBufferProcessor` captures audio by:
1. Collecting audio frames from both the user (input) and bot (output)
2. Emitting events with recorded audio data
3. Providing options for composite or separate track recordings
Add the processor to your pipeline after the `transport.output()` to capture
both the user audio and the bot audio as it's spoken.
## Audio Recording Options
The `AudioBufferProcessor` offers several configuration options:
- **Composite recording**: Combined audio from both user and bot
- `on_audio_data` event handler
- **Track-level recording**: Separate audio files for user and bot
- `on_track_audio_data` event handler
- **Turn-based recording**: Individual audio clips for each speaking turn
- `on_user_turn_audio_data` and `on_bot_turn_audio_data` event handlers
- **Mono or stereo output**: Single channel mixing or two-channel separation
- `num_channels=1` for mono; `num_channels=2` for stereo
## Basic Implementation
### Step 1: Create an Audio Buffer Processor
Initialize the audio buffer processor with your desired configuration:
```python
from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor
# Create audio buffer processor with default settings
audiobuffer = AudioBufferProcessor(
num_channels=1, # 1 for mono, 2 for stereo (user left, bot right)
enable_turn_audio=False, # Enable per-turn audio recording
auto_start_recording=False, # Start recording automatically when pipeline starts
)
```
### Step 2: Add to Your Pipeline
Place the processor in your pipeline after all audio-producing components:
```python
pipeline = Pipeline(
[
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
transport.output(),
audiobuffer, # Add after all audio components
context_aggregator.assistant(),
]
)
```
### Step 3: Start Recording
You have two options for starting recording:
**Option A: Automatic recording** (recommended for most use cases)
Set `auto_start_recording=True` when initializing the processor to begin recording as soon as the pipeline starts:
```python
audiobuffer = AudioBufferProcessor(
num_channels=1,
auto_start_recording=True, # Recording starts automatically
)
```
**Option B: Manual control**
Explicitly start recording when needed, typically when a session begins:
```python
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
logger.info(f"Client connected")
# Start recording explicitly
await audiobuffer.start_recording()
# Continue with session initialization...
```
Without `auto_start_recording=True`, you must call `start_recording()`
explicitly to begin capturing audio.
### Step 4: Handle Audio Data
Register an event handler to process audio data:
```python
@audiobuffer.event_handler("on_audio_data")
async def on_audio_data(buffer, audio, sample_rate, num_channels):
# Save or process the composite audio
timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
filename = f"recordings/conversation_{timestamp}.wav"
# Create the WAV file
with wave.open(filename, "wb") as wf:
wf.setnchannels(num_channels)
wf.setsampwidth(2) # 16-bit audio
wf.setframerate(sample_rate)
wf.writeframes(audio)
logger.info(f"Saved recording to {filename}")
```
If recording separate tracks, you can use the `on_track_audio_data` event
handler to save user and bot audio separately.
## Recording Longer Conversations
For conversations that last a few minutes, it may be sufficient to just buffer the audio in memory. However, for longer sessions, storing audio in memory poses two challenges:
1. **Memory Usage**: Long recordings can consume significant memory, leading to potential crashes or performance issues.
2. **Conversation Loss**: If the application crashes or the connection drops, you may lose all recorded audio.
Instead, use the `buffer_size` parameter to record audio in manageable segments. This allows you to periodically save audio data to disk or upload it to cloud storage, reducing memory usage and ensuring data persistence.
See an example of how to upload chunked audio to AWS cloud storage [here]().
### Chunked Recording
Set a reasonable `buffer_size` to trigger periodic uploads:
```python
# 30-second chunks (recommended for most use cases)
SAMPLE_RATE = 24000
CHUNK_DURATION = 30 # seconds
audiobuffer = AudioBufferProcessor(
sample_rate=SAMPLE_RATE,
buffer_size=SAMPLE_RATE * 2 * CHUNK_DURATION # 2 bytes per sample (16-bit)
)
chunk_counter = 0
@audiobuffer.event_handler("on_track_audio_data")
async def on_chunk_ready(buffer, user_audio, bot_audio, sample_rate, num_channels):
global chunk_counter
# Upload or save individual chunks
await upload_audio_chunk(f"user_chunk_{chunk_counter:03d}.wav", user_audio, sample_rate, 1)
await upload_audio_chunk(f"bot_chunk_{chunk_counter:03d}.wav", bot_audio, sample_rate, 1)
chunk_counter += 1
```
### Multipart Upload Strategy
For cloud storage, use multipart uploads to stream audio chunks. For example AWS cloud storage, use the [s3 multipart upload API](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html).
If you are rolling your own multipart upload code, consider the following:
**Conceptual Approach:**
1. **Initialize multipart upload** when recording starts
2. **Upload chunks as parts** when buffers fill (every ~30 seconds)
3. **Complete multipart upload** when recording ends
4. **Post-process** to create final WAV file(s), concatenate audio chunks
**Benefits:**
- Memory efficient for long sessions
- Fault tolerant (no data loss if connection drops)
- Enables real-time processing and analysis
- Parallel upload of multiple tracks
### [Optional] Post-Processing Pipeline
If not using a managed multipart upload framework like AWS s3 multipart upload, concatenate audio chunks together to create final audio files.
This can be done with tools like FFmpeg:
**Concatenating Audio Files:**
```bash
# Method 1: Simple concatenation (same format)
ffmpeg -i "concat:chunk_001.wav|chunk_002.wav|chunk_003.wav" -acodec copy final.wav
# Method 2: Using file list (recommended for many chunks)
# Create filelist.txt with format:
# file 'chunk_001.wav'
# file 'chunk_002.wav'
# ...
ffmpeg -f concat -safe 0 -i filelist.txt -c copy final_recording.wav
```
**Automation Considerations:**
- Use sequence numbers in chunk filenames for proper ordering
- Include metadata (sample rate, channels, duration) with each chunk
- Implement retry logic for failed uploads
- Consider using cloud functions/lambdas for automatic post-processing
## Next Steps
Explore a complete working example that demonstrates how to record and save
both composite and track-level audio with Pipecat.
Read the complete API reference documentation for advanced configuration
options and event handlers.
Consider implementing audio recording in your application for quality assurance, training data collection, or creating conversation archives. The recorded audio can be stored locally, uploaded to cloud storage, or processed in real-time for further analysis.
# Metrics
Source: https://docs.pipecat.ai/pipecat/fundamentals/metrics.md
Monitor Pipecat performance metrics: TTFB, processing time, and LLM and TTS usage across pipeline services.
When developing real-time, multimodal AI applications, monitoring two key
factors is crucial: performance (latency) and LLM/TTS usage. Performance impacts
user experience, while usage can affect operational costs. Pipecat offers
built-in metrics for both, which can be enabled with straightforward
configuration options.
## Enabling performance metrics
Set `enable_metrics=True` in `PipelineParams` when creating a worker:
```python Example config
worker = PipelineWorker(
pipeline,
params=PipelineParams(
...
enable_metrics=True,
...
),
)
```
Once enabled, Pipecat logs the following metrics:
| Metric | Description |
| ---------------- | -------------------------------------------------------------------------------- |
| TTFB | Time To First Byte in seconds |
| TTFA | Time To First Audio in seconds (TTS services only) |
| TTFAT | Time To First Answer Token in seconds (LLM services only) |
| Processing Time | Time taken by the service to respond in seconds |
| Text Aggregation | Time from the first LLM token to the first complete sentence (TTS services only) |
```console Sample output
AnthropicLLMService#0 TTFB: 0.8378312587738037
CartesiaTTSService#0 text aggregation time: 0.2134
CartesiaTTSService#0 TTFB: 0.17177796363830566
AnthropicLLMService#0 processing time: 2.4927797317504883
```
### Limiting TTFB responses
If you only want the **first** TTFB measurement for each service, you can
optionally pass `report_only_initial_ttfb=True` in `PipelineParams`:
```python Example config
worker = PipelineWorker(
pipeline,
params=PipelineParams(
...
enable_metrics=True,
report_only_initial_ttfb=True,
...
),
)
```
> **Note:** `enable_metrics=True` is required for this setting to have an
> effect.
### Disabling initial empty metrics
By default, Pipecat sends an initial `MetricsFrame` with zero values for all
services when the pipeline starts. To disable this behavior:
```python Example config
worker = PipelineWorker(
pipeline,
params=PipelineParams(
...
enable_metrics=True,
send_initial_empty_metrics=False,
...
),
)
```
## Enabling LLM/TTS Usage Metrics
Set `enable_usage_metrics=True` in PipelineParams when creating a worker:
```python Example config
worker = PipelineWorker(
pipeline,
params=PipelineParams(
...
enable_usage_metrics=True,
...
),
)
```
Pipecat will log the following as applicable:
| Metric | Description |
| --------- | ------------------------------------------- |
| LLM Usage | Number of prompt and completion tokens used |
| TTS Usage | Number of characters processed |
```console Sample output
CartesiaTTSService#0 usage characters: 65
AnthropicLLMService#0 prompt tokens: 104, completion tokens: 53
```
> **Note:** Usage metrics are recorded per interaction and do not represent
> running totals.
## Capturing Metrics Data
When metrics are enabled, Pipecat emits a `MetricsFrame` for each interaction. The `MetricsFrame` contains a list of metrics data objects, which can include:
- `TTFBMetricsData` β Time To First Byte
- `TTFAMetricsData` β Time To First Audio (TTS)
- `TTFATMetricsData` β Time To First Answer Token (LLM)
- `ProcessingMetricsData` β Processing time
- `LLMUsageMetricsData` β LLM token usage
- `TTSUsageMetricsData` β TTS character usage
- `TextAggregationMetricsData` β Sentence aggregation latency (TTS)
- `TurnMetricsData` β Turn completion predictions
You can access the metrics data by either adding a custom [FrameProcessor](/pipecat/fundamentals/custom-frame-processor) to your pipeline or adding an [observer](/api-reference/server/utilities/observers/observer-pattern) to monitor `MetricsFrame`s.
### Example: Using MetricsLogObserver
The simplest way to log metrics is with the built-in `MetricsLogObserver`. Pass it as an observer when creating your `PipelineWorker`:
```python
from pipecat.observers.loggers.metrics_log_observer import MetricsLogObserver
worker = PipelineWorker(
pipeline,
params=PipelineParams(enable_metrics=True, enable_usage_metrics=True),
observers=[MetricsLogObserver()],
)
```
You can filter which metrics types are logged by passing `include_metrics`:
```python
from pipecat.metrics.metrics import LLMUsageMetricsData, TTSUsageMetricsData
from pipecat.observers.loggers.metrics_log_observer import MetricsLogObserver
observers = [
MetricsLogObserver(
include_metrics={LLMUsageMetricsData, TTSUsageMetricsData}
)
]
```
### Example: Using a Custom FrameProcessor
Create a custom FrameProcessor to handle metrics data. Here's an example Metrics Processor that can be added to your pipeline after the TTS processor.
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Metrics Data Reference
All metrics data classes inherit from `MetricsData`, which includes `processor` (the name of the processor that generated the metric) and an optional `model` field.
### TTFBMetricsData
Time To First Byte β measures how long until the first byte of a response is received from a service.
| Field | Type | Description |
| ------- | ------- | --------------------------- |
| `value` | `float` | TTFB measurement in seconds |
### TTFAMetricsData
Time To First Audio β measures the time from a TTS request to the first audible audio sample. This includes the time to first byte plus any leading silence padding added by the service. `TTFAMetricsData` reports the latency breakdown directly, showing how much of the perceived latency is silence padding.
| Field | Type | Description |
| ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------- |
| `ttfa` | `float` | TTFA measurement in seconds (`ttfb` plus `leading_silence`) |
| `ttfb` | `float` | Time-to-first-byte in seconds. Mirrors the standalone `TTFBMetricsData` for convenience, not a separate measurement |
| `leading_silence` | `float` | Silence padding before the first audible sample, in seconds (`ttfa` minus `ttfb`) |
### TTFATMetricsData
Time To First Answer Token β measures the time from an LLM request to the first token of the answer the caller sees. This is time-to-first-byte plus any reasoning the model streamed first. `TTFATMetricsData` reports the latency breakdown, showing how much of the perceived latency is the model thinking rather than responding.
A turn that answers with a tool call rather than text ends the measurement at the call. Answering from a tool result takes a second inference, so such a turn reports twice β once for the call, once for the answer built from its result.
Reported only by LLM services that answer in text. Speech-to-speech services answer in audio, which has no answer token to measure to.
| Field | Type | Description |
| --------------- | ------- | ------------------------------------------------------------------------------------------------------------------- |
| `ttfat` | `float` | TTFAT measurement in seconds (`ttfb` plus `thinking_time`) |
| `ttfb` | `float` | Time-to-first-byte in seconds. Mirrors the standalone `TTFBMetricsData` for convenience, not a separate measurement |
| `thinking_time` | `float` | Time between the model's first output and the first answer token, in seconds (`ttfat` minus `ttfb`) |
### ProcessingMetricsData
Measures the total time taken by a service to process a request.
| Field | Type | Description |
| ------- | ------- | -------------------------------------- |
| `value` | `float` | Processing time measurement in seconds |
WebSocket-based TTS services (Cartesia, Deepgram, ElevenLabs WebSocket
variants) and `DeepgramSageMakerTTSService` do not report processing metrics;
TTFB and TTFA capture the meaningful latency for these services. Streaming STT
services (subclasses of `STTService`) don't report them either, because audio
arrives continuously and there is no discrete request whose duration a
processing window could measure; TTFB is the meaningful measurement there.
`SegmentedSTTService` subclasses do issue one request per utterance, so they
report both.
### TextAggregationMetricsData
Measures the time from the first LLM token to the first complete sentence, representing the latency cost of sentence aggregation in the TTS pipeline.
| Field | Type | Description |
| ------- | ------- | --------------------------- |
| `value` | `float` | Aggregation time in seconds |
### LLMUsageMetricsData
Token usage for an LLM interaction. The `value` field is an `LLMTokenUsage` object with:
| Field | Type | Description |
| ------------------------------- | --------------- | --------------------------------------------------- |
| `prompt_tokens` | `int` | Number of tokens in the input prompt |
| `completion_tokens` | `int` | Number of tokens in the generated completion |
| `total_tokens` | `int` | Total tokens used (prompt + completion) |
| `cache_read_input_tokens` | `Optional[int]` | Tokens read from cache, if applicable |
| `cache_creation_input_tokens` | `Optional[int]` | Tokens used to create cache entries |
| `reasoning_tokens` | `Optional[int]` | Reasoning tokens (for reasoning models) |
| `input_audio_tokens` | `Optional[int]` | Prompt tokens that were audio (realtime models) |
| `output_audio_tokens` | `Optional[int]` | Completion tokens that were audio (realtime models) |
| `cache_read_input_audio_tokens` | `Optional[int]` | Cache-read tokens that were audio (realtime models) |
### TTSUsageMetricsData
Character usage for a TTS interaction.
| Field | Type | Description |
| ------- | ----- | ------------------------------------- |
| `value` | `int` | Number of characters processed by TTS |
### TurnMetricsData
Metrics from turn completion prediction, emitted by turn analyzers like Krisp Viva Turn and Smart Turn.
| Field | Type | Description |
| ------------------------ | ------- | ------------------------------------------------------------------------------------------ |
| `is_complete` | `bool` | Whether the turn is predicted to be complete |
| `probability` | `float` | Confidence probability of the prediction |
| `e2e_processing_time_ms` | `float` | End-to-end processing time in ms, from VAD speech-to-silence transition to turn completion |
## Related Observers
In addition to `MetricsLogObserver`, Pipecat provides observers that track higher-level conversational metrics.
### StartupTimingObserver
Measures the time taken by each processor to start up.
```python
from pipecat.observers.startup_timing_observer import StartupTimingObserver
startup_observer = StartupTimingObserver()
@observer.event_handler("on_startup_timing_report")
async def on_startup_timing_report(observer, report):
print(f"Total startup: {report.total_duration_secs:.3f}s")
for timing in report.processor_timings:
print(f" {timing.processor_name}: {timing.duration_secs:.3f}s")
```
Additionally, it tracks the time taken to connect to the transport and the time taken to connect to the client.
```python
@observer.event_handler("on_transport_timing_report")
async def on_transport_timing_report(observer, report):
if report.bot_connected_secs is not None:
print(f"Bot connected: {report.bot_connected_secs:.3f}s")
print(f"Client connected: {report.client_connected_secs:.3f}s")
```
### UserBotLatencyObserver
Measures the time between when a user stops speaking and when the bot starts speaking.
```python
from pipecat.observers.user_bot_latency_observer import UserBotLatencyObserver
latency_observer = UserBotLatencyObserver()
@latency_observer.event_handler("on_latency_measured")
async def on_latency_measured(observer, latency_seconds):
print(f"User-to-bot latency: {latency_seconds:.3f}s")
worker = PipelineWorker(pipeline, observers=[latency_observer])
```
### TurnTrackingObserver
Tracks conversation turns, emitting events when turns start and end. Handles interruptions and configurable timeouts.
```python
from pipecat.observers.turn_tracking_observer import TurnTrackingObserver
turn_observer = TurnTrackingObserver(turn_end_timeout_secs=2.5)
@turn_observer.event_handler("on_turn_started")
async def on_turn_started(observer, turn_count):
print(f"Turn {turn_count} started")
@turn_observer.event_handler("on_turn_ended")
async def on_turn_ended(observer, turn_count, duration, was_interrupted):
status = "interrupted" if was_interrupted else "completed"
print(f"Turn {turn_count} {status} after {duration:.2f}s")
worker = PipelineWorker(pipeline, observers=[turn_observer])
```
# Voicemail Detection
Source: https://docs.pipecat.ai/pipecat/fundamentals/voicemail.md
Automatically classify outbound calls as conversation or voicemail and respond appropriately
## Overview
The VoicemailDetector classifies incoming communication as either live conversation or voicemail systems. This module is built primarily for voice AI bots that perform outbound calling, enabling them to respond appropriately based on whether a human answered or the call went to voicemail.
The detector is optimized for fast conversation response times, where TTS output is generated immediately but held in a gate until the classification decision is made. This ensures minimal latency for live conversations while preventing inappropriate responses to voicemail systems.
## How It Works
The VoicemailDetector uses a parallel pipeline architecture to perform real-time classification without interrupting conversation flow. It analyzes the initial response from the called party and determines whether it's a human greeting or an automated voicemail system.
Key features:
- **Real-time classification** - Determines conversation vs voicemail as soon as audio is received
- **TTS gating** - Holds generated audio until classification is complete
- **Event-driven** - Triggers custom handlers when voicemail is detected
- **Configurable timing** - Adjustable delay for voicemail response timing
## Basic Setup
### 1. Initialize the Detector
```python
from pipecat.extensions.voicemail.voicemail_detector import VoicemailDetector
from pipecat.services.openai.llm import OpenAILLMService
# Create an LLM for classification
classifier_llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
# Initialize the detector
voicemail_detector = VoicemailDetector(
llm=classifier_llm,
voicemail_response_delay=2.0 # Default: 2 seconds
)
```
The VoicemailDetector works with two LLMs: your main conversation LLM (which
must be text-based) and the classifier LLM (which can be either text-based or
realtime with a `text` output modality). Realtime LLMs are not compatible for
the main conversation LLM due to output control requirements.
### 2. Configure the Pipeline
The VoicemailDetector requires two components in your pipeline:
- `detector()`: between STT and user context aggregator
- `gate()`: immediately after TTS service
```python
pipeline = Pipeline([
transport.input(),
stt,
voicemail_detector.detector(), # Between STT and user context aggregator
context_aggregator.user(),
llm,
tts,
voicemail_detector.gate(), # Immediately after TTS service
transport.output(),
context_aggregator.assistant(),
])
```
### 3. Handle Voicemail Events
When a voicemail is detected, the `on_voicemail_detected` event is triggered. In your event handler, you have access to `processor`, which is a FrameProcessor instance, allowing you to push frames to the pipeline. For example, you may want to have the bot output a pre-canned message and then end the call.
```python
@voicemail_detector.event_handler("on_voicemail_detected")
async def handle_voicemail(processor):
logger.info("Voicemail detected! Leaving a message...")
# Leave a voicemail message
await processor.push_frame(
TTSSpeakFrame("Hello, this is Jamie calling about your appointment. Please call me back at 555-0123.")
)
# Optionally end the call after leaving the message
await processor.push_frame(EndWorkerFrame(), FrameDirection.UPSTREAM)
```
## Detecting a Conversation
When a conversation is detected, no additional processing is required by the VoicemailDetector. Once the VoicemailDetector classifies the call as a conversation, the TTSGate will be flushed, allowing the conversation to continue.
## Configuration Options
### Custom System Prompt
For specialized use cases, you can provide a custom classification prompt. The prompt must instruct the LLM to respond with exactly "CONVERSATION" or "VOICEMAIL":
```python
custom_prompt = """
Your custom classification logic here.
Consider factors like business hours, call patterns, etc.
""" + VoicemailDetector.CLASSIFIER_RESPONSE_INSTRUCTION
voicemail_detector = VoicemailDetector(
llm=classifier_llm,
custom_system_prompt=custom_prompt
)
```
The `CLASSIFIER_RESPONSE_INSTRUCTION` constant contains the required response
format: `'Respond with ONLY "CONVERSATION" if a person answered, or
"VOICEMAIL" if it\'s voicemail/recording.'`
For most use cases, the default classifier prompt will work effectively. If you need to customize the behavior, reference the built-in prompt as a starting point and modify from there.
### Response Timing
The `voicemail_response_delay` parameter controls how long to wait after the user stops speaking before triggering the voicemail event:
```python
voicemail_detector = VoicemailDetector(
llm=classifier_llm,
voicemail_response_delay=3.0 # Wait 3 seconds instead of default 2
)
```
This delay ensures:
- Voicemail greetings have finished playing
- The response occurs during the recording period
- Proper timing for different voicemail system behaviors
# IVR Navigation
Source: https://docs.pipecat.ai/pipecat/fundamentals/ivr.md
Automatically navigate phone system menus using AI-powered decision making
## Overview
The `IVRNavigator` enables your bot to automatically navigate Interactive Voice Response (IVR) phone systems to reach specific goals. Instead of manually programming navigation paths, you provide an end goal and the bot handles the complex decision-making required to traverse phone menus using DTMF tones and conversational responses.
## How IVR Navigation Works
The IVRNavigator combines several intelligent capabilities:
1. **Goal-oriented navigation**: You specify what you want to accomplish (e.g., "reach billing support")
2. **Automatic classification**: Detects whether incoming audio is an IVR system or human conversation
3. **Smart decision making**: Analyzes menu options and selects the best path toward your goal
4. **Multi-modal responses**: Uses both DTMF tones for menu selection, natural language for prompts, and waits when no input is appropriate
5. **Status tracking**: Monitors progress and reports completion, waiting, or stuck states
## Navigation Outcomes
The `IVRNavigator` can reach several outcomes during navigation:
### Completed β
The navigator successfully reaches its goal and is ready for the next step. Common actions:
- [Terminate the pipeline](/pipecat/learn/pipeline-termination) if the goal is complete
- Transfer the call to a human agent
- Allow your bot to start a conversation with the reached department
```python
@ivr_navigator.event_handler("on_ivr_status_changed")
async def handle_ivr_status(processor, status):
if status == IVRStatus.COMPLETED:
logger.info("Successfully navigated to target department")
# Start conversation
# Replace the context with new messages for the upcoming conversation
messages = [{"role": "developer", "content": "You are a helpful customer service assistant."}]
await worker.queue_frame(LLMMessagesUpdateFrame(messages))
# Adjust VAD stop_secs for conversation
vad_params = VADParams(stop_secs=0.8)
await worker.queue_frame(VADParamsUpdateFrame(vad_params))
```
### Stuck β οΈ
The navigator cannot find a path forward in the IVR system. This might happen when:
- Required information (account numbers, PINs) isn't available
- The menu options don't align with the stated goal
- The system encounters errors or invalid selections
```python
@ivr_navigator.event_handler("on_ivr_status_changed")
async def handle_ivr_status(processor, status):
if status == IVRStatus.STUCK:
logger.warning("IVR navigation stuck - terminating call")
# Log the issue and clean up
await log_navigation_failure()
await worker.queue_frame(EndFrame())
```
## Flexible Entry Points
One of the `IVRNavigator`'s key features is flexible entry point handling. When you dial a phone number, you might encounter either:
- An IVR system with menu options
- A direct connection to a human
The navigator automatically detects which scenario occurs and emits appropriate events:
### Human Conversation Detected
When a human answers instead of an IVR system, an `on_conversation_detected` event is emitted. You can handle that event to transition to a conversation.
```python
@ivr_navigator.event_handler("on_conversation_detected")
async def on_conversation_detected(processor, conversation_history):
# Set up conversation prompt and preserve conversation history
messages = [
{
"role": "developer",
"content": "You are an assistant calling to check on the prescription for John Smith, date of birth 01/01/1990.",
}
]
# Add preserved conversation history if available
if conversation_history:
messages.extend(conversation_history)
await worker.queue_frame(LLMMessagesUpdateFrame(messages=messages, run_llm=True))
```
Note that the `on_conversation_detected` event also emits a `conversation_history` parameter that contains the previous conversation history. This allows you to build a prompt that includes your conversation system prompt plus any conversation history up to that point in time.
### IVR System Detected
When an IVR system is detected, the IVR Navigator automatically transitions into navigation mode:
- **System prompt**: Updates to use your specified navigation goal
- **VAD timing**: Adjusts to `stop_secs=2.0` (or your custom `ivr_vad_params`) to allow time for complete menu announcements
- **Navigation logic**: Begins analyzing menu options and making decisions toward your goal
No additional code is required. The navigator handles this transition automatically.
**Optional Event Handling**
If you need to log the detection or perform custom actions, you can handle the `on_ivr_status_changed` event:
```python
@ivr_navigator.event_handler("on_ivr_status_changed")
async def on_ivr_status_changed(processor, status):
if status == IVRStatus.DETECTED:
logger.info("IVR system detected - beginning navigation")
# Optional: Add analytics tracking, custom setup, etc.
```
## Basic Implementation
### Step 1: Create the IVR Navigator
```python
from pipecat.extensions.ivr.ivr_navigator import IVRNavigator
from pipecat.audio.vad.vad_analyzer import VADParams
# Define your navigation goal
ivr_goal = "Navigate to the billing department to discuss my account balance"
# Create navigator with extended response time for IVR systems
ivr_vad_params = VADParams(stop_secs=2.0) # Longer wait for IVR menus
ivr_navigator = IVRNavigator(
llm=your_llm_service,
ivr_prompt=ivr_goal,
ivr_vad_params=ivr_vad_params
)
```
### Step 2: Set Up Event Handlers
```python
from pipecat.frames.frames import LLMMessagesUpdateFrame
from pipecat.extensions.ivr.ivr_navigator import IVRStatus
@ivr_navigator.event_handler("on_conversation_detected")
async def on_conversation_detected(processor, conversation_history):
"""Handle when a human conversation is detected instead of IVR"""
logger.info("Human conversation detected")
# Set up conversation context
messages = [
{"role": "developer", "content": "You are a customer service representative."}
]
if conversation_history:
messages.extend(conversation_history)
await worker.queue_frame(LLMMessagesUpdateFrame(messages=messages, run_llm=True))
@ivr_navigator.event_handler("on_ivr_status_changed")
async def on_ivr_status_changed(processor, status):
"""Handle IVR navigation status changes"""
if status == IVRStatus.COMPLETED:
logger.info("IVR navigation completed successfully")
# Your success handling logic here
elif status == IVRStatus.STUCK:
logger.warning("IVR navigation got stuck")
# Your error handling logic here
await handle_navigation_failure()
```
### Step 3: Add to Pipeline
Add the IVR Navigator to your pipeline in the place where you would normally add the LLM. The IVR Navigator contains your LLM and will perform the same functions as an LLM would, but in addition it will navigate the IVR system.
```python
from pipecat.pipeline.pipeline import Pipeline
pipeline = Pipeline([
transport.input(),
stt_service,
ivr_navigator, # Add the navigator to your pipeline
tts_service,
transport.output()
])
```
## VAD Parameter Optimization
The IVRNavigator automatically optimizes Voice Activity Detection (VAD) parameters for different scenarios:
### IVR Navigation Mode
- **Default**: `stop_secs=2.0`
- **Purpose**: Allows time to hear complete menu options before responding
- **Result**: Higher navigation success rates
### Conversation Mode
- **Recommended**: `stop_secs=0.8`
- **Purpose**: Enables natural conversation flow with quick responses
- **Implementation**: Push `VADParamsUpdateFrame` when transitioning to conversation
## Next Steps
Learn how to properly terminate pipelines when IVR navigation completes
Integrate structured conversation flows after successful IVR navigation
The IVRNavigator provides a powerful foundation for automating phone system interactions, allowing your bots to handle the complex task of menu navigation while you focus on the core conversation logic.
# Custom FrameProcessor
Source: https://docs.pipecat.ai/pipecat/fundamentals/custom-frame-processor.md
Write a custom Pipecat FrameProcessor: handle frames, push new ones downstream, and slot into a pipeline.
Pipecat's architecture is made up of a Pipeline, FrameProcessors, and Frames. See the [Core Concepts](/pipecat/learn/pipeline) for a full review. From that architecture, recall that FrameProcessors are the workers in the pipeline that receive frames and complete actions based on the frames received.
Pipecat comes with many FrameProcessors built in. These consist of services, like `OpenAILLMService` or `CartesiaTTSService`, utilities, like `LLMTextProcessor`, and other things. Largely, you can build most of your application with these built-in FrameProcessors, but commonly, your application code may require custom frame processing logic. For example, you may want to perform an action as a result of a frame that's pushed in the pipeline.
## Example: MetricsFrame logger
This custom FrameProcessor format and logs MetricsFrames:
```python
class MetricsFrameLogger(FrameProcessor):
"""MetricsFrameLogger formats and logs all MetericsFrames"""
def __init__(self):
super().__init__()
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, MetricsFrame):
logger.info(f"{frame.name}\n {format_metrics(frame.data)}")
await self.push_frame(frame, direction)
# ALWAYS push all frames
else:
# SUPER IMPORTANT: always push every frame!
await self.push_frame(frame, direction)
```
This frame processor looks for `MetricsFrames`. When it sees one, it formats the data and logs it.
It uses this `format_metrics` function:
```python
def format_metrics(metrics, indent=0):
lines = []
tab = "\t" * indent
for metric in metrics:
lines.append(tab + type(metric).__name__)
for field, value in vars(metric).items():
if hasattr(value, "__dict__") and not isinstance(
value, (str, int, float, bool, type(None))
):
lines.append(f"{tab}\t{field}={type(value).__name__}")
for k, v in vars(value).items():
lines.append(f"{tab}\t\t{k}={repr(v)}")
else:
lines.append(f"{tab}\t{field}={repr(value)}")
return "\n".join(lines)
```
See this [working
example](https://github.com/pipecat-ai/pipecat/blob/main/examples/features/features-custom-frame-processor.py)
using the `MetricsFrameLogger` FrameProcessor
## Add to a Pipeline
```python
# Create and initialize the custom FrameProcessor
metrics_frame_processor = MetricsFrameLogger()
pipeline = Pipeline(
[
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
metrics_frame_processor, # Our custom FrameProcessor that pretty prints metrics frames
]
)
```
With this positioning, the `MetricsFrameLogger` FrameProcessor will receive every MetericsFrame in the pipeline.
## Key Requirements
FrameProcessors must inherit from the base `FrameProcessor` class. This ensures that your custom FrameProcessor will correctly handle frames like `StartFrame`, `EndFrame`, `InterruptionFrame` without having to write custom logic for those frames. This inheritance also provides it with the ability to `process_frame()` and `push_frame()`:
- **`process_frame()`** is what allows the FrameProcessor to receive frames and add custom conditional logic based on the frames that are received.
- **`push_frame()`** allows the FrameProcessor to push frames to the pipeline. Normally, frames are pushed DOWNSTREAM, but based on which processors need the output, you can also push UPSTREAM or in both directions.
### Essential Implementation Details
To ensure proper base class inheritance, it's critical to include:
1. **`super().__init__()`** in your `__init__` method
2. **`await super().process_frame(frame, direction)`** in your `process_frame()` method
```python
class MyCustomProcessor(FrameProcessor):
def __init__(self, **kwargs):
super().__init__(**kwargs) # β
Required
# Your initialization code here
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction) # β
Required
# Your custom frame processing logic here
if isinstance(frame, SomeSpecificFrame):
# Handle the frame
pass
await self.push_frame(frame, direction) # β
Required - pass frame through
```
## Critical Responsibility: Frame Forwarding
FrameProcessors receive **all** frames that are pushed through the pipeline. This gives them a lot of power, but also a great responsibility. Critically, they must push all frames through the pipeline; if they don't, they block frames from moving through the Pipeline, which will cause issues in how your application functions.
As well as formatting and logging MetricsFrames, `MetricsFrameLogger` also has an `await self.push_frame(frame, direction)` which pushes the frame through to the next processor in the pipeline.
## Frame Direction
When pushing frames, you can specify the direction:
```python
# Push downstream (default)
await self.push_frame(frame, FrameDirection.DOWNSTREAM)
# Push upstream
await self.push_frame(frame, FrameDirection.UPSTREAM)
```
Most custom FrameProcessors will push frames downstream, but upstream can be useful for sending control frames or error notifications back up the pipeline.
## Best Practices
1. **Always call the parent methods**: Use `super().__init__()` and `await super().process_frame()`
2. **Forward all frames**: Make sure every frame is pushed through with `await self.push_frame(frame, direction)`
3. **Handle frames conditionally**: Use `isinstance()` checks to handle specific frame types
4. **Use proper error handling**: Wrap risky operations in try/catch blocks
5. **Position carefully in pipeline**: Consider where in the pipeline your processor needs to be to receive the right frames
With these patterns, you can create powerful custom FrameProcessors that extend Pipecat's capabilities for your specific use case.
# The Worker Bus
Source: https://docs.pipecat.ai/pipecat/fundamentals/agent-bus.md
How Pipecat workers communicate through the shared message bus: publish, subscribe, and message flow between agents.
## What is the bus?
The worker bus is the communication backbone of every multi-worker system. All workers connect to the same bus and exchange typed messages for frame routing, lifecycle events, and job coordination.
Think of it as an internal event bus -- workers publish messages and other workers receive them through their subscriptions. The bus handles priority queuing so that urgent messages (like cancellations) are delivered before queued data.
## Bus implementations
Pipecat provides three bus implementations:
### AsyncQueueBus (local)
The default bus, created automatically by `WorkerRunner` when you don't provide one. It uses asyncio queues for in-process communication with no serialization overhead.
```python
runner = WorkerRunner() # Creates AsyncQueueBus automatically
```
This is all you need for single-process applications where all workers run together.
### RedisBus (distributed)
For distributed setups where workers run in separate processes or on different machines, use `RedisBus`. It uses Redis pub/sub to relay messages across process boundaries.
```python
from redis.asyncio import Redis
from pipecat.bus.network.redis import RedisBus
redis = Redis.from_url("redis://localhost:6379")
bus = RedisBus(redis=redis, channel="pipecat:my-app")
runner = WorkerRunner(bus=bus)
```
All processes that share the same Redis channel can exchange messages. The programming model stays the same -- your agent code doesn't change between local and distributed setups.
`RedisBus` requires the `redis` extra: `uv add "pipecat-ai[redis]"`
### PgmqBus (distributed)
An alternative distributed bus backed by PGMQ (PostgreSQL Message Queue). Each instance creates its own queue and broadcasts to peer queues.
```python
from pgmq.async_queue import PGMQueue
from pipecat.bus.network.pgmq import PgmqBus
pgmq = PGMQueue(
host="localhost",
port="5432",
database="postgres",
username="postgres",
password="...",
pool_size=4,
)
await pgmq.init()
bus = PgmqBus(pgmq=pgmq, channel="pipecat:my-app")
runner = WorkerRunner(bus=bus)
```
`PgmqBus` requires the `pgmq` extra: `uv add "pipecat-ai[pgmq]"`
## Message types
Messages on the bus fall into four categories:
### Data messages
Normal-priority messages that carry data between agents. The most important one is `BusFrameMessage`, which wraps a Pipecat frame (audio, text, etc.) for transport across the bus.
### System messages
High-priority messages for lifecycle events: activation, deactivation, shutdown, and worker readiness. These are delivered before data messages in the queue.
### Job messages
Messages for coordinating work between agents: job requests, responses, progress updates, streaming, and cancellation.
### Local messages
Some messages are local-only and never cross process boundaries. For example, child agent errors stay local to the parent. This keeps internal state from leaking across distributed runners.
## Message routing
Messages have `source` and `target` fields:
- **Targeted messages** (with a specific `target`) are delivered only to the named agent
- **Broadcast messages** (with no `target`) are delivered to all subscribers
The bus handles this routing automatically. When you call `activate_worker("greeter")`, it sends a `BusActivateWorkerMessage` targeted at `"greeter"` -- only that agent receives it.
## The agent registry
The runner maintains a `WorkerRegistry` that tracks which agents are available. To get notified when an agent is ready, use the `@worker_ready` decorator (or call `watch_workers()` explicitly):
```python
from pipecat.pipeline.base_worker import BaseWorker
from pipecat.pipeline.worker_ready_decorator import worker_ready
from pipecat.registry.types import WorkerReadyData
class MainAgent(BaseWorker):
@worker_ready(name="greeter")
async def on_greeter_ready(self, data: WorkerReadyData) -> None:
await self.activate_worker("greeter")
```
The framework automatically calls `watch_workers()` for each `@worker_ready` handler when the agent starts. If the watched agent is already registered, the handler fires immediately, so you don't need to worry about race conditions.
# Agent Registry and Discovery
Source: https://docs.pipecat.ai/pipecat/fundamentals/agent-registry-and-discovery.md
How agents discover each other and get notified when other agents are ready.
## What is the agent registry?
The `WorkerRegistry` tracks all known agents across local and remote runners. It is owned by the runner and shared with all its agents. When an agent becomes ready, it registers itself, and other agents that are watching for it get notified.
You don't interact with the registry directly in most cases. Instead, you use the `@worker_ready` decorator or `watch_workers()` to express interest in a specific agent, and the registry handles the rest.
## Watching for agents
### The @worker_ready decorator
The most common way to watch for an agent. Decorate a method with the agent name, and it fires when that agent registers:
```python
from pipecat.pipeline.base_worker import BaseWorker
from pipecat.pipeline.worker_ready_decorator import worker_ready
from pipecat.registry.types import WorkerReadyData
class MainAgent(BaseWorker):
@worker_ready(name="greeter")
async def on_greeter_ready(self, data: WorkerReadyData) -> None:
await self.activate_worker("greeter")
```
The framework automatically calls `watch_workers()` for each `@worker_ready` handler when the agent starts. If the watched agent is already registered, the handler fires immediately.
### watch_workers()
For dynamic cases where you don't know the agent name at class definition time, call `watch_workers()` directly. The best place to do this is in `start()`, after calling `super().start()`:
```python
class MainAgent(BaseWorker):
async def start(self) -> None:
await super().start()
for name in self._dynamic_worker_names:
await self.watch_workers(name)
async def on_worker_ready(self, data: WorkerReadyData) -> None:
await super().on_worker_ready(data)
await self.activate_worker(data.worker_name)
```
This is useful when agent names come from configuration or are created at runtime. Watched agents that are not handled by a `@worker_ready` decorator dispatch to the `on_worker_ready` hook.
## How discovery works
### Local agents
When all agents run in the same process, discovery is straightforward. An agent registers in the shared registry when its pipeline is ready, and watchers are notified immediately.
### Distributed agents
In distributed setups (agents across different processes connected to the same bus), runners exchange registry snapshots automatically. When a remote agent becomes ready, its runner broadcasts a registry message over the bus. Other runners update their local registry, and any matching watchers fire.
Only root agents (added via `runner.add_workers()`) are discoverable across
runners. Child agents (added via `parent.add_workers()`) are not broadcast and
remain invisible to other runners.
## Agent readiness data
When a watcher fires, it receives a `WorkerReadyData` object:
```python
@worker_ready(name="greeter")
async def on_greeter_ready(self, data: WorkerReadyData) -> None:
print(data.worker_name) # "greeter"
print(data.runner) # Name of the runner managing the agent
```
## Uniqueness
Agent names must be unique within a registry. If the same name is registered from two different runners, the registry logs a warning. In distributed setups, choose unique names across all runners to avoid conflicts.
# Understanding the Bus Bridge
Source: https://docs.pipecat.ai/pipecat/fundamentals/understanding-the-bus-bridge.md
How BusBridgeProcessor routes frames between the transport pipeline and the agent bus.
## What is the bus bridge?
The `BusBridgeProcessor` is a pipeline processor that connects a transport pipeline to the agent bus. It sits in the main agent's pipeline where an LLM would normally go, routing frames to whichever agent is currently active.
Without the bridge, your main agent's pipeline would look like:
```
transport.input β STT β context_agg β LLM β TTS β transport.output
```
With the bridge, the LLM is replaced:
```
transport.input β STT β context_agg β BusBridge β TTS β transport.output
```
The bridge sends outgoing frames (like transcribed text) to the bus, and receives incoming frames (like LLM-generated text) from the active agent.
The examples above show a voice pipeline, but `BusBridgeProcessor` can be used
in any type of pipeline. It works with any frames, not just audio.
## How it works
The bridge operates in two directions:
**Outgoing** (pipeline to bus): Frames flowing downstream through the pipeline are captured by the bridge and published as `BusFrameMessage` on the bus. The active agent receives these frames.
**Incoming** (bus to pipeline): When an active agent sends frames back through the bus, the bridge pushes them into the pipeline. These frames continue downstream to TTS and the transport output.
Certain frames are never sent across the bus:
- Lifecycle frames (`StartFrame`, `EndFrame`, `CancelFrame`, `StopFrame`)
- Transport-urgent frames (these pass through the bridge locally)
## Basic usage
The main agent is a `PipelineWorker` wrapping a pipeline that contains a `BusBridgeProcessor`. The bridge gets its bus from `runner.bus`:
```python
from pipecat.bus import BusBridgeProcessor
from pipecat.pipeline.pipeline import Pipeline
from pipecat.workers.runner import WorkerRunner
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
MAIN_NAME = "acme"
runner = WorkerRunner()
bridge = BusBridgeProcessor(
bus=runner.bus,
worker_name=MAIN_NAME,
name=f"{MAIN_NAME}::BusBridge",
)
pipeline = Pipeline([
transport.input(),
stt,
context_aggregator.user(),
bridge,
tts,
transport.output(),
context_aggregator.assistant(),
])
main = PipelineWorker(pipeline, name=MAIN_NAME, params=PipelineParams())
await runner.add_workers(main)
```
The worker gets its bus from the runner when added with `runner.add_workers()`. You build the `BusBridgeProcessor` ahead of time with `runner.bus` and place it in the pipeline where an LLM would normally go.
## Named bridges
When you have multiple bridges in a system, you can name them to control which agents receive frames from which bridge. This is useful with [parallel pipelines](/server/pipeline/parallel-pipeline) where each branch has its own bridge (for example, separate audio and video branches):
```python
voice_bridge = BusBridgeProcessor(
bus=runner.bus,
worker_name=MAIN_NAME,
name=f"{MAIN_NAME}::VoiceBridge",
bridge="voice",
)
```
Agents control which bridges they listen to through the `bridged` argument when they are created:
```python
# Receives frames from all bridges
all_bridges_agent = MyLLMWorker("all", llm=llm, bridged=())
# Receives frames only from the "voice" bridge
voice_only_agent = MyLLMWorker("voice-only", llm=llm, bridged=("voice",))
```
## The bridged agent side
When an agent sets `bridged=()` (or a tuple of bridge names), the framework automatically wraps its pipeline with edge processors that handle bus frame conversion. An LLM agent subclasses `LLMWorker` and passes its own LLM service:
```python
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.workers.llm import LLMWorker, tool
class MyLLMWorker(LLMWorker):
@tool
async def my_function(self, params, arg: str):
...
llm = OpenAILLMService(api_key="...")
agent = MyLLMWorker("greeter", llm=llm, bridged=())
```
A worker never takes a `bus=` argument. It receives its bus from the runner when you register it with `runner.add_workers()`.
The resulting pipeline looks like:
```
BusEdgeProcessor (upstream) β [your pipeline] β BusEdgeProcessor (downstream)
```
The upstream edge processor receives `BusFrameMessage` from the bus and converts them back to regular Pipecat frames. The downstream edge processor captures output frames and sends them back through the bus. You don't need to manage this -- it happens automatically when `bridged` is set.
# Pipecat Evals
Source: https://docs.pipecat.ai/pipecat/evals/overview.md
Behavioral testing for your agents: scripted conversations, semantic assertions, and an LLM judge.
Pipecat Evals is the framework's built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, the same code) and tells you whether the expectation still holds.
```yaml capital_question.yaml
name: capital_question
turns:
- user: "What is the capital of Germany?"
expect:
- event: response
eval: "the response says the capital of Germany is Berlin"
```
```bash
pipecat eval run capital_question.yaml
```
## Why evals matter
Voice agents are probabilistic systems. The same agent can answer differently run to run, and a prompt tweak, a model upgrade, or a service swap can quietly break behavior that used to work: a function that no longer gets called, context that stops carrying across turns, an interruption that derails the conversation. Manual testing catches some of this, but it's slow, unrepeatable, and impractical to run on every change.
Evals make agent behavior testable the way unit tests make code testable:
- **Regression safety**: run your scenarios after every prompt, model, or pipeline change and catch breakage before users do.
- **Fast iteration**: text-mode evals skip STT and TTS entirely, so a full conversation test runs in seconds with no audio service cost.
- **Semantic assertions**: an LLM judge checks meaning ("the response says the capital is Berlin"), not exact strings, so tests don't break when wording changes.
- **A feedback signal for AI coding assistants**: evals give a coding assistant a command it can run and a pass/fail result it can read, closing the loop between writing agent code and verifying it. See [The Eval Loop](/pipecat/evals/the-eval-loop).
Pipecat itself relies on this framework: before every release, an eval suite drives 100+ example agents end to end.
## How it works
Pipecat Evals has two halves:
1. **The eval transport.** Your agent runs unchanged with the eval transport. If your agent uses `create_transport()` and the development runner, this is already built in: start it with `-t eval` and it hosts a local WebSocket server speaking RTVI, instead of connecting to Daily, WebRTC, or telephony.
2. **The eval harness.** The harness connects to that transport as an RTVI client, plays the scenario's user turns (as text, or as synthesized speech in audio mode), collects the events your agent emits, and asserts on them in order: transcriptions, LLM responses, spoken output, function calls, and timing.
When a scenario asserts on meaning rather than exact text, a **judge LLM** evaluates the agent's response against a natural-language criterion. The judge runs locally with [Ollama](https://ollama.com) by default, or against OpenAI or any OpenAI-compatible endpoint.
### Text and audio modes
Every scenario runs in one of two modes:
| Mode | User input | Agent output | Best for |
| ------------------ | -------------------------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------- |
| **Text** (default) | Sent as text, bypassing the STT | LLM text; TTS is skipped automatically | Fast, cheap iteration on prompts, logic, and function calling |
| **Audio** | Synthesized by a TTS the harness runs (local by default) | Real synthesized speech, transcribed by an STT the harness runs | True end-to-end coverage of the full STT, LLM, and TTS pipeline |
Text mode exercises your agent's actual pipeline and context handling while skipping the audio services, so it costs nothing in TTS or STT usage and runs fast. Audio mode synthesizes the user's voice, streams it through your agent's real STT, and transcribes the agent's actual spoken audio for judging, catching issues that only surface with real speech (turn detection, homophones, barge-in).
## What you can test
- **Response content**: substring checks (`text_contains`) or semantic judging (`eval`) of the agent's replies.
- **Multi-turn context**: verify the agent remembers earlier turns.
- **Function calling**: assert that specific tools were called, with specific arguments.
- **Interruptions**: barge in mid-response and verify the agent recovers (`send_after`).
- **Latency**: per-event budgets with `within_ms`.
- **Vision**: serve an image when the agent requests one and judge its description.
## YAML or Python
Scenarios are YAML files, so they're easy to write, review, and share. Everything is also available as a library: load and run scenarios programmatically, build them in code, inject a custom judge, or orchestrate whole suites from your own tooling. See [Using the Library](/pipecat/evals/library).
## Requirements
- **Pipecat CLI**: the `pipecat eval` commands ship with the CLI extra: `uv tool install "pipecat-ai[cli]"`. If you've added `pipecat-ai[cli]` to your project instead, run them with `uv run pipecat eval` (just like `uv run bot.py`). The same commands are also available as `python -m pipecat.evals`.
- **A judge LLM** (for `eval:` assertions): Ollama by default (`ollama pull gemma2:9b`), or point the scenario's `judge:` block at OpenAI or any OpenAI-compatible endpoint.
- **Audio services** (audio mode only): the harness needs a TTS to synthesize the user's voice and an STT to transcribe the agent's speech. Both can be local models or HTTP-based services; the defaults are local (Kokoro and Moonshine or Whisper, installed with `uv add "pipecat-ai[kokoro,moonshine]"` or `uv add "pipecat-ai[kokoro,whisper]"`), which download once on first use and run with no keys and no per-run cost. WebSocket-streaming services aren't supported here, which keeps the harness simple.
- **Your agent's own credentials**: the agent under test is your real agent, so it needs the same service API keys it normally would.
## Production evaluation
Pipecat Evals is built for development: fast, local, repeatable, and run on every change. Once your agent is deployed, third-party evaluation platforms complement it with testing and monitoring at production scale:
- **Simulations**: scripted or AI-driven test calls over API, WebSocket, or telephony, exercising multi-turn flows, edge cases, and real phone-network conditions before they reach users.
- **Observability**: continuous evaluation of live traffic, with automated quality scoring of calls and transcripts, and metrics tracked over time to catch quality drift.
Simulation, observability, and evaluation platform with native Pipecat Cloud
integration. Supports no-code API, WebSocket, and telephony testing.
Automated testing and monitoring platform with native Pipecat Integration for
WebRTC/Text based testing and support for Mock Tools, Custom Dynamic Variables
and more!
AI-native simulation and evaluation platform for voice agents, trusted by QA,
Engineering, Operations, AI, and Executive teams.
Simulation, observability, tracing, and metrics with native Pipecat Cloud
integration over the Daily transport β personas, flows, built-in metrics, and
a drop-in observer for capturing production calls.
Observability and online evaluation for voice agents. Auto-instrument Pipecat
with OpenInference (OpenTelemetry) to trace every turn, then run LLM-as-judge
evals on live traffic in Arize AX or open-source Phoenix.
Building an evaluation integration for Pipecat? We welcome contributions to
this page. Open a PR on the [docs
repository](https://github.com/pipecat-ai/docs).
Pipecat's other building blocks feed into any evaluation workflow: [Metrics](/pipecat/fundamentals/metrics) for TTFB, processing time, and usage; [Saving Transcripts](/pipecat/fundamentals/saving-transcripts) for offline analysis; [OpenTelemetry](/api-reference/server/utilities/opentelemetry) for latency traces; and [Observers](/api-reference/server/utilities/observers/observer-pattern) for custom instrumentation.
## Next steps
The full scenario format: turns, expectations, modalities, and the judge.
Spawn multiple agents and run many scenarios concurrently from a manifest.
Let a coding assistant write agent code, run evals, and iterate
automatically until the agent is better.
# Evals Lifecycle
Source: https://docs.pipecat.ai/pipecat/evals/lifecycle.md
How local Pipecat Evals fit with platform testing, monitoring, and team-scale evaluation.
Pipecat Evals gives you a durable first layer for agent quality: fast, local,
repeatable checks for the behavior your agent should preserve as code, prompts,
models, and tools change. As the agent moves toward production, the evaluation
surface expands: local scenarios stay active, and realistic simulations,
audio-signal metrics, trace-backed checks, submitted production calls, and
review workflows give product, QA, and operations teams shared quality signals.
Strong evaluation programs keep both layers active:
- **Pipecat Evals** for executable specifications close to the codebase.
- **An [evaluation platform](/pipecat/evals/overview#production-evaluation)**
for production-like conversations, richer voice analysis, dashboards,
monitoring, human review, and longitudinal quality trends.
## Lifecycle at a glance
## Start local
Use Pipecat Evals as soon as the agent has behavior worth preserving.
This usually starts before deployment, while the agent still runs on a laptop or
in a pull request.
Good local evals look like small executable specs:
- "The agent greets on connect."
- "The agent remembers the user's name two turns later."
- "The agent calls `lookup_order` before answering an order-status question."
- "The agent recovers when the user interrupts a long answer."
- "The first response starts within the expected latency budget."
This is where Pipecat Evals is strongest. Text mode keeps the loop fast and
cheap while you iterate on prompts, logic, and function calling. Audio mode adds
an end-to-end check for VAD, STT, TTS, turn-taking, and speech transcription
before you merge or release.
The [Eval Suites](/pipecat/evals/suites) page describes Pipecat's own release
evals as a manifest with 100+ example agents. The same pass/fail result is also
useful for coding assistants because it gives them a concrete command to run and
a clear failure artifact to fix against.
## Put local evals in CI
Once a few scenarios exist, run them on every meaningful change. A small set of
behavior-critical scenarios gives engineers and coding assistants a clear
pass/fail signal.
Use Pipecat Evals in CI when:
- The agent is still changing quickly.
- The question is "did this code or prompt change preserve an expected behavior?"
- The expected behavior can be expressed as a scripted conversation.
- The failure should block a pull request.
- The debug artifact should live next to the code as a log or trace file.
As a rule of thumb, local evals should cover the sharp edges that are easy to
state and expensive to rediscover manually.
## When to add a simulation and evaluation platform
Add a platform when the risk you need to test extends beyond a small local
scenario. Keep the Pipecat suite in place, then layer on broader coverage,
shared workflows, and production feedback.
| Signal | Outside Pipecat Evals' scope | What to add |
| --------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Realistic caller behavior matters | Scripted turns prove one path, not how varied users phrase, interrupt, or recover | Simulated callers with varied personas, edge cases, and abuse attempts |
| Voice behavior is part of the product | Transcripts hide TTS loops, clipping, dropouts, phoneme stretch, timbre drift, and odd pauses | Audio-signal metrics, a Speech Artifact Score, and an audio LLM judge that scores the audio itself |
| The deployed transport matters | The local eval transport isn't a real WebSocket, Pipecat Cloud, SIP, or telephony path | Tests over the deployed path users actually hit |
| Volume and concurrency matter | Local suites skip sustained load, burst traffic, queueing, and service limits | Load tests for concurrency, latency, error rates, and saturation |
| Hidden execution state matters | Transcripts don't show tool success, arguments, downstream errors, or timing | Trace-based checks on tool calls, span attributes, errors, timing, and custom metrics |
| Product, QA, or Ops need to participate | YAML and CI logs aren't a shared review workspace | Review queues, per-reviewer assignments, annotations, agreement scores, and dashboards |
| The agent is live with users | Pre-merge checks miss production drift and new real-world failures | Production monitoring that scores live calls and alerts on regressions |
| Deterministic audio regression | Synthesized audio can't replay the exact call or wording that broke | Exact transcripts and pre-recorded audio as fixed regression cases |
| Comparing releases, vendors, or configs | Local pass/fail can't do trend analysis or bake-offs | Persisted runs, metrics, recordings, traces, and score distributions |
## Choose the right layer
Think of local evals as **behavioral unit tests for the agent**. Think of an
evaluation platform as **system testing and quality operations for the agent in
the world**.
Local development, pull-request gates, coding-assistant loops, scripted
behavioral specs, function-call assertions, latency budgets, and fast text
mode iteration.
Multi-turn simulations, caller variation, realistic voice and telephony
paths, audio-signal metrics, trace metrics, submitted production calls,
dashboards, human review, scheduled runs, and agent-native CLI, MCP, or
skill-based workflows.
## Example: appointment booking
Suppose your Pipecat agent books appointments. Start with a local scenario that
protects the core behavior:
```yaml appointment_booking.yaml
name: appointment_booking
turns:
- user: "Can you book me for Tuesday at 3 PM Pacific?"
expect:
- event: function_call
calls:
- name: book_appointment
args:
day: "Tuesday"
time: "3 PM"
timezone: "America/Los_Angeles"
- event: response
eval: "confirms the appointment request with the user"
```
This is exactly the kind of check you want in your repository. If a prompt edit
stops the tool call, CI should fail.
Before this agent handles real users, the quality question gets broader:
- What happens when the caller changes the time three turns later?
- Can the agent handle an impatient caller who interrupts while it is checking
availability?
- Does the agent still work when the caller is in a noisy room?
- Did the booking API actually succeed, or did the agent only say that it did?
- Are users getting frustrated when the available slots are limited?
- Is the booking-success rate drifting after a model or voice-provider change?
Those are platform-level questions. The same booking flow can become a
simulation suite that varies caller goals, phrasing, interruptions, voice
conditions, transport paths, load, and tool outcomes before launch. After
launch, production conversations can feed monitoring, review, and future
regression coverage.
## Simulate before launch
Local scenarios are intentionally crisp. Platform simulations broaden the same
flow across realistic user variation, voice conditions, transport paths, load,
abuse attempts, and tool behavior.
For regressions that need exact replay, use fixed transcripts, scripted turns,
or pre-recorded audio. For tool-heavy flows, include traces so evaluation can
check what happened under the transcript: tool calls, arguments, span
attributes, errors, timing, and custom numerical metrics from
[OpenTelemetry](/api-reference/server/utilities/opentelemetry).
For the appointment agent, a transcript might show:
> "You're all set for Tuesday at 3 PM."
A simulation can also check that `book_appointment` was called with the
requested day, time, and timezone, that the tool returned success, and that later
spans stayed clear of errors, retry failures, and cancellations.
When judgment matters, route selected conversations to human review. Review
queues, labels, assignments, and agreement scores turn ambiguous calls into
better metrics, tighter ground truth, and better future cases.
## Monitor production conversations
Once users are live, send completed production conversations to the platform for
monitoring. Score the same quality signals over time, route ambiguous calls into
review, and track trends by agent version, transport, scenario, and customer
segment.
## Feed findings back into tests
Production monitoring and human review should create the next evaluation inputs:
new local Pipecat scenarios for crisp regressions, new simulation cases for
broader user behavior, new trace metrics for hidden tool failures, and new
monitoring metrics for recurring production patterns.
## Practical adoption path
Start small and grow in layers:
1. **Create 5-10 Pipecat scenarios** for the agent's most important scripted
behaviors.
2. **Run the suite in CI** and require it before merging prompt, model, tool, or
pipeline changes.
3. **Add audio-mode checks** before release for the flows most sensitive to STT,
TTS, VAD, and turn-taking.
4. **Add platform simulations** when you need realistic multi-turn behavior,
varied callers, telephony or WebSocket coverage, load testing, trace checks,
and audio-signal metrics.
5. **Instrument traces** for tool-heavy workflows so evaluations can verify what
happened under the hood.
6. **Expose platform controls to your coding assistant** through CLI, MCP, or
agent-skill surfaces so the same assistant that fixes local evals can launch
simulations, inspect failures, and triage monitoring results.
7. **Send production calls to monitoring** once users are live, then turn
recurring failures into new simulations, metrics, human-review projects, or
local scenarios.
## Next steps
Write and run the first local scenario against an existing agent.
Review the available platform integrations for simulation and monitoring.
Learn the YAML format for turns, expectations, function calls, interruptions,
latency budgets, and audio mode.
See one concrete setup path for simulations, monitoring, traces, and team
review workflows.
# Evals Quickstart
Source: https://docs.pipecat.ai/pipecat/evals/quickstart.md
Run your first Pipecat behavioral eval against an existing agent with pipecat eval and a simple scenario.
This guide takes an existing agent, starts it with the eval transport, and runs a two-turn scenario against it. Total time: a few minutes.
## Prerequisites
- A working Pipecat agent that uses `create_transport()` and the development runner (the standard pattern from the [quickstart](/pipecat/get-started/quickstart) and all Pipecat examples), with its usual service API keys in `.env`.
- The Pipecat CLI: `uv tool install "pipecat-ai[cli]"` (or add `pipecat-ai[cli]` to your project and run the commands below with `uv run pipecat eval`).
- A judge LLM. Either:
- **Ollama** (local, the default): install [Ollama](https://ollama.com) and run `ollama pull gemma2:9b`, or
- **OpenAI**: set `OPENAI_API_KEY` and point the scenario's `judge:` block at it (shown below).
If your agent uses `create_transport()`, it supports the eval transport with a one-line addition to its `transport_params`:
```python
from pipecat.transports.websocket.server import WebsocketServerParams
transport_params = {
"eval": lambda: WebsocketServerParams(
audio_in_enabled=True,
audio_out_enabled=True,
),
# ... your other transports (daily, webrtc, twilio, ...)
}
```
Then start the agent with `-t eval`:
```bash
uv run bot.py -t eval
```
```
π Bot ready! (eval transport on ws://localhost:7860)
```
Instead of connecting to Daily or WebRTC, the agent now hosts a local WebSocket server and waits for the eval harness to connect. Nothing else in the agent changes: same pipeline, same services, same event handlers.
The harness talks to your agent over RTVI. `PipelineWorker` adds an
`RTVIProcessor` and `RTVIObserver` automatically, so the standard agent
setup needs no extra wiring. All Pipecat example agents already include
the `"eval"` transport entry.
A scenario is a YAML file describing a scripted conversation and the behavior you expect. Save this as `scenarios/capital_question.yaml`:
```yaml
name: capital_question
turns:
# The agent greets on connect; wait for the greeting before speaking.
- expect:
- event: response
eval: "the bot opens the conversation with a greeting or an offer to help"
- user: "What is the capital of Germany?"
expect:
- event: response
eval: "the response says the capital of Germany is Berlin"
```
```yaml
name: capital_question
judge:
eval:
service: openai
model: gpt-4o-mini
turns:
# The agent greets on connect; wait for the greeting before speaking.
- expect:
- event: response
eval: "the bot opens the conversation with a greeting or an offer to help"
- user: "What is the capital of Germany?"
expect:
- event: response
eval: "the response says the capital of Germany is Berlin"
```
Each turn optionally sends a user utterance and lists the events expected in response. The `eval:` field is a natural-language criterion checked by the judge LLM, so the test passes whether the agent says "Berlin is the capital of Germany" or "That would be Berlin!".
This scenario runs in **text mode** (the default): the user turn is sent as text and the agent's TTS is skipped automatically, so the whole conversation costs nothing in audio services and finishes in seconds.
Ollama with `gemma2:9b` is the default judge, which is why the first tab
has no `judge:` block. To use a different judge LLM, add a `judge.eval:`
block as in the OpenAI tab.
With the agent still running, run the scenario from another terminal:
```bash
pipecat eval run scenarios/capital_question.yaml
```
The harness connects to `ws://localhost:7860` (override with `--bot-url`), drives the conversation, and reports the result. Pass `-v` to watch each turn resolve:
```
turn 0 β (observe)
β llm_response β "Hello! How can I help you today?"
turn 1 β "What is the capital of Germany?"
β llm_response β "The capital of Germany is Berlin."
β ws://localhost:7860 capital_question (3402ms)
1/1 passed Β· 3.4s
```
The command exits `0` when everything passes and `1` otherwise, so it slots directly into scripts and CI. Each scenario also writes a decision trace to `.eval.log`, which shows every event the harness saw and why each assertion passed or failed.
Change the criterion to something false, for example `"the response says the capital of Germany is Madrid"`, and run again:
```
β ws://localhost:7860 capital_question
Failed (1):
β ws://localhost:7860 capital_question
β’ turn 1 expectation 0 (llm_response): judge said no: the reply says the capital is Berlin, not Madrid
0/1 passed, 1 failed Β· 4.1s
```
A failing eval tells you which turn, which expectation, and why. That message (plus the `.eval.log` trace) is what you, or your AI coding assistant, iterate against.
## Where to go next
- Learn the full scenario format, including multi-turn conversations, function call assertions, interruptions, latency budgets, and text vs audio modes, in [Writing Scenarios](/pipecat/evals/scenarios).
- Have many scenarios or agents? Let Pipecat spawn the agents for you with [Eval Suites](/pipecat/evals/suites).
- Want your coding assistant to run these for you? See [The Eval Loop](/pipecat/evals/the-eval-loop).
# Writing Scenarios
Source: https://docs.pipecat.ai/pipecat/evals/scenarios.md
The scenario file format: configuration, turns, events, and assertions.
A scenario is a YAML file describing a scripted conversation and the events you expect your agent to emit. This page covers the full format. If you haven't run a scenario yet, start with the [quickstart](/pipecat/evals/quickstart).
## Anatomy of a scenario
```yaml
name: multi_turn # required: the eval's name
judge: # optional: judge modality and LLM (defaults shown below)
eval:
service: ollama
model: gemma2:9b
turns: # required: the conversation, in order
- user: "My name is Alex, and I'm planning a trip to Italy."
expect:
- event: response
eval: "acknowledges the user's message (the name Alex and/or the trip to Italy)"
- user: "Remind me, what's my name and where am I going?"
expect:
- event: response
eval: "recalls that the user's name is Alex and the destination is Italy"
```
Each turn optionally sends a user utterance (`user:`) and lists the events expected in response (`expect:`). Expected events must arrive in the order listed, but the agent may emit other events in between, so you don't have to enumerate everything it does.
The rest of this page is in four parts:
Scenario-wide setup: modalities, the judge, context, and lifecycle.
Drive each turn with an utterance, keypresses, an image, or timing.
The semantic events the agent emits, and what each one means.
Check an event's content or timing with `eval:`, `text_contains:`, and more.
## Configuration
Everything in this section is optional. A scenario with no configuration blocks runs entirely in [text mode](#text-and-audio-modes) with the default judge, which is the fastest way to start. To write the turns themselves, skip ahead to [User turns](#user-turns) and [Events](#events).
### Text and audio modes
Two top-level blocks control a scenario's modalities, and each has its own `modality:` field:
- `user:` sets how each turn's utterance is delivered to the agent: sent as text, bypassing its STT (`modality: text`), or synthesized into real speech (`modality: audio`).
- `judge:` sets what the judge evaluates: the agent's LLM text, with its TTS skipped (`modality: text`), or a transcription of its actual spoken audio (`modality: audio`).
When `modality:` isn't specified, or a block is omitted entirely, it defaults to `text`. The two sides are also independent: you can drive the agent with text while judging its real speech, or speak to it and judge the LLM text.
A scenario with neither block runs entirely in text mode. No audio flows on either side, so this is the fastest and cheapest way to test prompts, conversational logic, and function calling: no audio service cost, and a multi-turn scenario finishes in seconds. The judge LLM is the only service the harness itself needs (Ollama with `gemma2:9b` by default).
The top-level `user:` block here only configures delivery. Each turn's `user:`
field (see [User turns](#user-turns)) is the utterance itself, and is written
the same way in both modes.
### User delivery with `user:`
**Text (the default).** Each turn's utterance is sent to the agent as text, bypassing its STT. This needs no configuration; it's equivalent to:
```yaml
user:
modality: text
```
**Audio.** Each turn's utterance is synthesized by a TTS the harness runs and streamed into your agent's pipeline at real-time cadence, exercising its VAD, turn detection, and STT exactly as a live microphone would. Synthesized audio is cached across runs, so repeated turns don't re-synthesize. The `speech:` block (the TTS service and voice) is required:
```yaml
user:
modality: audio
speech:
service: kokoro # local TTS, no API key, no per-run cost
voice: af_heart
sample_rate: 16000
```
The built-in speech services are `kokoro`, a local model and the recommended
default, and `cartesia` (HTTP) when you want a cloud voice.
### Judging with `judge:`
**Text (the default).** The agent's TTS is skipped automatically, including any on-connect greeting, and the judge evaluates the LLM's text output. Fast and silent; equivalent to:
```yaml
judge:
modality: text
```
**Audio.** The agent speaks for real. The harness captures its synthesized audio, transcribes it with the configured STT, and the `response` event becomes that transcription, so the judge evaluates what a user would actually have heard. This is the true end-to-end check: STT in, LLM in the middle, TTS out. The `transcription:` block is required:
```yaml
judge:
modality: audio
transcription:
service: moonshine # STT for the agent's audio (or: whisper)
model: small-streaming
eval:
service: ollama # the judge LLM
model: gemma2:9b
```
The built-in transcribers are `moonshine` and `whisper`, both local models.
When `transcription.service:` is omitted, it defaults to `moonshine`.
In either modality, the `judge.eval:` block selects the judge LLM: `ollama` (the default, `gemma2:9b`), `openai`, or any OpenAI-compatible endpoint via `endpoint:`. This is the LLM that decides [`eval:`](#semantic-judging-with-eval) assertions.
### Custom services with `factory:`
To use a TTS or STT beyond the built-ins, both blocks accept a `factory:` escape hatch: a dotted path to a callable that receives the block's mapping and the resolved sample rate, and returns the service. Any extra keys you put in the block are passed through to your factory:
```yaml
user:
modality: audio
speech:
factory: "my_evals.services.make_tts"
voice: luna # available to your factory as speech_cfg["voice"]
judge:
modality: audio
transcription:
factory: "my_evals.services.make_stt"
```
```python my_evals/services.py
import os
from pipecat.services.fal.stt import FalSTTService
from pipecat.services.rime.tts import RimeHttpTTSService
def make_tts(speech_cfg, sample_rate):
return RimeHttpTTSService(
api_key=os.environ["RIME_API_KEY"],
settings=RimeHttpTTSService.Settings(voice=speech_cfg["voice"]),
sample_rate=sample_rate,
)
def make_stt(transcription_cfg, sample_rate):
return FalSTTService(api_key=os.environ["FAL_KEY"])
```
The service your factory returns must be a local model or an HTTP-based
service. WebSocket-streaming services aren't supported: they need a running
pipeline to manage their connection lifecycle, and keeping them out keeps the
evals code simple.
For a fully custom setup (your own caching, a pre-built service instance), construct `EvalSpeech` or `EvalTranscriber` directly and inject them through the [library](/pipecat/evals/library).
### Seeding the context with `context:`
By default the harness leaves the bot's LLM context alone: whatever the bot sets up for itself (for example, a system prompt added in its connect handler) is what the scenario runs against. Provide `context:` to replace that with messages of your own, which lets a scenario start mid-conversation:
```yaml
context:
- role: developer
content: "The user has already introduced themselves as Alex."
- role: assistant
content: "Nice to meet you, Alex! How can I help?"
```
The harness sends these right after the bot-ready handshake as an `LLMMessagesUpdateFrame` that replaces the bot's context wholesale. Omit `context:` and the harness sends nothing, leaving the bot's own context in place.
### Sharing config with `!include`
Any value can be pulled from another file with `!include`, resolved relative to the scenario file. This keeps per-scenario noise down when a whole directory of scenarios shares the same audio setup:
```yaml
name: capital_question
user: !include user_audio.yaml
judge: !include judge_audio.yaml
turns:
- user: "What is the capital of Germany?"
expect:
- event: response
eval: "the response says the capital of Germany is Berlin"
```
### Running scenarios back to back
By default the bot keeps running between scenarios. When a scenario ends its eval connection closes, but the eval transport suppresses the bot's `on_client_disconnected` handler, so the pipeline stays up to serve the next scenario. This is what lets `pipecat eval run a.yaml b.yaml c.yaml` drive a whole list against one bot instance with no reboot between them, which keeps a run fast.
The trade-off is that anything the bot accumulated in one scenario is still there for the next. For results to be independent, each scenario has to start from a clean slate, and clearing that state is split between the harness and your bot:
- **Conversation context**: seed or clear it per scenario with [`context:`](#seeding-the-context-with-context). The harness replaces the bot's LLM context with the messages you provide (via an `LLMMessagesUpdateFrame`); without it, the previous conversation carries forward, which is rarely what you want across independent scenarios.
- **Application state**: counters, flags, cached data, anything your bot holds outside the LLM context. The harness can't see this, so resetting it is your bot's job. A common place is the bot's connect handler, which runs again for each scenario's connection.
### Exercising the disconnect path
Some bots do meaningful work in `on_client_disconnected`, like a goodbye message, session teardown, or resource cleanup. Because the eval transport suppresses that handler by default, set `trigger_disconnect: true` on a scenario to fire it when that scenario ends:
```yaml
name: test_goodbye_on_disconnect
trigger_disconnect: true
turns:
- user: "Thanks for your help!"
expect:
- event: response
eval: "the agent acknowledges the thanks"
# on_client_disconnected fires after this turn, so the agent can
# send a goodbye message or clean up resources.
```
Bots often cancel their pipeline in `on_client_disconnected`, so a scenario with `trigger_disconnect: true` usually ends the bot process. Treat it as a terminal run, last in a list.
Enable it for every scenario in a run with `pipecat eval run
--trigger-disconnect`; a scenario's own `trigger_disconnect` field still takes
precedence. This is independent of `--stop-bot`, which tears the bot down via
an `eval-cancel` message regardless of the disconnect handler.
## User turns
Each turn drives the agent by speaking (a `user:` utterance) or pressing keys (a `dtmf:` sequence); the two are mutually exclusive. A turn can also register an `image:`, or be observation-only with no input. `send_after:` controls when the input is sent.
### Utterances with `user:`
Each turn's `user:` field is the user's utterance for that turn, a plain string. You write it the same way in both modes; whether it's delivered as text or synthesized into real speech is set once by the [`user:` block](#user-delivery-with-user), not per turn.
A turn without a `user:` field is observation-only: the harness just waits for the expected events. This is how you test agent-first behavior like an on-connect greeting:
```yaml
turns:
# No user input: just wait for the agent to speak first.
- expect:
- event: response
eval: "the bot opens the conversation with a greeting or an offer to help"
```
### DTMF keypresses with `dtmf:`
Instead of a `user:` utterance, a turn can press phone keypad keys with `dtmf:`. The two are mutually exclusive: a turn either speaks or presses keys. This drives keypad menus (IVR) and any agent that reacts to telephony tones:
```yaml
turns:
- dtmf: "123#"
expect:
- event: user_transcription
text_contains: "DTMF: 123#"
- event: response
eval: "confirms the entered digits"
```
Each character is sent as one `InputDTMFFrame`, the same path a telephony transport's keypress takes, regardless of the scenario's `user:`/`judge:` modality. Valid characters are the keypad entries `0`-`9`, `*`, and `#`; any other character is a parse error.
Quote the value in YAML (`dtmf: "123#"`). An unquoted `#` starts a YAML
comment, so `dtmf: 123#` would silently drop the `#`. An unquoted all-digit
sequence (`dtmf: 123`) is coerced to a string for you, but quoting is the safe
habit.
A bot running a [`DTMFAggregator`](/api-reference/server/utilities/dtmf-aggregator) accumulates the keys and flushes them into a `DTMF: ...` transcription, which (with the default transcription-based turn-start strategy) drives a full user turn: `user_started_speaking`, `user_transcription`, `user_stopped_speaking`, and the agent's response. So a `dtmf:` turn can assert on `user_transcription` and `response` just like a spoken turn.
The aggregator flushes either on the `#` terminator or on its idle timeout. To exercise the idle-timeout path, omit the `#` and pace the keys with a time-based [`send_after:`](#scheduling-with-send_after):
```yaml
turns:
# Flushes immediately on the '#' terminator.
- dtmf: "1#"
expect:
- event: response
eval: "states the business hours"
# No terminator: the aggregator flushes on its idle timeout instead.
- dtmf: "2"
expect:
- event: response
eval: "gives the office location"
```
Like any input turn, `expect:` is optional on a `dtmf:` turn: omit it for a turn that only presses keys, with the assertion living on a later turn.
### Vision with `image:`
A turn may register an image with `image:` (a path relative to the scenario file). When a vision agent requests a user image during the turn, the eval transport serves it:
```yaml
turns:
- user: "What do you see in this image?"
image: assets/cat.jpg
expect:
- event: response
eval: "the response describes a cat"
```
### Scheduling with `send_after:`
`send_after:` controls when a turn's input (its `user:` utterance or [`dtmf:`](#dtmf-keypresses-with-dtmf) keypresses) is sent, either relative to a prior event or after a plain delay. Anchoring it to an event is how you script barge-in tests:
```yaml
turns:
- user: "Tell me a long, detailed story about the history of Paris."
expect:
- event: llm_started
# Interrupt 2 seconds after the agent starts its long answer.
- user: "Actually, never mind that. What's the capital of Japan?"
send_after:
event: llm_started
delay_ms: 2000
expect:
- event: response
eval: "the response says the capital of Japan is Tokyo, instead of continuing the Paris story"
```
The `event:` anchor is optional. A bare `send_after: { delay_ms: 500 }` is a pure time delay measured from the previous turn's send, with no event to wait on. This is handy for pacing turns by time rather than off a bot event (for example, spacing out [DTMF keypresses](#dtmf-keypresses-with-dtmf) to exercise an aggregator's idle-timeout flush):
```yaml
turns:
- dtmf: "1"
# Wait 1.5s after the first keypress before sending the next.
- dtmf: "2"
send_after:
delay_ms: 1500
```
A `send_after:` with no `event:` and a zero `delay_ms` is rejected as a no-op: give it an `event:`, a positive `delay_ms`, or both.
## Events
Scenarios assert on a small set of semantic events, mapped from the RTVI messages the agent emits:
| Event | Meaning |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `response` | The agent's reply. In audio mode this is a transcription of the agent's actual synthesized speech; in text mode it resolves to `llm_response`. Prefer this for content checks. |
| `llm_response` | The LLM's text output for the turn. Available in both modes. |
| `tts_response` | The text the TTS reports speaking, one segment at a time. Audio mode only. |
| `llm_started` | The LLM began generating a response. |
| `function_call` | The LLM called a function. |
| `user_transcription` | The agent's STT finalized a transcription of the user. Audio mode only, except on [DTMF turns](#dtmf-keypresses-with-dtmf), where the aggregated keys become a transcription in either mode. |
| `user_started_speaking` | The agent's VAD detected the start of user speech. Audio mode, or a [DTMF turn](#dtmf-keypresses-with-dtmf) in either mode. |
| `user_stopped_speaking` | The agent's VAD detected the end of user speech. Audio mode, or a [DTMF turn](#dtmf-keypresses-with-dtmf) in either mode. |
Use `response` for the agent's reply unless you have a reason not to. It's
modality-agnostic: the same scenario judges LLM text in text mode and the
transcription of real spoken audio in audio mode, so one file covers both.
## Assertions
Each entry in `expect:` names an event and, optionally, asserts on its content or timing.
### Semantic judging with `eval:`
The `eval:` field is a natural-language criterion that the event's text must satisfy, decided by the [judge LLM](#judging-with-judge):
```yaml
- user: "What's 2 plus 2?"
expect:
- event: response
eval: "the response says the answer is four"
```
The judge sees the whole conversation so far, so it can resolve terse or context-dependent replies (like "That's four"). It also understands that audio-mode responses come from a speech-to-text pass and judges intended meaning rather than exact spelling, so "for" transcribed instead of "four" still passes.
The judge handles interim replies gracefully: if the agent says "Let me check on that." before the real answer, the harness keeps accumulating response text and re-judges until the criterion is met or the time budget runs out.
`eval:` only makes sense on the agent's text output (`response`, `llm_response`, `tts_response`).
### Substring checks with `text_contains:`
For exact content, `text_contains:` does a plain substring check, with no judge round-trip:
```yaml
- user: "What is the capital of France?"
expect:
- event: response
text_contains: "Paris"
```
### Latency budgets with `within_ms:`
`within_ms:` bounds how long after the turn's user send the event may arrive. All of a turn's expectations share that one anchor:
```yaml
- user: "What is the capital of France?"
expect:
- event: llm_started
within_ms: 2000 # the LLM must start responding within 2s
- event: response
text_contains: "Paris"
```
When omitted, an expectation defaults to a generous 60 second budget (configurable with `--timeout`), so timing is only asserted when you ask for it.
Because every deadline is measured from the send, time spent matching earlier expectations counts against later ones. In the example above, if `llm_started` arrives at 1.5 seconds, the `response` (with the default 60 second budget) has 58.5 seconds left, and a turn that stalls completely fails within a single budget rather than one per expectation.
### Function calls
A `function_call` expectation asserts that the turn invoked one or more tools. List the expected calls under `calls:`; they're matched by name in any order, and the expectation passes once all are found:
```yaml
- user: "What's the weather in San Francisco? And recommend a restaurant."
expect:
- event: function_call
calls:
- name: get_current_weather
args: { location: "San Francisco" }
- name: get_restaurant_recommendation
- event: response
eval: "describes the weather and recommends a restaurant"
```
`args` is a subset check: every listed key/value must be present in the call's arguments, and extra arguments are ignored. A single expected call can use the `name:`/`args:` shorthand directly on the expectation, and a bare `function_call` with neither just asserts that some call happened.
## Next steps
Let a coding assistant write agent code, run evals, and iterate
automatically until the agent is better.
# Eval Suites
Source: https://docs.pipecat.ai/pipecat/evals/suites.md
Spawn agents and run many scenarios concurrently from a single manifest.
`pipecat eval run` tests scenarios against an agent you started yourself. A **suite** goes one step further: you list agents and scenarios in a manifest, and `pipecat eval suite` spawns each agent with its eval transport on its own port, runs its scenarios, tears it down, and aggregates the results, several runs at a time.
Suites are the right tool when you have more than one agent, more than a handful of scenarios, or want a single command for CI. Pipecat's own release evals are a manifest with 100+ example agents plus this command.
## The manifest
```yaml manifest.yaml
concurrency: 4 # how many runs execute at once
runs_dir: eval-runs # logs + recordings go to //
record: false # record conversation audio (audio-mode scenarios)
scenarios_dir: scenarios # scenario names resolve to /.yaml
# How to start each agent. {python}, {bot}, and {port} are substituted per run.
spawn: "{python} {bot} -t eval --port {port}"
suite:
- bot: bots/support-agent.py
scenarios: [greeting, capital_question, multi_turn]
- bot: bots/sales-agent.py
scenarios: [greeting, weather_function_call]
- bot: bots/vision-agent.py
runner_body: scenarios/vision-body.json # optional --runner-body data
scenarios: [vision_describe]
```
Paths in the manifest (`bots_dir`, `scenarios_dir`, `runs_dir`, the `bot:` entries) resolve relative to the manifest file, so a manifest is portable: check it into your repo and run it from anywhere.
Scenarios are reusable across agents. One `greeting` scenario can cover every agent in the suite.
An optional `runner_body:` points at a JSON file passed to the agent as
`--runner-body`. It supplies session data the agent would normally receive in
a `/start` request body (for example, a vision agent's image path).
## Running a suite
```bash
pipecat eval suite manifest.yaml
```
In a terminal, a live dashboard shows each run's status, a running tally, and total time. When piped (in CI, or driven by a coding assistant), it streams one plain result line per run instead. The command exits `0` only if every run passes.
Useful flags:
```bash
pipecat eval suite manifest.yaml -p support # only bots whose path contains "support"
pipecat eval suite manifest.yaml -s greeting # only the greeting scenario
pipecat eval suite manifest.yaml -c 8 # 8 runs at a time
pipecat eval suite manifest.yaml -n nightly # output to eval-runs/nightly/
pipecat eval suite manifest.yaml -a # record conversation audio
pipecat eval suite manifest.yaml -d # save full per-pipeline debug logs
```
Everything except the `suite:` list can live in the manifest or be passed on the command line (the command line wins), so a manifest can be as minimal as a `suite:` list.
## Run output
Each invocation writes to `//` (a timestamp when `-n` is omitted):
```
eval-runs/20260610_142200/
logs/
bots_support-agent.py__greeting.log # the agent process output
bots_support-agent.py__greeting.eval.log # the harness's decision trace
bots_support-agent.py__greeting.debug.log # per-pipeline harness logs (-d only)
recordings/
bots_support-agent.py__greeting.wav # conversation audio (record: true or -a)
```
When a run fails, start with the `.eval.log` decision trace: it's a timestamped record of every event the harness saw, what it matched, what the judge said, and why an assertion failed. The agent's own log sits next to it.
## Testing one agent with many scenarios
If you just want to run a batch of scenarios against an agent you already have running, you don't need a manifest. `pipecat eval run` accepts multiple scenario files and shares the suite's dashboard and tally:
```bash
pipecat eval run scenarios/*.yaml --bot-url ws://localhost:7860
```
By default the agent is left running afterward so it can serve more evals; pass `--stop-bot` to shut it down when the batch finishes.
## Suites in CI
The exit code makes suites CI-ready with no extra glue:
```yaml
# e.g. GitHub Actions
- name: Run behavioral evals
run: pipecat eval suite manifest.yaml
```
For deterministic, key-free CI runs, prefer text-mode scenarios and an OpenAI-compatible judge endpoint you control. Audio-mode scenarios work in CI too, but need the harness's TTS and STT services available (local models by default, which also need more CPU).
# Using the Library
Source: https://docs.pipecat.ai/pipecat/evals/library.md
Run, build, and orchestrate evals from Python with the pipecat.evals API.
Everything the `pipecat eval` CLI does is available as a library under `pipecat.evals`. Use it to run evals from your own test runner (pytest, a CI script, a custom dashboard), to build scenarios in code instead of YAML, or to customize pieces like the judge LLM.
## Running a scenario
`EvalScenario.load()` parses a scenario file, and `EvalSession.from_scenario()` builds a ready-to-run session, constructing the judge, user speech, and transcriber the scenario calls for:
```python
import asyncio
from pipecat.evals.harness import EvalSession
from pipecat.evals.scenario import EvalScenario
async def main():
scenario = EvalScenario.load("scenarios/capital_question.yaml")
session = EvalSession.from_scenario(scenario, "ws://localhost:7860")
result = await session.run()
if result.passed:
print(f"PASS ({result.duration_ms}ms)")
else:
for failure in result.failures:
print(f" {failure}")
asyncio.run(main())
```
The agent must already be running with its eval transport (`python bot.py -t eval`), just as with `pipecat eval run`.
### The result
`run()` returns an `EvalResult`:
| Field | Description |
| --------------- | ------------------------------------------------------------------------------------------- |
| `scenario_name` | Name of the scenario that ran. |
| `passed` | Whether every assertion passed. |
| `failures` | The failed assertions, each with the turn index, expectation index, event name, and reason. |
| `duration_ms` | Wall-clock time the run took. |
| `events_seen` | Every semantic event observed, for diagnostics. |
| `debug_log` | The harness's timestamped decision trace (what the CLI writes to `.eval.log`). |
| `skipped` | Set (with a reason) when the scenario was not run; such a result is neither pass nor fail. |
This maps cleanly onto a pytest test:
```python
import pytest
from pipecat.evals.harness import EvalSession
from pipecat.evals.scenario import EvalScenario
@pytest.mark.asyncio
async def test_capital_question():
scenario = EvalScenario.load("scenarios/capital_question.yaml")
result = await EvalSession.from_scenario(scenario, "ws://localhost:7860").run()
assert result.passed, "\n".join(str(f) for f in result.failures)
```
## Building scenarios in code
Scenarios are plain dataclasses, so you can construct them programmatically, generating turns from a dataset, parameterizing a template, or skipping YAML entirely:
```python
from pipecat.evals.scenario import EvalExpectation, EvalScenario, EvalTurn
scenario = EvalScenario(
name="capital_question",
turns=[
EvalTurn(
user="What is the capital of Germany?",
expect=[
EvalExpectation(
event="llm_response",
eval="the response says the capital of Germany is Berlin",
)
],
)
],
)
```
The modality-agnostic `response` event is resolved while parsing YAML. When
constructing scenarios in code, use `llm_response` for text mode directly (or
`response` only when you also configure audio judging).
## Customizing the judge
`from_scenario()` builds the judge from the scenario's `judge:` block, but you can inject your own. `EvalJudge` works with any Pipecat LLM service backed by an OpenAI-compatible API:
```python
import os
from pipecat.evals.harness import EvalSession
from pipecat.evals.judge import EvalJudge
from pipecat.services.openai.llm import OpenAILLMService
llm = OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
settings=OpenAILLMService.Settings(model="gpt-4o-mini"),
)
session = EvalSession.from_scenario(
scenario,
"ws://localhost:7860",
judge=EvalJudge(llm),
)
```
The same injection points exist for the user's synthesized voice (`speech=`, wrapping any `TTSService` in an `EvalSpeech`) and the transcriber used for the agent's spoken audio (`transcriber=`, wrapping any `STTService` in an `EvalTranscriber`). The wrapped services can be local models or HTTP-based; WebSocket-streaming services are rejected, since they need a running pipeline to manage their connection lifecycle.
## Observing progress
Pass `on_progress` to get a callback as each turn and expectation resolves, which is how the CLI implements its `--verbose` output:
```python
from pipecat.evals.harness import EvalSession, EvalTurnProgress
def show(p: EvalTurnProgress):
print(f"turn {p.turn_index} [{p.status}] {p.event_name} {p.detail}")
session = EvalSession.from_scenario(scenario, url, on_progress=show)
```
## Orchestrating suites
`EvalManifest` and `EvalSuite` are the library behind `pipecat eval suite`: the suite spawns each agent with its eval transport on its own port, runs its scenarios, and executes several runs concurrently:
```python
import asyncio
from pathlib import Path
from pipecat.evals.suite import EvalManifest, EvalSuite
async def main():
manifest = EvalManifest.load("manifest.yaml")
suite = EvalSuite(manifest)
# Optionally narrow the runs, like the CLI's -p / -s flags.
suite.filter(pattern="support")
await suite.run(
Path("eval-runs/logs"),
on_update=lambda run: print(run.bot, run.scenario, run.status),
)
for run in suite.runs:
verdict = run.error or ("passed" if run.result and run.result.passed else "failed")
print(f"{run.bot} / {run.scenario}: {verdict}")
asyncio.run(main())
```
Each run is mutated in place as it executes (`status`, `result`, `error`, `duration_ms`), so a live display can render directly from `suite.runs`.
`EvalManifest.load()` accepts keyword overrides for every manifest value (`concurrency`, `base_port`, `spawn`, `scenarios_dir`, and so on), mirroring the CLI flags.
# The Eval Loop
Source: https://docs.pipecat.ai/pipecat/evals/the-eval-loop.md
Close the loop: Pipecat evals give an AI coding assistant a pass/fail signal it can read, so it writes agent code, runs evals, and iterates.
Evals turn agent quality into a signal an AI coding assistant can read. That closes the loop: instead of asking an assistant to "improve the prompt" and judging the result by hand, you describe the desired behavior as a scenario and let the assistant iterate until the eval passes, and the agent gets better with every pass.
Think of it as a REPL for agent behavior: the assistant writes a change, evals it, reads a pass/fail result, and loops, except the eval step already contains the judgment, so the cycle can close without a human reading the output.
## The loop
1. **Describe the behavior as a scenario.** A scenario file is an executable specification: the conversation, the expected events, and the criteria a response must meet.
2. **The assistant changes the agent.** A prompt edit, a new tool, a pipeline change.
3. **The assistant runs the evals.** One command, either against a running agent (`pipecat eval run`) or letting the suite spawn the agent itself (`pipecat eval suite`).
4. **The assistant reads the result.** A non-zero exit code, a per-assertion failure message ("turn 1 expectation 0 (llm_response): judge said no: ..."), and a full decision trace in `.eval.log`.
5. **Repeat until green.**
Steps 2 through 5 need no human in the loop. You review the final diff with the evidence that it works attached.
## Why this works well for coding assistants
The framework was built to be driven by tools, not just humans:
- **One command, one exit code.** `pipecat eval run scenarios/*.yaml` exits `0` on success and `1` on failure, so an assistant knows mechanically whether it's done.
- **Plain-text output when piped.** Outside a terminal the CLI streams one result line per scenario instead of rendering a live dashboard, which is exactly what an assistant running shell commands sees.
- **Actionable failures.** Failures name the turn, the expectation, and the reason, including what the judge said. The `.eval.log` decision trace shows every event the harness observed, so "why did this fail" is answerable from files.
- **Suites are self-contained.** `pipecat eval suite` spawns the agents itself, so an autonomous loop doesn't need to manage processes: edit, run one command, read the result.
- **Text mode is fast and cheap.** Iterating on prompts and logic skips STT and TTS entirely, so an assistant can afford to run the evals after every change.
## Setting up your project
Keep scenarios in the repo next to the agent and tell your assistant how to run them. For example, in your project's `CLAUDE.md` or `AGENTS.md`:
```markdown
## Behavioral evals
Evals live in `scenarios/`. To verify any change to the agent's behavior:
1. Start the agent: `uv run bot.py -t eval` (serves ws://localhost:7860)
2. Run the evals: `pipecat eval run scenarios/*.yaml`
The command exits non-zero on failure and prints each failed assertion.
Each scenario writes a decision trace to `.eval.log`; read it
to understand a failure before changing code.
When you add or change agent behavior, add or update a scenario in
`scenarios/` to cover it.
```
With that in place, a request like this becomes fully verifiable:
> Add a `get_order_status` tool to the agent and make sure it gets called when the user asks where their order is. Add a scenario for it and run the evals until they pass.
The assistant writes the tool, writes the scenario (a `function_call` assertion plus a judged response), runs `pipecat eval run`, reads any failure, and fixes its own work.
## Evals as acceptance criteria
You can also run the loop in the other direction: write the scenario first, watch it fail, and hand the failure to the assistant. The scenario is the spec, and "make this pass" is the task.
```yaml order_status.yaml
name: order_status
turns:
- user: "Where's my order? The number is 12345."
expect:
- event: function_call
calls:
- name: get_order_status
args: { order_id: "12345" }
- event: response
eval: "tells the user the status of their order"
```
This is test-driven development for agent behavior, with the judge LLM absorbing the fuzziness that makes conversational output hard to assert on with string matching.
## Guardrails
A few practices keep autonomous loops honest:
- **Review scenario changes like code.** An assistant that can edit scenarios can also weaken them. Failing evals should usually be fixed in the agent, not in the scenario.
- **Keep a regression set.** As behaviors accumulate, so should scenarios. Run the full set (or a suite) before merging, not just the scenario being worked on.
- **Gate merges in CI.** `pipecat eval suite manifest.yaml` in CI makes "the evals pass" a property of the branch, whoever (or whatever) wrote it. See [Eval Suites](/pipecat/evals/suites).
- **Use audio mode for the final check.** Iterate in text mode for speed, then run the audio variants before release to cover the full STT, LLM, and TTS path.
## Next steps
Give your coding assistant access to Pipecat docs and source context.
Layer in simulation platforms and observability once your agent is
deployed.
# Arize
Source: https://docs.pipecat.ai/pipecat/evals/platforms/arize.md
Observability and online evaluation for Pipecat voice agents, powered by OpenInference auto-instrumentation and OpenTelemetry.
## Overview
[Arize AI](https://arize.com/?utm_source=pipecat-docs&utm_medium=partner&utm_campaign=partner-docs&utm_content=evals-arize) provides AI observability and evaluation for agents in development and production. Use [Arize AX](https://arize.com/products/ax/) for the full-featured platform built for production teams, AI-native companies, and enterprises, available as managed cloud or enterprise self-hosted deployment. Use [Arize Phoenix](https://arize.com/phoenix/) when you need an open-source path for local development, experimentation, or single-container self-hosting.
Arize maintains a Pipecat instrumentor, [`openinference-instrumentation-pipecat`](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-pipecat), that auto-traces a running pipeline. It's built on [OpenInference](https://github.com/Arize-ai/openinference), a set of OpenTelemetry-compatible semantic conventions for AI, so spans land in Arize AX, Arize Phoenix, or any OpenTelemetry backend, complementing Pipecat's built-in [OpenTelemetry tracing](/api-reference/server/utilities/opentelemetry). See the [Pipecat tracing guide](https://arize.com/docs/ax/integrations/python-agent-frameworks/pipecat/pipecat-tracing) for the full integration.
With Arize, you can:
- Auto-instrument a Pipecat agent with a few lines at startup, no manual span code required
- Trace every turn, with STT, LLM, TTS, and tool spans grouped by conversation
- Align transcripts, tool calls, and per-stage latency in a single timeline to find bottlenecks
- Run LLM-as-a-judge evaluations (hallucination, correctness, relevance, task completion) over live traffic
- Track quality over time with dashboards and monitors, and alert on drift or regressions
## Connect your Pipecat agent
Install the instrumentor plus the OTel SDK for your backend (`arize-otel` for Arize AX, or `arize-phoenix-otel` for Phoenix):
```bash
# Arize AX
pip install openinference-instrumentation-pipecat pipecat-ai arize-otel
# Phoenix (open source)
pip install openinference-instrumentation-pipecat pipecat-ai arize-phoenix-otel
```
Register a tracer provider and instrument Pipecat once at application startup, before you build your pipeline. Pass a `conversation_id` to `PipelineWorker` so spans are grouped per session.
```python Arize AX
import os
from arize.otel import register
from openinference.instrumentation.pipecat import PipecatInstrumentor
# Send traces to Arize AX
tracer_provider = register(
space_id=os.environ["ARIZE_SPACE_ID"],
api_key=os.environ["ARIZE_API_KEY"],
project_name="my-voice-agent",
)
PipecatInstrumentor().instrument(tracer_provider=tracer_provider)
# Build your pipeline as usual; spans now export to Arize AX.
pipeline = Pipeline(...)
worker = PipelineWorker(pipeline, conversation_id=conversation_id)
```
```python Phoenix (open source)
from phoenix.otel import register
from openinference.instrumentation.pipecat import PipecatInstrumentor
# Send traces to Phoenix (local or self-hosted)
tracer_provider = register(project_name="my-voice-agent")
PipecatInstrumentor().instrument(tracer_provider=tracer_provider)
pipeline = Pipeline(...)
worker = PipelineWorker(pipeline, conversation_id=conversation_id)
```
That's it. Run your agent and conversations show up in your Arize project. Because the instrumentor speaks OpenTelemetry, you can also point it at any other OTel-compatible collector by configuring the tracer provider accordingly.
The instrumentor requires `pipecat-ai>=1.3` and Python 3.11+. Instrument
before the pipeline is constructed so worker spans are captured from the
first turn.
## What gets traced
The instrumentor converts Pipecat's pipeline activity into OpenInference spans, so each conversation becomes a structured trace in Arize. As described in the [Pipecat tracing guide](https://arize.com/docs/ax/integrations/python-agent-frameworks/pipecat/pipecat-tracing), it captures:
- **Conversation sessions**, grouping all turns that share a `conversation_id`
- **Turn boundaries**, with each user-to-assistant exchange as a parent span
- **LLM calls** with prompts, responses, token counts, and model metadata
- **Speech-to-text and text-to-speech** spans with their input/output and latency
- **Tool and function calls** with inputs, outputs, and duration
- **End-to-end and per-stage latency**, with failures surfaced as span errors
## Online evaluation
Beyond tracing, Arize runs evaluations on the traces it collects, the "evals" part of the workflow. You define an LLM-as-judge (a prompt plus an output label), and Arize scores spans automatically as traffic flows in:
- Pre-built and custom judges for hallucination, correctness, relevance, and task completion
- Continuous evaluation of live traffic, with scores attached back to the originating spans
- Dashboards and monitors that track eval scores over time and alert on quality drift
This complements Pipecat Evals: use Pipecat Evals for fast, scripted, pre-merge behavioral checks, and Arize for production-scale observability and online scoring of real conversations.
For a broader production workflow, see Arize's [agent evaluation guide](https://arize.com/guides/ai-agent-handbook/agent-evaluation/) and [LLM evaluation guide](https://arize.com/resources/llm-evaluation/) for examples of turning traces into regression checks and continuous quality signals.
## Next steps
Arize's official guide to tracing a Pipecat agent, including setup and what
gets captured.
Set up AX projects, tracing, online evals, dashboards, and monitors for
production deployments.
Self-host the open-source version for local tracing and evaluation of your
Pipecat agent.
The OpenTelemetry-compatible semantic conventions behind Arize's
instrumentation.
# Bluejay
Source: https://docs.pipecat.ai/pipecat/evals/platforms/bluejay.md
Simulation, observability, and evaluation platform for voice AI agents with native Pipecat integration.
## Overview
[Bluejay](https://getbluejay.ai) is a simulation, observability, and evaluation platform purpose-built for voice AI agents. It provides no-code simulation testing and production call monitoring that integrate directly with Pipecat, whether you're running on Pipecat Cloud or self-hosting.
With Bluejay, you can:
- Run automated simulations that call your agent and evaluate its responses
- Define test scenarios covering edge cases like interruptions, unexpected input, and multi-turn flows
- Monitor every production call with automated quality scoring
- Track evaluation metrics over time to catch regressions early
## Pipecat Cloud integration
If your agent is deployed on [Pipecat Cloud](/pipecat-cloud/introduction), Bluejay offers two zero-configuration integration paths:
Enter your Pipecat Cloud API key and agent name in Bluejay's dashboard.
Bluejay connects directly to your agent's API to spin up simulation sessions
with no code changes required.
Enter your agent's phone number into Bluejay and start running simulations
immediately. Bluejay calls your agent just like a real user would, testing
end-to-end behavior over telephony.
The telephony integration tests the full call stack, from phone network to
your agent and back, making it ideal for catching issues that only surface in
real call conditions.
## Self-hosted integration
If you're running Pipecat on your own infrastructure, Bluejay integrates via a WebSocket connection. Point Bluejay at your agent's WebSocket endpoint and it will establish a session to run simulations against your agent directly.
See the [Bluejay WebSocket integration guide](https://docs.getbluejay.ai/simulation-integrations/websockets) for setup instructions.
## Observability
Simulations cover pre-deployment testing, but observability ensures your agent maintains quality with real users. Bluejay's [Evaluate API](https://docs.getbluejay.ai/api-reference/endpoint/evaluate) lets you submit any production call for automated evaluation.
```python
import requests
url = "https://api.getbluejay.ai/v1/evaluate"
headers = {"X-API-Key": ""}
payload = {
"agent_id": "",
"start_time_utc": "2025-03-31T18:30:00Z",
"participants": [
{"role": "AGENT", "name": "Healthcare Agent Harry"},
{"role": "USER", "name": "John Doe"},
],
"recording_url": "https://s3.amazonaws.com/my-recordings/call-123.wav",
}
response = requests.post(url, json=payload, headers=headers)
```
Integrate the evaluate endpoint into your agent's session cleanup logic to
automatically evaluate every production call without manual intervention.
## Traces
Bluejay supports [tracing](https://docs.getbluejay.ai/core-concepts/traces) to monitor and observe your agent's execution flow, latency, and performance in real-time. Traces conform to the [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) standard, so you can use any compatible instrumentation library, including OpenInference, Langfuse, and OpenLLMetry.
To send traces to Bluejay:
1. Instrument your application to export traces to Bluejay's OTLP endpoint
2. Link traces to call evaluations by including the `trace_id` in your [Evaluate API](https://docs.getbluejay.ai/api-reference/endpoint/evaluate) requests
3. View traces alongside your call evaluations in the Bluejay dashboard
### Example: OpenTelemetry setup
Configure the OpenTelemetry SDK to export traces to Bluejay:
```python
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
endpoint = "https://otlp.getbluejay.ai/v1/traces"
resource = Resource.create({SERVICE_NAME: "my-pipecat-agent"})
tracer_provider = trace_sdk.TracerProvider(resource=resource)
headers = {
"X-API-KEY": "",
}
tracer_provider.add_span_processor(
SimpleSpanProcessor(OTLPSpanExporter(endpoint, headers=headers))
)
```
Once the tracer provider is configured, use it with any OpenTelemetry-compatible instrumentation. For example, to automatically trace LLM calls:
```python
from openinference.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
```
## Next steps
Full setup guides, API reference, and configuration options.
Step-by-step guide for connecting Bluejay to your Pipecat agent.
Pipecat's built-in behavioral testing framework.
Capture conversation transcripts to use with evaluation tools.
# Cekura
Source: https://docs.pipecat.ai/pipecat/evals/platforms/cekura.md
Simulation and production monitoring platform for Pipecat agents with flexible connection methods and metric-driven QA.
## Overview
[Cekura](https://www.cekura.ai) is a simulation and production monitoring platform for voice AI agents. The broad idea is to use the same evaluation system across the full lifecycle of a Pipecat agent: simulate conversations before deployment, validate infrastructure continuously in CI, monitor real calls after deployment with metrics, dashboards, and alerts, and turn production issues back into new simulations so you can fix regressions and verify them before shipping again.
With Cekura, you can:
- Treat simulations and monitoring as one continuous QA workflow
- Connect to Pipecat through telephony, SIP, or native Pipecat WebRTC flows
- Run scenario-based tests from the dashboard or API with automated or manual session control
- Track quality with predefined metrics and custom metrics tailored to your agent's workflow
## Simulations
Testing in Cekura is built around scenarios, metrics, and repeatable runs. The main design choice is how you want Cekura to connect to your Pipecat agent:
- Use **telephony or SIP** when you want to validate the full voice and network path
- Use **native Pipecat connections** when you want faster, more direct WebRTC-based testing
After you choose the connection method, the workflow is straightforward: create or select scenarios, run them from the dashboard or your CI pipeline, and review transcripts, recordings, run status, and metrics in Cekura. Telephony and SIP are useful when you want the full call path in the loop; native Pipecat connections are better when you want faster and more direct WebRTC-based simulation against your Pipecat runtime.
### Automated Pipecat connection
For Pipecat Cloud agents, configure the following fields in Cekura:
- **Provider**: Select `Pipecat` as the voice integration provider
- **Assistant ID**: Optional Pipecat assistant identifier
- **Pipecat Cloud API Key**: Used by Cekura to create and manage sessions
- **Pipecat Agent Name**: Identifier used for your agent in Pipecat
- **Agent Configuration (JSON)**: Optional request body sent to Pipecat's start endpoint
- **Room Properties (JSON)**: Optional Daily room configuration for the WebRTC session
After setup, you can run scenarios from the dashboard or API and let Cekura handle session creation, lifecycle management, execution, and cleanup automatically. This is the simplest path for repeatable Pipecat Cloud simulations.
Configure Pipecat Cloud credentials once and let Cekura create and manage
sessions for each test run.
Provide your own room URL and token when you want explicit control over
Pipecat session creation and lifecycle.
### Manual Pipecat connection
If you want to control the session yourself, Cekura also supports a [manual Pipecat connection flow](https://docs.cekura.ai/documentation/integrations/pipecat/manual). In this mode, you provide the Pipecat room URL and token for each run, and Cekura joins the session you created instead of provisioning one automatically.
This mode is useful when you need custom session orchestration, want to test nonstandard environments, or already manage Pipecat rooms in your own infrastructure.
The automated flow is the best default for Pipecat Cloud. Use the manual flow
when you need explicit control over room creation and access tokens.
### Infrastructure Suite
For infrastructure-focused validation, Cekura provides an [Infrastructure Suite](https://docs.cekura.ai/documentation/guides/testing-agents/infrastructure-suite) with 18+ pre-built scenarios for latency, audio quality, interruption handling, language support, and failure cases such as packet loss and rapid-fire speech.
Instead of building these tests from scratch, you can add the suite to your project from the Cekura dashboard and run the scenarios as a group. This makes it a good fit for validating the parts of your stack around the LLM itself, such as transport behavior, turn-taking, silence handling, and voice pipeline stability.
Add Cekura's predefined infrastructure scenarios to your project with the
appropriate metric mappings already set up.
Run tagged infrastructure scenarios automatically on pull requests, pushes,
or scheduled workflows.
Cekura documents this suite as a good fit for CI/CD. A practical setup is to tag those scenarios as `infrastructure-suite` and execute them on every pull request:
```yaml
name: Cekura Infrastructure Tests
on:
pull_request:
types: [opened, synchronize]
jobs:
infrastructure-tests:
runs-on: ubuntu-latest
steps:
- name: Run Cekura Infrastructure Suite
uses: cekura-ai/cekura-github-actions@v1.0.0
with:
agent_id: ${{ vars.AGENT_ID }}
tags: "infrastructure-suite"
api_key: ${{ secrets.CEKURA_API_KEY }}
```
This gives you a straightforward CI gate for infrastructure regressions before you ship changes to your Pipecat agent.
## Production observability
Cekura can also monitor production calls after they complete. For Pipecat, the core pattern is simple: send completed calls to Cekura, use metrics and dashboards to track quality, infrastructure health, and workflow performance over time, then turn failed or interesting real calls into new simulations so the same issues are covered in your future test suite.
### Send calls to Cekura
If you want observability for Pipecat traffic, send recordings and transcripts to Cekura's observability API:
```bash
POST https://api.cekura.ai/observability/v1/observe/
X-CEKURA-API-KEY:
Content-Type: application/json
```
The observability API accepts `transcript_type: "pipecat"` along with either an `agent` ID or `assistant_id`, plus a `voice_recording` or `voice_recording_url`.
### Example: send a Pipecat production call
```python
import requests
headers = {
"X-CEKURA-API-KEY": "",
"Content-Type": "application/json",
}
payload = {
"call_id": "call_123",
"agent": 2421,
"transcript_type": "pipecat",
"voice_recording_url": "https://storage.example.com/recordings/call_123.mp3",
"transcript_json": [
{
"role": "user",
"content": "I need help with my order",
"timestamp": "2026-04-03T20:15:01Z",
},
{
"role": "assistant",
"content": "Sure, let me look that up for you.",
"timestamp": "2026-04-03T20:15:03Z",
},
],
"call_ended_reason": "customer-ended-call",
"timestamp": "2026-04-03T20:15:45Z",
}
response = requests.post(
"https://api.cekura.ai/observability/v1/observe/",
headers=headers,
json=payload,
)
result = response.json()
print(result)
```
Once calls are ingested, Cekura stores them for review in its observability workflow, where you can inspect call logs, transcripts, recordings, and evaluation output.
### Metrics and performance insights
Cekura's metrics system can be used in both simulations and production monitoring. For reference, see the [metrics overview](https://docs.cekura.ai/documentation/key-concepts/metrics/overview), [predefined metrics](https://docs.cekura.ai/documentation/key-concepts/metrics/pre-defined-metrics), and [custom metrics](https://docs.cekura.ai/documentation/key-concepts/metrics/custom-metrics).
Relevant predefined metrics for production monitoring include:
- **Latency** with percentile statistics for response-time monitoring
- **Infrastructure Issues** to detect failures to respond within a configured timeout
- **Transcription Accuracy** for STT quality checks
- **AI Interrupting User** and **Stop Time After User Interruption** for turn-taking quality
- **Gibberish Detection**, **Average Pitch**, and other speech-quality metrics for audio analysis
- **CSAT**, **Sentiment**, and **Dropoff Node** for agent-performance and user-experience insights
Beyond the predefined set, Cekura also supports custom metrics so you can evaluate whether your agent followed the right workflow, completed key business logic, or handled domain-specific tasks correctly. That is useful for measuring the logical performance of the agent itself, not just infrastructure or audio quality.
For production monitoring, audio recordings improve what Cekura can measure.
Several of the speech-quality and turn-taking metrics in their docs require
audio, and some require stereo recordings for the most precise results.
For day-to-day operations, Cekura's observability tooling is designed around dashboards, detailed call logs, and alerting for critical issues and anomalies. In practice, that gives you voice-quality metrics, agent-performance insights, and infrastructure-failure visibility in one monitoring loop.
## Next steps
Step-by-step setup for running automated Pipecat tests in Cekura.
Start with Cekura's broader guide to scenario creation, execution, and testing
workflows.
Learn how Cekura structures production monitoring, observability, and call
review workflows.
Review predefined metrics and custom metrics for workflow, logic, speech,
and infrastructure evaluation.
# Coval
Source: https://docs.pipecat.ai/pipecat/evals/platforms/coval.md
AI-native simulation and evaluation platform for voice agents, trusted by QA, Engineering, Operations, AI, and Executive teams.
## Overview
[Coval](https://www.coval.dev) is an AI-native simulation, evaluation, and production monitoring platform for voice agents β trusted by QA, Engineering, Operations, AI, and Executive teams to test and improve voice AI before and after it ships.
With Coval, you can:
- Spin up realistic voice simulations against your Pipecat agent and score them on voice-grade metrics
- Drive evals end-to-end from Claude Code, Cursor, or any MCP-compatible client
- Monitor production calls with the same metric suite you use in simulations
- Catch regressions with scheduled runs, CI integration, and live dashboards
## Agent-native by design
Coval is built so your AI coding assistant can run, inspect, and iterate on evals on your behalf, without you stepping out of your editor. Three surfaces work together:
Drive every Coval resource from your terminal with structured JSON output. Built for scripting and CI pipelines.
Expose Coval as MCP tools to Claude Code, Cursor, and any MCP-compatible
client. Your agent reads state, launches runs, and surfaces failures inline.
Pre-built skills your coding agent invokes to onboard a new Pipecat agent, configure metrics, and triage failing conversations.
In practice, your agent can create personas, launch a run against a new Pipecat build, identify which conversations failed which metrics, fetch transcripts and audio for the failures, and propose prompt fixes β all without you leaving your editor. Prefer a web UI? The same workflows are available in the [Coval dashboard](https://app.coval.dev) for teammates who'd rather click than type.
## Connect your Pipecat agent
Coval supports two paths into Pipecat, depending on how you're deployed:
- **Pipecat Cloud agents** β Coval's first-class [Pipecat Cloud connection](https://docs.coval.dev/concepts/agents/connections/pipecat) authenticates with your Pipecat API key, calls your agent directly, and runs simulations end-to-end. Configuration-only β no code changes required.
- **Self-hosted Pipecat agents** β Expose your agent over a [WebSocket transport](https://docs.coval.dev/concepts/agents/connections/websocket) and point Coval at the endpoint.
Either way, the workflow is the same once you're connected: pick a persona, point at a test set, and launch a run from the dashboard, CLI, MCP, or Agent Skills.
## Personas and scenarios
Realistic voice evaluations need realistic users. Coval ships a [persona system](https://docs.coval.dev/concepts/personas/overview) with configurable voices, background environments, interruption rates, and emotional progression β so you can stress-test your agent against an impatient caller in a noisy cafΓ© as easily as a calm one in a quiet room.
[Test sets](https://docs.coval.dev/concepts/test-sets/overview) tell your simulated users what to do, say, and how to behave. Author them as freeform prompts for generative simulations, or as exact transcripts and pre-recorded audio for deterministic regression runs.
## Voice-native metrics
Coval's [built-in metric catalog](https://docs.coval.dev/concepts/metrics/built-in-metrics) is purpose-built for voice. Evaluate conversations on:
- **Latency** β LLM, STT, and TTS time-to-first-byte, plus end-to-end turn latency
- **Turn-taking** β interruption rate, agent-fails-to-respond, repeated turns
- **Audio quality** β natural tone detection, pause analysis, speech tempo
- **STT accuracy** β Word Error Rate against a reference transcript
- **Sentiment** β audio sentiment per segment and transcript-level tone
- **Conversation flow** β end-reason classification (completed, hangup, max-turns, error)
- **[Trace metrics](https://docs.coval.dev/concepts/metrics/built-in-metrics#trace-metrics)** β LLM, STT, and TTS span data, tool call counts, and token usage extracted from your OpenTelemetry traces
- **LLM Judge & Composite Evaluation** β behavioral checks against your own criteria
- **[Custom metrics](https://docs.coval.dev/concepts/metrics/prompting) and [custom trace metrics](https://docs.coval.dev/concepts/metrics/custom-trace-metrics)** β author your own LLM-judge criteria or pull numerical values straight from any OTel span
## CI and scheduled runs
Wire Coval into your release process so every Pipecat change is evaluated before it ships. Coval ships a [GitHub Actions integration](https://docs.coval.dev/getting-started/github-actions-tutorial) for running eval suites on pull requests, pushes, or merges, plus [scheduled runs](https://docs.coval.dev/guides/scheduled-runs) for recurring nightly or weekly regression coverage. Both are driven by the [Coval CLI](https://docs.coval.dev/cli/overview), so the same commands work locally, in CI, and from your AI coding agent.
## Production monitoring and alerts
Coval [monitors](https://docs.coval.ai/agents/overview) live Pipecat traffic with the same metric suite you use in simulations. Push completed conversations β transcript-only or with stereo audio β and Coval scores them automatically. Configure default metrics that run on every call, or conditional rules that fire additional metrics when results or metadata look off. Send [OpenTelemetry traces alongside your monitoring calls](https://docs.coval.dev/concepts/simulations/traces/opentelemetry#tracing-for-monitoring-calls) and the same trace metrics light up on real production traffic.
When a monitor crosses an alert threshold, Coval can notify your team in Slack or Microsoft Teams, open a ticket in Linear for the failing conversation, or POST to any HTTP endpoint via [webhooks](https://docs.coval.ai/welcome). Failing production calls flow straight back into your test sets, so the same regression is covered the next time you ship.
Stereo recordings (agent on the left channel, user on the right) unlock the
full audio metric set. See [Coval's monitoring
guide](https://docs.coval.ai/agents/overview) for ingestion options.
## Human review and continuous improvement
Automated metrics catch most issues; the long tail needs a human in the loop. Coval ships a [human review workflow](https://docs.coval.dev/concepts/metrics/human-review/human-review) for QA and Operations teams to label conversations, correct automated scoring, and feed ground truth back into your metrics:
- Manual annotation with binary, numerical, categorical, sentiment, and per-message labels
- Collaborative review queues or per-reviewer assignments, with keyboard shortcuts for high-volume triage
- Programmatic review project creation via the [Human Review API](https://docs.coval.dev/guides/human-review-api)
- Annotated calls flow back into your metric definitions, tightening agreement scores over time
QA, Engineering, Operations, AI, and Executive teams align around a shared feedback loop: automated scoring drives human correction, which improves metrics and strengthens regression coverage.
## Next steps
Full setup guides, API reference, and configuration options.
Connect a Pipecat Cloud agent to Coval and run end-to-end voice simulations.
Wire up Coval as MCP tools in Claude Code, Cursor, or any MCP client.
Label conversations, correct automated scoring, and feed ground truth back into your metrics.
# Roark Eval Platform
Source: https://docs.pipecat.ai/pipecat/evals/platforms/roark.md
Test and monitor Pipecat voice agents with Roark: automated simulations with personas, plus observability, tracing, and metrics.
## Overview
[Roark](https://roark.ai) is an end-to-end platform for testing and monitoring voice AI agents. It runs automated simulations against your [Pipecat Cloud](/pipecat-cloud/introduction) agent over the Daily transport, captures every production call, traces each turn end to end, and scores everything with the same library of metrics β so the behavior you test before shipping is the behavior you measure in production.
With Roark, you can:
- Run automated simulations that call your agent over the Daily transport using lifelike **personas** and branching **flows**
- Score every call β simulated or live β against **built-in metrics** plus your own LLM-graded custom metrics
- **Trace** each turn (STT β LLM β TTS, tool calls, latency) via OpenTelemetry and inspect it on the call
- Monitor production traffic continuously and turn any real call into a repeatable regression test
## How the integration works
Roark connects to Pipecat through a single observer, then layers testing and monitoring on top:
1. **Sync and capture with the observer.** The [`pipecat-roark`](https://pypi.org/project/pipecat-roark/) package adds a `RoarkObserver` to your pipeline. It registers ("syncs") your agent to Roark the first time it runs, then streams every call's transcript, tool calls, and a stereo recording to Roark. See the [Roark analytics observer](/api-reference/server/services/analytics/roark) reference for full setup.
2. **Simulate over the Daily transport.** Once your Pipecat Cloud credentials are connected, Roark starts your agent on Pipecat Cloud, joins the Daily room as a simulated caller, drives your flow with a chosen persona, and collects the transcript, recording, traces, and metrics automatically β no code changes beyond the observer.
## Getting started
Sign up at [roark.ai](https://roark.ai) and create an API key on the **API
keys** page in your project. You'll pass it as `api_key` in the steps below.
Add the [`pipecat-roark`](https://pypi.org/project/pipecat-roark/) package and
drop `RoarkObserver` into your pipeline. The first call registers your agent
with Roark automatically, so it appears under the Pipecat source filter and
becomes available for simulations and reports.
```bash
pip install pipecat-roark
```
```python
import os
from pipecat.pipeline.worker import PipelineWorker
from pipecat_roark import RoarkObserver
roark = RoarkObserver(
api_key=os.environ["ROARK_API_KEY"],
agent_id="support-bot-v3",
agent_name="Support Bot v3",
agent_prompt=SYSTEM_PROMPT,
)
pipeline = Pipeline([
transport.input(), stt, context_aggregator.user(), llm, tts,
transport.output(),
roark.audio_processor, # after transport.output() β L=user, R=bot
context_aggregator.assistant(),
])
worker = PipelineWorker(pipeline, observers=[roark])
```
See the [Roark analytics observer](/api-reference/server/services/analytics/roark)
reference for the full configuration, including audio capture and the
OpenTelemetry correlation used by [Tracing](#tracing).
In the Roark dashboard, create a Pipecat integration and enter your
**Pipecat Cloud API key** and the **Pipecat agent name** (the name of your
deployed Pipecat Cloud agent β this is what Roark starts for each run,
distinct from the Roark agent name you set on the observer). Roark uses
these to start your agent and target it for simulation runs.
Assemble a run plan β a test matrix of flows, personas, and metrics β then
run it from the dashboard or on a schedule. Roark provisions each simulation
automatically and handles session creation, execution, and cleanup for every
run.
## Simulations
When you start a run, Roark starts your agent on Pipecat Cloud, which boots it into a Daily room, and Roark joins that room as a simulated caller over the Daily transport. Because Pipecat Cloud returns the room URL and token up front, Roark connects without any SDP handshake and manages the full session lifecycle for you.
A run is defined by a **run plan** β a test matrix of flows Γ personas Γ metrics. You can run 1β100 iterations per test case, run on a schedule, and compare runs over time to catch regressions.
### Personas
Personas are the simulated callers that drive each conversation β configurable characters that let you test how your agent handles a realistic range of real-world callers, not just the happy path:
- **Voice**: language and accent (US, British, Indian, and more), gender, and optional background noise (e.g. office)
- **Speech**: pace, clarity (clear, vague, rambling), and natural disfluencies like "um" and "uh"
- **Behavior**: base emotion (neutral, cheerful, confused, frustrated, skeptical, rushed), intent clarity, confirmation style, and memory reliability
- **Context**: an optional backstory and custom key-value properties (account number, membership status, and so on)
Start with 3β5 core personas that mirror your most common callers, then reuse them across flows.
### Flows
A flow defines the conversation path a simulation follows β what the caller says, how the agent is expected to respond, and where the conversation can branch. Flows come in two modes:
- **Scripted** β a visual graph of nodes and edges that lays out an exact path. Nodes cover customer turns, agent turns, silence, DTMF keypad input, and voicemail; edges let the conversation branch or loop back, so one flow can cover many outcomes.
- **Improv** β a plain-language brief the simulated caller improvises from, for open-ended conversations you don't want to script turn by turn.
Build a flow three ways:
- **Generate from your agent prompt** β Roark drafts flows from a description of what your agent does
- **Generate from a real call** β turn a production conversation into a repeatable flow
- **Build it yourself** β lay out the graph (scripted) or write the brief (improv) in the visual editor
Each flow can carry **variants** β edge-case variations like an interrupting, confused, or hostile caller β that reuse the flow while swapping in a different persona.
## Metrics
Every call β simulated or live β is scored against the same metric library, so a fix you verify in simulation is measured the same way in production.
- **Built-in metrics** are collected automatically with no configuration, spanning latency and response time, interruptions and overlap, sentiment and emotion, speech quality (DNSMOS), and compliance checks like PII handling and prompt-injection resistance.
- **Custom metrics** are defined in natural language and graded by Roark Prism, Roark's model built for scoring voice conversations β for example identity verification (boolean), empathy (scale), call reason (classification), or upsell attempts (count).
- **Thresholds** turn any metric into a pass/fail outcome (e.g. `Response Time < 1000ms`), applied per call or aggregated across a run.
Because simulations use the same `RoarkObserver` capture path as production
calls, the two are scored identically β making it straightforward to reproduce
a production issue as a repeatable, metric-gated test.
## Tracing
Roark ingests [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) traces from Pipecat's built-in tracing, so each call's full turn tree β STT, LLM, TTS, and tool calls, with latency at every span β shows up on the **Tracing** tab of the call.
Configure the tracer provider **before** constructing `PipelineWorker` (Pipecat grabs the provider at task-init time), and pass the **same** call ID to both `RoarkObserver(pipecat_call_id=...)` and `PipelineWorker(conversation_id=...)` so Roark can link the trace to the call.
```python
import os
import uuid
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from pipecat.pipeline.worker import PipelineWorker
from pipecat_roark import RoarkObserver
def setup_roark_tracing() -> None:
"""Must run BEFORE constructing PipelineWorker."""
resource = Resource.create({
"deployment.environment": os.getenv("ENVIRONMENT", "production"),
# Optional: tag spans for the trace explorer.
"roark.project.tag.env": os.getenv("ENVIRONMENT", "production"),
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint="https://api.roark.ai/v1/traces",
headers={"Authorization": f"Bearer {os.environ['ROARK_API_KEY']}"},
timeout=30,
)
)
)
trace.set_tracer_provider(provider)
setup_roark_tracing()
# Generate ONE id and pass it to both sides so Roark can correlate.
call_id = str(uuid.uuid4())
roark = RoarkObserver(
api_key=os.environ["ROARK_API_KEY"],
agent_id="support-bot-v3",
pipecat_call_id=call_id, # appears on the Roark record as pipecatCallId
)
worker = PipelineWorker(
pipeline,
observers=[roark],
enable_tracing=True,
conversation_id=call_id, # set as the conversation.id span attribute
)
```
If the two IDs don't match, the call still records β but its **Tracing** tab
will be empty because Roark can't link the trace to the call.
## Observability
Simulations cover pre-deployment testing; the `RoarkObserver` keeps that coverage running against real traffic. Because the observer is already in your pipeline, every production call flows to Roark automatically β transcript, tool calls, stereo recording, traces, and metrics β with no extra wiring. Watch calls live, track metrics over time on dashboards, and turn any interesting production call into a new flow so the same case is covered in your next run.
See the [Roark analytics observer](/api-reference/server/services/analytics/roark) reference for capture details and configuration.
## Next steps
Full setup guides, API reference, and configuration options.
Step-by-step guide for connecting Roark to your Pipecat agent.
Personas, flows, run plans, and how simulations execute.
The `RoarkObserver` reference for capturing production calls.
# Krisp VIVA
Source: https://docs.pipecat.ai/pipecat/features/krisp-viva.md
Integrate Krisp VIVA into Pipecat: voice isolation, noise filtering, and turn detection from the Krisp VIVA SDK.
## Overview
Krisp's VIVA SDK provides four capabilities for Pipecat applications:
- **Voice Isolation** β Filter out background noise and voices from the user's audio input stream, yielding clearer audio for fewer false interruptions and better transcription.
- **Turn Detection** β Determine when a user has finished speaking using Krisp's streaming turn detection model, as an alternative to the [Smart Turn model](/api-reference/server/utilities/turn-detection/smart-turn-overview).
- **Interruption Prediction** β Distinguish genuine user interruptions from backchannels (e.g. "uh-huh", "yeah"), preventing the bot from being interrupted by brief acknowledgements.
- **Voice Activity Detection** β Detect speech in audio streams using Krisp's VAD model, supporting sample rates from 8kHz to 48kHz.
You can use any combination of these features together.
API reference for voice isolation
API reference for turn detection
API reference for interruption prediction
API reference for voice activity detection
Complete example with Krisp features
Get the Krisp SDK and API key
## Prerequisites
To complete this setup, you will need access to a Krisp developers account, where you can download the Python SDK, models, and generate an API key.
Get started on the [Krisp developers website](https://krisp.ai/developers).
## Setup
### Download the Python SDK and Models
1. Log in to the [Krisp developer portal](https://sdk.krisp.ai/)
2. Navigate to the `Server SDK Version` Tab
3. Find the latest version of the Python SDK:
- Download the SDK
- Download the Voice Isolation models (for voice isolation)
- Download the Turn Detection models (for turn detection)
### Install the Python wheel file
1. First, unzip the SDK files you downloaded in the previous step. In the unzipped folder, you will find a `dist` folder containing the Python wheel file you will need to install.
2. Install the Python wheel file that corresponds to your platform. For example, a macOS ARM64 platform running Python 3.12 would install the following:
```bash
uv pip install /PATH_TO_DOWNLOADED_SDK/krisp-viva-uar-python-sdk-1.8.0/dist/krisp_audio-1.8.0-cp312-cp312-macosx_12_0_arm64.whl
```
### Generate an API key
1. In the [Krisp developer portal](https://sdk.krisp.ai/), generate an API key for your application.
The `KRISP_VIVA_API_KEY` is required for Krisp SDK v1.6.1 and later. For older
SDK versions, this is not required.
### Set up environment variables
1. Unzip the models you downloaded in the first step.
2. For voice isolation, choose a model:
- `krisp-viva-pro`: Mobile, Desktop, Browser (WebRTC, up to 32kHz)
- `krisp-viva-tel`: Telephony, Cellular, Landline, Mobile, Desktop, Browser (up to 16kHz)
Note: the full model name will be in the format of `krisp-viva-tel-v2.kef`.
3. In your .env file, add the environment variables for the features you're using:
```bash
# Krisp SDK API key (required for SDK v1.6.1+)
KRISP_VIVA_API_KEY=your_api_key_here
# Voice isolation model path
KRISP_VIVA_FILTER_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-vi-tel-v2.kef
# Turn detection model path
KRISP_VIVA_TURN_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-tp-v3.kef
# Interruption prediction model path
KRISP_VIVA_IP_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-ip-v1.kef
# Voice activity detection model path (optional)
KRISP_VIVA_VAD_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-vad-v2.kef
```
Each feature uses a **different model**. Set `KRISP_VIVA_FILTER_MODEL_PATH`
for voice isolation, `KRISP_VIVA_TURN_MODEL_PATH` for turn detection,
`KRISP_VIVA_IP_MODEL_PATH` for interruption prediction, and
`KRISP_VIVA_VAD_MODEL_PATH` for voice activity detection.
## Test the integration
You're ready to test the integration! Try running the [Krisp VIVA foundation example](https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-krisp-viva.py), which demonstrates both voice isolation and turn detection together.
Learn how to [run foundational
examples](https://github.com/pipecat-ai/pipecat/blob/main/examples/README.md)
in Pipecat.
## Voice Isolation
`KrispVivaFilter` isolates the user's voice by filtering out background noise and other voices in real-time audio streams. Add it to any transport via the `audio_in_filter` parameter.
```python
from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter
from pipecat.transports.base_transport import TransportParams
transport = SmallWebRTCTransport(
webrtc_connection=webrtc_connection,
params=TransportParams(
audio_in_enabled=True,
audio_in_filter=KrispVivaFilter(), # Enable Krisp voice isolation
audio_out_enabled=True,
),
)
```
See the [KrispVivaFilter reference](/api-reference/server/services/audio-filters/krisp-viva-filter) for configuration options.
## Turn Detection
`KrispVivaTurn` uses Krisp's streaming turn detection model to determine when a user has finished speaking. Unlike the [Smart Turn model](/api-reference/server/utilities/turn-detection/smart-turn-overview) which analyzes audio in batches, `KrispVivaTurn` processes each audio frame in real time.
Configure it as a user turn stop strategy:
```python
from pipecat.audio.turn.krisp_viva_turn import KrispVivaTurn
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
user_turn_strategies=UserTurnStrategies(
stop=[TurnAnalyzerUserTurnStopStrategy(
turn_analyzer=KrispVivaTurn()
)]
),
vad_analyzer=SileroVADAnalyzer(),
),
)
```
See the [KrispVivaTurn reference](/api-reference/server/utilities/turn-detection/krisp-viva-turn) for configuration options.
## Interruption Prediction
`KrispVivaIPUserTurnStartStrategy` uses Krisp's Interruption Prediction (IP) model to distinguish genuine user interruptions from backchannels. When VAD detects user speech, the IP model analyzes the audio and outputs a probability indicating whether the speech is a real interruption or a brief acknowledgement (e.g., "uh-huh", "yeah").
This prevents the bot from being interrupted unnecessarily by short utterances. Configure it as a user turn start strategy:
```python
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
from pipecat.turns.user_start import (
KrispVivaIPUserTurnStartStrategy,
TranscriptionUserTurnStartStrategy,
)
from pipecat.turns.user_turn_strategies import UserTurnStrategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
user_turn_strategies=UserTurnStrategies(
start=[
KrispVivaIPUserTurnStartStrategy(threshold=0.5),
TranscriptionUserTurnStartStrategy(), # Fallback
],
),
vad_analyzer=SileroVADAnalyzer(),
),
)
```
See the [KrispVivaIPUserTurnStartStrategy reference](/api-reference/server/utilities/turn-management/user-turn-strategies#krispvivaipuserturnstartstrategy) for configuration options.
## Voice Activity Detection
`KrispVivaVadAnalyzer` detects speech in audio streams using Krisp's VAD model. It supports sample rates from 8kHz to 48kHz, making it suitable for a wide range of applications including telephony and high-quality audio.
Configure it as a VAD analyzer:
```python
from pipecat.audio.vad.krisp_viva_vad import KrispVivaVadAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
vad_analyzer=KrispVivaVadAnalyzer(params=VADParams(stop_secs=0.2)),
),
)
```
See the [KrispVivaVadAnalyzer reference](/api-reference/server/services/vad/krisp-viva-vad-analyzer) for configuration options.
# WhatsApp Business Calling API
Source: https://docs.pipecat.ai/pipecat/features/whatsapp.md
Receive WhatsApp voice calls in a Pipecat application using the WhatsApp Business Calling API and WhatsAppTransport.
## Overview
This guide explains how to integrate WhatsApp Business voice calling into your Pipecat application using the **WhatsApp Cloud API** and **SmallWebRTC**. Once configured, your Pipecat bot will be able to receive and handle real-time voice calls directly from WhatsApp users.
## Prerequisites
### WhatsApp Business Setup
Before proceeding, ensure you have completed the following steps:
1. **Facebook Account** β Create one at [facebook.com](https://facebook.com)
2. **Facebook Developer Account** β Register at [developers.facebook.com](https://developers.facebook.com)
3. **WhatsApp Business App** β Create a new app via [Meta for Developers](https://developers.facebook.com/apps)
4. **Phone Number** β Add and verify a WhatsApp Business phone number
5. **Business Verification** β Required for production environments
6. **Webhook Configuration** β Youβll need a public webhook URL to receive call events
> **Note**: Voice calling is only available for WhatsApp Business API numbers with calling capability enabled.
For reference:
- [WhatsApp Cloud API Getting Started](https://developers.facebook.com/docs/whatsapp/cloud-api/get-started/)
- [Voice Calling API Documentation](https://developers.facebook.com/docs/whatsapp/cloud-api/calling/)
- [Webhook Setup](https://developers.facebook.com/docs/whatsapp/webhooks/)
## WhatsApp Configuration
> **Note**: For these settings you must be logged into your **Meta Developer Console** and inside your **App**.
### 1. Enable Voice Calls
To