{"owner":"livekit","repo":"agents","hasSkills":true,"totalSkillsCount":85,"totalTokensCount":52748,"categories":["root-instruction","claude-rule","plugin-manifest","cursor-rule"],"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","CLAUDE.md","livekit-plugins/livekit-durable/README.md","livekit-plugins/livekit-plugins-anam/README.md","livekit-plugins/livekit-plugins-anthropic/README.md","livekit-plugins/livekit-plugins-assemblyai/README.md","livekit-plugins/livekit-plugins-asyncai/README.md","livekit-plugins/livekit-plugins-avatario/README.md","livekit-plugins/livekit-plugins-avatartalk/README.md","livekit-plugins/livekit-plugins-aws/README.md","livekit-plugins/livekit-plugins-azure/README.md","livekit-plugins/livekit-plugins-baseten/README.md","livekit-plugins/livekit-plugins-bey/README.md","livekit-plugins/livekit-plugins-bithuman/README.md","livekit-plugins/livekit-plugins-bland/README.md","livekit-plugins/livekit-plugins-boson-avatar/README.md","livekit-plugins/livekit-plugins-browser/README.md","livekit-plugins/livekit-plugins-cambai/README.md","livekit-plugins/livekit-plugins-cartesia/README.md","livekit-plugins/livekit-plugins-cerebras/README.md","livekit-plugins/livekit-plugins-clova/README.md","livekit-plugins/livekit-plugins-deepgram/README.md","livekit-plugins/livekit-plugins-did/README.md","livekit-plugins/livekit-plugins-elevenlabs/README.md","livekit-plugins/livekit-plugins-fal/README.md","livekit-plugins/livekit-plugins-fireworksai/README.md","livekit-plugins/livekit-plugins-fishaudio/README.md","livekit-plugins/livekit-plugins-gladia/README.md","livekit-plugins/livekit-plugins-gnani/README.md","livekit-plugins/livekit-plugins-google/README.md","livekit-plugins/livekit-plugins-gradium/README.md","livekit-plugins/livekit-plugins-groq/README.md","livekit-plugins/livekit-plugins-hamming/README.md","livekit-plugins/livekit-plugins-hume/README.md","livekit-plugins/livekit-plugins-inworld/README.md","livekit-plugins/livekit-plugins-keyframe/README.md","livekit-plugins/livekit-plugins-krisp/README.md","livekit-plugins/livekit-plugins-langchain/README.md","livekit-plugins/livekit-plugins-lemonslice/README.md","livekit-plugins/livekit-plugins-liveavatar/README.md","livekit-plugins/livekit-plugins-lmnt/README.md","livekit-plugins/livekit-plugins-meta/README.md","livekit-plugins/livekit-plugins-microsoft-ai/README.md","livekit-plugins/livekit-plugins-minimal/README.md","livekit-plugins/livekit-plugins-minimax/README.md","livekit-plugins/livekit-plugins-mistralai/README.md","livekit-plugins/livekit-plugins-murf/README.md","livekit-plugins/livekit-plugins-nabrah/README.md","livekit-plugins/livekit-plugins-neuphonic/README.md","livekit-plugins/livekit-plugins-nltk/README.md","livekit-plugins/livekit-plugins-nvidia/README.md","livekit-plugins/livekit-plugins-openai/README.md","livekit-plugins/livekit-plugins-palabra/README.md","livekit-plugins/livekit-plugins-perplexity/README.md","livekit-plugins/livekit-plugins-phonic/README.md","livekit-plugins/livekit-plugins-protoface/README.md","livekit-plugins/livekit-plugins-resemble/README.md","livekit-plugins/livekit-plugins-reson8/README.md","livekit-plugins/livekit-plugins-respeecher/README.md","livekit-plugins/livekit-plugins-rime/README.md","livekit-plugins/livekit-plugins-rtzr/README.md","livekit-plugins/livekit-plugins-runway/README.md","livekit-plugins/livekit-plugins-sarvam/README.md","livekit-plugins/livekit-plugins-silero/README.md","livekit-plugins/livekit-plugins-simli/README.md","livekit-plugins/livekit-plugins-simplismart/README.md","livekit-plugins/livekit-plugins-slng/README.md","livekit-plugins/livekit-plugins-smallestai/README.md","livekit-plugins/livekit-plugins-soniox/README.md","livekit-plugins/livekit-plugins-spatius/README.md","livekit-plugins/livekit-plugins-speechify/README.md","livekit-plugins/livekit-plugins-speechmatics/README.md","livekit-plugins/livekit-plugins-spitch/README.md","livekit-plugins/livekit-plugins-synthesia/README.md","livekit-plugins/livekit-plugins-tavus/README.md","livekit-plugins/livekit-plugins-telnyx/README.md","livekit-plugins/livekit-plugins-thunderphone/README.md","livekit-plugins/livekit-plugins-trugen/README.md","livekit-plugins/livekit-plugins-turn-detector/README.md","livekit-plugins/livekit-plugins-typesafe/README.md","livekit-plugins/livekit-plugins-ultravox/README.md","livekit-plugins/livekit-plugins-upliftai/README.md","livekit-plugins/livekit-plugins-vakyam/README.md","livekit-plugins/livekit-plugins-xai/README.md",".cursorrules (ai-agent-specialist.mdc)"],"skills":{"AGENTS.md":"# AGENTS.md\n\n## Build and Development Commands\n\nThis project uses **uv** as the package manager. All commands run from the repository root.\n\n### Installation\n```bash\nmake install # Install all dependencies with dev extras (uv sync --all-extras --dev)\n```\n\n### Versioning\n\n- Versions change only in release PRs (`livekit-agents@X.Y.Z`). Other PRs never edit `version.py` files or the `livekit-agents` version floor in plugin `pyproject.toml` files.\n- Use a `patch` version bump by default.\n- Do not use a `minor` or `major` version bump unless a human explicitly confirms the bump level.\n\n### Code Quality\n```bash\nmake format # Format code with ruff\nmake lint # Run ruff linter\nmake lint-fix # Run ruff linter and auto-fix issues\nmake type-check # Run mypy type checker (strict mode)\nmake check # Run all checks (format-check, lint, type-check)\n```\n\n### Testing\n```bash\nuv run pytest --unit # Run all unit tests\nuv run pytest tests/test_tools.py # Run a single test file\nmake unit-tests # Run unit tests that don't require cloud accounts\n```\n\n#### Test categories\n\nEvery test module declares exactly one category via a module-level marker, and\neach category has a matching `--` selection flag. Selection happens\n*before* import, so a category run never imports (or fails on) modules outside\nit.\n\n| Marker | Flag | Meaning |\n|--------|------|---------|\n| `pytest.mark.unit` | `--unit` | fast, hermetic, no external providers/credentials/network |\n| `pytest.mark.audio_eot` | `--audio_eot` | hermetic audio end-of-turn / turn-detection suite |\n| `pytest.mark.plugin(\"name\")` | `--plugin [name]` | provider integration test (needs that provider's deps/keys) |\n| `pytest.mark.stt` | `--stt` | cross-provider speech-to-text suite (`tests/test_stt.py`) |\n| `pytest.mark.tts` | `--tts` | cross-provider text-to-speech suite (`tests/test_tts.py`) |\n| `pytest.mark.realtime(\"name\")` | `--realtime [name]` | realtime-model test |\n| `pytest.mark.evals` | `--evals` | behavioral evals against the LiveKit inference gateway |\n| `pytest.mark.docs` | `--docs` | tests for the docs-build tooling under `.github/` |\n\n```bash\nuv run pytest --unit --audio_eot # the CI unit gate (no cloud accounts)\nuv run pytest --plugin openai # only the openai provider tests\nuv run pytest --list-categories # list every module grouped by category, then exit\n```\n\n**Adding a test:** give the new module a category marker (`pytestmark =\npytest.mark.unit`, etc.) — collection fails with a hint if it lacks one. Run\npytest with the `--allow-uncategorized` option to temporarily disable this rule\n(CI keeps it on by default).\n\n### Running Agents\n```bash\npython myagent.py console # Terminal mode with local audio I/O (no server needed)\npython myagent.py dev # Development mode with hot reload (connects to LiveKit)\npython myagent.py start # Production mode\npython myagent.py connect --room --identity # Connect to existing room\n```\n\n### Linking Local python-rtc (for SDK development)\n```bash\nmake link-rtc # Link to local python-rtc with downloaded FFI artifacts\nmake link-rtc-local # Build and link local rust SDK from source (requires cargo)\nmake unlink-rtc # Restore PyPI version\nmake status # Show current linking status\nmake doctor # Check development environment health\n```\n\n## Architecture Overview\n\n### Core Concepts\n- **AgentServer** (formerly known as **Worker**) (`worker.py`): Main process coordinating job scheduling, launches agents for user sessions\n- **JobContext** (`job.py`): Context provided to entrypoint functions for connecting to LiveKit rooms\n- **Agent** (`voice/agent.py`): LLM-based application with instructions, tools, and model integrations\n- **AgentSession** (`voice/agent_session.py`): Container managing interactions between agents and end users\n\n### Key Directories\n```\nlivekit-agents/livekit/agents/\n├── voice/ # Core voice agent: AgentSession, Agent, room I/O, transcription\n├── llm/ # LLM and realtime engines: chat context, tools, MCP, provider protocols\n├── stt/ # Speech-to-text with fallback and stream adapters\n├── tts/ # Text-to-speech with fallback and stream pacing\n├── ipc/ # Inter-process communication for distributed job execution\n├── cli/ # CLI commands (console, dev, start, connect)\n├── inference/ # Hosted model inference (LLM, STT, TTS, realtime)\n├── telemetry/ # OpenTelemetry traces and Prometheus metrics\n└── utils/ # Audio processing, codecs, HTTP, async utilities\n\nlivekit-plugins/ # 50+ provider plugins (openai, anthropic, google, deepgram, etc.)\ntests/ # Test suite with mock implementations (fake_stt.py, fake_vad.py)\nexamples/ # Example agents and use cases\n```\n\n### Plugin System\nPlugins in `livekit-plugins/` provide STT, TTS, LLM, and specialized services. Each plugin is a separate package following the pattern `livekit-plugins-`. Plugins register via the `Plugin` base class in `plugin.py`.\n\n### Model Interface Pattern\nSTT, TTS, LLM, Realtime models have provider-agnostic interfaces with:\n- Base classes defining the interface (`stt/stt.py`, `tts/tts.py`, `llm/llm.py`, `llm/realtime.py`)\n- Fallback adapters for resilience\n- Stream adapters for different streaming patterns\n\n### Job Execution Flow\n1. Worker receives job request from LiveKit server\n2. Job is dispatched to process/thread pool (`ipc/proc_pool.py`)\n3. Entrypoint function receives `JobContext`\n4. Agent connects to room via `ctx.connect()`\n5. `AgentSession` manages the conversation lifecycle\n\n## Environment Variables\n- `LIVEKIT_URL`: WebSocket URL of LiveKit server\n- `LIVEKIT_API_KEY`: API key for authentication\n- `LIVEKIT_API_SECRET`: API secret for authentication\n- `LIVEKIT_AGENT_NAME`: Agent name for explicit dispatch (optional)\n- Provider-specific keys: `OPENAI_API_KEY`, `DEEPGRAM_API_KEY`, `ANTHROPIC_API_KEY`, etc.\n\n## Code Style\n- Line length: 100 characters\n- Python 3.10+ compatibility required\n- Google-style docstrings\n- Strict mypy type checking enabled\n- Use `make check` and `make fix` before committing\n\n\n\n\n## LiveKit documentation\n\nAlways refer to the latest documentation when working with this project. LiveKit offers both a CLI (`lk docs`) and an MCP server for browsing and searching its documentation.\n\n### LiveKit CLI\n\nInstall or update the CLI:\n\n- **macOS:** `brew install livekit-cli` (update: `brew update && brew upgrade livekit-cli`)\n- **Linux:** `curl -sSL https://get.livekit.io/cli | bash`\n- **Windows:** `winget install LiveKit.LiveKitCLI` (update: `winget upgrade LiveKit.LiveKitCLI`)\n\nRun `lk docs` before your first docs lookup. Its help text explains how to use each command and how to interpret results.\n\n### LiveKit docs MCP server\n\nFor deeper coding agent integration, LiveKit also offers an [MCP server](https://docs.livekit.io/reference/developer-tools/docs-mcp/).\n\n### Submit feedback to LiveKit\n\nWhen you run into any problem with LiveKit, add it to your to-do list, then submit it with `lk docs submit-feedback` or the MCP `submit_docs_feedback` tool once you finish your other tasks. Useful feedback includes incomplete documentation, broken examples, missing steps, broken features or implementations, and any other gap or issue in the LiveKit ecosystem.\n\n\n\n\n\n## LiveKit agent skills\n\nLiveKit publishes agent skills for each stage of the work: reading the docs, building, debugging, testing, writing scenarios, running simulations, and operating in production. They defer to the live documentation for API details. The developer installs them with `lk skills install`, which also adds the LiveKit Docs MCP server to your config, or with the LiveKit Claude Code plugin. If no LiveKit skills are available to you (such as `reading-livekit-docs`), recommend that the developer install them. Check the skills you have, not only the `.agents/skills/` and `.claude/skills/` directories, since the plugin and `lk skills install --global` install them elsewhere. Don't recommend a second install method on top of one the developer already uses: installing the same skills twice, for example as the plugin and with `lk skills install`, loads both copies, so every skill competes with itself. If your tool doesn't load skills automatically, read the matching `.agents/skills//SKILL.md` before you start that kind of task.\n\n","CLAUDE.md":"# CLAUDE.md\n\n@AGENTS.md\n","livekit-plugins/livekit-durable/README.md":"# Durable generator/coroutine tools for LiveKit Agents","livekit-plugins/livekit-plugins-anam/README.md":"# Anam plugin for LiveKit Agents\n\nSupport for the [Anam](https://anam.ai/) virtual avatar.\n\nSee the [Anam integration docs](https://docs.livekit.io/agents/models/avatar/plugins/anam/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-anam\n```\n\n## Pre-requisites\n\nYou'll need an API key from Anam. It can be set as an environment variable: `ANAM_API_KEY`\n","livekit-plugins/livekit-plugins-anthropic/README.md":"# Anthropic plugin for LiveKit Agents\n\nSupport for the Claude family of LLMs from Anthropic.\n\nSee [https://docs.livekit.io/agents/integrations/llm/anthropic/](https://docs.livekit.io/agents/integrations/llm/anthropic/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-anthropic\n```\n\n## Pre-requisites\n\nYou'll need an API key from Anthropic. It can be set as an environment variable: `ANTHROPIC_API_KEY`\n","livekit-plugins/livekit-plugins-assemblyai/README.md":"# AssemblyAI plugin for LiveKit Agents\n\nSupport for Streaming Speech-to-Text and Streaming Text-to-Speech from AssemblyAI.\n\nMore information is available in the docs for the [STT](https://docs.livekit.io/agents/integrations/stt/assemblyai/) integration and for AssemblyAI's [Streaming TTS](https://www.assemblyai.com/docs/tts/getting-started/quickstart).\n\n## Installation\n\n```bash\npip install livekit-plugins-assemblyai\n```\n\n## Pre-requisites\n\nYou'll need to specify an AssemblyAI API Key. It can be set as environment variable: `ASSEMBLYAI_API_KEY`.\n\n## Usage\n\n```python\nfrom livekit.agents import AgentSession, TurnHandlingOptions\nfrom livekit.plugins import assemblyai\n\nsession = AgentSession(\n stt=assemblyai.STT(model=\"universal-3-6-pro\", mode=\"balanced\"),\n tts=assemblyai.TTS(\n model=\"assemblyai-tts-1\",\n voice=\"49c443a4-8a1d-5f7e-b70b-98bdbd9823c3\", # Audrey\n ),\n # let AssemblyAI decide when the user has finished speaking\n turn_handling=TurnHandlingOptions(turn_detection=\"stt\"),\n # use the TTS word timings for the transcript and for what an interrupted reply said\n use_tts_aligned_transcript=True,\n # ... llm, vad, etc.\n)\n```\n\n### Speech-to-text\n\n`assemblyai.STT` streams Universal-3.6 Pro by default (`model`). Useful options:\n\n- `mode`: `\"min_latency\"`, `\"balanced\"` or `\"max_accuracy\"`.\n- `language_codes`: steer transcription toward one or more expected languages (`\"es\"` or\n `[\"en\", \"es\"]`).\n- `prompt`: context about the audio, such as its domain or scenario, to help recognize likely\n vocabulary (up to 1,750 characters).\n- `keyterms_prompt`: names and terms the model should get right.\n\n### Text-to-speech\n\n`assemblyai.TTS` streams speech over a pooled WebSocket and cancels in-flight audio when the user\ninterrupts. Useful options:\n\n- `voice`: a voice's id, from the [voice library](https://www.assemblyai.com/docs/tts/voices)\n or a custom (cloned) voice; defaults to Audrey. Slugs such as `\"aai-linda-3\"` also work, but\n can change, while ids don't.\n- `language`: a code such as `\"es\"`, set independently of the voice.\n- `model`: defaults to `assemblyai-tts-1`.\n- `base_url`: the US data zone by default; `wss://tts.eu.assemblyai.com` keeps data in the EU.\n- `word_timestamps`: per-word timings (default `True`). With\n `AgentSession(use_tts_aligned_transcript=True)`, LiveKit uses them to sync the transcript and to\n record an interrupted reply up to the word the user heard.\n\nCall `tts.prewarm()` to open the session before the first reply.\n","livekit-plugins/livekit-plugins-asyncai/README.md":"# Async plugin for LiveKit Agents\n\nSupport for text-to-speech with [Async](https://async.com/).\n\nSee [AsyncAI TTS docs](https://docs.livekit.io/agents/integrations/tts/asyncai/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-asyncai\n```\n\n## Pre-requisites\n\nYou'll need an API key from Async. It can be set as an environment variable: `ASYNCAI_API_KEY`","livekit-plugins/livekit-plugins-avatario/README.md":"# Avatario plugin for LiveKit Agents\n\nSupport for the [Avatario](https://avatario.ai/) virtual avatar.\n\nSee the [Avatario integration docs](https://docs.livekit.io/agents/models/avatar/plugins/avatario/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-avatario\n```\n\n## Pre-requisites\n\nYou'll need an API key from Avatario. It can be set as an environment variable: `AVATARIO_API_KEY`\n","livekit-plugins/livekit-plugins-avatartalk/README.md":"# AvatarTalk plugin for LiveKit Agents\n\nSupport for the [AvatarTalk](https://avatartalk.ai/) virtual avatar.\n\n## Installation\n\n```bash\npip install livekit-plugins-avatartalk\n```\n\n## Pre-requisites\n\nYou'll need an API key from AvatarTalk. It can be set as an environment variable: `AVATARTALK_API_KEY`\n","livekit-plugins/livekit-plugins-aws/README.md":"# AWS Plugin for LiveKit Agents\n\nComplete AWS AI integration for LiveKit Agents, including Bedrock, Polly, Transcribe, and realtime speech-to-speech support for Amazon Nova Sonic\n\n**What's included:**\n- **RealtimeModel** - Amazon Nova 2.5 Sonic, Nova 2 Sonic, and Nova Sonic 1.0 for speech-to-speech\n- **LLM** - Powered by Amazon Bedrock, defaults to Nova 2 Lite\n- **STT** - Powered by Amazon Transcribe\n- **TTS** - Powered by Amazon Polly\n\nSee [https://docs.livekit.io/agents/integrations/aws/](https://docs.livekit.io/agents/integrations/aws/) for more information.\n\n## ⚠️ Breaking Change\n\n**Default model changed to Nova 2 Sonic**: `RealtimeModel()` now defaults to `amazon.nova-2-sonic-v1:0` with `modalities=\"mixed\"` (was `amazon.nova-sonic-v1:0` with `modalities=\"audio\"`).\n\nIf you need the previous behavior, explicitly specify Nova Sonic 1.0:\n```python\nmodel = aws.realtime.RealtimeModel.with_nova_sonic_1()\n# or\nmodel = aws.realtime.RealtimeModel(\n model=\"amazon.nova-sonic-v1:0\",\n modalities=\"audio\"\n)\n```\n\n## Installation\n\n```bash\npip install livekit-plugins-aws\n\n# For Nova Sonic realtime models\npip install livekit-plugins-aws[realtime]\n```\n\n## Prerequisites\n\n### AWS Credentials\n\nYou'll need AWS credentials with access to Amazon Bedrock. Set them as environment variables:\n\n```bash\nexport AWS_ACCESS_KEY_ID=\nexport AWS_SECRET_ACCESS_KEY=\nexport AWS_DEFAULT_REGION=us-east-1 # or your preferred region\n```\n\n### Getting Temporary Credentials from SSO (Local Testing)\n\nIf you use AWS SSO for authentication, get temporary credentials for local testing:\n\n```bash\n# Login to your SSO profile\naws sso login --profile your-profile-name\n\n# Export credentials from your SSO session\neval $(aws configure export-credentials --profile your-profile-name --format env)\n\n# Verify credentials are set\naws sts get-caller-identity\n```\n\nAlternatively, add this to your shell profile for automatic credential export:\n\n```bash\n# Add to ~/.bashrc or ~/.zshrc\nfunction aws-creds() {\n eval $(aws configure export-credentials --profile $1 --format env)\n}\n\n# Usage: aws-creds your-profile-name\n```\n\n## Features\n\n### Nova 2 Sonic Capabilities\n\nAmazon Nova 2 Sonic is a unified speech-to-speech foundation model that delivers:\n\n- **Realtime bidirectional streaming** - Low-latency, natural conversations\n- **Multilingual support** - English, French, Italian, German, Spanish, Portuguese, and Hindi\n- **Automatic language mirroring** - Responds in the user's spoken language\n- **Polyglot voices** - Matthew and Tiffany can seamlessly switch between languages within a single conversation, ideal for multilingual applications\n- **18 expressive voices** - Multiple voices per language with natural prosody\n- **Function calling** - Built-in tool use and agentic workflows\n- **Interruption handling** - Graceful handling without losing context\n- **Noise robustness** - Works in real-world environments\n- **Text input support** - Programmatic text prompting\n\n### Model Selection\n\n```python\nfrom livekit.plugins import aws\n\n# Nova 2.5 Sonic (audio + text input, latest)\nmodel = aws.realtime.RealtimeModel.with_nova_sonic_2_5()\n\n# Nova 2 Sonic (audio + text input, default)\nmodel = aws.realtime.RealtimeModel.with_nova_sonic_2()\n\n# Nova Sonic 1.0 (audio-only, original model)\nmodel = aws.realtime.RealtimeModel.with_nova_sonic_1()\n```\n\n### Voice Selection\n\nVoices are specified as lowercase strings. Import `SONIC1_VOICES` or `SONIC2_VOICES` type hints for IDE autocomplete.\n\n```python\nfrom livekit.plugins.aws.experimental.realtime import SONIC2_VOICES\n\nmodel = aws.realtime.RealtimeModel.with_nova_sonic_2(\n voice=\"carolina\" # Portuguese, feminine\n)\n```\n\n#### Nova 2 Sonic Voice IDs (18 voices)\n\nSee [official documentation](https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-language-support.html) for most up-to-date list and IDs.\n\n- **English (US)**: `tiffany` (polyglot), `matthew` (polyglot)\n- **English (UK)**: `amy`\n- **English (Australia)**: `olivia`\n- **English (India)**: `kiara`, `arjun`\n- **French**: `ambre`, `florian`\n- **Italian**: `beatrice`, `lorenzo`\n- **German**: `tina`, `lennart`\n- **Spanish (US)**: `lupe`, `carlos`\n- **Portuguese (Brazil)**: `carolina`, `leo`\n- **Hindi**: `kiara`, `arjun`\n\n**Note**: Tiffany abd Matthew in Nova 2 Sonic support polyglot mode, seamlessly switching between languages within a single conversation.\n\n#### Nova Sonic 1.0 Voice IDs (11 voices)\n\nSee [official documentation](https://docs.aws.amazon.com/nova/latest/userguide/available-voices.html) for most up-to-date list and IDs.\n\n- **English (US)**: `tiffany`, `matthew`\n- **English (UK)**: `amy`\n- **French**: `ambre`, `florian`\n- **Italian**: `beatrice`, `lorenzo`\n- **German**: `greta`, `lennart`\n- **Spanish**: `lupe`, `carlos`\n\n### Text Prompting with `generate_reply()`\n\nNova 2 Sonic supports programmatic text input. This can be used to trigger agent responses or to mix speech and text input within a UI in the same conversation:\n\n```python\nclass Assistant(Agent):\n async def on_enter(self):\n # Make the agent speak first with a greeting\n await self.session.generate_reply(\n instructions=\"Greet the user and introduce your capabilities\"\n )\n```\n\n#### `instructions` vs `user_input`\n\nThe `generate_reply()` method accepts two parameters with different behaviors:\n\n**`instructions`** - System-level commands (recommended):\n```python\nawait session.generate_reply(\n instructions=\"Greet the user warmly and ask how you can help\"\n)\n```\n- Sent as a system prompt/command to the model\n- Triggers immediate generation\n- Does not appear in conversation history as user message\n- Use for: Agent-initiated speech, prompting specific behaviors\n\n**`user_input`** - Simulated user messages:\n```python\nawait session.generate_reply(\n user_input=\"Hello, I need help with my account\"\n)\n```\n- Sent as interactive USER role content\n- Added to Nova's conversation context\n- Triggers generation as if user spoke\n- Use for: Testing, simulating user input, programmatic conversations\n\n**When to use each:**\n- **Agent greetings**: Use `instructions` - agent should speak without user input\n- **Guided responses**: Use `instructions` - direct the agent's next action\n- **Simulated conversations**: Use `user_input` - test multi-turn dialogs\n- **Programmatic user input**: Use `user_input` - inject text as if user spoke\n\n### Turn-Taking Sensitivity\n\nControl how quickly the agent responds to pauses:\n\n```python\nmodel = aws.realtime.RealtimeModel.with_nova_sonic_2(\n turn_detection=\"MEDIUM\" # HIGH, MEDIUM (default), LOW\n)\n```\n\n- **HIGH**: Fastest response time, optimized for latency. May interrupt slower speakers\n- **MEDIUM**: Balanced approach with moderate response time. Reduces false positives while maintaining responsiveness (recommended)\n- **LOW**: Slowest response time with maximum patience, better for hesitant speakers\n\n### Complete Example\n\n```python\nfrom livekit import agents\nfrom livekit.agents import Agent, AgentSession\nfrom livekit.plugins import aws\nfrom dotenv import load_dotenv\n\n\nload_dotenv()\n\nclass Assistant(Agent):\n def __init__(self):\n super().__init__(\n instructions=\"You are a helpful voice assistant powered by Amazon Nova 2 Sonic.\"\n )\n \n async def on_enter(self):\n await self.session.generate_reply(\n instructions=\"Greet the user and offer assistance\"\n )\n\nserver = agents.AgentServer()\n\n@server.rtc_session()\nasync def entrypoint(ctx: agents.JobContext):\n await ctx.connect()\n \n session = AgentSession(\n llm=aws.realtime.RealtimeModel.with_nova_sonic_2(\n voice=\"matthew\",\n turn_detection=\"MEDIUM\",\n tool_choice=\"auto\"\n )\n )\n \n await session.start(room=ctx.room, agent=Assistant())\n\nif __name__ == \"__main__\":\n agents.cli.run_app(server)\n```\n\n## Pipeline Mode (STT + LLM + TTS)\n\nFor more control over individual components, use pipeline mode:\n\n```python\nfrom livekit.agents import inference\nfrom livekit.plugins import aws\n\nsession = AgentSession(\n stt=aws.STT(), # Amazon Transcribe\n llm=aws.LLM(), # Nova 2 Lite (default)\n tts=aws.TTS(), # Amazon Polly\n vad=inference.VAD(),\n)\n```\n\n### Nova 2 Lite\n\nAmazon Nova 2 Lite is a fast, cost-effective reasoning model optimized for everyday AI workloads:\n\n- **Lightning-fast processing** - Very low latency for real-time conversations\n- **Cost-effective** - Industry-leading price-performance\n- **Multimodal inputs** - Text, image, and video ([documentation](https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-multimodal-models.html))\n- **1 million token context window** - Handle long conversations and complex context ([source](https://aws.amazon.com/blogs/aws/introducing-amazon-nova-2-lite-a-fast-cost-effective-reasoning-model/))\n- **Agentic workflows** - RAG systems, function calling, tool use\n- **Fine-tuning support** - Customize for your specific use case\n\nIdeal for pipeline mode where you need fast, accurate LLM responses in voice applications.\n\n## Resources\n\n- [LiveKit Agents Documentation](https://docs.livekit.io/agents/)\n- [Amazon Nova Documentation](https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-conversational-speech.html)\n","livekit-plugins/livekit-plugins-azure/README.md":"# Azure Plugin for LiveKit Agents\n\nComplete Azure AI integration for LiveKit Agents, including Azure Speech Services and Azure Voice Live Realtime API.\n\n**What's included:**\n- **RealtimeModel** - Azure Voice Live for speech-to-speech\n- **STT** - Powered by Azure Speech Services\n- **TTS** - Powered by Azure Speech Services with neural voices\n\nFor Azure OpenAI LLM (non-realtime), see the [OpenAI plugin](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-openai) which supports `LLM.with_azure()`.\n\nSee [https://docs.livekit.io/agents/integrations/azure/](https://docs.livekit.io/agents/integrations/azure/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-azure\n\n# For the Azure Voice Live realtime model\npip install \"livekit-plugins-azure[realtime]\"\n```\n\n## Realtime Mode (Azure Voice Live)\n\n### Azure Credentials\n\nFor the realtime speech-to-speech model:\n\n```bash\nexport AZURE_VOICE_LIVE_ENDPOINT=https://.api.cognitive.microsoft.com/\nexport AZURE_VOICE_LIVE_API_KEY=\nexport AZURE_VOICE_LIVE_MODEL= # e.g., gpt-realtime, gpt-4o, gpt-4o-mini, etc.\n```\n\nTo authenticate with Microsoft Entra ID instead of an API key, pass `use_default_credential=True`, which uses [`DefaultAzureCredential`](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.aio.defaultazurecredential).\n\nTo power the intelligence of your voice agent, you have flexibility and choice in the generative AI model between GPT-Realtime, GPT-5, GPT-4.1, Phi, and more options. For supported models and regions, see [Supported models and regions](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live#supported-models-and-regions).\n\n\n\n### Quick Start\nAzure Voice Live provides end-to-end speech-to-speech:\n\n```python\nfrom livekit import agents\nfrom livekit.agents import Agent, AgentServer, AgentSession\nfrom livekit.plugins import azure\n\nclass Assistant(Agent):\n def __init__(self):\n super().__init__(\n instructions=\"You are a helpful voice assistant powered by Azure.\"\n )\n\n async def on_enter(self):\n await self.session.generate_reply(\n instructions=\"Greet the user and offer assistance\"\n )\n\nserver = AgentServer()\n\n@server.rtc_session()\nasync def entrypoint(ctx: agents.JobContext):\n session = AgentSession(\n llm=azure.realtime.RealtimeModel(\n voice=\"en-US-AvaMultilingualNeural\",\n )\n )\n\n await session.start(room=ctx.room, agent=Assistant())\n\nif __name__ == \"__main__\":\n agents.cli.run_app(server)\n```\n\n### Advanced Configuration\n\nFor full control over Azure Voice Live settings:\n\n```python\nimport os\nfrom azure.ai.voicelive.models import AzureSemanticVadEn, AudioInputTranscriptionOptions\nfrom livekit.agents import AgentSession\nfrom livekit.plugins import azure\n\n# Configure Semantic VAD for intelligent turn detection\n# Options:\n# - AzureSemanticVad: Default semantic VAD (multilingual)\n# - AzureSemanticVadEn: English-only, optimized for English\n# - AzureSemanticVadMultilingual: Explicit multilingual support\nturn_detection = AzureSemanticVadEn(\n threshold=0.5, # Voice activity detection threshold (0.0-1.0)\n silence_duration_ms=500, # Silence duration before turn ends\n prefix_padding_ms=300, # Audio padding before speech\n speech_duration_ms=200, # Minimum speech duration to trigger detection\n remove_filler_words=True, # Remove filler words like \"um\", \"uh\"\n)\n\n# Configure input audio transcription with language constraint\n# This helps prevent language misidentification\n# Use azure-speech with text models such as gpt-4o, whisper-1 with gpt-realtime or gpt-realtime-mini\ninput_audio_transcription = AudioInputTranscriptionOptions(\n model=\"azure-speech\",\n language=\"en-US\", # Constrain to English for reliable detection\n)\n\nsession = AgentSession(\n llm=azure.realtime.RealtimeModel(\n endpoint=os.getenv(\"AZURE_VOICE_LIVE_ENDPOINT\"),\n api_key=os.getenv(\"AZURE_VOICE_LIVE_API_KEY\"),\n model=os.getenv(\"AZURE_VOICE_LIVE_MODEL\", \"gpt-4o\"),\n voice=os.getenv(\"AZURE_VOICE_LIVE_VOICE\", \"en-US-AvaMultilingualNeural\"),\n input_audio_transcription=input_audio_transcription,\n turn_detection=turn_detection,\n tool_choice=\"auto\", # Enable function calling\n )\n)\n```\n\n### RealtimeModel Parameters\n\n| Parameter | Description | Default |\n|-----------|-------------|---------|\n| `endpoint` | Azure Voice Live endpoint URL | `AZURE_VOICE_LIVE_ENDPOINT` env var |\n| `api_key` | Azure API key | `AZURE_VOICE_LIVE_API_KEY` env var |\n| `use_default_credential` | Authenticate with `DefaultAzureCredential` instead of an API key | `False` |\n| `model` | Model name | `AZURE_VOICE_LIVE_MODEL` env var, or `gpt-realtime` |\n| `voice` | Azure neural voice name | `en-US-AvaMultilingualNeural` |\n| `modalities` | Output modalities, `[\"text\"]` for text-only responses | `[\"text\", \"audio\"]` |\n| `input_audio_transcription` | Audio transcription config (model, language), `None` to disable | `azure-speech` for non-multimodal models (e.g. `gpt-4.1`) and `phi4-mm-realtime`, `whisper-1` for the other models, with auto-detect |\n| `turn_detection` | VAD configuration object | `ServerVad(threshold=0.5)` |\n| `tool_choice` | Function calling mode (\"auto\", \"none\", etc.) | \"auto\" |\n| `temperature` | Sampling temperature | `0.8` |\n| `max_output_tokens` | Maximum output tokens per response | `4096` |\n\n### Turn Detection Options\n\n| VAD Type | Description |\n|----------|-------------|\n| `AzureSemanticVad` | Default semantic VAD (multilingual) |\n| `AzureSemanticVadEn` | English-only, optimized for English |\n| `AzureSemanticVadMultilingual` | Explicit multilingual support |\n\n### VAD Parameters\n\n| Parameter | Description | Range |\n|-----------|-------------|-------|\n| `threshold` | Voice activity detection sensitivity | 0.0-1.0 |\n| `silence_duration_ms` | Silence duration before turn ends | milliseconds |\n| `prefix_padding_ms` | Audio padding before detected speech | milliseconds |\n| `speech_duration_ms` | Minimum speech duration to trigger | milliseconds |\n| `remove_filler_words` | Remove \"um\", \"uh\", etc. | boolean |\n\n### Azure Voice Live Capabilities\n\n- **Low-latency conversations** - Realtime bidirectional streaming\n- **Multilingual support** - Works with Azure's multilingual neural voices\n- **Function calling** - Built-in tool use for agentic workflows\n- **Interruption handling** - Graceful handling of user interruptions\n\n## Pipeline Mode (STT + LLM + TTS)\n\n\n### Azure Credentials\n\n**Azure Speech Services (for STT/TTS)**\n\n```bash\nexport AZURE_SPEECH_KEY=\nexport AZURE_SPEECH_REGION= # e.g., eastus, westus2\n```\n\n**Azure OpenAI (for LLM)**\n\n```bash\nexport AZURE_OPENAI_ENDPOINT=https://.openai.azure.com\nexport AZURE_OPENAI_API_KEY=\nexport AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o\nexport AZURE_OPENAI_API_VERSION=2024-10-01-preview\n```\n\n### Quick Start\n\nFor more control over individual components:\n\n```python\nfrom livekit.agents import Agent, AgentSession\nfrom livekit.plugins import azure, openai, silero\n\nsession = AgentSession(\n stt=azure.STT(),\n llm=openai.LLM.with_azure(\n model=\"gpt-4o\",\n azure_endpoint=\"https://your-resource.openai.azure.com\",\n api_key=\"your-api-key\",\n api_version=\"2024-10-01-preview\",\n ),\n tts=azure.TTS(voice=\"en-US-JennyNeural\"),\n vad=silero.VAD.load(),\n)\n\nawait session.start(room=ctx.room, agent=Agent(instructions=\"You are helpful.\"))\n```\n\n## Voice Selection\n\nAzure provides a wide variety of neural voices. Use multilingual voices for multi-language support:\n\n```python\n# Multilingual voices (recommended)\nmodel = azure.realtime.RealtimeModel(\n voice=\"en-US-AvaMultilingualNeural\"\n)\n\n# Or use TTS with specific voices\ntts = azure.TTS(voice=\"en-US-JennyNeural\")\n```\n\n**Popular voices:**\n- `en-US-AvaMultilingualNeural` - Multilingual, feminine\n- `en-US-AndrewMultilingualNeural` - Multilingual, masculine\n- `en-US-JennyNeural` - English, feminine\n- `en-US-GuyNeural` - English, masculine\n\nSee [Azure Voice Gallery](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts) for full list.\n\n### HD Voices\n\nFor higher quality audio, use DragonHD voices:\n\n```python\ntts = azure.TTS(voice=\"en-US-Ava:DragonHDLatestNeural\")\n```\n\n## Resources\n\n- [LiveKit Agents Documentation](https://docs.livekit.io/agents/)\n- [LiveKit Azure Integration Guide](https://docs.livekit.io/agents/integrations/azure/)\n- [Azure Voice Live Documentation](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live)\n- [Azure Voice Live Supported Models and Regions](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/voice-live#supported-models-and-regions)\n- [Azure Speech Service Documentation](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/)\n- [Azure OpenAI Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/)\n- [Azure Voice Gallery](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts)\n- [Azure Portal](https://portal.azure.com)\n- [OpenAI Plugin (for Azure OpenAI LLM)](https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-openai)\n","livekit-plugins/livekit-plugins-baseten/README.md":"# Baseten plugin for LiveKit Agents\n\nSupport for [Baseten](https://baseten.co/)-hosted models in LiveKit Agents, including **STT** (Speech-to-Text), **TTS** (Text-to-Speech), and **LLM** (Large Language Model) integrations.\n\n## Installation\n\n```bash\npip install livekit-plugins-baseten\n```\n\n## Pre-requisites\n\nYou'll need an API key from Baseten. It can be set as an environment variable: `BASETEN_API_KEY`\n\nYou also need to deploy a model to Baseten and will need your model endpoint to configure the plugin.\n\n## STT (Speech-to-Text)\n\nThe STT plugin connects to Baseten's [Whisper Streaming](https://docs.baseten.co/reference/inference-api/predict-endpoints/streaming-transcription-api) WebSocket endpoint for real-time transcription. It works with both **truss** and **chain** deployments.\n\n### Recommended model\n\n[Whisper Large V3 Turbo](https://www.baseten.co/library/whisper-large-turbo/) (real-time deployment)\n\n### Endpoint URL formats\n\n| Deployment type | URL pattern |\n|---|---|\n| **Truss** | `wss://model-{model_id}.api.baseten.co/environments/production/websocket` |\n| **Chain** | `wss://chain-{chain_id}.api.baseten.co/environments/production/websocket` |\n\n### Basic usage\n\nYou can specify the endpoint in three ways:\n\n```python\nfrom livekit.plugins import baseten\n\n# 1. Using a truss model ID (recommended for truss deployments)\nstt = baseten.STT(\n api_key=\"your-baseten-api-key\", # or set BASETEN_API_KEY env var\n model_id=\"your-model-id\",\n language=\"en\",\n)\n\n# 2. Using a chain ID (recommended for chain deployments)\nstt = baseten.STT(\n api_key=\"your-baseten-api-key\",\n chain_id=\"your-chain-id\",\n language=\"en\",\n)\n\n# 3. Using a full endpoint URL (for custom routing or deployment URLs)\nstt = baseten.STT(\n api_key=\"your-baseten-api-key\",\n model_endpoint=\"wss://model-{model_id}.api.baseten.co/environments/production/websocket\",\n language=\"en\",\n)\n```\n\n### Configuration options\n\n| Parameter | Default | Description |\n|---|---|---|\n| `api_key` | `BASETEN_API_KEY` env var | Baseten API key |\n| `model_endpoint` | `BASETEN_MODEL_ENDPOINT` env var | Full WebSocket URL (takes priority over `model_id`/`chain_id`) |\n| `model_id` | — | Baseten truss model ID; auto-constructs the endpoint URL |\n| `chain_id` | — | Baseten chain ID; auto-constructs the endpoint URL |\n| `language` | `\"en\"` | BCP-47 language code (use `\"auto\"` for auto-detection) |\n| `language_options` | `[]` | Restrict auto-detection to these codes, e.g. `[\"en\", \"de\"]`. More reliable than `\"auto\"` on short telephony utterances. Requires Whisper runtime v0.5.0+ |\n| `encoding` | `\"pcm_s16le\"` | Audio encoding (`pcm_s16le` or `pcm_mulaw`) |\n| `sample_rate` | `16000` | Audio sample rate in Hz |\n| `enable_partial_transcripts` | `True` | Emit interim transcripts while the speaker is talking |\n| `partial_transcript_interval_s` | `1.0` | Interval (seconds) between partial transcript updates |\n| `final_transcript_max_duration_s` | `30` | Max seconds of audio before forcing a final transcript |\n| `show_word_timestamps` | `True` | Include word-level timestamps in results |\n| `vad_threshold` | `0.5` | Server-side VAD speech probability threshold (0.0–1.0) |\n| `vad_min_silence_duration_ms` | `300` | Minimum silence (ms) to mark end of speech |\n| `vad_speech_pad_ms` | `30` | Padding (ms) added around detected speech |\n\n### Full voice pipeline example\n\n```python\nimport os\nfrom livekit import agents\nfrom livekit.agents import AgentSession, Agent, RoomInputOptions, inference\nfrom livekit.plugins import baseten, openai, noise_cancellation\nfrom livekit.agents.inference import TurnDetector\n\nBASETEN_API_KEY = os.getenv(\"BASETEN_API_KEY\")\nwhisper_model_id = \"your-whisper-model-id\" # or use chain_id for chain deployments\norpheus_model_id = \"your-orpheus-model-id\"\n\n\nclass Assistant(Agent):\n def __init__(self) -> None:\n super().__init__(instructions=\"You are a helpful voice AI assistant.\")\n\n\nasync def entrypoint(ctx: agents.JobContext):\n session = AgentSession(\n stt=baseten.STT(\n api_key=BASETEN_API_KEY,\n model_id=whisper_model_id, # or chain_id=\"your-chain-id\"\n language=\"en\",\n enable_partial_transcripts=True,\n ),\n llm=openai.LLM(\n api_key=BASETEN_API_KEY,\n base_url=\"https://inference.baseten.co/v1\",\n model=\"openai/gpt-oss-120b\",\n ),\n tts=baseten.TTS(\n api_key=BASETEN_API_KEY,\n model_endpoint=(\n f\"https://model-{orpheus_model_id}\"\n \".api.baseten.co/environments/production/predict\"\n ),\n ),\n vad=inference.VAD(),\n turn_detection=TurnDetector(),\n )\n\n await session.start(\n room=ctx.room,\n agent=Assistant(),\n room_input_options=RoomInputOptions(\n noise_cancellation=noise_cancellation.BVC(),\n ),\n )\n\n await session.generate_reply(\n instructions=\"Greet the user and offer your assistance.\"\n )\n\n\nif __name__ == \"__main__\":\n agents.cli.run_app(agents.WorkerOptions(entrypoint_fnc=entrypoint))\n```\n\n## TTS (Text-to-Speech)\n\nThe TTS plugin calls Baseten-hosted TTS models (e.g. [Orpheus 3B](https://www.baseten.co/library/orpheus-tts/)) over HTTP.\n\n```python\ntts = baseten.TTS(\n api_key=\"your-baseten-api-key\",\n model_endpoint=\"https://model-{model_id}.api.baseten.co/environments/production/predict\",\n voice=\"tara\",\n language=\"en\",\n)\n```\n\n## Qwen3 (STT + TTS)\n\nBaseten also hosts **Qwen3-ASR** and **Qwen3-TTS**. They speak different wire\nprotocols from the Whisper/Orpheus trusses, so select them with `model` — the\nsame `baseten.STT` / `baseten.TTS` classes handle both.\n\n```python\nfrom livekit.agents import AgentSession\nfrom livekit.plugins import baseten\n\nsession = AgentSession(\n stt=baseten.STT(model=\"qwen3-asr\", model_id=\"your-qwen3-asr-model-id\"),\n tts=baseten.TTS(model=\"qwen3-tts\", model_id=\"your-qwen3-tts-model-id\",\n voice=\"your-voice\"),\n # llm=...\n)\n```\n\n| | `model=\"whisper\"` / `\"orpheus\"` (default) | `model=\"qwen3-asr\"` / `\"qwen3-tts\"` |\n|---|---|---|\n| STT audio | raw binary PCM | base64 `input_audio_buffer.append` |\n| STT results | `message_type` / `transcript` | `type: \"transcription\"` / `segments[].text` |\n| TTS transport | HTTP, or WS with an `__END__` sentinel | `session.config` → `input.text` → `input.done` |\n| TTS voices | preset names (`tara`) | registered voice clones |\n\nSeveral defaults follow the selected model, matching each deployment's own:\n\n| Parameter | `whisper` / `orpheus` | `qwen3-asr` / `qwen3-tts` |\n|---|---|---|\n| `language` | `en` | `auto` (STT) / `Auto` (TTS) |\n| `partial_transcript_interval_s` | `1.0` | `0.5` |\n| `vad_min_silence_duration_ms` | `300` | `500` |\n| `vad_speech_pad_ms` | `30` | `100` |\n| `show_word_timestamps` | on | off (needs `STREAM_ALIGNER=mms` and `STREAM_ALIGNER_DEVICE=cuda` on the deployment) |\n| `voice` | `tara` | required — a registered clone |\n\n### Qwen3 TTS voices\n\nQwen3-TTS Base ships **no built-in speakers** — there is no `tara` equivalent.\nRegister a clone from 10–20s of clean speech, then pass its name as `voice`:\n\n```python\nfrom livekit.plugins.baseten import list_voices, register_voice\n\nawait register_voice(\n model_endpoint=\"wss://model-{model_id}.api.baseten.co/environments/production/websocket\",\n name=\"my_voice\",\n ref_audio_path=\"./reference.wav\",\n ref_text=\"Transcript of the reference audio.\",\n)\nawait list_voices(model_endpoint=...) # {\"voices\": [...], \"uploaded_voices\": [...]}\n```\n\nThe server stores uploaded voices on the container's local disk, so a voice\nregistered at runtime lives on **one replica** and is lost when that container\nrestarts. For anything beyond single-replica testing, bake the reference audio\ninto the deployment (`REQUIRED_VOICES`) so every replica starts with it, or pass\n`ref_audio`/`ref_text` to clone inline on each session.\n\n### Qwen3-only options\n\n`task_type` (`Base` cloning, or `CustomVoice`/`VoiceDesign` deployments),\n`instructions`, `max_new_tokens`, `initial_codec_chunk_frames`,\n`x_vector_only_mode`, `ref_audio`/`ref_text`, and `extra_config` (merged into\n`session.config` for server fields newer than this plugin) apply to\n`model=\"qwen3-tts\"` only.\n\n## LLM (Large Language Model)\n\nThe LLM plugin wraps Baseten's OpenAI-compatible inference endpoint.\n\n```python\nllm = baseten.LLM(\n api_key=\"your-baseten-api-key\",\n model=\"openai/gpt-oss-120b\",\n)\n```\n\n## Documentation\n\n- [LiveKit STT integration guide](https://docs.livekit.io/agents/integrations/stt/baseten/)\n- [LiveKit TTS integration guide](https://docs.livekit.io/agents/integrations/tts/baseten/)\n- [Baseten Whisper Streaming docs](https://docs.baseten.co/reference/inference-api/predict-endpoints/streaming-transcription-api)\n- [Baseten Model Library](https://www.baseten.co/library/)\n","livekit-plugins/livekit-plugins-bey/README.md":"# Beyond Presence plugin for LiveKit Agents\n\nSupport for [Beyond Presence](https://docs.bey.dev) virtual avatars.\n\nSee [https://docs.livekit.io/agents/integrations/avatar/bey/](https://docs.livekit.io/agents/integrations/avatar/bey/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-bey\n```\n\n## Pre-requisites\n\nCreate a developer API key from the [creator dashboard](https://app.bey.chat) and set the `BEY_API_KEY` environment variable with it:\n\n```bash\nexport BEY_API_KEY=\n```\n","livekit-plugins/livekit-plugins-bithuman/README.md":"# bitHuman plugin for LiveKit Agents\n\nSupport for [bitHuman](https://www.bithuman.ai/) virtual avatars in LiveKit Agents. The avatar renders either locally, inside your agent's process from a bitHuman `.imx` model file (no GPU required), or in the bitHuman cloud, where it joins the room as a participant.\n\nSee the [bitHuman integration docs](https://docs.livekit.io/agents/models/avatar/plugins/bithuman/) for more information, and bitHuman's [LiveKit guide](https://docs.bithuman.ai/platforms/livekit) for a complete agent.\n\n## Installation\n\n```bash\npip install livekit-plugins-bithuman\n```\n\nTo render Expression 2 models locally, also install the SDK's Expression 2 extra:\n\n```bash\npip install \"bithuman[expression-2]\"\n```\n\n## Pre-requisites\n\nYou'll need an API secret from bitHuman ([create one](https://www.bithuman.ai/developer/api-keys)). Store it under a name the plugin doesn't read, such as `BITHUMAN_MASTER_SECRET`, and pass it as `api_secret=`. If `api_secret` isn't passed, the plugin falls back to `BITHUMAN_API_SECRET`.\n\nIn cloud mode the plugin puts `api_secret` in the avatar participant's attributes, which everyone in the room can read. Pass a one-hour token minted for one agent and one room (`POST https://api.bithuman.ai/v1/runtime-tokens/mint`, [how](https://docs.bithuman.ai/platforms/livekit#authenticate)) instead of the secret, and keep `BITHUMAN_API_SECRET` unset. Cloud mode also uses `LIVEKIT_URL`, `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET`.\n\n## Usage\n\nLocal mode: the avatar renders in this process.\n\n```python\nimport os\n\nfrom livekit.plugins import bithuman\n\navatar = bithuman.AvatarSession(\n model_path=\"./agent.imx\", # or set BITHUMAN_MODEL_PATH\n api_secret=os.environ[\"BITHUMAN_MASTER_SECRET\"],\n)\nawait avatar.start(session, room=ctx.room)\n```\n\nCloud mode: the avatar renders on bitHuman's servers.\n\n```python\navatar = bithuman.AvatarSession(\n avatar_id=\"A23WJF0199\", # your bitHuman agent code (A23WJF0199 is the wise-pup sample)\n api_secret=minted_token, # from POST /v1/runtime-tokens/mint\n)\nawait avatar.start(session, room=ctx.room)\n```\n\nIn both modes the avatar publishes the agent's audio, so start the session with `room_options=room_io.RoomOptions(audio_output=False)`.\n\n## Models\n\n- **Essence 2** (`essence-2`): a photoreal person from one portrait.\n- **Expression 2** (`expression-2`): any character, including people, animals and cartoons, from one portrait.\n- `essence` / `expression`: the first-generation Essence 1 and Expression 1. Expression 1 runs in the cloud only.\n\nIn local mode the `.imx` file decides the model. In cloud mode, leave `model` unset to use the model the agent was created with, or set it to pin one; bitHuman refuses a model the avatar wasn't prepared for. A photo passed as `avatar_image` (no `avatar_id`) works only with Expression 1. Pricing: [docs.bithuman.ai/pricing](https://docs.bithuman.ai/pricing).\n","livekit-plugins/livekit-plugins-bland/README.md":"# Bland plugin for LiveKit Agents\n\nSupport for voice synthesis with the [Bland](https://bland.ai/) API.\n\nVoice agents can stream text through Bland's realtime WebSocket. Use `synthesize()` for complete strings over HTTP, or pass `streaming=False` to use HTTP for everything. See the [Bland realtime TTS reference](https://docs.bland.ai/api-v2/post/tts-ws).\n\n## Installation\n\n```bash\npip install livekit-plugins-bland\n```\n\n## Prerequisites\n\nYou'll need an API key from Bland. It can be set as an environment variable: `BLAND_API_KEY`\n","livekit-plugins/livekit-plugins-boson-avatar/README.md":"# Boson Higgs Avatar plugin for LiveKit Agents\n\nUse Boson's Higgs Audio-Driven Avatar as the video output for a LiveKit voice\nagent. This is an Avatar plugin: it composes with your existing voice/LLM\nplugin and does not replace or fork it.\n\nThe plugin has no dependency on Boson Voice, Boson Audio, or a particular TTS\nprovider. Its input is the standard audio output of a LiveKit `AgentSession`,\nso it works with any TTS plugin, realtime model, or custom source that produces\nLiveKit audio frames. Only Avatar rendering and Avatar session lifecycle are\nBoson-specific.\n\n## Installation\n\n```shell\npip install livekit-plugins-boson-avatar\n```\n\nSet the credentials used by your LiveKit Agent Worker:\n\n```shell\nexport BOSON_API_KEY=\"...\"\nexport BOSON_AVATAR_API_URL=\"https://your-avatar-session-service.example/v1\"\nexport LIVEKIT_URL=\"wss://...\"\nexport LIVEKIT_API_KEY=\"...\"\nexport LIVEKIT_API_SECRET=\"...\"\n```\n\nThe plugin deliberately has no hard-coded provider endpoint. Your application\nor deployment environment supplies the base URL exposed by its Boson Avatar\ndeployment/operator. The plugin appends `POST /sessions` when starting an\nAvatar and `DELETE /sessions/{id}` during cleanup, so do not include\n`/sessions` itself in `BOSON_AVATAR_API_URL`.\n\nLoad the project-scoped Avatar catalog on your application server and use it\nto populate the face picker. The returned `avatar_id` is passed unchanged to\n`AvatarSession`; browser users never need to type or remember it:\n\n```python\nfrom livekit.plugins import boson_avatar\n\n\nasync def avatar_options():\n # Returns [AvatarInfo(avatar_id=\"...\", name=\"...\")]\n return await boson_avatar.list_avatars()\n```\n\nThis calls `GET {BOSON_AVATAR_API_URL}/avatars` with `BOSON_API_KEY`. Keep the\nkey server-side and cache the result according to your application's needs.\n\n## Usage\n\nCreate the voice `AgentSession` with the audio provider of your choice, then\nstart the Avatar before starting the agent session:\n\n```python\nfrom livekit.agents import Agent, AgentSession, JobContext, inference\nfrom livekit.plugins import boson_avatar\n\n\nasync def entrypoint(ctx: JobContext) -> None:\n await ctx.connect()\n\n session = AgentSession(\n stt=inference.STT(\"deepgram/nova-3\"),\n llm=inference.LLM(\"openai/gpt-4.1-mini\"),\n tts=inference.TTS(\"cartesia/sonic-3\"),\n )\n avatar = boson_avatar.AvatarSession(\n avatar_id=\"your-avatar-id\",\n )\n\n await avatar.start(session, room=ctx.room)\n await session.start(\n agent=Agent(instructions=\"You are a helpful assistant.\"),\n room=ctx.room,\n )\n```\n\n`BOSON_AVATAR_ID` can supply `avatar_id` instead. `AvatarSession` also accepts\noptional `width`, `height`, `max_duration_seconds`,\n`avatar_participant_identity`, `idempotency_key`, and `APIConnectOptions`.\n`max_duration_seconds` must be an integer from 1 through 14400.\n\nInside a LiveKit Agent job, provider-session creation automatically derives a\nstable UUID idempotency key from the LiveKit job ID and Avatar session binding.\nIf LiveKit redelivers that job after a worker crash, the plugin recovers the\nexisting provider session instead of allocating another one. The standard\nmodel is one Avatar lifecycle per LiveKit job. If one job intentionally starts\nanother Avatar after closing the first, pass a new explicit UUID\n`idempotency_key` for that lifecycle.\n\nThe `avatar_id` is the value returned by `list_avatars()` behind the Avatar\nselected in your application; end users do not need to type or remember it.\n\nThe plugin handles the provider API call, LiveKit participant token, PCM data\nstream routing, interruption buffer clears, and provider-session cleanup. A\ndeveloper does not need to call Boson's Avatar REST API or combine voice and\nAvatar responses in an application server. The application server only needs\nits normal responsibility: create a LiveKit room and dispatch the Agent Worker.\n\nWhen started inside a LiveKit job, cleanup is registered automatically. When\nusing the plugin in a standalone script or test, open LiveKit's HTTP context and\nclose the Avatar explicitly:\n\n```python\nfrom livekit.agents import utils\n\n\nasync with utils.http_context.open():\n await avatar.start(session, room)\n try:\n # Run the standalone session.\n ...\n finally:\n await avatar.aclose()\n```\n","livekit-plugins/livekit-plugins-browser/README.md":"# Browser plugin for LiveKit Agents\n\nChromium Embedded Framework (CEF) for LiveKit Agents.\n\n## Installation\n\n```bash\npip install livekit-plugins-browser\n```\n\nCEF binaries are downloaded automatically on first use via `livekit.browser.download()`.\n","livekit-plugins/livekit-plugins-cambai/README.md":"# Camb.ai Plugin for LiveKit Agents\n\nText-to-Speech plugin for [Camb.ai](https://camb.ai) TTS API, powered by MARS technology.\n\n## Features\n\n- High-quality neural text-to-speech with MARS series models\n- Multiple model variants (mars-flash, mars-pro)\n- Enhanced pronunciation for names and places\n- Support for 140+ languages\n- Real-time HTTP streaming\n- Pre-built voice library\n\n## Installation\n\n```bash\npip install livekit-plugins-cambai\n```\n\n## Prerequisites\n\nYou'll need a Camb.ai API key. Set it as an environment variable:\n\n```bash\nexport CAMB_API_KEY=your_api_key_here\n```\n\nOr obtain it from [Camb.ai Studio](https://studio.camb.ai/public/onboarding).\n\n## Quick Start\n\n```python\nimport asyncio\nfrom livekit.plugins.cambai import TTS\n\nasync def main():\n # Initialize TTS (uses CAMB_API_KEY env var)\n tts = TTS()\n\n # Synthesize speech\n stream = tts.synthesize(\"Hello from Camb.ai!\")\n audio_frame = await stream.collect()\n\n # Save to file\n with open(\"output.wav\", \"wb\") as f:\n f.write(audio_frame.to_wav_bytes())\n\nasyncio.run(main())\n```\n\n## List Available Voices\n\n```python\nimport asyncio\nfrom livekit.plugins.cambai import list_voices\n\nasync def main():\n voices = await list_voices()\n for voice in voices:\n print(f\"{voice['name']} ({voice['id']}): {voice['gender']}, {voice['language']}\")\n\nasyncio.run(main())\n```\n\n## Select a Specific Voice\n\n```python\ntts = TTS(voice_id=147320)\nstream = tts.synthesize(\"Using a specific voice!\")\n```\n\n## Model Selection\n\nCamb.ai offers multiple MARS models for different use cases:\n\n```python\n# Faster inference, 22050 Hz (default)\ntts = TTS(model=\"mars-flash\")\n\n# Higher quality, 48000 Hz\ntts = TTS(model=\"mars-pro\")\n```\n\n## Advanced Configuration\n\n```python\ntts = TTS(\n api_key=\"your-api-key\", # Or use CAMB_API_KEY env var\n voice_id=147320, # Voice ID from list-voices\n language=\"en-us\", # BCP-47 locale\n model=\"mars-pro\", # MARS model variant\n output_format=\"pcm_s16le\", # Audio format\n enhance_named_entities=True, # Better pronunciation for names/places\n)\n```\n\n## Usage with LiveKit Agents\n\n```python\nfrom livekit import agents\nfrom livekit.plugins.cambai import TTS\n\nasync def entrypoint(ctx: agents.JobContext):\n # Connect to room\n await ctx.connect()\n\n # Initialize TTS\n tts = TTS(language=\"en-us\")\n\n # Synthesize and publish\n stream = tts.synthesize(\"Hello from LiveKit with Camb.ai!\")\n audio_frame = await stream.collect()\n\n # Publish to room\n source = agents.AudioSource(tts.sample_rate, tts.num_channels)\n track = agents.LocalAudioTrack.create_audio_track(\"tts\", source)\n await ctx.room.local_participant.publish_track(track)\n await source.capture_frame(audio_frame)\n```\n\n## Configuration Options\n\n### TTS Constructor Parameters\n\n- **api_key** (str | None): Camb.ai API key\n- **voice_id** (int): Voice ID to use (default: 147320)\n- **language** (str): BCP-47 locale (default: \"en-us\")\n- **model** (SpeechModel): MARS model variant (default: \"mars-flash\")\n- **output_format** (OutputFormat): Audio format (default: \"pcm_s16le\")\n- **enhance_named_entities** (bool): Enhanced pronunciation (default: False)\n- **sample_rate** (int | None): Audio sample rate (auto-detected from model if None)\n- **base_url** (str): API base URL\n- **http_session** (aiohttp.ClientSession | None): Reusable HTTP session\n\n### Available Models\n\n- **mars-flash**: Faster inference, 22050 Hz (default)\n- **mars-pro**: Higher quality synthesis, 48000 Hz\n\n### Output Formats\n\n- **pcm_s16le**: 16-bit PCM (recommended for streaming)\n- **pcm_s32le**: 32-bit PCM (highest quality)\n- **wav**: WAV with headers\n- **flac**: Lossless compression\n- **adts**: ADTS streaming format\n\n## API Reference\n\n### TTS Class\n\nMain text-to-speech interface.\n\n**Methods:**\n- `synthesize(text: str) -> ChunkedStream`: Synthesize text to speech\n- `update_options(**kwargs)`: Update voice settings dynamically\n- `aclose()`: Clean up resources\n\n**Properties:**\n- `model` (str): Current MARS model name\n- `provider` (str): Provider name (\"Camb.ai\")\n- `sample_rate` (int): Audio sample rate (22050 or 48000 Hz depending on model)\n- `num_channels` (int): Number of audio channels (1)\n\n### list_voices Function\n\n```python\nasync def list_voices(\n api_key: str | None = None,\n base_url: str = \"https://client.camb.ai/apis\",\n) -> list[dict]\n```\n\nReturns list of voice dicts with: id, name, gender, age, language.\n\n## Multi-Language Support\n\nCamb.ai supports 140+ languages. Specify using BCP-47 locales:\n\n```python\n# French\ntts = TTS(language=\"fr-fr\", voice_id=...)\n\n# Spanish\ntts = TTS(language=\"es-es\", voice_id=...)\n\n# Japanese\ntts = TTS(language=\"ja-jp\", voice_id=...)\n```\n\n## Dynamic Options\n\nUpdate TTS settings without recreating the instance:\n\n```python\ntts = TTS()\n\n# Change voice\ntts.update_options(voice_id=12345)\n\n# Change model\ntts.update_options(model=\"mars-pro\")\n```\n\n## Error Handling\n\nThe plugin handles errors according to LiveKit conventions:\n\n```python\nfrom livekit.agents import APIStatusError, APIConnectionError, APITimeoutError\n\ntry:\n stream = tts.synthesize(\"Hello!\")\n audio = await stream.collect()\nexcept APIStatusError as e:\n print(f\"API error: {e.status_code} - {e.message}\")\nexcept APIConnectionError as e:\n print(f\"Connection error: {e}\")\nexcept APITimeoutError as e:\n print(f\"Request timed out: {e}\")\n```\n\n## Future Features\n\nComing soon:\n- GCP Vertex AI integration\n- Voice cloning via custom voice creation\n- Voice generation from text descriptions\n- WebSocket streaming for real-time applications\n\n## Links\n\n- [Camb.ai Documentation](https://docs.camb.ai/)\n- [LiveKit Agents Documentation](https://docs.livekit.io/agents/)\n- [GitHub Repository](https://github.com/livekit/agents)\n\n## License\n\nApache License 2.0\n","livekit-plugins/livekit-plugins-cartesia/README.md":"# Cartesia plugin for LiveKit Agents\n\nSupport for [Cartesia](https://cartesia.ai/)'s voice AI services in LiveKit Agents.\n\nMore information is available in the docs for the [STT](https://docs.livekit.io/agents/integrations/stt/cartesia/) and [TTS](https://docs.livekit.io/agents/integrations/tts/cartesia/) integrations.\n\n## Installation\n\n```bash\npip install livekit-plugins-cartesia\n```\n\n## Pre-requisites\n\nYou'll need an API key from Cartesia. It can be set as an environment variable: `CARTESIA_API_KEY`\n","livekit-plugins/livekit-plugins-cerebras/README.md":"# Cerebras plugin for LiveKit Agents\n\nSupport for LLM with [Cerebras](https://cerebras.ai/) fast inference.\n\nSee [https://docs.livekit.io/agents/integrations/llm/](https://docs.livekit.io/agents/integrations/llm/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-cerebras\n```\n\n## Pre-requisites\n\nFor credentials, you'll need a Cerebras account and API key. Credentials can be passed directly or via `CEREBRAS_API_KEY` environment variable.\n","livekit-plugins/livekit-plugins-clova/README.md":"# Clova plugin for LiveKit Agents\n\nSupport for speech-to-text with [Clova](https://api.ncloud-docs.com/docs/).\n\nSee https://docs.livekit.io/agents/integrations/stt/clova/ for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-clova\n```\n\n## Pre-requisites\n\nYou need invoke url and secret key from Naver cloud platform -> Clova Speech and set as environment variables: `CLOVA_STT_INVOKE_URL` & `CLOVA_STT_SECRET_KEY`\n","livekit-plugins/livekit-plugins-deepgram/README.md":"# Deepgram plugin for LiveKit Agents\n\nSupport for [Deepgram](https://deepgram.com/)'s voice AI services in LiveKit Agents.\n\nMore information is available in the docs for the [STT](https://docs.livekit.io/agents/integrations/stt/deepgram/) and [TTS](https://docs.livekit.io/agents/integrations/tts/deepgram/) integrations.\n\n## Installation\n\n```bash\npip install livekit-plugins-deepgram\n```\n\n## Pre-requisites\n\nYou'll need an API key from DeepGram. It can be set as an environment variable: `DEEPGRAM_API_KEY`\n","livekit-plugins/livekit-plugins-did/README.md":"# D-ID plugin for LiveKit Agents\n\nSupport for the [D-ID](https://d-id.com/) virtual avatar.\n\nSee the [D-ID integration docs](https://docs.livekit.io/agents/models/avatar/plugins/did/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-did\n```\n\n## Pre-requisites\n\nYou'll need an API key from D-ID. It can be set as an environment variable: `DID_API_KEY`\n\n## Supported avatars\n\nThis plugin only supports **v4 avatars** (type: `expressive`). Earlier avatar versions are not compatible. See the [D-ID Create Agent API](https://docs.d-id.com/reference/createagent) for details on creating a compatible agent.\n\nExample — creating an expressive agent via the D-ID API:\n```bash\ncurl -X POST https://api.d-id.com/agents \\\n -H \"Authorization: Basic \" \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"presenter\": {\n \"type\": \"expressive\",\n \"presenter_id\": \"public_mia_elegant@avt_TJ0Tq5\"\n },\n \"preview_name\": \"My Expressive Agent\"\n }'\n```\n\nUse the agent ID from the response as the `agent_id` parameter in the plugin.\n","livekit-plugins/livekit-plugins-elevenlabs/README.md":"# ElevenLabs plugin for LiveKit Agents\n\nSupport for voice synthesis with [ElevenLabs](https://elevenlabs.io/).\n\nSee [https://docs.livekit.io/agents/integrations/tts/elevenlabs/](https://docs.livekit.io/agents/integrations/tts/elevenlabs/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-elevenlabs\n```\n\n## Pre-requisites\n\nYou'll need an API key from ElevenLabs. It can be set as an environment variable: `ELEVEN_API_KEY`\n\n## Realtime speech-to-text audio chunks\n\nConfigure the outgoing audio chunk duration when creating the STT instance:\n\n```python\nfrom livekit.plugins import elevenlabs\n\nstt = elevenlabs.STT(\n model=\"scribe_v2_realtime\",\n audio_chunk_duration_ms=100,\n)\n```\n\n`audio_chunk_duration_ms` accepts a positive integer in milliseconds and defaults\nto 50. At 100 ms, continuous audio produces approximately 10 audio messages per\nsecond instead of 20, at the cost of up to 50 ms additional buffering. Larger\nchunks do not pace reconnect backlogs or guarantee that provider queue errors\nare avoided. Flushes send any remaining partial chunk before the commit.\n\nThis option applies only to realtime STT and is set at construction time.\n","livekit-plugins/livekit-plugins-fal/README.md":"# fal plugin for LiveKit Agents\n\nSupport for speech-to-text with [fal.ai](https://fal.ai/).\n\nSee [https://docs.livekit.io/agents/integrations/stt/fal/](https://docs.livekit.io/agents/integrations/stt/fal/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-fal\n```\n\n## Pre-requisites\n\nYou'll need an API key from fal. It can be set as an environment variable: `FAL_KEY`\n","livekit-plugins/livekit-plugins-fireworksai/README.md":"# Fireworks AI plugin for LiveKit Agents\n\nSupport for speech-to-text api with [Fireworks AI](https://fireworks.ai/).\n\n## Installation\n\n```bash\npip install livekit-plugins-fireworksai\n```\n\n## Pre-requisites\n\nYou'll need an API key from Fireworks AI. It can be set as an environment variable: `FIREWORKS_API_KEY` \n","livekit-plugins/livekit-plugins-fishaudio/README.md":"# Fish Audio plugin for LiveKit Agents\n\nSupport for voice synthesis with [Fish Audio](https://fish.audio/).\n\n- Docs: `https://docs.fish.audio/`\n\n## Installation\n\n```bash\npip install livekit-plugins-fishaudio\n```\n\n## Prerequisites\n\nObtain an API key from Fish Audio.\n\nSet the API key as an environment variable:\n\n```\nFISH_API_KEY=\n```\n\n## Usage\n\n```python\nfrom livekit.agents import AgentSession\nfrom livekit.plugins import fishaudio\n\n# Basic usage with env-based credentials\ntts = fishaudio.TTS()\n\nsession = AgentSession(\n tts=tts,\n # ... stt, llm, etc.\n)\n```\n","livekit-plugins/livekit-plugins-gladia/README.md":"# Gladia plugin for LiveKit Agents\n\nSupport for speech-to-text with [Gladia](https://gladia.io/).\n\nSee [https://docs.livekit.io/agents/integrations/stt/gladia/](https://docs.livekit.io/agents/integrations/stt/gladia/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-gladia\n```\n\n## Pre-requisites\n\nYou'll need an API key from Gladia. It can be set as an environment variable: `GLADIA_API_KEY`\n\n## Features\n\n- Streaming speech-to-text\n- Multi-language support\n- Code-switching between languages\n- Interim results (partial transcriptions)\n- Voice activity detection with energy filtering\n- Optional real-time translation\n- Customizable audio parameters (sample rate, bit depth, channels, encoding)\n\n## Example Usage\n\n```python\nfrom livekit.stt import STT\nfrom livekit.plugins.gladia.stt import STT as GladiaSTT\n\n# Basic initialization\nstt = GladiaSTT(\n api_key=\"your-api-key-here\", # or use GLADIA_API_KEY env var\n interim_results=True\n)\n\n# With more options\nstt = GladiaSTT(\n languages=[\"en\", \"fr\"], # Specify languages or let Gladia auto-detect\n code_switching=True, # Allow switching between languages during recognition\n sample_rate=16000, # Audio sample rate in Hz\n bit_depth=16, # Audio bit depth\n channels=1, # Number of audio channels\n region=\"eu-west\" # Specify Region to use for the Gladia API\n encoding=\"wav/pcm\", # Audio encoding format\n energy_filter=True, # Enable voice activity detection\n translation_enabled=True,\n translation_target_languages=[\"en\"],\n translation_model=\"base\",\n translation_match_original_utterances=True\n translation_context_adaptation= False, # Enable context-aware translation\n translation_context= None, # Context input to guide translation\n translation_informal=False, # Use informal tone in translation\n pre_processing_audio_enhancer=False, # Apply pre-processing to the audio stream to enhance the quality\n pre_processing_speech_threshold=0.6, # Sensitivity for speech detection; closer to 1 = stricter, less background noise\n\n # Custom_vocabulary exemple\n custom_vocabulary=[\n \"Westeros\",\n {\"value\": \"Stark\"},\n {\n \"value\": \"Night's Watch\",\n \"pronunciations\": [\"Nightz Watch\"],\n \"intensity\": 0.4,\n \"language\": \"en\"\n }\n ],\n\n # Custom_spelling exemple\n custom_spelling={\n \"Gorish\": [\"ghorish\", \"gaurish\", \"gaureish\"],\n \"Data Science\": [\"data-science\", \"data science\"],\n \".\": [\"period\", \"full stop\"],\n \"SQL\": [\"sequel\"]\n }\n)\n\n# Update options after initialization\nstt.update_options(\n languages=[\"ja\", \"en\"],\n translation_enabled=True,\n translation_target_languages=[\"fr\"]\n)\n```\n\n## Using with LiveKit Agents Framework\n\n```python\nfrom livekit.agents import Agent\nfrom livekit.plugins.gladia.stt import STT as GladiaSTT\n\nagent = Agent(\n stt=GladiaSTT(\n api_key=\"your-api-key-here\",\n languages=[\"en\"],\n translation_enabled=True,\n translation_target_languages=[\"es\"]\n )\n)\n\n# Rest of your agent setup...\n```\n","livekit-plugins/livekit-plugins-gnani/README.md":"# livekit-plugins-gnani\n\n[![PyPI](https://img.shields.io/pypi/v/livekit-plugins-gnani)](https://pypi.org/project/livekit-plugins-gnani/)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](../../LICENSE)\n\n[LiveKit Agents](https://github.com/livekit/agents) plugin for **[Gnani](https://gnani.ai/)** — high-accuracy Speech-to-Text (Prisma) and low-latency Text-to-Speech (Timbre) for Indian languages.\n\n> **Gnani** is a production-ready speech AI platform supporting 10+ Indian languages, real-time streaming, and multilingual transcription.\n\n> This integration is maintained by [Gnani.ai](https://gnani.ai/).\n\n## Installation\n\n```bash\npip install livekit-plugins-gnani\n```\n\nOr with [uv](https://docs.astral.sh/uv/):\n\n```bash\nuv add livekit-plugins-gnani\n```\n\nThis will also install the [`websockets`](https://pypi.org/project/websockets/) and [`livekit-agents`](https://pypi.org/project/livekit-agents/) packages as dependencies.\n\nInstall with the LiveKit Agents Gnani extra:\n\n```bash\nuv add \"livekit-agents[gnani]\"\n```\n\n## Prerequisites\n\nYou need a Gnani API key. [Gnani APIs](https://app.gnani.ai/voice) have this.\n\nSet your credentials as environment variables:\n\n```bash\nexport GNANI_API_KEY=\"your-api-key\"\n```\n\n**Or pass the key in the constructor:**\n\n```python\nstt = STT(api_key=\"your-api-key\", language=\"hi-IN\")\ntts = TTS(api_key=\"your-api-key\")\n```\n\n## Environment variables\n\n| Variable | Purpose |\n|----------|---------|\n| `GNANI_API_KEY` | API key for Gnani Vachana STT and TTS |\n\n## Quick Start — AgentSession snippet\n\nThe snippet below shows core `AgentSession` wiring with Gnani WebSocket STT/TTS.\n\n```python\nfrom livekit.agents import AgentSession, room_io\nfrom livekit.plugins import gnani, groq, silero\n\nsession = AgentSession(\n stt=gnani.STT(\n language=\"en-IN\",\n use_streaming=True,\n ),\n llm=groq.LLM(model=\"llama-3.1-8b-instant\"),\n tts=gnani.TTS(\n voice=\"Nalini\",\n model=\"timbre-v2.5\",\n language=\"en-IN\",\n synthesize_method=\"websocket\",\n ),\n vad=silero.VAD.load(),\n)\n\nawait session.start(\n agent=MyAgent(),\n room=ctx.room,\n room_options=room_io.RoomOptions(\n audio_input=room_io.AudioInputOptions(sample_rate=16000),\n audio_output=room_io.AudioOutputOptions(sample_rate=16000),\n ),\n)\n```\n\nSet `use_streaming=False` (STT) or swap `synthesize_method` (TTS) for REST variants — see below. WebSocket STT + TTS is the default for lowest latency.\n\n## Service Construction\n\n### Speech-to-Text (REST)\n\n```python\nfrom livekit.plugins.gnani import STT\n\nstt = STT(\n language=\"hi-IN\",\n use_streaming=False,\n)\n```\n\nREST mode requires a VAD in the pipeline. LiveKit wraps the STT with `stt.StreamAdapter` automatically when `use_streaming=False`.\n\n### Speech-to-Text (Streaming WebSocket)\n\n```python\nfrom livekit.plugins.gnani import STT\n\nstt = STT(\n language=\"hi-IN\",\n use_streaming=True,\n sample_rate=16000,\n)\n```\n\n### Text-to-Speech (REST)\n\n```python\nfrom livekit.plugins.gnani import TTS\n\ntts = TTS(\n voice=\"Pranav\",\n model=\"timbre-v2.0\",\n synthesize_method=\"rest\",\n)\n```\n\n### Text-to-Speech (SSE Streaming)\n\n```python\nfrom livekit.plugins.gnani import TTS\n\ntts = TTS(\n voice=\"Pranav\",\n synthesize_method=\"sse\",\n)\n```\n\n### Text-to-Speech (WebSocket Streaming)\n\n```python\nfrom livekit.plugins.gnani import TTS\n\ntts = TTS(\n voice=\"Nalini\",\n model=\"timbre-v2.5\",\n language=\"hi-IN\",\n synthesize_method=\"websocket\",\n)\n```\n\nThe `stream()` method always uses WebSocket regardless of `synthesize_method`.\n\n## Services\n\n### STT\n\n| Mode | Parameter | Transport | Description |\n|------|-----------|-----------|-------------|\n| REST | `use_streaming=False` | POST `/stt/v3` | File/buffer transcription. Requires VAD. |\n| WebSocket | `use_streaming=True` | `wss://api.vachana.ai/stt/v3/stream` | Real-time streaming with VAD. Default. |\n\n#### Streaming PCM Specification\n\nAll streaming audio must be sent as **raw PCM binary frames** — no container format (WAV, MP3) mid-stream.\n\n| Property | 16 kHz | 8 kHz |\n|-------------------|-------------------------------------------|-------------------------------------------|\n| Encoding | PCM signed 16-bit little-endian | PCM signed 16-bit little-endian |\n| Sample Rate | 16,000 Hz | 8,000 Hz |\n| Channels | 1 (mono) | 1 (mono) |\n| Samples per chunk | 512 | 512 |\n| **Bytes per frame** | **1,024 bytes** (512 samples × 2 bytes) | **1,024 bytes** (512 samples × 2 bytes) |\n| Frame duration | 32 ms | 64 ms |\n\nFrames must be sent at **real-time cadence**. See **[STT Realtime — PCM Specification](https://docs.gnani.ai/api/STT/stt-websocket#pcm-specification)** for full details.\n\n### TTS\n\n| Mode | Parameter | Transport | Description |\n|------|-----------|-----------|-------------|\n| REST | `synthesize_method=\"rest\"` | POST `/api/v1/tts/inference` | Single-request batch synthesis. Default for `synthesize()`. |\n| SSE | `synthesize_method=\"sse\"` | POST `/api/v1/tts/sse` | Chunked synthesis via Server-Sent Events. |\n| WebSocket | `synthesize_method=\"websocket\"` or `stream()` | `wss://api.vachana.ai/api/v1/tts` | Lowest latency; `stream()` always uses WebSocket. |\n\n## Full Constructor Reference\n\n### STT — All parameters\n\n```python\nfrom livekit.plugins.gnani import STT\n\nstt = STT(\n language=\"en-IN\", # Default: \"en-IN\"\n sample_rate=16000, # Default: 16000 (also: 8000, 44100, 48000)\n format=\"verbatim\", # Default: \"verbatim\" (also: \"transcribe\" for ITN)\n itn_native_numerals=False, # Default: False\n use_streaming=True, # Default: True (WebSocket); False = REST + VAD\n api_key=None, # Default: None (reads GNANI_API_KEY env var)\n base_url=\"https://api.vachana.ai\",\n)\n```\n\n### TTS — All parameters\n\n```python\nfrom livekit.plugins.gnani import TTS\n\ntts = TTS(\n voice=\"Pranav\", # Default: \"Pranav\" (timbre-v2.0: Kaveri, Shubhra, Deepak)\n model=\"timbre-v2.0\", # Default: \"timbre-v2.0\" (also: \"timbre-v2.5\" with 42 voices)\n language=None, # timbre-v2.5 only — e.g. \"hi-IN\", \"en-IN\"\n sample_rate=16000, # Default: 16000 (also: 8000, 22050, 44100)\n encoding=\"linear_pcm\", # Default: \"linear_pcm\" (also: \"oggopus\")\n container=\"wav\", # Default: \"wav\" (also: \"raw\", \"mp3\", \"ogg\")\n num_channels=1, # Default: 1\n bitrate=None, # Default: None (also: \"96k\", \"128k\", \"192k\")\n synthesize_method=\"rest\", # Default: \"rest\" (also: \"sse\", \"websocket\")\n api_key=None, # Default: None (reads GNANI_API_KEY env var)\n base_url=\"https://api.vachana.ai\",\n)\n```\n\n## Supported Languages\n\n### STT Languages (Prisma)\n\nSTT uses BCP-47 locale codes (e.g. `hi-IN`, `bn-IN`).\n\nFor the full list of supported languages, see:\n\n- **[STT REST — Supported Languages](https://docs.gnani.ai/api/STT/speech-to-text#supported-languages)**\n- **[STT Realtime — Supported Languages](https://docs.gnani.ai/api/STT/stt-websocket#supported-languages)**\n\n### TTS Languages (Timbre)\n\nThe optional `language` parameter is supported for **`timbre-v2.5` only**. For the full list, see **[TTS — Supported Languages](https://docs.gnani.ai/api/TTS/tts-inference#supported-languages)**.\n\n> **Migration:** The former model name `vachana-voice-v3` has been renamed to **`timbre-v2.0`**. Update any `model=\"vachana-voice-v3\"` calls to `model=\"timbre-v2.0\"` (or omit `model` to use the default).\n\n## Available Voices\n\nSee the [official voice list](https://docs.gnani.ai/api/TTS/tts-sse#available-voices) for the latest supported voices.\n\n### timbre-v2.0 (4 voices)\n\n| Voice | ID | Gender | Description |\n|---------|-----------|--------|--------------------------|\n| Pranav | `Pranav` | Male | Bold, Trustworthy |\n| Kaveri | `Kaveri` | Female | Confident, Bright |\n| Shubhra | `Shubhra` | Female | Gentle, Expressive |\n| Deepak | `Deepak` | Male | Grounded, Conversational |\n\n### timbre-v2.5 (42 voices)\n\nThe expanded catalog includes voices across Hindi, English, Tamil, Telugu, Kannada, Malayalam, Marathi, Bengali, Gujarati, Punjabi, and Hinglish. Use `model=\"timbre-v2.5\"` with an optional `language` parameter (e.g. `language=\"hi-IN\"`).\n\n```python\nfrom livekit.plugins.gnani import TTS\n\ntts = TTS(model=\"timbre-v2.5\", voice=\"Nalini\", language=\"hi-IN\")\n```\n\n## Architecture\n\n```\nlivekit-plugins-gnani ← This package (LiveKit Agents adapter)\n ├── STT: REST + WebSocket\n └── TTS: REST + SSE + WebSocket\n```\n\nThis plugin directly implements the Gnani REST and WebSocket APIs using `aiohttp` (for REST STT/TTS) and `websockets` (for streaming STT/TTS), adapting them into LiveKit's `stt.STT` and `tts.TTS` base classes. It uses the **Prisma** model for speech-to-text and the **Timbre** model for text-to-speech. No external SDK is required — all connection logic, authentication, and audio format handling is self-contained. Authentication uses a single `api_key` passed via the `X-API-Key-ID` header.\n\n## Documentation\n\n- [Gnani API Docs](https://docs.gnani.ai/)\n- [LiveKit Agents Docs](https://docs.livekit.io/agents/)\n- [Gnani STT Plugin Guide](https://docs.livekit.io/agents/integrations/stt/gnani/)\n- [Gnani TTS Plugin Guide](https://docs.livekit.io/agents/integrations/tts/gnani/)\n- [STT REST API](https://docs.gnani.ai/api/STT/speech-to-text)\n- [STT Realtime WebSocket](https://docs.gnani.ai/api/STT/stt-websocket)\n- [TTS REST API](https://docs.gnani.ai/api/TTS/tts-inference)\n- [TTS Streaming (SSE)](https://docs.gnani.ai/api/TTS/tts-sse)\n- [TTS Realtime WebSocket](https://docs.gnani.ai/api/TTS/tts-websocket)\n\n## LiveKit Compatibility\n\nTested with **LiveKit Agents v1.6.x**.\n\n## License\n\nApache 2.0 — see [LICENSE](../../LICENSE).\n","livekit-plugins/livekit-plugins-google/README.md":"# Google AI plugin for LiveKit Agents\n\nSupport for Gemini, Gemini Live, Cloud Speech-to-Text, and Cloud Text-to-Speech.\n\nSee [https://docs.livekit.io/agents/integrations/google/](https://docs.livekit.io/agents/integrations/google/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-google\n```\n\n## Pre-requisites\n\nFor credentials, you'll need a Google Cloud account and obtain the correct credentials. Credentials can be passed directly or via Application Default Credentials as specified in [How Application Default Credentials works](https://cloud.google.com/docs/authentication/application-default-credentials).\n\nTo use the STT and TTS API, you'll need to enable the respective services for your Google Cloud project.\n\n- Cloud Speech-to-Text API\n- Cloud Text-to-Speech API\n\n## Live API model support\n\nLiveKit supports the Gemini Live API through both the Gemini Developer API and Vertex AI. Model availability and behavior differ between APIs. Some models, such as `gemini-3.8-live`, are available on both.\n\nFor model/API pairs that appear incompatible, the plugin logs a warning and lets the API decide whether the model is available.\n\nThe following models are supported by Gemini Developer API:\n\n- gemini-3.8-live\n- gemini-3.8-live-extended-thinking\n- gemini-3.1-flash-live-preview\n- gemini-2.5-flash-native-audio-preview-09-2025\n- gemini-2.5-flash-native-audio-preview-12-2025\n\nOn the Gemini Developer API, `gemini-3.8-live` does not support `thinking_config.thinking_level`. Use `gemini-3.8-live-extended-thinking` for configurable thinking.\n\nAnd these on Vertex AI:\n\n- gemini-3.8-live\n- gemini-live-2.5-flash-native-audio\n\nReferences:\n\n- [Gemini API Models](https://ai.google.dev/gemini-api/docs/models)\n- [Gemini Live API thinking](https://ai.google.dev/gemini-api/docs/live-api/thinking)\n- [Vertex Live API](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/live-api)\n","livekit-plugins/livekit-plugins-gradium/README.md":"# Gradium plugin for LiveKit Agents\n\nSupport for [Gradium](https://gradium.ai/)-hosted models in LiveKit Agents.\n\nMore information is available in the docs for the\n[TTS](https://docs.livekit.io/agents/integrations/tts/gradium/) and [STT](https://docs.livekit.io/agents/integrations/stt/gradium/) integrations.\n\n## Installation\n\n```bash\npip install livekit-plugins-gradium\n```\n\n## Pre-requisites\n\nYou'll need an API key from Gradium. It can be set as an environment variable: `GRADIUM_API_KEY`.\n\nYou also need to deploy a model to Gradium and will need your model endpoint to configure the plugin.\n","livekit-plugins/livekit-plugins-groq/README.md":"# Groq plugin for LiveKit Agents\n\nSupport for STT, TTS, and LLM with [Groq](https://www.groq.com/) fast inference.\n\nSee [https://docs.livekit.io/agents/integrations/groq/](https://docs.livekit.io/agents/integrations/groq/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-groq\n```\n\n## Pre-requisites\n\nFor credentials, you'll need a Groq Cloud account and obtain the correct credentials. Credentials can be passed directly or via GROQ_API_KEY environment variable\n","livekit-plugins/livekit-plugins-hamming/README.md":"# Hamming plugin for LiveKit Agents\n\nPost-call monitoring and session export for LiveKit Agents with Hamming.\n\n## Installation\n\nInstall the plugin from a checked out repository or extracted source archive:\n\n```bash\npython -m pip install ./livekit-plugins/livekit-plugins-hamming\n```\n\n## Pre-requisites\n\nYou will need Hamming credentials before configuring the plugin.\n\n- `HAMMING_API_KEY`\n- `HAMMING_EXTERNAL_AGENT_ID`\n\nCredentials can be passed directly or through environment variables.\n\n## Usage\n\n```python\nimport os\n\nfrom livekit.agents import AgentSession, JobContext\nfrom livekit.plugins import hamming\n\n\nasync def entrypoint(ctx: JobContext) -> None:\n await ctx.connect()\n\n hamming.configure_hamming(\n api_key=os.environ[\"HAMMING_API_KEY\"],\n external_agent_id=os.environ[\"HAMMING_EXTERNAL_AGENT_ID\"],\n recording={\"mode\": \"session_audio\"},\n )\n\n session = AgentSession()\n hamming.attach_session(\n session,\n job_ctx=ctx,\n customer_metadata={\n \"deployment\": {\n \"environment\": \"prod\",\n \"prompt_version\": \"v17\",\n },\n \"experiment\": {\n \"variant\": \"B\",\n },\n },\n external_links=[\n {\n \"label\": \"CRM Contact\",\n \"url\": \"https://crm.example.com/contact/123\",\n \"source\": \"salesforce\",\n }\n ],\n session_mode=\"testing\",\n )\n```\n\n## Unified Responsibilities\n\nThe Hamming plugin is intentionally focused on one responsibility:\n\n- post-call session export for Call Review and testing workflows\n\nUse `attach_session(...)` for per-call review/testing context:\n\n- `customer_metadata`\n- `external_links`\n- `session_mode`\n\n## Call Review And Testing Fields\n\n### `customer_metadata`\n\nUse `customer_metadata` for filterable customer-owned metadata:\n\n```python\nhamming.attach_session(\n session,\n job_ctx=ctx,\n customer_metadata={\n \"deployment\": {\"environment\": \"prod\"},\n \"experiment\": {\"variant\": \"B\"},\n },\n)\n```\n\n### `external_links`\n\nUse `external_links` for click-through operator links:\n\n```python\nhamming.attach_session(\n session,\n job_ctx=ctx,\n external_links=[\n {\n \"label\": \"CRM Contact\",\n \"url\": \"https://crm.example.com/contact/123\",\n \"source\": \"salesforce\",\n \"description\": \"Primary customer record\",\n }\n ],\n)\n```\n\n### `session_mode`\n\nUse `session_mode=\"testing\"` when the call is part of a testing workflow:\n\n```python\nhamming.attach_session(\n session,\n job_ctx=ctx,\n session_mode=\"testing\",\n)\n```\n\n## Recording Modes\n\nThe plugin supports explicit recording modes through `configure_hamming(recording=...)`.\nIf `recording` is omitted, the plugin exports monitoring payloads only and does not start or resolve recordings.\n\n- `none`\n - Send monitoring payload only.\n- `session_audio`\n - Use LiveKit session recording and inline `recording_capture` on the final `/collect` payload.\n- `participant_egress`\n - Start managed LiveKit participant egress and send dual-track recording URLs derived from deterministic output paths.\n- `room_composite`\n - Start managed LiveKit room composite egress and send a single `recording_url` derived from a deterministic output path.\n\n### Session Audio (Default Simple Path)\n\n```python\nhamming.configure_hamming(\n api_key=os.environ[\"HAMMING_API_KEY\"],\n external_agent_id=os.environ[\"HAMMING_EXTERNAL_AGENT_ID\"],\n recording={\"mode\": \"session_audio\"},\n)\n```\n\n### Managed Participant Egress\n\n```python\nhamming.configure_hamming(\n api_key=os.environ[\"HAMMING_API_KEY\"],\n external_agent_id=os.environ[\"HAMMING_EXTERNAL_AGENT_ID\"],\n recording={\n \"mode\": \"participant_egress\",\n \"livekit\": {\n \"url\": os.environ[\"LIVEKIT_URL\"],\n \"api_key\": os.environ[\"LIVEKIT_API_KEY\"],\n \"api_secret\": os.environ[\"LIVEKIT_API_SECRET\"],\n },\n \"s3\": {\n # Required for deterministic artifact URL resolution.\n \"public_url_base\": os.environ.get(\"LIVEKIT_RECORDING_PUBLIC_URL_BASE\", \"\"),\n # Optional. Include upload credentials only if the plugin should write\n # directly to S3 instead of relying on LiveKit project defaults.\n \"access_key\": os.environ.get(\"LIVEKIT_RECORDING_S3_ACCESS_KEY\", \"\"),\n \"secret\": os.environ.get(\"LIVEKIT_RECORDING_S3_SECRET\", \"\"),\n \"region\": os.environ.get(\"LIVEKIT_RECORDING_S3_REGION\", \"\"),\n \"bucket\": os.environ.get(\"LIVEKIT_RECORDING_S3_BUCKET\", \"\"),\n },\n },\n)\n```\n\n### Managed Room Composite\n\n```python\nhamming.configure_hamming(\n api_key=os.environ[\"HAMMING_API_KEY\"],\n external_agent_id=os.environ[\"HAMMING_EXTERNAL_AGENT_ID\"],\n recording={\n \"mode\": \"room_composite\",\n \"livekit\": {\n \"url\": os.environ[\"LIVEKIT_URL\"],\n \"api_key\": os.environ[\"LIVEKIT_API_KEY\"],\n \"api_secret\": os.environ[\"LIVEKIT_API_SECRET\"],\n },\n \"audio_only\": True,\n \"file_type\": \"ogg\",\n },\n)\n```\n\nNotes:\n\n- Managed remote recording modes require LiveKit server API credentials.\n- Managed remote recording modes also require `job_ctx` when calling `hamming.attach_session(...)`.\n- Managed remote recording modes require deterministic artifact URL resolution via `recording.s3.public_url_base` or `recording.s3.bucket` + `recording.s3.region`.\n- If you provide full S3 credentials, the plugin uses those upload settings for egress output.\n- If you omit S3 upload credentials, your LiveKit project must already have default file output storage configured.\n- The plugin does not poll LiveKit for completed artifact locations; it derives the final public URLs from the configured output path.\n\n## Verify configuration\n\n```python\nimport os\n\nfrom livekit.plugins import hamming\n\nreport = hamming.doctor(api_key=os.environ[\"HAMMING_API_KEY\"])\nprint(report.to_dict())\n```\n\n## Notes\n\n- Sessions are exported when attached through `hamming.attach_session(...)`.\n- Recording is opt-in. Omitting `recording=...` sends monitoring payloads without recording artifacts.\n- `auto_record_audio=True` is still supported as a backward-compatible alias for `recording={\"mode\": \"session_audio\"}`.\n- Managed remote recording modes stop egress on close and send deterministic artifact URLs through the same `/api/rest/v2/collect` ingestion path.\n\n## Troubleshooting\n\n- `RuntimeError: hamming is not configured`\n - Call `hamming.configure_hamming(...)` before `hamming.attach_session(...)`.\n- `ValueError: Hamming API key required`\n - Set `HAMMING_API_KEY` or pass `api_key=...`.\n- `external_agent_id is required`\n - Set `HAMMING_EXTERNAL_AGENT_ID` or pass `external_agent_id=...`.\n- `Unsupported recording mode`\n - Use `none`, `session_audio`, `participant_egress`, or `room_composite`.\n- `recording mode 'participant_egress' requires LiveKit server credentials`\n - Provide `recording.livekit` or set `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET`.\n- `recording mode 'participant_egress' requires deterministic artifact URL resolution`\n - Provide `recording.s3.public_url_base` or `recording.s3.bucket` + `recording.s3.region`.\n- `recording mode 'participant_egress' requires JobContext`\n - Pass `job_ctx=ctx` to `hamming.attach_session(...)` or call it from inside the active LiveKit job.\n","livekit-plugins/livekit-plugins-hume/README.md":"# Hume AI TTS plugin for LiveKit Agents\n\nSupport for text-to-speech with [Hume](https://www.hume.ai/).\n\nSee [https://docs.livekit.io/agents/integrations/tts/hume/](https://docs.livekit.io/agents/integrations/tts/hume/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-hume\n```\n\nYou will need an API Key from Hume, it can be set as an environment variable: `HUME_API_KEY`. You can get it from [here](https://platform.hume.ai/settings/keys)\n","livekit-plugins/livekit-plugins-inworld/README.md":"# Inworld plugin for LiveKit Agents\n\nSupport for voice synthesis and speech-to-text with [Inworld TTS](https://docs.inworld.ai/tts/tts) and [Inworld STT](https://docs.inworld.ai/stt/overview).\n\nSee [Inworld TTS](https://docs.livekit.io/agents/integrations/tts/inworld/) and [Inworld STT](https://docs.livekit.io/agents/models/stt/inworld/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-inworld\n```\n\n## Authentication\n\nSet `INWORLD_API_KEY` in your `.env` file ([get one here](https://platform.inworld.ai/login)).\n\n## Usage\n\n### TTS\n\nUse Inworld TTS within an `AgentSession` or as a standalone speech generator.\n\n```python\nfrom livekit.plugins import inworld\n\ntts = inworld.TTS()\n```\n\nOr with options:\n\n```python\nfrom livekit.plugins import inworld\n\ntts = inworld.TTS(\n voice=\"Hades\", # voice ID (default or custom cloned voice)\n model=\"inworld-tts-1\", # or \"inworld-tts-1-max\"\n encoding=\"OGG_OPUS\", # LINEAR16, MP3, OGG_OPUS, ALAW, MULAW, FLAC\n sample_rate=48000, # 8000-48000 Hz\n bit_rate=64000, # bits per second (for compressed formats)\n speaking_rate=1.0, # 0.5-1.5\n temperature=1.1, # 0-2\n timestamp_type=\"WORD\", # WORD, CHARACTER, or TIMESTAMP_TYPE_UNSPECIFIED\n text_normalization=\"OFF\", # ON, OFF, or APPLY_TEXT_NORMALIZATION_UNSPECIFIED\n)\n```\n\n### TTS Streaming\n\nInworld TTS supports WebSocket streaming for lower latency real-time synthesis. Use the\n`stream()` method for streaming text as it's generated:\n\n```python\nfrom livekit.plugins import inworld\n\ntts = inworld.TTS(\n voice=\"Hades\",\n model=\"inworld-tts-1\",\n buffer_char_threshold=100, # chars before triggering synthesis (default: 250)\n max_buffer_delay_ms=3000, # max buffer time in ms (default: 3000)\n)\n\n# Create a stream for real-time synthesis\nstream = tts.stream()\n\n# Push text incrementally\nstream.push_text(\"Hello, \")\nstream.push_text(\"how are you today?\")\nstream.flush() # Flush any remaining buffered text\nstream.end_input() # Signal end of input\n\n# Consume audio as it's generated\nasync for audio in stream:\n # Process audio frames\n pass\n```\n\n### STT\n\nUse Inworld STT for streaming speech-to-text. Multiple models are supported.\n\n```python\nfrom livekit.plugins import inworld\n\nsession = AgentSession(\n stt=inworld.STT()\n # ... llm, tts, etc.\n)\n```\n\nWith a specific model and voice profile detection:\n\n```python\nfrom livekit.plugins import inworld\n\nsession = AgentSession(\n stt=inworld.STT(\n model=\"inworld/inworld-stt-1\",\n enable_voice_profile=True,\n )\n # ... llm, tts, etc.\n)\n```\n\n### Example\n\nA full voice agent using Inworld for both STT and TTS:\n\n```python\n\"\"\"Inworld STT + TTS voice agent example.\n\nDemonstrates using Inworld for both speech-to-text and text-to-speech\nin a LiveKit voice agent. Save this as ``inworld_agent.py`` and run:\n\n uv run inworld_agent.py console # local console mode\n uv run inworld_agent.py dev # LiveKit Cloud (requires LIVEKIT_URL,\n # LIVEKIT_API_KEY, LIVEKIT_API_SECRET)\n\nThen connect via https://agents-playground.livekit.io\n\"\"\"\n\nimport logging\n\nfrom dotenv import load_dotenv\n\nfrom livekit.agents import (\n Agent,\n AgentServer,\n AgentSession,\n JobContext,\n cli,\n inference,\n metrics,\n room_io,\n)\nfrom livekit.agents.inference import TurnDetector\nfrom livekit.plugins import inworld\n\nlogger = logging.getLogger(\"inworld-agent\")\n\nload_dotenv()\n\n\nclass InworldAgent(Agent):\n def __init__(self) -> None:\n super().__init__(\n instructions=(\n \"Your name is Nova. You interact with users via voice. \"\n \"Keep your responses concise and to the point. \"\n \"Do not use emojis, asterisks, markdown, or other special characters. \"\n \"You are helpful, curious, and friendly.\"\n ),\n )\n\n async def on_enter(self):\n self.session.generate_reply()\n\n\nserver = AgentServer()\n\n\n@server.rtc_session()\nasync def entrypoint(ctx: JobContext):\n ctx.log_context_fields = {\"room\": ctx.room.name}\n\n session = AgentSession(\n stt=inworld.STT(model=\"inworld/inworld-stt-1\"),\n llm=\"openai/gpt-4.1-mini\",\n tts=inworld.TTS(voice=\"Clive\"),\n turn_detection=TurnDetector(),\n vad=inference.VAD(),\n )\n\n usage_collector = metrics.UsageCollector()\n\n @session.on(\"metrics_collected\")\n def _on_metrics(ev):\n metrics.log_metrics(ev.metrics)\n usage_collector.collect(ev.metrics)\n\n async def log_usage():\n logger.info(f\"Usage: {usage_collector.get_summary()}\")\n\n ctx.add_shutdown_callback(log_usage)\n\n await session.start(\n agent=InworldAgent(),\n room=ctx.room,\n room_options=room_io.RoomOptions(),\n )\n\n\nif __name__ == \"__main__\":\n cli.run_app(server)\n```\n\n### Combined TTS + STT\n\n```python\nfrom livekit.plugins import inworld\n\nsession = AgentSession(\n tts=inworld.TTS(voice=\"Hades\"),\n stt=inworld.STT(),\n # ... llm, etc.\n)\n```\n","livekit-plugins/livekit-plugins-keyframe/README.md":"# livekit-plugins-keyframe\n\nAgent Framework plugin for [Keyframe Labs](https://keyframelabs.com) avatars.\n\n## Installation\n\n```bash\npip install livekit-plugins-keyframe\n```\n\n## Usage\n\n```python\nfrom livekit.agents import AgentSession\nfrom livekit.plugins import keyframe\n\nsession = AgentSession(stt=..., llm=..., tts=...)\n\navatar = keyframe.AvatarSession(\n persona_slug=\"public:cosmo-2.0-live-preview\", # or persona_id=\"...\"\n api_key=\"keyframe_sk_live_...\", # or set KEYFRAME_API_KEY env var\n)\n\nawait avatar.start(session, room=ctx.room)\nawait session.start(room=ctx.room, agent=my_agent)\n```\n\n## Authentication\n\nSet the following environment variables:\n\n- `KEYFRAME_API_KEY` - Your Keyframe API key\n- `LIVEKIT_URL` - LiveKit server URL\n- `LIVEKIT_API_KEY` - LiveKit API key\n- `LIVEKIT_API_SECRET` - LiveKit API secret\n\n## Emotions\n\n`persona-2.0-live` avatars support eleven expressions (`keyframe.EMOTIONS`):\n`neutral`, `happy`, `sad`, `angry`, `surprised`, `concerned`, `confused`, `disgusted`, `excited`, `scared`, `skeptical`.\nAn expression holds until the next change.\n\n### Sync expressions to speech\n\nOverride `tts_node` with `keyframe.emotion_tts_node` on every agent in the session, and add `keyframe.emotion_instructions()` to the prompt.\nThe LLM writes emotion markers into its reply, and the avatar changes expression on the marked word.\n\n```python\nclass MyAgent(Agent):\n def tts_node(self, text, model_settings):\n return keyframe.emotion_tts_node(self, text, model_settings)\n```\n\nUse the configuration that matches your TTS:\n\n- LiveKit Inference ([expressive mode](https://docs.livekit.io/agents/models/tts/expressive/)), where one `` marker steers both the voice and the face:\n\n ```python\n session = AgentSession(\n tts=inference.TTS(\"cartesia/sonic-3.6\", extra_kwargs={\"add_timestamps\": True}),\n expressive={\"tts_instructions_append\": keyframe.emotion_instructions()},\n ...\n )\n ```\n\n- ElevenLabs v3/v4 plugin, where one `[happy]` audio tag steers both the voice and the face. Pass `style=\"brackets\"` to both functions, and set `use_tts_aligned_transcript=True` to keep tags out of transcripts:\n\n ```python\n class MyAgent(Agent):\n def tts_node(self, text, model_settings):\n return keyframe.emotion_tts_node(self, text, model_settings, style=\"brackets\")\n\n agent = MyAgent(\n instructions=f\"{MY_INSTRUCTIONS}\\n\\n{keyframe.emotion_instructions(style='brackets')}\",\n )\n session = AgentSession(\n tts=elevenlabs.TTS(model=\"eleven_v3\"),\n use_tts_aligned_transcript=True,\n ...\n )\n ```\n\n- Any other TTS, where `` markers steer the face and are stripped before synthesis:\n\n ```python\n agent = MyAgent(instructions=f\"{MY_INSTRUCTIONS}\\n\\n{keyframe.emotion_instructions()}\")\n ```\n\n### Set an expression directly\n\nUse `set_emotion` for tool calls, vision events, or `persona-1.5-live` avatars (which support `neutral`, `happy`, `sad`, and `angry`):\n\n```python\nawait avatar.set_emotion(\"happy\")\n```\n\nThe avatar switches on the next video frame that it renders.\nDuring a reply, the avatar renders ahead of playback.\nWhen a change must land on a specific word, use markers instead.\n\n
\nTTS support and timing details\n\nTiming accuracy depends on what the TTS reports:\n\n| TTS | When the expression changes |\n|-----|-----------------------------|\n| Streaming TTS with word timings: Cartesia (`add_timestamps=True`, plugin default), Inworld (`timestamp_type=\"WORD\"`), ElevenLabs v3/v4 (plugin), Rime (WebSocket), Speechify, Smallest.ai, Baseten (`word_timestamps=True`) | On the marked word |\n| Non-streaming TTS (Gemini, OpenAI, Hume) | At the start of the sentence |\n| Streaming TTS without word timings (Deepgram, xAI, or ElevenLabs, Fish Audio, and Gemini through Inference) | Before the first word only (later markers are ignored) |\n\nProvider notes:\n\n- Inworld expressive mode accepts free-form labels: `\"warm, concerned\"` steers both the voice and the face, while `\"sympathetic\"` steers the voice only.\n- Cartesia expressive mode covers nine of the eleven expressions (`concerned` and `skeptical` change the face only).\n- SpaceXAI expressive mode has no expression labels, so markers change the face and are stripped before synthesis.\n- In `style=\"brackets\"`, only exact expression names (`[happy]`) change the face. Other audio tags (`[whispers]`, `[warmly]`) steer the voice only. In the default `\"expr\"` style, square brackets are plain text.\n\n
\n\n
\nCustom emotion sets and custom TTS nodes\n\nPass `emotions=` to `emotion_tts_node` and `emotion_instructions` to limit which expressions a marker can select.\n\nA custom `tts_node` can attach `keyframe.EmotionCue` items directly to audio frames.\n`offset` is the time in seconds from the first sample of the frame and can be negative.\n\n```python\nframe.userdata[keyframe.USERDATA_EMOTION_CUES] = [\n keyframe.EmotionCue(\"happy\", offset=0.25),\n]\n```\n\n
\n","livekit-plugins/livekit-plugins-krisp/README.md":"# Krisp VIVA Plugin for LiveKit Agents\n\nReal-time noise reduction for LiveKit voice agents using [Krisp's VIVA SDK](https://krisp.ai).\n\n## Features\n\n- **`voice_isolation()`**: Real-time voice isolation and noise reduction `FrameProcessor`\n- **`voice_isolation_telephony()`**: Voice isolation tuned for telephony audio (for example, SIP participants)\n\n## Installation\n\n```bash\npip install livekit-plugins-krisp\n```\n\nThat's it. The default backend is bundled with the plugin and authenticates through\nLiveKit Cloud using the room's credentials — no separate SDK download, license key,\nor model file is required.\n\n> Using your own [Krisp license](#alternative-krisp-license-auth) instead?\n> That path has additional prerequisites — see below.\n\n## Quick Start\n\nBy default, `krisp.voice_isolation()` uses **LiveKit Cloud** authentication: the\nbundled backend ships the voice isolation model and authenticates against LiveKit Cloud\nusing the room JWT the agent framework hands to the `FrameProcessor` automatically.\n\n```python\nfrom livekit.agents import AgentSession, Agent, JobContext, inference, room_io\nfrom livekit.plugins import krisp, openai\n\n@server.rtc_session()\nasync def entrypoint(ctx: JobContext):\n # Default: LiveKit Cloud auth + bundled model. No keys or model files.\n noise_cancellation = krisp.voice_isolation(\n noise_suppression_level=75, # 0-100\n )\n\n session = AgentSession(\n vad=inference.VAD(),\n stt=openai.STT(),\n llm=openai.LLM(model=\"gpt-4o-mini\"),\n tts=openai.TTS(),\n )\n\n # Start session with RoomIO and pass the FrameProcessor directly\n await session.start(\n agent=MyAgent(),\n room=ctx.room,\n room_options=room_io.RoomOptions(\n audio_input=room_io.AudioInputOptions(\n noise_cancellation=noise_cancellation, # Pass the FrameProcessor directly\n ),\n ),\n )\n```\n\nFor telephony audio, use `krisp.voice_isolation_telephony()` instead — it takes the\nsame options and behaves identically, but selects a voice isolation model tuned for\ntelephony.\n\n**Audio Pipeline:** `Room → RoomIO (with voice_isolation) → VAD → STT → LLM`\n\n## Configuration\n\n### `voice_isolation()` / `voice_isolation_telephony()` options\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `auth_provider` | `LiveKitCloudAuthProvider \\| KrispLicenseAuthProvider` | `LiveKitCloudAuthProvider` | Authentication backend. Defaults to LiveKit Cloud. See [the alternative](#alternative-krisp-license-auth). |\n| `noise_suppression_level` | int | 75 | Noise reduction intensity (0-100) |\n\nInput frames of any size and sample rate are buffered and adapted automatically.\n\n### Runtime control\n\nBoth factory functions return a `KrispVivaFilterFrameProcessor`. Adjust it while the\nsession is running:\n\n```python\nnoise_cancellation.enabled = False # pass audio through unmodified\nnoise_cancellation.noise_suppression_level = 50 # adjust 0-100 on the fly\nnoise_cancellation.close() # free resources when done\n```\n\n\n## Alternative: Krisp License Auth\n\n> **Most users should use the default LiveKit Cloud path above.** This alternative is for\n> running the public Krisp SDK directly with your own Krisp license — for example, when\n> using Livekit OSS server.\n\nThis path uses the public `krisp_audio` wheel together with a Krisp license key and a\n`.kef` model file that you obtain from Krisp.\n\n### Prerequisites\n\n1. **Krisp Audio SDK** — proprietary, not on public PyPI. Obtain and install it\n separately from [Krisp](https://krisp.ai/developers/):\n ```bash\n pip install krisp-audio\n ```\n2. **License key**:\n ```bash\n export KRISP_VIVA_SDK_LICENSE_KEY=your-license-key-here\n ```\n3. **Noise-reduction model** — a `.kef` model file from Krisp:\n ```bash\n export KRISP_VIVA_FILTER_MODEL_PATH=/path/to/noise_model.kef\n ```\n\n### Usage\n\nSelect the license backend by passing `auth_provider`:\n\n```python\nfrom livekit.plugins import krisp\n\nnoise_cancellation = krisp.voice_isolation(\n auth_provider=krisp.auth.krisp_license(\n license_key=\"...\", # or KRISP_VIVA_SDK_LICENSE_KEY\n model_path=\"/path/to/noise_model.kef\", # or KRISP_VIVA_FILTER_MODEL_PATH\n ),\n noise_suppression_level=75,\n)\n```\n\n`license_key` and `model_path` fall back to the environment variables above when omitted.\n\n## Troubleshooting\n\n### `RuntimeError`: bundled backend missing\nIf the default (LiveKit Cloud) backend reports a missing wheel, the install is likely\nbroken. Reinstall the plugin:\n```bash\npip install --force-reinstall livekit-plugins-krisp\n```\nAlternatively, fall back to the [Krisp license auth](#alternative-krisp-license-auth) path.\n\n### \"Krisp SDK initialization failed\" or licensing errors *(license auth only)*\nMake sure the license key is set:\n```bash\nexport KRISP_VIVA_SDK_LICENSE_KEY=your-license-key-here\n```\n\n### \"Model path must be provided\" *(license auth only)*\n```bash\nexport KRISP_VIVA_FILTER_MODEL_PATH=/path/to/model.kef\n```\n\n### \"Unsupported sample rate\"\nSupported: 8000, 16000, 24000, 32000, 44100, 48000 Hz\n\n### Silent output\n- Verify the model file is valid *(license auth only)*\n- Test with known noisy audio\n\n## License\n\nThe source code in this package (`livekit-plugins-krisp`) is licensed under the\n**Apache-2.0** license.\n\nThe **default backend** is a separate, closed-source wheel (`livekit-plugins-krisp-internal`)\nthat is installed automatically as a dependency. It is **proprietary** and distributed under\nthe [LiveKit Terms of Service](https://livekit.io/legal/terms-of-service). That wheel bundles\nthe Krisp VIVA SDK along with its third-party open-source components, whose attribution\nnotices are shipped inside the wheel.\n\nThe **Krisp license alternative** (`KrispLicenseAuthProvider`) instead needs a manual install of the proprietary Krisp\nAudio SDK together with your own Krisp license key and model file, governed by your agreement\nwith [Krisp](https://krisp.ai).\n","livekit-plugins/livekit-plugins-langchain/README.md":"# LangChain plugin for LiveKit Agents\n\nThis plugin integrates capabilites from LangChain within LiveKit Agents\n\n## Installation\n\n```bash\npip install livekit-plugins-langchain\n```\n\n## Usage\n\n### Using LangGraph workflows\n\nYou can bring over any existing workflow in LangGraph as an Agents LLM with `langchain.LLMAdapter`. For example:\n\n```python\nfrom langgraph.graph import StateGraph\nfrom livekit.agents import Agent, AgentSession, JobContext\nfrom livekit.plugins import langchain\n\n...\n\ndef entrypoint(ctx: JobContext):\n graph = StateGraph(...).compile()\n\n session = AgentSession(\n vad=...,\n stt=...,\n tts=...,\n )\n\n await session.start(\n agent=Agent(llm=langchain.LLMAdapter(graph)),\n )\n ...\n```\n","livekit-plugins/livekit-plugins-lemonslice/README.md":"# LemonSlice plugin for LiveKit Agents\n\nSupport for the [LemonSlice](https://www.lemonslice.com/) virtual avatar.\n\nSee the [LemonSlice integration docs](https://docs.livekit.io/agents/models/avatar/plugins/lemonslice/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-lemonslice\n```\n\n## Pre-requisites\n\nYou'll need an API key from LemonSlice. It can be set as an environment variable: `LEMONSLICE_API_KEY`\n","livekit-plugins/livekit-plugins-liveavatar/README.md":"# LiveAvatar plugin for LiveKit Agents\n\nSupport for the [HeyGen LiveAvatar](https://www.liveavatar.com/) virtual avatar.\n\nSee the [LiveAvatar integration docs](https://docs.livekit.io/agents/models/avatar/plugins/liveavatar/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-liveavatar\n```\n\n## Pre-requisites\n\nYou'll need an API key from LiveAvatar. It can be set as an environment variable: `LIVEAVATAR_API_KEY`\n","livekit-plugins/livekit-plugins-lmnt/README.md":"# LMNT plugin for LiveKit Agents\n\nSupport for voice synthesis with [LMNT](https://lmnt.com/) in LiveKit Agents.\n\nSee the [LMNT TTS docs](https://docs.livekit.io/agents/integrations/tts/lmnt/) for more information.\n\n## Installation\n\n```bash\npip install livekit-plugins-lmnt\n```\n\n## Pre-requisites\n\nYou'll need an API key from LMNT. It can be set as an environment variable: `LMNT_API_KEY`. You can get it from [here](https://app.lmnt.com/account#api-keys)\n","livekit-plugins/livekit-plugins-meta/README.md":"# Meta plugin for LiveKit Agents\n\nSupport for Meta [Muse Voice Transcribe](https://dev.meta.ai/docs/speech-to-text) using its [realtime streaming speech-to-text interface](https://dev.meta.ai/docs/api-reference/voice/realtime).\n\n## Installation\n\n```bash\npip install livekit-plugins-meta\n```\n\n## Pre-requisites\n\nSet a Meta Model API key in your environment using either variable:\n\n```bash\nMODEL_API_KEY=\n# Or: META_API_KEY=\n```\n\nYou can also pass the key directly with `api_key=`. The resolution order is `api_key`, `MODEL_API_KEY`, then `META_API_KEY`.\n\n## Usage\n\n```python\nfrom livekit.agents import AgentSession\nfrom livekit.plugins import meta\n\nsession = AgentSession(\n stt=meta.STT(\n keywords=[\"LiveKit\", \"Muse\"],\n language_bias=[\"English\"],\n ),\n # ... llm, tts, etc.\n)\n```\n\nThe plugin supports streaming recognition with server-side endpointing, cumulative interim transcripts, and mono PCM16 audio at 24 kHz. Connected streams must keep sending real-time PCM, including silence. `keywords` and `language_bias` are static hints applied during the initial handshake and cannot be changed on an active stream. Omit `language_bias` for automatic language detection.\n\nSupported language names are: Arabic, Bengali, Dutch, English, French, German, Hebrew, Hindi, Indonesian, Italian, Japanese, Kannada, Korean, Malay, Mandarin Chinese, Marathi, Polish, Portuguese, Spanish, Tagalog, Tamil, Telugu, Thai, Turkish, and Vietnamese. The per-stream `language=` argument also accepts corresponding language codes and locales, such as `en-US`, `pt-BR`, and `zh-CN`, and maps them to the documented names.\n\nMuse realtime sessions have a maximum duration of 60 minutes. This plugin reports the provider close and does not rotate an active LiveKit speech stream automatically; start a new stream to continue. Batch recognition, diarization, detected-language metadata, and active-stream keyterm updates are not supported.\n","livekit-plugins/livekit-plugins-microsoft-ai/README.md":"# Microsoft AI plugin for LiveKit Agents\n\n## Service availability\n\n**TTS** uses Microsoft's public, documented\n[MAI-Voice](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/mai-voices)\npath: SSML posted to the Azure Speech REST synthesis endpoint with a Speech\nresource key. It has been verified with `en-US-Harper:MAI-Voice-2-Flash` and\n`en-US-Harper:MAI-Voice-2.1-Flash` (PCM16 mono at 24 kHz) on an `eastus` Speech\nresource, including playback through a local LiveKit room. MAI voices are in\npublic preview.\n\n**STT** has no publicly documented Microsoft endpoint. The plugin speaks a\nrealtime streaming transcription websocket protocol (described below), but\nMicrosoft documents\n[MAI-Transcribe](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/mai-transcribe)\nonly through:\n\n- the Speech **fast transcription** REST API\n (`speechtotext/transcriptions:transcribe` with `enhancedMode.model`), which\n is file-based, not streaming, and is **not** used by this plugin; and\n- **Voice Live**, as input-audio transcription inside a Voice Live session.\n\nNeither is a streaming endpoint that `microsoft_ai.STT` can connect to. Use\n`microsoft_ai.STT` only with a MAI realtime transcription endpoint that\nMicrosoft has provided to you directly, and confirm its contract against the\nprotocol below. **Do not point it at an Azure OpenAI endpoint**\n(`*.openai.azure.com` or `/openai/v1/realtime`): MAI-Transcribe is not\ndeployable there, so transcription fails with `DeploymentNotFound`.\n\nSupported MAI-Transcribe models are `MAI-Transcribe-2` and `MAI-Transcribe-1.5`;\n`MAI-Transcribe-1` was deprecated on August 20, 2026.\n\n**Decisions** use\n[Microsoft-Decision-1](https://ai.azure.com/catalog/models/Microsoft-Decision-1)\ndeployed in Microsoft Foundry, through the resource's System One endpoint\n(`POST /providers/microsoft/v1/systemone`). See\n[Decision model contract](#decision-model-contract).\n\nNone of this establishes access in every resource or region, recognition\naccuracy across inputs/languages, every backend tail boundary, long-session\nreliability, physical microphone behavior, or subjective voice quality.\nHermetic tests cover the client lifecycle and VAD ordering. None of these\nresults is a model-latency benchmark. An Azure Speech TTS resource/key does\n**not** establish access to any STT endpoint.\n\nThe package requires `livekit-agents>=1.8.3`.\n\nThere is no LLM, speech-to-speech realtime model, Azure OpenAI convenience\nconstructor, provider catalog, token minting, or OpenAI credential/model default.\n\nThe implementation adapts the Apache-2.0-licensed LiveKit OpenAI plugin's package\nand STT/TTS interfaces, and the Azure plugin's REST transport pattern. It does\nnot depend on either plugin. Transport, transcription finalization, and the\nescaped SSML/WAV mapping are specific to this integration. See `LICENSE` and `NOTICE`.\n\n## Local installation\n\nFrom this repository, install the workspace package rather than assuming a\npublished distribution exists:\n\n```sh\nuv sync --package livekit-plugins-microsoft-ai --no-default-groups\n```\n\n## Configuration\n\nPass constructor arguments or set the following environment variables. There\nare deliberately no default endpoints, models, voice IDs, or TTS sample rate.\nObtain these values and the exact contract from your deployment owner.\n\n| Environment variable | Constructor argument |\n| --- | --- |\n| `MICROSOFT_AI_STT_URL` | `STT(url=...)` |\n| `MICROSOFT_AI_STT_API_KEY` | `STT(api_key=...)` |\n| `MICROSOFT_AI_STT_AUTH_HEADER` | `STT(auth_header=...)`: `Authorization` or `api-key` |\n| `MICROSOFT_AI_STT_MODEL` | `STT(model=...)` |\n| `MICROSOFT_AI_STT_LANGUAGE` | `STT(language=...)` (optional) |\n| `MICROSOFT_AI_TTS_URL` | `TTS(url=...)` |\n| `MICROSOFT_AI_TTS_REGION` | `TTS(region=...)` (when no URL is configured) |\n| `MICROSOFT_AI_TTS_API_KEY` | `TTS(api_key=...)` |\n| `MICROSOFT_AI_TTS_MODEL` | `TTS(model=...)` (optional with a full voice ID) |\n| `MICROSOFT_AI_TTS_VOICE` | `TTS(voice=...)` |\n| `MICROSOFT_AI_TTS_SAMPLE_RATE` | `TTS(sample_rate=...)` |\n| `MICROSOFT_AI_DECISION_URL` | `DecisionModel(url=...)` |\n| `MICROSOFT_AI_DECISION_API_KEY` | `DecisionModel(api_key=...)` |\n| `MICROSOFT_AI_DECISION_AUTH_HEADER` | `DecisionModel(auth_header=...)`: `api-key` or `Authorization` |\n| `MICROSOFT_AI_DECISION_MODEL` | `DecisionModel(model=...)` (Foundry deployment name) |\n| `MICROSOFT_AI_ENV_FILE` | `STT(env_file=...)`, `TTS(env_file=...)`, `DecisionModel(env_file=...)` |\n\nURLs are complete endpoints, including any required path and query string. For\nexample, `wss://stt.example.invalid/v1/realtime?intent=transcription` is a **dummy**,\nnot a Microsoft service address. No path or model query parameter is appended.\nTLS is required except for loopback development endpoints. For TTS, a configured\nfull URL wins over `region`, regardless of which configuration source provides\neach. Only when the URL is unset does an explicitly supplied region select\nthe standard public-cloud endpoint. An explicit or configured empty/whitespace\nURL is an error, not permission to fall back to a region. Other required empty\nvalues also fail rather than falling back silently.\n\nTTS sends the Azure Speech resource key as `Ocp-Apim-Subscription-Key`, **not**\nas a raw-key Bearer token. STT preserves its `Authorization: Bearer ...` default.\nFor an STT endpoint that expects a raw resource key, explicitly set\n`MICROSOFT_AI_STT_AUTH_HEADER=api-key` (or `auth_header=\"api-key\"`); it sends the\nraw credential from `MICROSOFT_AI_STT_API_KEY` as the `api-key` header, with no\nBearer prefix. The selector accepts only the exact values `Authorization` and\n`api-key`; an empty/unknown value is an error. No auth fallback or automatic\nscheme detection occurs, and credentials are never added to the URL.\n\nExplicit `headers` (including `{}`) override the STT selector and credential\nenvironment settings and cannot be combined with `api_key` or `auth_header`\nconstructor arguments. Use them only for a confirmed alternate authentication\nscheme. No credentials are read from OpenAI/Azure variables. Caller-supplied\n`http_session` objects are borrowed; otherwise STT and TTS own lazy sessions and\nclose them in `aclose()`, and `DecisionModel` uses the job's HTTP session (outside\na job, wrap calls in `async with utils.http_context.open()`).\n\nKeep actual connection information in your environment or a user-selected\nlocal dotenv file **outside the checkout** (or a deliberately ignored file).\nThe providers load a file only when `env_file` or `MICROSOFT_AI_ENV_FILE` selects\none; there is no automatic `.env` discovery. Files are parsed with python-dotenv,\nwithout shell sourcing, variable interpolation, environment mutation or value\nlogging. Precedence is constructor argument, then process environment, then the\nselected file. An empty required value fails instead of falling back silently.\nUse owner-only file permissions and never copy this file into commits.\n\n## STT contract and lifecycle\n\nSee [Service availability](#service-availability) first: there is no publicly\ndocumented MAI streaming STT endpoint, and Azure OpenAI endpoints do not serve\nMAI-Transcribe. Configure the full URL of the endpoint Microsoft provided in\n`MICROSOFT_AI_STT_URL` and its model identifier in `MICROSOFT_AI_STT_MODEL`.\nThe model is sent in `session.audio.input.transcription.model`, not added as a\nURL query parameter. The plugin sends the URL unchanged; it does not add a\n`model=`, `deployment` or `api-version` query. Do not put a key in the URL.\nValidate the endpoint's audio-rate and transcript-event compatibility against\nthe contract below before relying on it.\n\n```python\nfrom livekit.agents import inference\nfrom livekit.plugins import microsoft_ai\n\ndetector = inference.VAD(model=\"silero\")\nspeech_to_text = microsoft_ai.STT(vad=detector, language=\"en\")\ntext_to_speech = microsoft_ai.TTS()\n```\n\n`vad` is explicit. Pass a LiveKit VAD with ordered `INFERENCE_DONE` events even\nduring silence, input-relative timestamps, and `START_OF_SPEECH.frames`\ncontaining the detected onset and prefix through that timestamp (the bundled\nSilero VAD provides these). Empty\nor incompatible start frames fail explicitly rather than clipping the onset.\nAlternatively, pass `vad=None` and call `flush()` / `end_input()` yourself.\n**Configuring only AgentSession's VAD is insufficient:** it does not commit\nnative STT streams.\n\nThe client protocol is:\n\n1. Await `session.created`, send `session.update`, await `session.updated`.\n2. Configure a transcription session with `audio.input.format` equal to\n `{\"type\": \"audio/pcm\", \"rate\": 16000}`, a required transcription `model`,\n optional `language`, and `turn_detection: null`, `noise_reduction: null`.\n3. Send base64 PCM16 little-endian mono audio as `input_audio_buffer.append`.\n4. For an item identified by `item_id`, an\n `conversation.item.input_audio_transcription.intermediate` event's\n `intermediate` replaces the revisable hypothesis. A `.delta` event's `delta`\n appends finalized text and clears the hypothesis. Both produce LiveKit\n interim results, not final utterances.\n5. Drain audio, send `input_audio_buffer.commit`, await\n `input_audio_buffer.committed` with `item_id`, then `.completed` with the\n authoritative `transcript`. Only this emits a LiveKit final transcript,\n followed by end-of-speech. The completion may correct or retract earlier\n hypotheses; an empty completion finishes the item without inventing user\n text. The socket stays open for subsequent utterances.\n\nNew deployments must be verified to implement these event fields and\nhandshake/commit ordering. A short manual-commit smoke does not exercise every\ninterim revision or VAD boundary. HTTP statuses are preserved; WebSocket\n`error` / transcription `.failed` events are terminal unless a pre-audio\ntransient status is supplied. The initial error mapping recognizes\n`error.status_code`, `invalid_api_key`, `rate_limit_exceeded`, `content_filter`\nand `safety_violation`; it never disables or works around safety checks.\n\nVAD inference timestamps serialize audio and turn boundaries, so later audio\ncannot overtake an earlier commit even if VAD processing or uploads are slow.\nMono input at other sample rates is resampled by the SDK. Stereo is rejected.\nTransport frames are 50 ms; the final shorter frame and resampler/VAD remainder\nare sent without rounding away samples or adding synthetic padding.\n\nWith VAD, idle inference windows are discarded locally, not uploaded to an\nuncommitted server buffer. At speech start, the VAD's actual frames restore\nthe complete detected onset/prefix; no guessed pre-roll duration or private\nVAD settings are used. Only overlap with a previously committed turn is removed.\nThe prefix is framed and flushed before subsequent audio so it is not counted\ntwice for backpressure. Speech and the VAD's observed end-of-speech silence are\nuploaded in order, then committed; prolonged inter-turn silence sends neither\naudio nor empty commits. No provider clear/keepalive events are invented.\n\n**Tail limitation:** sending every byte and receiving `.completed` proves\ntransport completion, not that the backend decoded an incomplete model chunk.\nThere is no invented padding rule. The acknowledged completion is authoritative,\neven when it removes an interim hypothesis. Comparing final and revisable text\ncannot distinguish legitimate revision from recognition or audio-tail loss;\ndiscarded hypotheses are never appended to the final. Verify the full expected\ntranscript, particularly the last word, for both a short clip and a\nnon-chunk-aligned tail. Obtain a documented backend drain/flush mechanism if\ncommit does not decode the tail.\n\nAfter VAD detects speech, `flush()` drains its real audio tail and commits,\nwaiting internally before processing subsequent input; it leaves the socket\nopen. `end_input()` also waits for the acknowledged final and closes the socket.\nFlushing or ending idle VAD input produces no empty turn. To send a finite clip\nregardless of whether a VAD detects speech, use `vad=None`; that manual mode\ncontinues forwarding all input audio and requires caller-managed commits.\n`aclose()` cancels immediately without committing or exposing buffered events.\nBatch `recognize()` is unsupported and `offline_recognize=False`.\n\n`APIConnectOptions.timeout` bounds connection, handshake, writes and\nfinalization. `max_retry` is a finite connection-only retry budget: after any\naudio is consumed, a disconnect/error is surfaced without replay or hidden\nreconnection. Reopening the stream is the caller's decision. Input is bounded\nby `max_buffered_audio` (default 5 seconds) and 1,024 queued entries; overflow\nfails explicitly. Each VAD start prefix is separately capped by the same\nduration and fails rather than being truncated if oversized. The adapter keeps\nno additional idle history; the VAD owns its bounded onset/prefix buffer.\nIdle samples count as processed, so ordinary silence does not consume the\nqueued-audio allowance indefinitely. These bounds cover client lag/prefix\nretention, not the length of an active utterance at the provider. Pace\nprerecorded input rather than enqueueing entire files. Idle gating and delayed\nonset recovery are covered by hermetic tests, not an additional live-service\naccuracy or tail guarantee.\n\n## Azure Speech TTS contract\n\nThis implementation follows Microsoft's public\n[MAI voice documentation](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/mai-voices)\nand [Speech REST reference](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/rest-text-to-speech).\nIt posts SSML to the **exact configured synthesis URL**, with\n`Content-Type: application/ssml+xml`, `Ocp-Apim-Subscription-Key`,\n`X-Microsoft-OutputFormat`, `User-Agent`, and `Accept: audio/wav`.\n\nSupply the full synthesis endpoint, including its documented\n`cognitiveservices/v1` path and any resource-specific routing. A generic Azure\nresource URL or SDK endpoint may not be a usable REST synthesis URL. The plugin\ndoes not append a path, infer a region, rewrite the host, or follow redirects.\nConfirm that the provided URL is correct before live validation.\n\nAlternatively, supply `region` / `MICROSOFT_AI_TTS_REGION` without a URL. It\nconstructs `https://.tts.speech.microsoft.com/cognitiveservices/v1`,\nfollowing the standard public-cloud Azure Speech convention. The plugin does\nnot check region availability: MAI voices are offered only in some Speech\nregions (see the **MAI voices** column of\n[Speech service regions](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/regions?tabs=tts)),\nand the resource key must belong to the selected region. Sovereign clouds and\ncustom/private deployments require an explicit full URL. There is no automatic\nregion detection, failover or redirection to a different region.\n\nThe SSML voice ID is `name:model`, such as `en-US-Harper:MAI-Voice-2-Flash`.\nSet `voice` to the name (`en-US-Harper`) and `model` to the model\n(`MAI-Voice-2-Flash`), and the plugin joins them. A full ID in `voice` is also\naccepted; `model` is then optional, and when set it is checked against the ID's\nsuffix, case-insensitively. Availability still depends on the resource and\nregion. **Voice-2.1-Flash and Voice-2-Flash are not treated as aliases.**\nSSML language defaults to `en-US`; override the `language` constructor argument\nfor other locales.\n\nText and attributes are XML-escaped structurally. Input text is always plain\ntext, not caller-supplied SSML; it cannot inject `