### README
## Generate HTML from reStructuredText files
To generate docs execute following command,
```sh
sh build_docs.sh
```
## Requirements
This build system requires [docker](https://docs.docker.com/engine/install/) to be intsalled and running locally.
---
### Source/Concepts/Llm Providers/Client Libraries
.. _client_libraries:
Client Libraries
================
Plano provides a unified interface that works seamlessly with multiple client libraries and tools. You can use your preferred client library without changing your existing code - just point it to Plano's gateway endpoints.
Supported Clients
------------------
- **OpenAI SDK** - Full compatibility with OpenAI's official client
- **Anthropic SDK** - Native support for Anthropic's client library
- **cURL** - Direct HTTP requests for any programming language
- **Custom HTTP Clients** - Any HTTP client that supports REST APIs
Gateway Endpoints
-----------------
Plano exposes three main endpoints:
.. list-table::
:header-rows: 1
:widths: 40 60
* - Endpoint
- Purpose
* - ``http://127.0.0.1:12000/v1/chat/completions``
- OpenAI-compatible chat completions (LLM Gateway)
* - ``http://127.0.0.1:12000/v1/responses``
- OpenAI Responses API with :ref:`conversational state management ` (LLM Gateway)
* - ``http://127.0.0.1:12000/v1/messages``
- Anthropic-compatible messages (LLM Gateway)
OpenAI (Python) SDK
-------------------
The OpenAI SDK works with any provider through Plano's OpenAI-compatible endpoint.
**Installation:**
.. code-block:: bash
pip install openai
**Basic Usage:**
.. code-block:: python
from openai import OpenAI
# Point to Plano's LLM Gateway
client = OpenAI(
api_key="test-key", # Can be any value for local testing
base_url="http://127.0.0.1:12000/v1"
)
# Use any model configured in your plano_config.yaml
completion = client.chat.completions.create(
model="gpt-4o-mini", # Or use :ref:`model aliases ` like "fast-model"
max_tokens=50,
messages=[
{
"role": "user",
"content": "Hello, how are you?"
}
]
)
print(completion.choices[0].message.content)
**Streaming Responses:**
.. code-block:: python
from openai import OpenAI
client = OpenAI(
api_key="test-key",
base_url="http://127.0.0.1:12000/v1"
)
stream = client.chat.completions.create(
model="gpt-4o-mini",
max_tokens=50,
messages=[
{
"role": "user",
"content": "Tell me a short story"
}
],
stream=True
)
# Collect streaming chunks
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
**Using with Non-OpenAI Models:**
The OpenAI SDK can be used with any provider configured in Plano:
.. code-block:: python
# Using Claude model through OpenAI SDK
completion = client.chat.completions.create(
model="claude-3-5-sonnet-20241022",
max_tokens=50,
messages=[
{
"role": "user",
"content": "Explain quantum computing briefly"
}
]
)
# Using Ollama model through OpenAI SDK
completion = client.chat.completions.create(
model="llama3.1",
max_tokens=50,
messages=[
{
"role": "user",
"content": "What's the capital of France?"
}
]
)
OpenAI Responses API (Conversational State)
-------------------------------------------
The OpenAI Responses API (``v1/responses``) enables multi-turn conversations with automatic state management. Plano handles conversation history for you, so you don't need to manually include previous messages in each request.
See :ref:`managing_conversational_state` for detailed configuration and storage backend options.
**Installation:**
.. code-block:: bash
pip install openai
**Basic Multi-Turn Conversation:**
.. code-block:: python
from openai import OpenAI
# Point to Plano's LLM Gateway
client = OpenAI(
api_key="test-key",
base_url="http://127.0.0.1:12000/v1"
)
# First turn - creates a new conversation
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": "My name is Alice"}
]
)
# Extract response_id for conversation continuity
response_id = response.id
print(f"Assistant: {response.choices[0].message.content}")
# Second turn - continues the conversation
# Plano automatically retrieves and merges previous context
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "user", "content": "What's my name?"}
],
metadata={"response_id": response_id} # Reference previous conversation
)
print(f"Assistant: {response.choices[0].message.content}")
# Output: "Your name is Alice"
**Using with Any Provider:**
The Responses API works with any LLM provider configured in Plano:
.. code-block:: python
# Multi-turn conversation with Claude
response = client.chat.completions.create(
model="claude-3-5-sonnet-20241022",
messages=[
{"role": "user", "content": "Let's discuss quantum physics"}
]
)
response_id = response.id
# Continue conversation - Plano manages state regardless of provider
response = client.chat.completions.create(
model="claude-3-5-sonnet-20241022",
messages=[
{"role": "user", "content": "Tell me more about entanglement"}
],
metadata={"response_id": response_id}
)
**Key Benefits:**
* **Reduced payload size**: No need to send full conversation history in each request
* **Provider flexibility**: Use any configured LLM provider with state management
* **Automatic context merging**: Plano handles conversation continuity behind the scenes
* **Production-ready storage**: Configure :ref:`PostgreSQL or memory storage ` based on your needs
Anthropic (Python) SDK
----------------------
The Anthropic SDK works with any provider through Plano's Anthropic-compatible endpoint.
**Installation:**
.. code-block:: bash
pip install anthropic
**Basic Usage:**
.. code-block:: python
import anthropic
# Point to Plano's LLM Gateway
client = anthropic.Anthropic(
api_key="test-key", # Can be any value for local testing
base_url="http://127.0.0.1:12000"
)
# Use any model configured in your plano_config.yaml
message = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=50,
messages=[
{
"role": "user",
"content": "Hello, please respond briefly!"
}
]
)
print(message.content[0].text)
**Streaming Responses:**
.. code-block:: python
import anthropic
client = anthropic.Anthropic(
api_key="test-key",
base_url="http://127.0.0.1:12000"
)
with client.messages.stream(
model="claude-3-5-sonnet-20241022",
max_tokens=50,
messages=[
{
"role": "user",
"content": "Tell me about artificial intelligence"
}
]
) as stream:
# Collect text deltas
for text in stream.text_stream:
print(text, end="")
# Get final assembled message
final_message = stream.get_final_message()
final_text = "".join(block.text for block in final_message.content if block.type == "text")
**Using with Non-Anthropic Models:**
The Anthropic SDK can be used with any provider configured in Plano:
.. code-block:: python
# Using OpenAI model through Anthropic SDK
message = client.messages.create(
model="gpt-4o-mini",
max_tokens=50,
messages=[
{
"role": "user",
"content": "Explain machine learning in simple terms"
}
]
)
# Using Ollama model through Anthropic SDK
message = client.messages.create(
model="llama3.1",
max_tokens=50,
messages=[
{
"role": "user",
"content": "What is Python programming?"
}
]
)
cURL Examples
-------------
For direct HTTP requests or integration with any programming language:
**OpenAI-Compatible Endpoint:**
.. code-block:: bash
# Basic request
curl -X POST http://127.0.0.1:12000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer test-key" \
-d '{
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "content": "Hello!"}
],
"max_tokens": 50
}'
# Using :ref:`model aliases `
curl -X POST http://127.0.0.1:12000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "fast-model",
"messages": [
{"role": "user", "content": "Summarize this text..."}
],
"max_tokens": 100
}'
# Streaming request
curl -X POST http://127.0.0.1:12000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini",
"messages": [
{"role": "user", "content": "Tell me a story"}
],
"stream": true,
"max_tokens": 200
}'
**Anthropic-Compatible Endpoint:**
.. code-block:: bash
# Basic request
curl -X POST http://127.0.0.1:12000/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: test-key" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "claude-3-5-sonnet-20241022",
"max_tokens": 50,
"messages": [
{"role": "user", "content": "Hello Claude!"}
]
}'
Cross-Client Compatibility
--------------------------
One of Plano's key features is cross-client compatibility. You can:
**Use OpenAI SDK with Claude Models:**
.. code-block:: python
# OpenAI client calling Claude model
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:12000/v1", api_key="test")
response = client.chat.completions.create(
model="claude-3-5-sonnet-20241022", # Claude model
messages=[{"role": "user", "content": "Hello"}]
)
**Use Anthropic SDK with OpenAI Models:**
.. code-block:: python
# Anthropic client calling OpenAI model
import anthropic
client = anthropic.Anthropic(base_url="http://127.0.0.1:12000", api_key="test")
response = client.messages.create(
model="gpt-4o-mini", # OpenAI model
max_tokens=50,
messages=[{"role": "user", "content": "Hello"}]
)
**Mix and Match with** :ref:`Model Aliases `:
.. code-block:: python
# Same code works with different underlying models
def ask_question(client, question):
return client.chat.completions.create(
model="reasoning-model", # Alias could point to any provider
messages=[{"role": "user", "content": question}]
)
# Works regardless of what "reasoning-model" actually points to
openai_client = OpenAI(base_url="http://127.0.0.1:12000/v1", api_key="test")
response = ask_question(openai_client, "Solve this math problem...")
Error Handling
--------------
**OpenAI SDK Error Handling:**
.. code-block:: python
from openai import OpenAI
import openai
client = OpenAI(base_url="http://127.0.0.1:12000/v1", api_key="test")
try:
completion = client.chat.completions.create(
model="nonexistent-model",
messages=[{"role": "user", "content": "Hello"}]
)
except openai.NotFoundError as e:
print(f"Model not found: {e}")
except openai.APIError as e:
print(f"API error: {e}")
**Anthropic SDK Error Handling:**
.. code-block:: python
import anthropic
client = anthropic.Anthropic(base_url="http://127.0.0.1:12000", api_key="test")
try:
message = client.messages.create(
model="nonexistent-model",
max_tokens=50,
messages=[{"role": "user", "content": "Hello"}]
)
except anthropic.NotFoundError as e:
print(f"Model not found: {e}")
except anthropic.APIError as e:
print(f"API error: {e}")
Best Practices
--------------
**Use** :ref:`Model Aliases `:
Instead of hardcoding provider-specific model names, use semantic aliases:
.. code-block:: python
# Good - uses semantic alias
model = "fast-model"
# Less ideal - hardcoded provider model
model = "openai/gpt-4o-mini"
**Environment-Based Configuration:**
Use different :ref:`model aliases ` for different environments:
.. code-block:: python
import os
# Development uses cheaper/faster models
model = os.getenv("MODEL_ALIAS", "dev.chat.v1")
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Hello"}]
)
**Graceful Fallbacks:**
Implement fallback logic for better reliability:
.. code-block:: python
def chat_with_fallback(client, messages, primary_model="smart-model", fallback_model="fast-model"):
try:
return client.chat.completions.create(model=primary_model, messages=messages)
except Exception as e:
print(f"Primary model failed, trying fallback: {e}")
return client.chat.completions.create(model=fallback_model, messages=messages)
See Also
--------
- :ref:`supported_providers` - Configure your providers and see available models
- :ref:`model_aliases` - Create semantic model names
- :ref:`llm_router` - Intelligent routing capabilities
---
### Source/Concepts/Llm Providers/Llm Providers
.. _llm_providers:
Model (LLM) Providers
=====================
**Model Providers** are a top-level primitive in Plano, helping developers centrally define, secure, observe,
and manage the usage of their models. Plano builds on Envoy's reliable `cluster subsystem `_ to manage egress traffic to models, which includes intelligent routing, retry and fail-over mechanisms,
ensuring high availability and fault tolerance. This abstraction also enables developers to seamlessly switch between model providers or upgrade model versions, simplifying the integration and scaling of models across applications.
Today, we are enable you to connect to 15+ different AI providers through a unified interface with advanced routing and management capabilities.
Whether you're using OpenAI, Anthropic, Azure OpenAI, local Ollama models, or any OpenAI-compatible provider, Plano provides seamless integration with enterprise-grade features.
.. note::
Please refer to the quickstart guide :ref:`here ` to configure and use LLM providers via common client libraries like OpenAI and Anthropic Python SDKs, or via direct HTTP/cURL requests.
Core Capabilities
-----------------
**Multi-Provider Support**
Connect to any combination of providers simultaneously (see :ref:`supported_providers` for full details):
- First-Class Providers: Native integrations with OpenAI, Anthropic, DeepSeek, Mistral, Groq, Google Gemini, Together AI, xAI, Azure OpenAI, and Ollama
- OpenAI-Compatible Providers: Any provider implementing the OpenAI Chat Completions API standard
- Wildcard Model Configuration: Automatically configure all models from a provider using ``provider/*`` syntax
**Intelligent Routing**
Three powerful routing approaches to optimize model selection:
- Model-based Routing: Direct routing to specific models using provider/model names (see :ref:`supported_providers`)
- Alias-based Routing: Semantic routing using custom aliases (see :ref:`model_aliases`)
- Preference-aligned Routing: Intelligent routing using the Plano-Router model (see :ref:`preference_aligned_routing`)
**Unified Client Interface**
Use your preferred client library without changing existing code (see :ref:`client_libraries` for details):
- OpenAI Python SDK: Full compatibility with all providers
- Anthropic Python SDK: Native support with cross-provider capabilities
- cURL & HTTP Clients: Direct REST API access for any programming language
- Custom Integrations: Standard HTTP interfaces for seamless integration
Key Benefits
------------
- **Provider Flexibility**: Switch between providers without changing client code
- **Three Routing Methods**: Choose from model-based, alias-based, or preference-aligned routing (using `Plano-Router-1.5B `_) strategies
- **Cost Optimization**: Route requests to cost-effective models based on complexity
- **Performance Optimization**: Use fast models for simple tasks, powerful models for complex reasoning
- **Environment Management**: Configure different models for different environments
- **Future-Proof**: Easy to add new providers and upgrade models
Common Use Cases
----------------
**Development Teams**
- Use aliases like ``dev.chat.v1`` and ``prod.chat.v1`` for environment-specific models
- Route simple queries to fast/cheap models, complex tasks to powerful models
- Test new models safely using canary deployments (coming soon)
**Production Applications**
- Implement fallback strategies across multiple providers for reliability
- Use intelligent routing to optimize cost and performance automatically
- Monitor usage patterns and model performance across providers
**Enterprise Deployments**
- Connect to both cloud providers and on-premises models (Ollama, custom deployments)
- Apply consistent security and governance policies across all providers
- Scale across regions using different provider endpoints
Advanced Features
-----------------
- :ref:`preference_aligned_routing` - Learn about preference-aligned dynamic routing and intelligent model selection
Getting Started
---------------
Dive into specific areas based on your needs:
.. toctree::
:maxdepth: 2
supported_providers
client_libraries
model_aliases
---
### Source/Concepts/Llm Providers/Model Aliases
.. _model_aliases:
Model Aliases
=============
Model aliases provide semantic, version-controlled names for your models, enabling cleaner client code, easier model management, and advanced routing capabilities. Instead of using provider-specific model names like ``gpt-4o-mini`` or ``claude-3-5-sonnet-20241022``, you can create meaningful aliases like ``fast-model`` or ``arch.summarize.v1``.
**Benefits of Model Aliases:**
- **Semantic Naming**: Use descriptive names that reflect the model's purpose
- **Version Control**: Implement versioning schemes (e.g., ``v1``, ``v2``) for model upgrades
- **Environment Management**: Different aliases can point to different models across environments
- **Client Simplification**: Clients use consistent, meaningful names regardless of underlying provider
- **Advanced Routing (Coming Soon)**: Enable guardrails, fallbacks, and traffic splitting at the alias level
Basic Configuration
-------------------
**Simple Alias Mapping**
.. code-block:: yaml
:caption: Basic Model Aliases
llm_providers:
- model: openai/gpt-4o-mini
access_key: $OPENAI_API_KEY
- model: openai/gpt-4o
access_key: $OPENAI_API_KEY
- model: anthropic/claude-3-5-sonnet-20241022
access_key: $ANTHROPIC_API_KEY
- model: ollama/llama3.1
base_url: http://localhost:11434
# Define aliases that map to the models above
model_aliases:
# Semantic versioning approach
arch.summarize.v1:
target: gpt-4o-mini
arch.reasoning.v1:
target: gpt-4o
arch.creative.v1:
target: claude-3-5-sonnet-20241022
# Functional aliases
fast-model:
target: gpt-4o-mini
smart-model:
target: gpt-4o
creative-model:
target: claude-3-5-sonnet-20241022
# Local model alias
local-chat:
target: llama3.1
Using Aliases
-------------
**Client Code Examples**
Once aliases are configured, clients can use semantic names instead of provider-specific model names:
.. code-block:: python
:caption: Python Client Usage
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:12000/")
# Use semantic alias instead of provider model name
response = client.chat.completions.create(
model="arch.summarize.v1", # Points to gpt-4o-mini
messages=[{"role": "user", "content": "Summarize this document..."}]
)
# Switch to a different capability
response = client.chat.completions.create(
model="arch.reasoning.v1", # Points to gpt-4o
messages=[{"role": "user", "content": "Solve this complex problem..."}]
)
.. code-block:: bash
:caption: cURL Example
curl -X POST http://127.0.0.1:12000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "fast-model",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Naming Best Practices
---------------------
**Semantic Versioning**
Use version numbers for backward compatibility and gradual model upgrades:
.. code-block:: yaml
model_aliases:
# Current production version
arch.summarize.v1:
target: gpt-4o-mini
# Beta version for testing
arch.summarize.v2:
target: gpt-4o
# Stable alias that always points to latest
arch.summarize.latest:
target: gpt-4o-mini
**Purpose-Based Naming**
Create aliases that reflect the intended use case:
.. code-block:: yaml
model_aliases:
# Task-specific
code-reviewer:
target: gpt-4o
document-summarizer:
target: gpt-4o-mini
creative-writer:
target: claude-3-5-sonnet-20241022
data-analyst:
target: gpt-4o
**Environment-Specific Aliases**
Different environments can use different underlying models:
.. code-block:: yaml
model_aliases:
# Development environment - use faster/cheaper models
dev.chat.v1:
target: gpt-4o-mini
# Production environment - use more capable models
prod.chat.v1:
target: gpt-4o
# Staging environment - test new models
staging.chat.v1:
target: claude-3-5-sonnet-20241022
Advanced Features (Coming Soon)
--------------------------------
The following features are planned for future releases of model aliases:
**Guardrails Integration**
Apply safety, cost, or latency rules at the alias level:
.. code-block:: yaml
:caption: Future Feature - Guardrails
model_aliases:
arch.reasoning.v1:
target: gpt-oss-120b
guardrails:
max_latency: 5s
max_cost_per_request: 0.10
block_categories: ["jailbreak", "PII"]
content_filters:
- type: "profanity"
- type: "sensitive_data"
**Fallback Chains**
Provide a chain of models if the primary target fails or hits quota limits:
.. code-block:: yaml
:caption: Future Feature - Fallbacks
model_aliases:
arch.summarize.v1:
target: gpt-4o-mini
fallbacks:
- target: llama3.1
conditions: ["quota_exceeded", "timeout"]
- target: claude-3-haiku-20240307
conditions: ["primary_and_first_fallback_failed"]
**Traffic Splitting & Canary Deployments**
Distribute traffic across multiple models for A/B testing or gradual rollouts:
.. code-block:: yaml
:caption: Future Feature - Traffic Splitting
model_aliases:
arch.v1:
targets:
- model: llama3.1
weight: 80
- model: gpt-4o-mini
weight: 20
# Canary deployment
arch.experimental.v1:
targets:
- model: gpt-4o # Current stable
weight: 95
- model: o1-preview # New model being tested
weight: 5
**Load Balancing**
Distribute requests across multiple instances of the same model:
.. code-block:: yaml
:caption: Future Feature - Load Balancing
model_aliases:
high-throughput-chat:
load_balance:
algorithm: "round_robin" # or "least_connections", "weighted"
targets:
- model: gpt-4o-mini
endpoint: "https://api-1.example.com"
- model: gpt-4o-mini
endpoint: "https://api-2.example.com"
- model: gpt-4o-mini
endpoint: "https://api-3.example.com"
Validation Rules
----------------
- Alias names must be valid identifiers (alphanumeric, dots, hyphens, underscores)
- Target models must be defined in the ``llm_providers`` section
- Circular references between aliases are not allowed
- Weights in traffic splitting must sum to 100
See Also
--------
- :ref:`llm_providers` - Learn about configuring LLM providers
- :ref:`llm_router` - Understand how aliases work with intelligent routing
---
### Source/Concepts/Llm Providers/Supported Providers
.. _supported_providers:
Supported Providers & Configuration
===================================
Plano provides first-class support for multiple LLM providers through native integrations and OpenAI-compatible interfaces. This comprehensive guide covers all supported providers, their available chat models, and detailed configuration instructions.
.. note::
**Model Support:** Plano supports all chat models from each provider, not just the examples shown in this guide. The configurations below demonstrate common models for reference, but you can use any chat model available from your chosen provider.
Please refer to the quickstart guide :ref:`here ` to configure and use LLM providers via common client libraries like OpenAI and Anthropic Python SDKs, or via direct HTTP/cURL requests.
Configuration Structure
-----------------------
All providers are configured in the ``llm_providers`` section of your ``plano_config.yaml`` file:
.. code-block:: yaml
llm_providers:
# Provider configurations go here
- model: provider/model-name
access_key: $API_KEY
# Additional provider-specific options
**Common Configuration Fields:**
- ``model``: Provider prefix and model name (format: ``provider/model-name`` or ``provider/*`` for wildcard expansion)
- ``access_key``: API key for authentication (supports environment variables)
- ``default``: Mark a model as the default (optional, boolean)
- ``name``: Custom name for the provider instance (optional)
- ``base_url``: Custom endpoint URL (required for some providers, optional for others - see :ref:`base_url_details`)
Provider Categories
-------------------
**First-Class Providers**
Native integrations with built-in support for provider-specific features and authentication.
**OpenAI-Compatible Providers**
Any provider that implements the OpenAI API interface can be configured using custom endpoints.
Supported API Endpoints
------------------------
Plano supports the following standardized endpoints across providers:
.. list-table::
:header-rows: 1
:widths: 30 30 40
* - Endpoint
- Purpose
- Supported Clients
* - ``/v1/chat/completions``
- OpenAI-style chat completions
- OpenAI SDK, cURL, custom clients
* - ``/v1/messages``
- Anthropic-style messages
- Anthropic SDK, cURL, custom clients
* - ``/v1/responses``
- Unified response endpoint for agentic apps
- All SDKs, cURL, custom clients
First-Class Providers
---------------------
OpenAI
~~~~~~
**Provider Prefix:** ``openai/``
**API Endpoint:** ``/v1/chat/completions``
**Authentication:** API Key - Get your OpenAI API key from `OpenAI Platform `_.
**Supported Chat Models:** All OpenAI chat models including GPT-5.2, GPT-5, GPT-4o, and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - GPT-5.2
- ``openai/gpt-5.2``
- Next-generation model (use any model name from OpenAI's API)
* - GPT-5
- ``openai/gpt-5``
- Latest multimodal model
* - GPT-4o mini
- ``openai/gpt-4o-mini``
- Fast, cost-effective model
* - GPT-4o
- ``openai/gpt-4o``
- High-capability reasoning model
* - o3-mini
- ``openai/o3-mini``
- Reasoning-focused model (preview)
* - o3
- ``openai/o3``
- Advanced reasoning model (preview)
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
# Configure all OpenAI models with wildcard
- model: openai/*
access_key: $OPENAI_API_KEY
# Or configure specific models
- model: openai/gpt-5.2
access_key: $OPENAI_API_KEY
default: true
- model: openai/gpt-5
access_key: $OPENAI_API_KEY
- model: openai/gpt-4o
access_key: $OPENAI_API_KEY
Anthropic
~~~~~~~~~
**Provider Prefix:** ``anthropic/``
**API Endpoint:** ``/v1/messages``
**Authentication:** API Key - Get your Anthropic API key from `Anthropic Console `_.
**Supported Chat Models:** All Anthropic Claude models including Claude Sonnet 4.5, Claude Opus 4.5, Claude Haiku 4.5, and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - Claude Opus 4.5
- ``anthropic/claude-opus-4-5``
- Most capable model for complex tasks
* - Claude Sonnet 4.5
- ``anthropic/claude-sonnet-4-5``
- Balanced performance model
* - Claude Haiku 4.5
- ``anthropic/claude-haiku-4-5``
- Fast and efficient model
* - Claude Sonnet 3.5
- ``anthropic/claude-sonnet-3-5``
- Complex agents and coding
**Configuration Examples:**
.. code-block:: yaml
version: v0.4.0
model_providers:
# Configure all Anthropic models with wildcard
- model: anthropic/*
access_key: $ANTHROPIC_API_KEY
# Or configure specific models
- model: anthropic/claude-opus-4-5
access_key: $ANTHROPIC_API_KEY
- model: anthropic/claude-sonnet-4-5
access_key: $ANTHROPIC_API_KEY
- model: anthropic/claude-haiku-4-5
access_key: $ANTHROPIC_API_KEY
# Override specific model with custom routing
- model: anthropic/*
access_key: $ANTHROPIC_API_KEY
- model: anthropic/claude-sonnet-4-6
access_key: $ANTHROPIC_PROD_API_KEY
routing_preferences:
- name: code_generation
description: generating new code snippets, functions, or boilerplate based on user prompts or requirements
models:
- anthropic/claude-sonnet-4-6
DeepSeek
~~~~~~~~
**Provider Prefix:** ``deepseek/``
**API Endpoint:** ``/v1/chat/completions``
**Authentication:** API Key - Get your DeepSeek API key from `DeepSeek Platform `_.
**Supported Chat Models:** All DeepSeek chat models including DeepSeek-Chat, DeepSeek-Coder, and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - DeepSeek Chat
- ``deepseek/deepseek-chat``
- General purpose chat model
* - DeepSeek Coder
- ``deepseek/deepseek-coder``
- Code-specialized model
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
- model: deepseek/deepseek-chat
access_key: $DEEPSEEK_API_KEY
- model: deepseek/deepseek-coder
access_key: $DEEPSEEK_API_KEY
Mistral AI
~~~~~~~~~~
**Provider Prefix:** ``mistral/``
**API Endpoint:** ``/v1/chat/completions``
**Authentication:** API Key - Get your Mistral API key from `Mistral AI Console `_.
**Supported Chat Models:** All Mistral chat models including Mistral Large, Mistral Small, Ministral, and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - Mistral Large
- ``mistral/mistral-large-latest``
- Most capable model
* - Mistral Medium
- ``mistral/mistral-medium-latest``
- Balanced performance
* - Mistral Small
- ``mistral/mistral-small-latest``
- Fast and efficient
* - Ministral 3B
- ``mistral/ministral-3b-latest``
- Compact model
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
- model: mistral/mistral-large-latest
access_key: $MISTRAL_API_KEY
- model: mistral/mistral-small-latest
access_key: $MISTRAL_API_KEY
Groq
~~~~
**Provider Prefix:** ``groq/``
**API Endpoint:** ``/openai/v1/chat/completions`` (transformed internally)
**Authentication:** API Key - Get your Groq API key from `Groq Console `_.
**Supported Chat Models:** All Groq chat models including Llama 4, GPT OSS, Mixtral, Gemma, and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - Llama 4 Maverick 17B
- ``groq/llama-4-maverick-17b-128e-instruct``
- Fast inference Llama model
* - Llama 4 Scout 8B
- ``groq/llama-4-scout-8b-128e-instruct``
- Smaller Llama model
* - GPT OSS 20B
- ``groq/gpt-oss-20b``
- Open source GPT model
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
- model: groq/llama-4-maverick-17b-128e-instruct
access_key: $GROQ_API_KEY
- model: groq/llama-4-scout-8b-128e-instruct
access_key: $GROQ_API_KEY
- model: groq/gpt-oss-20b
access_key: $GROQ_API_KEY
Google Gemini
~~~~~~~~~~~~~
**Provider Prefix:** ``gemini/``
**API Endpoint:** ``/v1beta/openai/chat/completions`` (transformed internally)
**Authentication:** API Key - Get your Google AI API key from `Google AI Studio `_.
**Supported Chat Models:** All Google Gemini chat models including Gemini 3 Pro, Gemini 3 Flash, and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - Gemini 3 Pro
- ``gemini/gemini-3-pro``
- Advanced reasoning and creativity
* - Gemini 3 Flash
- ``gemini/gemini-3-flash``
- Fast and efficient model
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
- model: gemini/gemini-3-pro
access_key: $GOOGLE_API_KEY
- model: gemini/gemini-3-flash
access_key: $GOOGLE_API_KEY
Together AI
~~~~~~~~~~~
**Provider Prefix:** ``together_ai/``
**API Endpoint:** ``/v1/chat/completions``
**Authentication:** API Key - Get your Together AI API key from `Together AI Settings `_.
**Supported Chat Models:** All Together AI chat models including Llama, CodeLlama, Mixtral, Qwen, and hundreds of other open-source models.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - Meta Llama 2 7B
- ``together_ai/meta-llama/Llama-2-7b-chat-hf``
- Open source chat model
* - Meta Llama 2 13B
- ``together_ai/meta-llama/Llama-2-13b-chat-hf``
- Larger open source model
* - Code Llama 34B
- ``together_ai/codellama/CodeLlama-34b-Instruct-hf``
- Code-specialized model
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
- model: together_ai/meta-llama/Llama-2-7b-chat-hf
access_key: $TOGETHER_API_KEY
- model: together_ai/codellama/CodeLlama-34b-Instruct-hf
access_key: $TOGETHER_API_KEY
xAI
~~~
**Provider Prefix:** ``xai/``
**API Endpoint:** ``/v1/chat/completions``
**Authentication:** API Key - Get your xAI API key from `xAI Console `_.
**Supported Chat Models:** All xAI chat models including Grok Beta and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - Grok Beta
- ``xai/grok-beta``
- Conversational AI model
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
- model: xai/grok-beta
access_key: $XAI_API_KEY
Moonshot AI
~~~~~~~~~~~
**Provider Prefix:** ``moonshotai/``
**API Endpoint:** ``/v1/chat/completions``
**Authentication:** API Key - Get your Moonshot AI API key from `Moonshot AI Platform `_.
**Supported Chat Models:** All Moonshot AI chat models including Kimi K3, Kimi K2, Moonshot v1, and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - Kimi K3
- ``moonshotai/kimi-k3``
- Flagship multimodal reasoning model with a 1M-token context window
* - Kimi for Coding
- ``moonshotai/kimi-for-coding``
- Kimi Code API model for agentic coding (use with ``base_url: https://api.kimi.com/coding/v1``)
* - Kimi K2.6
- ``moonshotai/kimi-k2.6``
- Latest K2-line foundation model optimized for agentic tasks
* - Kimi K2.5
- ``moonshotai/kimi-k2.5``
- Previous K2-line foundation model optimized for agentic tasks
* - Moonshot v1 32K
- ``moonshotai/moonshot-v1-32k``
- Extended context model with 32K tokens
* - Moonshot v1 128K
- ``moonshotai/moonshot-v1-128k``
- Long context model with 128K tokens
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
# Flagship reasoning model with a 1M-token context window
- model: moonshotai/kimi-k3
access_key: $MOONSHOTAI_API_KEY
# Kimi Code API (Claude Code / agentic clients via Plano translation)
- model: moonshotai/kimi-for-coding
access_key: $MOONSHOTAI_API_KEY
base_url: https://api.kimi.com/coding/v1
headers:
User-Agent: "KimiCLI/1.3"
# Latest K2 models for agentic tasks
- model: moonshotai/kimi-k2.6
access_key: $MOONSHOTAI_API_KEY
# V1 models with different context lengths
- model: moonshotai/moonshot-v1-32k
access_key: $MOONSHOTAI_API_KEY
- model: moonshotai/moonshot-v1-128k
access_key: $MOONSHOTAI_API_KEY
.. note::
Kimi K3 always runs in thinking mode and pins ``temperature``, ``top_p``, ``n``,
``presence_penalty``, and ``frequency_penalty`` to fixed values. Plano strips these
fields from requests sent to Moonshot's API so clients that set them do not fail.
Use the ``reasoning_effort`` field to control reasoning depth instead. K3 accepts
``low``, ``high``, and ``max`` (default ``max``); Plano maps other OpenAI-style
values onto the nearest supported level, so ``none`` and ``minimal`` become ``low``
and ``medium`` becomes ``high``.
Zhipu AI
~~~~~~~~
**Provider Prefix:** ``zhipu/``
**API Endpoint:** ``/api/paas/v4/chat/completions``
**Authentication:** API Key - Get your Zhipu AI API key from `Zhipu AI Platform `_.
**Supported Chat Models:** All Zhipu AI GLM models including GLM-4, GLM-4 Flash, and all future releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - GLM-4.6
- ``zhipu/glm-4.6``
- Latest and most capable GLM model with enhanced reasoning abilities
* - GLM-4.5
- ``zhipu/glm-4.5``
- High-performance model with multimodal capabilities
* - GLM-4.5 Air
- ``zhipu/glm-4.5-air``
- Lightweight and fast model optimized for efficiency
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
# Latest GLM models
- model: zhipu/glm-4.6
access_key: $ZHIPU_API_KEY
- model: zhipu/glm-4.5
access_key: $ZHIPU_API_KEY
- model: zhipu/glm-4.5-air
access_key: $ZHIPU_API_KEY
Xiaomi MiMo
~~~~~~~~~~~
**Provider Prefix:** ``xiaomi/``
**API Endpoint:** ``/v1/chat/completions``
**Authentication:** API Key - Create your key in the `Xiaomi MiMo API Open Platform `_ and set ``MIMO_API_KEY``.
**Supported Chat Models:** All Xiaomi MiMo chat models including mimo-v2-pro, mimo-v2-omni, mimo-v2-flash, and future chat model releases.
.. list-table::
:header-rows: 1
:widths: 30 20 50
* - Model Name
- Model ID for Config
- Description
* - MiMo V2 Pro
- ``xiaomi/mimo-v2-pro``
- Highest capability general model
* - MiMo V2 Omni
- ``xiaomi/mimo-v2-omni``
- Multimodal-capable assistant model
* - MiMo V2 Flash
- ``xiaomi/mimo-v2-flash``
- Faster, lower-latency model
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
# Configure all known Xiaomi models with wildcard expansion
- model: xiaomi/*
access_key: $MIMO_API_KEY
# Or configure specific models
- model: xiaomi/mimo-v2-pro
access_key: $MIMO_API_KEY
default: true
- model: xiaomi/mimo-v2-omni
access_key: $MIMO_API_KEY
Providers Requiring Base URL
----------------------------
The following providers require a ``base_url`` parameter to be configured. For detailed information on base URL configuration including path prefix behavior and examples, see :ref:`base_url_details`.
Azure OpenAI
~~~~~~~~~~~~
**Provider Prefix:** ``azure_openai/``
**API Endpoint:** ``/openai/deployments/{deployment-name}/chat/completions`` (constructed automatically)
**Authentication:** API Key + Base URL - Get your Azure OpenAI API key from `Azure Portal `_ → Your OpenAI Resource → Keys and Endpoint.
**Supported Chat Models:** All Azure OpenAI chat models including GPT-4o, GPT-4, GPT-3.5-turbo deployed in your Azure subscription.
.. code-block:: yaml
llm_providers:
# Single deployment
- model: azure_openai/gpt-4o
access_key: $AZURE_OPENAI_API_KEY
base_url: https://your-resource.openai.azure.com
# Multiple deployments
- model: azure_openai/gpt-4o-mini
access_key: $AZURE_OPENAI_API_KEY
base_url: https://your-resource.openai.azure.com
Amazon Bedrock
~~~~~~~~~~~~~~
**Provider Prefix:** ``amazon_bedrock/``
**API Endpoint:** Plano automatically constructs the endpoint as:
- Non-streaming: ``/model/{model-id}/converse``
- Streaming: ``/model/{model-id}/converse-stream``
**Authentication:** AWS Bearer Token + Base URL - Get your API Keys from `AWS Bedrock Console `_ → Discover → API Keys.
**Supported Chat Models:** All Amazon Bedrock foundation models including Claude (Anthropic), Nova (Amazon), Llama (Meta), Mistral AI, and Cohere Command models.
.. code-block:: yaml
llm_providers:
# Amazon Nova models
- model: amazon_bedrock/us.amazon.nova-premier-v1:0
access_key: $AWS_BEARER_TOKEN_BEDROCK
base_url: https://bedrock-runtime.us-west-2.amazonaws.com
default: true
- model: amazon_bedrock/us.amazon.nova-pro-v1:0
access_key: $AWS_BEARER_TOKEN_BEDROCK
base_url: https://bedrock-runtime.us-west-2.amazonaws.com
# Claude on Bedrock
- model: amazon_bedrock/us.anthropic.claude-3-5-sonnet-20241022-v2:0
access_key: $AWS_BEARER_TOKEN_BEDROCK
base_url: https://bedrock-runtime.us-west-2.amazonaws.com
Qwen (Alibaba)
~~~~~~~~~~~~~~
**Provider Prefix:** ``qwen/``
**API Endpoint:** ``/v1/chat/completions``
**Authentication:** API Key + Base URL - Get your Qwen API key from `Qwen Portal `_ → Your Qwen Resource → Keys and Endpoint.
**Supported Chat Models:** All Qwen chat models including Qwen3, Qwen3-Coder and all future releases.
.. code-block:: yaml
llm_providers:
# Single deployment
- model: qwen/qwen3
access_key: $DASHSCOPE_API_KEY
base_url: https://dashscope.aliyuncs.com
# Multiple deployments
- model: qwen/qwen3-coder
access_key: $DASHSCOPE_API_KEY
base_url: "https://dashscope-intl.aliyuncs.com"
Ollama
~~~~~~
**Provider Prefix:** ``ollama/``
**API Endpoint:** ``/v1/chat/completions`` (Ollama's OpenAI-compatible endpoint)
**Authentication:** None (Base URL only) - Install Ollama from `Ollama.com `_ and pull your desired models.
**Supported Chat Models:** All chat models available in your local Ollama installation. Use ``ollama list`` to see installed models.
.. code-block:: yaml
llm_providers:
# Local Ollama installation
- model: ollama/llama3.1
base_url: http://localhost:11434
# Ollama running locally
- model: ollama/codellama
base_url: http://localhost:11434
OpenAI-Compatible Providers
~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Supported Models:** Any chat models from providers that implement the OpenAI Chat Completions API standard.
For providers that implement the OpenAI API but aren't natively supported:
.. code-block:: yaml
llm_providers:
# Generic OpenAI-compatible provider
- model: custom-provider/custom-model
base_url: https://api.customprovider.com
provider_interface: openai
access_key: $CUSTOM_API_KEY
# Local deployment
- model: local/llama2-7b
base_url: http://localhost:8000
provider_interface: openai
.. _base_url_details:
Base URL Configuration
----------------------
The ``base_url`` parameter allows you to specify custom endpoints for model providers. It supports both hostname and path components, enabling flexible routing to different API endpoints.
**Format:** ``://[:][/]``
**Components:**
- ``scheme``: ``http`` or ``https``
- ``hostname``: API server hostname or IP address
- ``port``: Optional, defaults to 80 for http, 443 for https
- ``path``: Optional path prefix that **replaces** the provider's default API path
**How Path Prefixes Work:**
When you include a path in ``base_url``, it replaces the provider's default path prefix while preserving the endpoint suffix:
- **Without path prefix**: Uses the provider's default path structure
- **With path prefix**: Your custom path replaces the provider's default prefix, then the endpoint suffix is appended
**Configuration Examples:**
.. code-block:: yaml
llm_providers:
# Simple hostname only - uses provider's default path
- model: zhipu/glm-4.6
access_key: $ZHIPU_API_KEY
base_url: https://api.z.ai
# Results in: https://api.z.ai/api/paas/v4/chat/completions
# With custom path prefix - replaces provider's default path
- model: zhipu/glm-4.6
access_key: $ZHIPU_API_KEY
base_url: https://api.z.ai/api/coding/paas/v4
# Results in: https://api.z.ai/api/coding/paas/v4/chat/completions
# Azure with custom path
- model: azure_openai/gpt-4
access_key: $AZURE_API_KEY
base_url: https://mycompany.openai.azure.com/custom/deployment/path
# Results in: https://mycompany.openai.azure.com/custom/deployment/path/chat/completions
# Behind a proxy or API gateway
- model: openai/gpt-4o
access_key: $OPENAI_API_KEY
base_url: https://proxy.company.com/ai-gateway/openai
# Results in: https://proxy.company.com/ai-gateway/openai/chat/completions
# Local endpoint with custom port
- model: ollama/llama3.1
base_url: http://localhost:8080
# Results in: http://localhost:8080/v1/chat/completions
# Custom provider with path prefix
- model: vllm/custom-model
access_key: $VLLM_API_KEY
base_url: https://vllm.example.com/models/v2
provider_interface: openai
# Results in: https://vllm.example.com/models/v2/chat/completions
Advanced Configuration
----------------------
Multiple Provider Instances
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Configure multiple instances of the same provider:
.. code-block:: yaml
llm_providers:
# Production OpenAI
- model: openai/gpt-4o
access_key: $OPENAI_PROD_KEY
name: openai-prod
# Development OpenAI (different key/quota)
- model: openai/gpt-4o-mini
access_key: $OPENAI_DEV_KEY
name: openai-dev
Wildcard Model Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Automatically configure all available models from a provider using wildcard patterns. Plano expands wildcards at configuration load time to include all known models from the provider's registry.
**Basic Wildcard Usage:**
.. code-block:: yaml
llm_providers:
# Expand to all OpenAI models
- model: openai/*
access_key: $OPENAI_API_KEY
# Expand to all Anthropic Claude models
- model: anthropic/*
access_key: $ANTHROPIC_API_KEY
# Expand to all Mistral models
- model: mistral/*
access_key: $MISTRAL_API_KEY
**How Wildcards Work:**
1. **Known Providers** (OpenAI, Anthropic, DeepSeek, Mistral, Groq, Gemini, Together AI, xAI, Moonshot, Zhipu, Xiaomi):
- Expands at config load time to all models in Plano's provider registry
- Creates entries for both canonical (``openai/gpt-4``) and short names (``gpt-4``)
- Enables the ``/models/list`` endpoint to list all available models
- **View complete model list**: `provider_models.yaml <../../includes/provider_models.yaml>`_
2. **Unknown/Custom Providers** (e.g., ``custom-provider/*``):
- Stores as a wildcard pattern for runtime matching
- Requires ``base_url`` and ``provider_interface`` configuration
- Matches model requests dynamically (e.g., ``custom-provider/any-model-name``)
- Does not appear in ``/models/list`` endpoint
**Overriding Wildcard Models:**
You can configure specific models with custom settings even when using wildcards. Specific configurations take precedence and are excluded from wildcard expansion:
.. code-block:: yaml
version: v0.4.0
model_providers:
# Expand to all Anthropic models
- model: anthropic/*
access_key: $ANTHROPIC_API_KEY
# Override specific model with custom settings
# This model will NOT be included in the wildcard expansion above
- model: anthropic/claude-sonnet-4-6
access_key: $ANTHROPIC_PROD_API_KEY
# Another specific override
- model: anthropic/claude-3-haiku-20240307
access_key: $ANTHROPIC_DEV_API_KEY
routing_preferences:
- name: code_generation
description: generating new code snippets, functions, or boilerplate based on user prompts or requirements
models:
- anthropic/claude-sonnet-4-6
**Custom Provider Wildcards:**
For providers not in Plano's registry, wildcards enable dynamic model routing:
.. code-block:: yaml
llm_providers:
# Custom LiteLLM deployment
- model: litellm/*
base_url: https://litellm.example.com
provider_interface: openai
passthrough_auth: true
# Custom provider with all models
- model: custom-provider/*
access_key: $CUSTOM_API_KEY
base_url: https://api.custom-provider.com
provider_interface: openai
**Benefits:**
- **Simplified Configuration**: One line instead of listing dozens of models
- **Future-Proof**: Automatically includes new models as they're released
- **Flexible Overrides**: Customize specific models while using wildcards for others
- **Selective Expansion**: Control which models get custom configurations
Default Model Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Mark one model as the default for fallback scenarios:
.. code-block:: yaml
llm_providers:
- model: openai/gpt-4o-mini
access_key: $OPENAI_API_KEY
default: true # Used when no specific model is requested
Routing Preferences
~~~~~~~~~~~~~~~~~~~
Starting in ``v0.4.0``, configure routing preferences at the top level of the config. Each preference declares an ordered ``models`` candidate pool; the first entry is primary and the rest are fallbacks the client tries on ``429``/``5xx`` errors. Multiple providers can serve the same route — just list them all under ``models``. See :doc:`/guides/llm_router` for the full routing model.
.. code-block:: yaml
version: v0.4.0
model_providers:
- model: openai/gpt-5.2
access_key: $OPENAI_API_KEY
- model: anthropic/claude-sonnet-4-5
access_key: $ANTHROPIC_API_KEY
routing_preferences:
- name: complex_reasoning
description: deep analysis, mathematical problem solving, and logical reasoning
models:
- openai/gpt-5.2
- anthropic/claude-sonnet-4-5
- name: code_review
description: reviewing and analyzing existing code for bugs and improvements
models:
- openai/gpt-5.2
- name: creative_writing
description: creative content generation, storytelling, and writing assistance
models:
- anthropic/claude-sonnet-4-5
.. note::
``v0.3.0`` configs that declare ``routing_preferences`` inline under each ``model_provider`` are auto-migrated to this top-level shape by the Plano CLI at compile time, with a deprecation warning. Update to the form above to silence the warning and gain the multi-model fallback behavior.
.. _passthrough_auth:
Passthrough Authentication
~~~~~~~~~~~~~~~~~~~~~~~~~~
When deploying Plano in front of LLM proxy services that manage their own API key validation (such as LiteLLM, OpenRouter, or custom gateways), you may want to forward the client's original ``Authorization`` header instead of replacing it with a configured ``access_key``.
The ``passthrough_auth`` option enables this behavior:
.. code-block:: yaml
llm_providers:
# Forward client's Authorization header to LiteLLM
- model: openai/gpt-4o-litellm
base_url: https://litellm.example.com
passthrough_auth: true
default: true
# Forward to OpenRouter
- model: openai/claude-3-opus
base_url: https://openrouter.ai/api/v1
passthrough_auth: true
**How it works:**
1. Client sends a request with ``Authorization: Bearer ``
2. Plano preserves this header instead of replacing it with ``access_key``
3. The upstream service (e.g., LiteLLM) validates the virtual key
4. Response flows back through Plano to the client
**Use Cases:**
- **LiteLLM Integration**: Route requests to LiteLLM which manages virtual keys and rate limits
- **OpenRouter**: Forward requests to OpenRouter with per-user API keys
- **Custom API Gateways**: Integrate with internal gateways that have their own authentication
- **Multi-tenant Deployments**: Allow different clients to use their own credentials
**Important Notes:**
- When ``passthrough_auth: true`` is set, the ``access_key`` field is ignored (a warning is logged if both are configured)
- If the client doesn't provide an ``Authorization`` header, the request is forwarded without authentication (upstream will likely return 401)
- The ``base_url`` is typically required when using ``passthrough_auth``
**Configuration with LiteLLM example:**
.. code-block:: yaml
# plano_config.yaml
version: v0.3.0
listeners:
- name: llm
type: model
port: 10000
model_providers:
- model: openai/gpt-4o
base_url: https://litellm.example.com
passthrough_auth: true
default: true
.. code-block:: bash
# Client request - virtual key is forwarded to upstream
curl http://localhost:10000/v1/chat/completions \
-H "Authorization: Bearer sk-litellm-virtual-key-abc123" \
-H "Content-Type: application/json" \
-d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'
Model Selection Guidelines
--------------------------
**For Production Applications:**
- **High Performance**: OpenAI GPT-5.2, Anthropic Claude Sonnet 4.5
- **Cost-Effective**: OpenAI GPT-5, Anthropic Claude Haiku 4.5
- **Code Tasks**: DeepSeek Coder, Together AI Code Llama
- **Local Deployment**: Ollama with Llama 3.1 or Code Llama
**For Development/Testing:**
- **Fast Iteration**: Groq models (optimized inference)
- **Local Testing**: Ollama models
- **Cost Control**: Smaller models like GPT-4o or Mistral Small
See Also
--------
- :ref:`client_libraries` - Using different client libraries with providers
- :ref:`model_aliases` - Creating semantic model names
- :ref:`llm_router` - Setting up intelligent routing
- :ref:`client_libraries` - Using different client libraries
- :ref:`model_aliases` - Creating semantic model names
---
### Source/Concepts/Agents
.. _agents:
Agents
======
Agents are autonomous systems that handle wide-ranging, open-ended tasks by calling models in a loop until the work is complete. Unlike deterministic :ref:`prompt targets `, agents have access to tools, reason about which actions to take, and adapt their behavior based on intermediate results—making them ideal for complex workflows that require multi-step reasoning, external API calls, and dynamic decision-making.
Plano helps developers build and scale multi-agent systems by managing the orchestration layer—deciding which agent(s) or LLM(s) should handle each request, and in what sequence—while developers focus on implementing agent logic in any language or framework they choose.
Agent Orchestration
-------------------
**Plano-Orchestrator** is a family of state-of-the-art routing and orchestration models that decide which agent(s) should handle each request, and in what sequence. Built for real-world multi-agent deployments, it analyzes user intent and conversation context to make precise routing and orchestration decisions while remaining efficient enough for low-latency production use across general chat, coding, and long-context multi-turn conversations.
This allows development teams to:
* **Scale multi-agent systems**: Route requests across multiple specialized agents without hardcoding routing logic in application code.
* **Improve performance**: Direct requests to the most appropriate agent based on intent, reducing unnecessary handoffs and improving response quality.
* **Enhance debuggability**: Centralized routing decisions are observable through Plano's tracing and logging, making it easier to understand why a particular agent was selected.
Inner Loop vs. Outer Loop
--------------------------
Plano distinguishes between the **inner loop** (agent implementation logic) and the **outer loop** (orchestration and routing):
Inner Loop (Agent Logic)
^^^^^^^^^^^^^^^^^^^^^^^^^
The inner loop is where your agent lives—the business logic that decides which tools to call, how to interpret results, and when the task is complete. You implement this in any language or framework:
* **Python agents**: Using frameworks like LangChain, LlamaIndex, CrewAI, or custom Python code.
* **JavaScript/TypeScript agents**: Using frameworks like LangChain.js or custom Node.js implementations.
* **Any other AI famreowkr**: Agents are just HTTP services that Plano can route to.
Your agent controls:
* Which tools or APIs to call in response to a prompt.
* How to interpret tool results and decide next steps.
* When to call the LLM for reasoning or summarization.
* When the task is complete and what response to return.
.. note::
**Making LLM Calls from Agents**
When your agent needs to call an LLM for reasoning, summarization, or completion, you should route those calls through Plano's Model Proxy rather than calling LLM providers directly. This gives you:
* **Consistent responses**: Normalized response formats across all :ref:`LLM providers `, whether you're using OpenAI, Anthropic, Azure OpenAI, or any OpenAI-compatible provider.
* **Rich agentic signals**: Automatic capture of function calls, tool usage, reasoning steps, and model behavior—surfaced through traces and metrics without instrumenting your agent code.
* **Smart model routing**: Leverage :ref:`model-based, alias-based, or preference-aligned routing ` to dynamically select the best model for each task based on cost, performance, or custom policies.
By routing LLM calls through the Model Proxy, your agents remain decoupled from specific providers and can benefit from centralized policy enforcement, observability, and intelligent routing—all managed in the outer loop. For a step-by-step guide, see :ref:`llm_router` in the LLM Router guide.
Outer Loop (Orchestration)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
The outer loop is Plano's orchestration layer—it manages the lifecycle of requests across agents and LLMs:
* **Intent analysis**: Plano-Orchestrator analyzes incoming prompts to determine user intent and conversation context.
* **Routing decisions**: Routes requests to the appropriate agent(s) or LLM(s) based on capabilities, context, and availability.
* **Sequencing**: Determines whether multiple agents need to collaborate and in what order.
* **Lifecycle management**: Handles retries, failover, circuit breaking, and load balancing across agent instances.
By managing the outer loop, Plano allows you to:
* Add new agents without changing routing logic in existing agents.
* Run multiple versions or variants of agents for A/B testing or canary deployments.
* Apply consistent :ref:`filter chains ` (guardrails, context enrichment) before requests reach agents.
* Monitor and debug multi-agent workflows through centralized observability.
Key Benefits
------------
* **Language and framework agnostic**: Write agents in any language; Plano orchestrates them via HTTP.
* **Reduced complexity**: Agents focus on task logic; Plano handles routing, retries, and cross-cutting concerns.
* **Better observability**: Centralized tracing shows which agents were called, in what sequence, and why.
* **Easier scaling**: Add more agent instances or new agent types without refactoring existing code.
---
### Source/Concepts/Filter Chain
.. _filter_chain:
Filter Chains
==============
Filter chains are Plano's way of capturing **reusable workflow steps** in the dataplane, without duplication and coupling logic into application code. A filter chain is an ordered list of **mutations** that a request flows through before reaching its final destination —such as an agent, an LLM, or a tool backend. Each filter is a network-addressable service/path that can:
1. Inspect the incoming prompt, metadata, and conversation state.
2. Mutate or enrich the request (for example, rewrite queries or build context).
3. Short-circuit the flow and return a response early (for example, block a request on a compliance failure).
4. Emit structured logs and traces so you can debug and continuously improve your agents.
In other words, filter chains provide a lightweight programming model over HTTP for building reusable steps
in your agent architectures.
Typical Use Cases
-----------------
Without a dataplane programming model, teams tend to spread logic like query rewriting, compliance checks,
context building, and routing decisions across many agents and frameworks. This quickly becomes hard to reason
about and even harder to evolve.
Filter chains show up most often in patterns like:
* **Guardrails and Compliance**: Enforcing content policies, stripping or masking sensitive data, and blocking obviously unsafe or off-topic requests before they reach an agent.
* **Query rewriting, RAG, and Memory**: Rewriting user queries for retrieval, normalizing entities, and assembling RAG context envelopes while pulling in relevant memory (for example, conversation history, user profiles, or prior tool results) before calling a model or tool.
* **Cross-cutting Observability**: Injecting correlation IDs, sampling traces, or logging enriched request metadata at consistent points in the request path.
Because these behaviors live in the dataplane rather than inside individual agents, you define them once, attach them to many agents and prompt targets, and can add, remove, or reorder them without changing application code.
Configuration example
---------------------
Agent listener filter chain
^^^^^^^^^^^^^^^^^^^^^^^^^^^
The example below shows a configuration where an agent uses a filter chain with two filters: a query rewriter,
and a context builder that prepares retrieval context before the agent runs.
.. literalinclude:: ../../source/resources/includes/plano_config_agents_filters.yaml
:language: yaml
:linenos:
:emphasize-lines: 7-14, 37-39
:caption: Example Configuration
In this setup:
* The ``filters`` section defines the reusable filters, each running as its own HTTP/MCP service.
* The ``listeners`` section wires the ``rag_agent`` behind an ``agent`` listener and attaches a ``filter_chain`` with ``query_rewriter`` followed by ``context_builder``.
* When a request arrives at ``agent_1``, Plano executes the filters in order before handing control to ``rag_agent``.
Model listener filter chain
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Filter chains can also be attached directly to a **model listener**. This lets you run input guardrails on
direct LLM proxy requests (``/v1/chat/completions``, ``/v1/responses``, etc.) without an agent layer in between.
.. code-block:: yaml
:caption: Model listener with a content-safety filter chain
filters:
- id: content_guard
url: http://content-guard:10500
type: http
model_providers:
- model: openai/gpt-4o-mini
access_key: $OPENAI_API_KEY
default: true
listeners:
- type: model
name: llm_gateway
port: 12000
filter_chain:
- content_guard
In this setup:
* The ``filter_chain`` is declared at the listener level (not per-agent).
* When a request arrives at the model listener, Plano executes the filters in order before forwarding the request to the upstream LLM provider.
* If a filter rejects the request (HTTP 4xx), the error is returned to the caller and the LLM is never called.
Filter Chain Programming Model (HTTP and MCP)
---------------------------------------------
Filters are implemented as simple RESTful endpoints reachable via HTTP. If you want to use the `Model Context Protocol (MCP) `_, you can configure that as well, which makes it easy to write filters in any language. However, you can also write a filter as a plain HTTP service.
When defining a filter in Plano configuration, the following fields are optional:
* ``type``: Controls the filter runtime. Use ``mcp`` for Model Context Protocol filters, or ``http`` for plain HTTP filters. Defaults to ``mcp``.
* ``transport``: Controls how Plano talks to the filter (defaults to ``streamable-http`` for efficient streaming interactions over HTTP). You can omit this for standard HTTP transport.
* ``tool``: Names the MCP tool Plano will invoke (by default, the filter ``id``). You can omit this if the tool name matches your filter id.
In practice, you typically only need to specify ``id`` and ``url`` to get started. Plano's sensible defaults mean a filter can be as simple as an HTTP endpoint. If you want to customize the runtime or protocol, those fields are there, but they're optional.
Filters communicate the outcome of their work via HTTP status codes:
* **HTTP 200 (Success)**: The filter successfully processed the request. If the filter mutated the request (e.g., rewrote a query or enriched context), those mutations are passed downstream.
* **HTTP 4xx (User Error)**: The request violates a filter's rules or constraints—for example, content moderation policies or compliance checks. The request is terminated, and the error is returned to the caller. This is *not* a fatal error; it represents expected user-facing policy enforcement.
* **HTTP 5xx (Fatal Error)**: An unexpected failure in the filter itself (for example, a crash or misconfiguration). Plano will surface the error back to the caller and record it in logs and traces.
This semantics allows filters to enforce guardrails and policies (4xx) without blocking the entire system, while still surfacing critical failures (5xx) for investigation.
If any filter fails or decides to terminate the request early (for example, after a policy violation), Plano will
surface that outcome back to the caller and record it in logs and traces. This makes filter chains a safe and
powerful abstraction for evolving your agent workflows over time.
---
### Source/Concepts/Listeners
.. _plano_overview_listeners:
Listeners
---------
**Listeners** are a top-level primitive in Plano that bind network traffic to the dataplane. They simplify the
configuration required to accept incoming connections from downstream clients (edge) and to expose a unified egress
endpoint for calls from your applications to upstream LLMs.
Plano builds on Envoy's Listener subsystem to streamline connection management for developers. It hides most of
Envoy's complexity behind sensible defaults and a focused configuration surface, so you can bind listeners without
deep knowledge of Envoy’s configuration model while still getting secure, reliable, and performant connections.
Listeners are modular building blocks: you can configure only inbound listeners (for edge proxying and guardrails),
only outbound/model-proxy listeners (for LLM routing from your services), or both together. This lets you fit Plano
cleanly into existing architectures, whether you need it at the edge, behind the firewall, or across the full
request path.
Network Topology
^^^^^^^^^^^^^^^^
The diagram below shows how inbound and outbound traffic flow through Plano and how listeners relate to agents,
prompt targets, and upstream LLMs:
.. image:: /_static/img/network-topology-ingress-egress.png
:width: 100%
:align: center
Inbound (Agent & Prompt Target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Developers configure **inbound listeners** to accept connections from clients such as web frontends, backend
services, or other gateways. An inbound listener acts as the primary entry point for prompt traffic, handling
initial connection setup, TLS termination, guardrails, and forwarding incoming traffic to the appropriate prompt
targets or agents.
There are two primary types of inbound connections exposed via listeners:
* **Agent Inbound (Edge)**: Clients (web/mobile apps or other services) connect to Plano, send prompts, and receive
responses. This is typically your public/edge listener where Plano applies guardrails, routing, and orchestration
before returning results to the caller.
* **Prompt Target Inbound (Edge)**: Your application server calls Plano's internal listener targeting
:ref:`prompt targets ` that can invoke tools and LLMs directly on its behalf.
Inbound listeners are where you attach :ref:`Filter Chains ` so that safety and context-building happen
consistently at the edge.
Outbound (Model Proxy & Egress)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Plano also exposes an **egress listener** that your applications call when sending requests to upstream LLM providers
or self-hosted models. From your application's perspective this looks like a single OpenAI-compatible HTTP endpoint
(for example, ``http://127.0.0.1:12000/v1``), while Plano handles provider selection, retries, and failover behind
the scenes.
Under the hood, Plano opens outbound HTTP(S) connections to upstream LLM providers using its unified API surface and
smart model routing. For more details on how Plano talks to models and how providers are configured, see
:ref:`LLM providers `.
Model listeners also support :ref:`Filter Chains `. By adding a ``filter_chain`` to a model listener
you can run input guardrails, content-safety checks, or other preprocessing on direct LLM requests before they reach
the upstream provider — without requiring an agent layer.
Configure Listeners
^^^^^^^^^^^^^^^^^^^
Listeners are configured via the ``listeners`` block in your Plano configuration. You can define one or more inbound
listeners (for example, ``type:edge``) or one or more outbound/model listeners (for example, ``type:model``), or both
in the same deployment.
To configure an inbound (edge) listener, add a ``listeners`` block to your configuration file and define at least one
listener with address, port, and protocol details:
.. literalinclude:: ./includes/plano_config.yaml
:language: yaml
:linenos:
:lines: 1-13
:emphasize-lines: 3-7
:caption: Example Configuration
When you start Plano, you specify a listener address/port that you want to bind downstream. Plano also exposes a
predefined internal listener (``127.0.0.1:12000``) that you can use to proxy egress calls originating from your
application to LLMs (API-based or hosted) via prompt targets.
---
### Source/Concepts/Prompt Target
.. _prompt_target:
Prompt Target
=============
.. deprecated:: v0.4.22
**Prompt Targets are deprecated and no longer actively maintained.** This concept is
retained for existing users on older Plano configurations, but new applications should
not adopt it. For deterministic, task-specific workloads, use :ref:`Agents `
together with :ref:`Function Calling ` instead. The
``prompt_targets`` configuration block and related CLI commands will continue to
function for now, but may be removed in a future release.
A Prompt Target is a deterministic, task-specific backend function or API endpoint that your application calls via Plano.
Unlike agents (which handle wide-ranging, open-ended tasks), prompt targets are designed for focused, specific workloads where Plano can add value through input clarification and validation.
Plano helps by:
* **Clarifying and validating input**: Plano enriches incoming prompts with metadata (e.g., detecting follow-ups or clarifying requests) and can extract structured parameters from natural language before passing them to your backend.
* **Enabling high determinism**: Since the task is specific and well-defined, Plano can reliably extract the information your backend needs without ambiguity.
* **Reducing backend work**: Your backend receives clean, validated, structured inputs—so you can focus on business logic instead of parsing and validation.
For example, a prompt target might be "schedule a meeting" (specific task, deterministic inputs like date, time, attendees) or "retrieve documents" (well-defined RAG query with clear intent). Prompt targets are typically called from your application code via Plano's internal listener.
.. table::
:width: 100%
==================== ============================================
**Capability** **Description**
==================== ============================================
Intent Recognition Identify the purpose of a user prompt.
Parameter Extraction Extract necessary data from the prompt.
Invocation Call relevant backend agents or tools (APIs).
Response Handling Process and return responses to the user.
==================== ============================================
Key Features
~~~~~~~~~~~~
Below are the key features of prompt targets that empower developers to build efficient, scalable, and personalized GenAI solutions:
- **Design Scenarios**: Define prompt targets to effectively handle specific agentic scenarios.
- **Input Management**: Specify required and optional parameters for each target.
- **Tools Integration**: Seamlessly connect prompts to backend APIs or functions.
- **Error Handling**: Direct errors to designated handlers for streamlined troubleshooting.
- **Multi-Turn Support**: Manage follow-up prompts and clarifications in conversational flows.
Basic Configuration
~~~~~~~~~~~~~~~~~~~
Configuring prompt targets involves defining them in Plano's configuration file. Each Prompt target specifies how a particular type of prompt should be handled, including the endpoint to invoke and any parameters required. A prompt target configuration includes the following elements:
.. vale Vale.Spelling = NO
- ``name``: A unique identifier for the prompt target.
- ``description``: A brief explanation of what the prompt target does.
- ``endpoint``: Required if you want to call a tool or specific API. ``name`` and ``path`` ``http_method`` are the three attributes of the endpoint.
- ``parameters`` (Optional): A list of parameters to extract from the prompt.
.. _defining_prompt_target_parameters:
Defining Parameters
~~~~~~~~~~~~~~~~~~~
Parameters are the pieces of information that Plano needs to extract from the user's prompt to perform the desired action.
Each parameter can be marked as required or optional. Here is a full list of parameter attributes that Plano can support:
.. table::
:width: 100%
======================== ============================================================================
**Attribute** **Description**
======================== ============================================================================
``name (req.)`` Specifies name of the parameter.
``description (req.)`` Provides a human-readable explanation of the parameter's purpose.
``type (req.)`` Specifies the data type. Supported types include: **int**, **str**, **float**, **bool**, **list**, **set**, **dict**, **tuple**
``in_path`` Indicates whether the parameter is part of the path in the endpoint url. Valid values: **true** or **false**
``default`` Specifies a default value for the parameter if not provided by the user.
``format`` Specifies a format for the parameter value. For example: `2019-12-31` for a date value.
``enum`` Lists of allowable values for the parameter with data type matching the ``type`` attribute. **Usage Example**: ``enum: ["celsius`", "fahrenheit"]``
``items`` Specifies the attribute of the elements when type equals **list**, **set**, **dict**, **tuple**. **Usage Example**: ``items: {"type": "str"}``
``required`` Indicates whether the parameter is mandatory or optional. Valid values: **true** or **false**
======================== ============================================================================
Example Configuration For Tools
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: yaml
:caption: Tools and Function Calling Configuration Example
prompt_targets:
- name: get_weather
description: Get the current weather for a location
parameters:
- name: location
description: The city and state, e.g. San Francisco, New York
type: str
required: true
- name: unit
description: The unit of temperature
type: str
default: fahrenheit
enum: [celsius, fahrenheit]
endpoint:
name: api_server
path: /weather
.. _plano_multi_turn_guide:
Multi-Turn
~~~~~~~~~~
Developers often `struggle `_ to efficiently handle
``follow-up`` or ``clarification`` questions. Specifically, when users ask for changes or additions to previous responses, it requires developers to
re-write prompts using LLMs with precise prompt engineering techniques. This process is slow, manual, error prone and adds latency and token cost for
common scenarios that can be managed more efficiently.
Plano is highly capable of accurately detecting and processing prompts in multi-turn scenarios so that you can buil fast and accurate agents in minutes.
Below are some cnversational examples that you can build via Plano. Each example is enriched with annotations (via ** [Plano] ** ) that illustrates how Plano
processess conversational messages on your behalf.
Example 1: Adjusting Retrieval
.. code-block:: text
User: What are the benefits of renewable energy?
**[Plano]**: Check if there is an available that can handle this user query.
**[Plano]**: Found "get_info_for_energy_source" prompt_target in plano_config.yaml. Forward prompt to the endpoint configured in "get_info_for_energy_source"
...
Assistant: Renewable energy reduces greenhouse gas emissions, lowers air pollution, and provides sustainable power sources like solar and wind.
User: Include cost considerations in the response.
**[Plano]**: Follow-up detected. Forward prompt history to the "get_info_for_energy_source" prompt_target and post the following parameters consideration="cost"
...
Assistant: Renewable energy reduces greenhouse gas emissions, lowers air pollution, and provides sustainable power sources like solar and wind. While the initial setup costs can be high, long-term savings from reduced fuel expenses and government incentives make it cost-effective.
Example 2: Switching Intent
---------------------------
.. code-block:: text
User: What are the symptoms of diabetes?
**[Plano]**: Check if there is an available that can handle this user query.
**[Plano]**: Found "diseases_symptoms" prompt_target in plano_config.yaml. Forward disease=diabeteres to "diseases_symptoms" prompt target
...
Assistant: Common symptoms include frequent urination, excessive thirst, fatigue, and blurry vision.
User: How is it diagnosed?
**[Plano]**: New intent detected.
**[Plano]**: Found "disease_diagnoses" prompt_target in plano_config.yaml. Forward disease=diabeteres to "disease_diagnoses" prompt target
...
Assistant: Diabetes is diagnosed through blood tests like fasting blood sugar, A1C, or an oral glucose tolerance test.
Build Multi-Turn RAG Apps
-------------------------
The following section describes how you can easilly add support for multi-turn scenarios via Plano. You process and manage multi-turn prompts
just like you manage single-turn ones. Plano handles the conpleixity of detecting the correct intent based on the last user prompt and
the covnersational history, extracts relevant parameters needed by downstream APIs, and dipatches calls to any upstream LLMs to summarize the
response from your APIs.
.. _multi_turn_subsection_prompt_target:
Step 1: Define Plano Config
---------------------------
.. literalinclude:: ../build_with_plano/includes/multi_turn/prompt_targets_multi_turn.yaml
:language: yaml
:caption: Plano Config
:linenos:
Step 2: Process Request in Flask
--------------------------------
Once the prompt targets are configured as above, handle parameters across multi-turn as if its a single-turn request
.. literalinclude:: ../build_with_plano/includes/multi_turn/multi_turn_rag.py
:language: python
:caption: Parameter handling with Flask
:linenos:
Demo App
--------
For your convenience, we've built a `demo app `_
that you can test and modify locally for multi-turn RAG scenarios.
.. figure:: ../build_with_plano/includes/multi_turn/mutli-turn-example.png
:width: 100%
:align: center
Example multi-turn user conversation showing adjusting retrieval
Summary
~~~~~~~
By carefully designing prompt targets as deterministic, task-specific entry points, you ensure that prompts are routed to the right workload, necessary parameters are cleanly extracted and validated, and backend services are invoked with structured inputs. This clear separation between prompt handling and business logic simplifies your architecture, makes behavior more predictable and testable, and improves the scalability and maintainability of your agentic applications.
---
### Source/Concepts/Signals
.. -*- coding: utf-8 -*-
========
Signals™
========
Agentic Signals are lightweight, model-free behavioral indicators computed
from live interaction trajectories and attached to your existing
OpenTelemetry traces. They are the instrumentation layer of a closed-loop
improvement flywheel for agents — turning raw production traffic into
prioritized data that can drive prompt, routing, and model updates without
running an LLM-as-judge on every session.
The framework implemented here follows the taxonomy and detector design in
*Signals: Trajectory Sampling and Triage for Agentic Interactions*
(`Chen et al., 2026 `_). All detectors
are computed without model calls; the entire pipeline attaches structured
attributes and span events to existing spans so your dashboards and alerts
work unmodified.
Why Signals Matter: The Improvement Flywheel
============================================
Agentic applications are increasingly deployed at scale, yet improving them
after deployment remains difficult. Production trajectories are long,
numerous, and non-deterministic, making exhaustive human review infeasible
and auxiliary LLM evaluation expensive. As a result, teams face a
bottleneck: they cannot score every response, inspect every trace, or
reliably identify which failures and successes should inform the next model
update. Without a low-cost triage layer, the feedback loop from production
behavior to model improvement remains incomplete.
Signals close this loop by cheaply identifying which interactions among
millions are worth inspecting:
1. **Instrument.** Live trajectories are scored with model-free signals
attached as structured attributes on existing OpenTelemetry spans,
organized under a fixed taxonomy of interaction, execution, and
environment signals. This requires no additional model calls,
infrastructure, or changes to online agent behavior.
2. **Sample & triage.** Signal attributes act as filters: they surface
severe failures, retrieve representative exemplars, and exclude the
uninformative middle. In our experiments, signal-based sampling
achieves 82% informativeness on :math:`\tau`-bench, compared with 54%
for random sampling, yielding a 1.52× efficiency gain per informative
trajectory.
3. **Data Construction.** The triaged subset becomes targeted input for
constructing preference datasets or supervised fine-tuning datasets
from production trajectories.
4. **Model Optimization.** The resulting preference or supervised
fine-tuning data is used to update the model through methods such as
DPO, RLHF, or supervised fine-tuning, so optimization is driven by
targeted production behavior rather than undifferentiated trace noise.
5. **Deploy.** The improved model is deployed and immediately
re-instrumented with the same signals, enabling teams to measure
whether the change improved production behavior and to feed the next
iteration.
This loop depends on the first step being nearly free. The framework is
therefore designed around fixed-taxonomy, model-free detectors with
:math:`O(\text{messages})` cost, no online behavior change, and no
dependence on expensive evaluator models. By making production traces
searchable and sampleable at scale, signals turn raw agent telemetry into a
practical model-optimization flywheel.
What Are Behavioral Signals?
============================
Behavioral signals are canaries in the coal mine — early, objective
indicators that something may have gone wrong (or gone exceptionally well).
They don't explain *why* an agent failed, but they reliably signal *where*
attention is needed.
These signals emerge naturally from the rhythm of interaction:
- A user rephrasing or correcting the same request
- Sharp increases in conversation length
- Negative stance markers ("this doesn't work", ALL CAPS, excessive !!! or ???)
- Agent repetition or tool-call loops
- Expressions of gratitude, confirmation, or task success
- Requests for a human agent or explicit quit intent
- Tool errors, timeouts, rate limits, and context-window exhaustion
Individually, these clues are shallow; together, they form a fingerprint of
agent performance. Embedded directly into traces, they make it easy to spot
friction as it happens: where users struggle, where agents loop, where tool
failures cluster, and where escalations occur.
Signal Taxonomy
===============
Signals are organized into three top-level **layers**, each with its own
intent. Every detected signal belongs to exactly one leaf type under one of
seven categories. The per-category summaries and leaf-type descriptions
below are borrowed verbatim from the reference implementation at
`katanemo/signals `_ to keep the
documentation and the detector contract in sync.
Interaction — user ↔ agent conversational quality
-------------------------------------------------
**Misalignment** — Misalignment signals capture semantic or intent mismatch
between the user and the agent, such as rephrasing, corrections,
clarifications, and restated constraints. These signals do not assert that
either party is "wrong"; they only indicate that shared understanding has
not yet been established.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Leaf signal type
- Description
* - ``misalignment.correction``
- Explicit corrections, negations, mistake acknowledgments.
* - ``misalignment.rephrase``
- Rephrasing indicators, alternative explanations.
* - ``misalignment.clarification``
- Confusion expressions, requests for clarification.
**Stagnation** — Stagnation signals capture cases where the discourse
continues but fails to make visible progress. This includes near-duplicate
assistant responses, circular explanations, repeated scaffolding, and other
forms of linguistic degeneration.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Leaf signal type
- Description
* - ``stagnation.dragging``
- Excessive turn count, conversation not progressing efficiently.
* - ``stagnation.repetition``
- Near-duplicate or repetitive assistant responses.
**Disengagement** — Disengagement signals mark the withdrawal of
cooperative intent from the interaction. These include explicit requests to
exit the agent flow (e.g., "talk to a human"), strong negative stances, and
abandonment markers.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Leaf signal type
- Description
* - ``disengagement.escalation``
- Requests for human agent or support.
* - ``disengagement.quit``
- Notification to quit or leave.
* - ``disengagement.negative_stance``
- Complaints, frustration, negative sentiment.
**Satisfaction** — Satisfaction signals indicate explicit stabilization and
completion of the interaction. These include expressions of gratitude,
success confirmations, and closing utterances. We use these signals to
sample exemplar traces rather than to assign quality scores.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Leaf signal type
- Description
* - ``satisfaction.gratitude``
- Expressions of thanks and appreciation.
* - ``satisfaction.confirmation``
- Explicit satisfaction expressions.
* - ``satisfaction.success``
- Confirmation of task completion or understanding.
Execution — agent-caused action quality
---------------------------------------
**Failure** — Detects agent-caused failures in tool/function usage. These
are issues the agent is responsible for (as opposed to environment failures
which are external system issues). Requires tool-call traces
(``function_call`` / ``observation``) to fire.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Leaf signal type
- Description
* - ``execution.failure.invalid_args``
- Wrong type, missing required field.
* - ``execution.failure.bad_query``
- Empty results due to overly narrow/wrong query.
* - ``execution.failure.tool_not_found``
- Agent called non-existent tool.
* - ``execution.failure.auth_misuse``
- Agent didn't pass credentials correctly.
* - ``execution.failure.state_error``
- Tool called in wrong state/order.
**Loops** — Detects behavioral patterns where the agent gets stuck
repeating tool calls. These are distinct from
``interaction.stagnation`` (conversation text repetition) and
``execution.failure`` (single tool errors) — these detect tool-level
behavioral loops.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Leaf signal type
- Description
* - ``execution.loops.retry``
- Same tool with identical args ≥3 times.
* - ``execution.loops.parameter_drift``
- Same tool with varied args ≥3 times.
* - ``execution.loops.oscillation``
- Multi-tool A→B→A→B pattern ≥3 cycles.
Environment — external system / boundary conditions
---------------------------------------------------
**Exhaustion** — Detects failures and constraints arising from the
surrounding system rather than the agent's internal policy or reasoning.
These are external issues the agent cannot control.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Leaf signal type
- Description
* - ``environment.exhaustion.api_error``
- 5xx errors, service unavailable.
* - ``environment.exhaustion.timeout``
- Connection/read timeouts.
* - ``environment.exhaustion.rate_limit``
- 429, quota exceeded.
* - ``environment.exhaustion.network``
- Connection refused, DNS errors.
* - ``environment.exhaustion.malformed_response``
- Invalid JSON, unexpected schema.
* - ``environment.exhaustion.context_overflow``
- Token/context limit exceeded.
How It Works
============
Signals are computed automatically by the gateway after each assistant
response and emitted as **OpenTelemetry trace attributes** and **span events**
on your existing spans. No additional libraries or instrumentation are
required — just configure your OTEL collector endpoint as usual.
Each conversation trace is enriched with layered signal attributes
(category-level counts and severities) plus one span event per detected
signal instance (with confidence, snippet, and per-detector metadata).
.. note::
Signal analysis is enabled by default and runs on the request path. It
does **not** affect the response sent to the client. Set
``overrides.disable_signals: true`` in your Plano config to skip this
CPU-heavy analysis (see the configuration reference).
OTel Span Attributes
====================
Signal data is exported as structured OTel attributes. There are two tiers:
**top-level** attributes (always emitted on spans that carry signal
analysis) and **layered** attributes (emitted only when the corresponding
category has at least one signal instance).
Top-level attributes
--------------------
Always emitted once signals are computed.
.. list-table::
:header-rows: 1
:widths: 40 15 45
* - Attribute
- Type
- Value
* - ``signals.quality``
- string
- One of ``excellent``, ``good``, ``neutral``, ``poor``, ``severe``.
* - ``signals.quality_score``
- float
- Numeric score 0.0 – 100.0 that feeds the quality bucket.
* - ``signals.turn_count``
- int
- Total number of user + assistant turns in the interaction.
* - ``signals.efficiency_score``
- float
- Efficiency metric 0.0 – 1.0 (stays at 1.0 up to baseline turns,
then decays: ``1 / (1 + 0.3 * (turns - baseline))``).
Layered attributes
------------------
Emitted per category, only when ``count > 0``. One ``.count`` and one
``.severity`` attribute per category. Severity is a 0–3 bucket (see
`Severity levels`_ below).
.. list-table::
:header-rows: 1
:widths: 50 50
* - Attribute (emitted when fired)
- Source
* - ``signals.interaction.misalignment.count``
- Any ``misalignment.*`` leaf type
* - ``signals.interaction.misalignment.severity``
- "
* - ``signals.interaction.stagnation.count``
- Any ``stagnation.*`` leaf type
* - ``signals.interaction.stagnation.severity``
- "
* - ``signals.interaction.disengagement.count``
- Any ``disengagement.*`` leaf type
* - ``signals.interaction.disengagement.severity``
- "
* - ``signals.interaction.satisfaction.count``
- Any ``satisfaction.*`` leaf type
* - ``signals.interaction.satisfaction.severity``
- "
* - ``signals.execution.failure.count``
- Any ``failure.*`` leaf type
* - ``signals.execution.failure.severity``
- "
* - ``signals.execution.loops.count``
- Any ``loops.*`` leaf type
* - ``signals.execution.loops.severity``
- "
* - ``signals.environment.exhaustion.count``
- Any ``exhaustion.*`` leaf type
* - ``signals.environment.exhaustion.severity``
- "
Span Events
===========
In addition to span attributes, every detected signal instance is emitted as
a span event named ``signal.`` (e.g.
``signal.interaction.satisfaction.gratitude``). Each event carries:
.. list-table::
:header-rows: 1
:widths: 30 15 55
* - Event attribute
- Type
- Description
* - ``signal.type``
- string
- Full dotted signal type (same as the event name suffix).
* - ``signal.message_index``
- int
- Zero-based index of the message that triggered the signal.
* - ``signal.confidence``
- float
- Detector confidence in [0.0, 1.0].
* - ``signal.snippet``
- string
- Matched substring from the source message (when available).
* - ``signal.metadata``
- string (JSON)
- Per-detector metadata (pattern name, ratio values, etc.).
Span events are the right surface for drill-down: attribute filters narrow
traces, then events tell you *which messages* fired *which signals* with
*what evidence*.
Visual Flag Marker
------------------
When concerning signals are detected (disengagement present, stagnation
count > 2, any execution failure / loop, or overall quality ``poor``/
``severe``), the marker 🚩 (U+1F6A9) is appended to the span's operation
name.
This makes flagged sessions immediately visible in trace UIs without
requiring attribute filtering.
Querying in Your Observability Platform
---------------------------------------
Example queries against the layered keys::
signals.quality = "severe"
signals.turn_count > 10
signals.efficiency_score < 0.5
signals.interaction.disengagement.severity >= 2
signals.interaction.misalignment.count > 3
signals.interaction.satisfaction.count > 0 AND signals.quality = "good"
signals.execution.failure.count > 0
signals.environment.exhaustion.count > 0
For flagged sessions, search for 🚩 in span names.
.. image:: /_static/img/signals_trace.png
:width: 100%
:align: center
Severity Levels
===============
Every category aggregates its leaf signal counts into a severity bucket used
by both the layered ``.severity`` attribute and the overall quality score.
- **None (0)**: 0 instances
- **Mild (1)**: 1–2 instances
- **Moderate (2)**: 3–4 instances
- **Severe (3)**: 5+ instances
Severity is always computed per-category. For example, three instances of
``misalignment.rephrase`` plus two of ``misalignment.correction`` yield
``signals.interaction.misalignment.severity = 3`` (5 instances total).
Overall Quality Assessment
==========================
Signals are aggregated into an overall interaction quality on a 5-point
scale. The scoring model starts at 50.0 (neutral), adds positive weight for
satisfaction, and subtracts weight for disengagement, misalignment (when
ratio > 30% of user turns), stagnation (when count > 2), execution failures,
execution loops, and environment exhaustion.
The resulting numeric score maps to the bucket emitted in ``signals.quality``:
**Excellent (75 – 100)**
Strong positive signals, efficient resolution, low friction.
**Good (60 – 74)**
Mostly positive with minor clarifications; some back-and-forth but
successful.
**Neutral (40 – 59)**
Mixed signals; neither clearly good nor bad.
**Poor (25 – 39)**
Concerning negative patterns (high friction, multiple misalignments,
moderate disengagement, tool failures). High abandonment risk.
**Severe (0 – 24)**
Critical issues — escalation requested, severe disengagement, severe
stagnation, or compounding failures. Requires immediate attention.
The raw numeric score is available under ``signals.quality_score``.
Sampling and Prioritization
===========================
In production, trace data is overwhelming. Signals provide a lightweight
first layer of triage to select the small fraction of trajectories that are
most likely to be informative. Per the paper, signal-based sampling reaches
82% informativeness on τ-bench versus 54% for random sampling — a 1.52×
efficiency gain per informative trajectory.
Workflow:
1. Gateway captures conversation messages and computes signals
2. Signal attributes and per-instance events are emitted to OTEL spans
3. Your observability platform ingests and indexes the attributes
4. Query / filter by signal attributes to surface outliers and exemplars
5. Review high-information traces to identify improvement opportunities
6. Update prompts, routing, or policies based on findings
7. Redeploy and monitor signal metrics to validate improvements
This creates a reinforcement loop where traces become both diagnostic data
and training signal for prompt engineering, routing policies, and
preference-data construction.
.. note::
An in-gateway triage sampler that selects informative trajectories
inline — with configurable per-category weights and budgets — is planned
as a follow-up to this release. Today, sampling is consumer-side: your
observability platform filters on the signal attributes described above.
Example Span
============
A concerning session, showing both layered attributes and a per-instance
event::
# Span name: "POST /v1/chat/completions gpt-5.2 🚩"
# Top-level
signals.quality = "severe"
signals.quality_score = 0.0
signals.turn_count = 4
signals.efficiency_score = 1.0
# Layered (only non-zero categories are emitted)
signals.interaction.disengagement.count = 6
signals.interaction.disengagement.severity = 3
# Per-instance span events
event: signal.interaction.disengagement.escalation
signal.type = "interaction.disengagement.escalation"
signal.message_index = 6
signal.confidence = 1.0
signal.snippet = "get me a human"
signal.metadata = {"pattern_type":"escalation"}
Building Dashboards
===================
Use signal attributes to build monitoring dashboards in Grafana, Honeycomb,
Datadog, etc. The layered keys align with the paper taxonomy.
- **Quality distribution**: Count of traces by ``signals.quality``
- **P95 turn count**: 95th percentile of ``signals.turn_count``
- **Average efficiency**: Mean of ``signals.efficiency_score``
- **High misalignment rate**: Percentage where
``signals.interaction.misalignment.count > 3``
- **Disengagement rate**: Percentage where
``signals.interaction.disengagement.severity >= 2``
- **Satisfaction rate**: Percentage where
``signals.interaction.satisfaction.count >= 1``
- **Escalation rate**: Percentage where a ``disengagement.escalation`` or
``disengagement.quit`` event fired (via span-event filter)
- **Tool-failure rate**: Percentage where
``signals.execution.failure.count > 0``
- **Environment issue rate**: Percentage where
``signals.environment.exhaustion.count > 0``
Creating Alerts
===============
Set up alerts based on signal thresholds:
- Alert when ``signals.quality = "severe"`` count exceeds threshold in a
1-hour window
- Alert on sudden spike in
``signals.interaction.disengagement.severity >= 2`` (>2× baseline)
- Alert on sustained ``signals.execution.failure.count > 0`` — agent-caused
tool issues
- Alert on spikes in ``signals.environment.exhaustion.count`` — external
system degradation
- Alert on degraded efficiency (P95 ``signals.turn_count`` up > 50%)
Best Practices
==============
Start simple:
- Alert or page on ``severe`` sessions (or on spikes in ``severe`` rate)
- Review ``poor`` sessions within 24 hours
- Sample ``excellent`` sessions as exemplars
Combine multiple signals to infer failure modes:
- **Silent loop**: ``signals.interaction.stagnation.severity >= 2`` +
``signals.turn_count`` above baseline
- **User giving up**: ``signals.interaction.disengagement.severity >= 2`` +
any escalation event
- **Misunderstood intent**:
``signals.interaction.misalignment.count / user_turns > 0.3``
- **Agent-caused friction**: ``signals.execution.failure.count > 0`` +
``signals.interaction.misalignment.count > 0``
- **External degradation, not agent fault**:
``signals.environment.exhaustion.count > 0`` while
``signals.execution.failure.count = 0``
- **Working well**: ``signals.interaction.satisfaction.count >= 1`` +
``signals.efficiency_score > 0.8`` + no disengagement
Limitations and Considerations
==============================
Signals don't capture:
- Task completion / real outcomes
- Factual or domain correctness
- Silent abandonment (user leaves without expressing frustration)
- Non-English nuance (pattern libraries are English-oriented)
Mitigation strategies:
- Periodically sample flagged sessions and measure false positives / negatives
- Tune baselines per use case and user population
- Add domain-specific phrase libraries where needed
- Combine signals with non-text metrics (tool failures, disconnects, latency)
.. note::
Behavioral signals complement — but do not replace — domain-specific
response quality evaluation. Use signals to prioritize which traces to
inspect, then apply domain expertise and outcome checks to diagnose root
causes.
.. tip::
The 🚩 marker in the span name provides instant visual feedback in
trace UIs, while the structured attributes (``signals.quality``,
``signals.interaction.disengagement.severity``, etc.) and per-instance
span events enable powerful querying and drill-down in your observability
platform.
See Also
========
- `Signals: Trajectory Sampling and Triage for Agentic Interactions
`_ — the paper this framework implements
- :doc:`../guides/observability/tracing` — Distributed tracing for agent
systems
- :doc:`../guides/observability/monitoring` — Metrics and dashboards
- :doc:`../guides/observability/access_logging` — Request / response logging
- :doc:`../guides/observability/observability` — Complete observability guide
---
### Source/Get Started/Intro To Plano
.. _intro_to_plano:
Intro to Plano
==============
Building agentic demos is easy. Delivering agentic applications safely, reliably, and repeatably to production is hard. After a quick hack, you end up building the "hidden AI middleware" to reach production: routing logic to reach the right agent, guardrail hooks for safety and moderation, evaluation and observability glue for continuous learning, and model/provider quirks — scattered across frameworks and application code.
Plano solves this by moving core delivery concerns into a unified, out-of-process dataplane. Core capabilities:
- **🚦 Orchestration:** Low-latency orchestration between agents, and add new agents without changing app code. When routing lives inside app code, it becomes hard to evolve and easy to duplicate. Moving orchestration into a centrally managed dataplane lets you change strategies without touching your agents, improving performance and reducing maintenance burden while avoiding tight coupling.
- **🛡️ Guardrails & Memory Hooks:** Apply jailbreak protection, content policies, and context workflows (e.g., rewriting, retrieval, redaction) once via :ref:`Filter Chains ` at the dataplane. Instead of re-implementing these in every agentic service, you get centralized governance, reduced code duplication, and consistent behavior across your stack.
- **🔗 Model Agility:** Route by model, alias (semantic names), or automatically via preferences so agents stay decoupled from specific providers. Swap or add models without refactoring prompts, tool-calling, or streaming handlers throughout your codebase by using Plano's smart routing and unified API.
- **🕵 Agentic Signals™:** Zero-code capture of behavior signals, traces, and metrics consistently across every agent. Rather than stitching together logging and metrics per framework, Plano surfaces traces, token usage, and learning signals in one place so you can iterate safely.
Built by core contributors to the widely adopted Envoy Proxy _, Plano gives you a production‑grade foundation for agentic applications. It helps **developers** stay focused on the core logic of their agents, helps **product teams** shorten feedback loops for learning, and helps **engineering teams** standardize policy and safety across agents and LLMs. Plano is grounded in open protocols (de facto: OpenAI‑style v1/responses, de jure: MCP) and proven patterns like sidecar deployments, so it plugs in cleanly while remaining robust, scalable, and flexible.
In practice, achieving the above goal is incredibly difficult. Plano attempts to do so by providing the following high level features:
.. figure:: /_static/img/plano_network_diagram_high_level.png
:width: 100%
:align: center
High-level network flow of where Plano sits in your agentic stack. Designed for both ingress and egress prompt traffic.
**Engineered with Task-Specific LLMs (TLMs):** Plano is engineered with specialized LLMs that are designed for fast, cost-effective and accurate handling of prompts.
These LLMs are designed to be best-in-class for critical tasks like:
* **Agent Orchestration:** `Plano-Orchestrator `_ is a family of state-of-the-art routing and orchestration models that decide which agent(s) or LLM(s) should handle each request, and in what sequence. Built for real-world multi-agent deployments, it analyzes user intent and conversation context to make precise routing and orchestration decisions while remaining efficient enough for low-latency production use across general chat, coding, and long-context multi-turn conversations.
* **Function Calling:** Plano lets you expose application-specific (API) operations as tools so that your agents can update records, fetch data, or trigger determininistic workflows via prompts. Under the hood this is backed by Arch-Function-Chat; for more details, read :ref:`Function Calling `.
* **Guardrails:** Plano helps you improve the safety of your application by applying prompt guardrails in a centralized way for better governance hygiene.
With prompt guardrails you can prevent ``jailbreak attempts`` present in user's prompts without having to write a single line of code.
To learn more about how to configure guardrails available in Plano, read :ref:`Prompt Guard `.
**Model Proxy:** Plano offers several capabilities for LLM calls originating from your applications, including smart retries on errors from upstream LLMs and automatic cut-over to other LLMs configured in Plano for continuous availability and disaster recovery scenarios. From your application's perspective you keep using an OpenAI-compatible API, while Plano owns resiliency and failover policies in one place.
Plano extends Envoy's `cluster subsystem `_ to manage upstream connections to LLMs so that you can build resilient, provider-agnostic AI applications.
**Edge Proxy:** There is substantial benefit in using the same software at the edge (observability, traffic shaping algorithms, applying guardrails, etc.) as for outbound LLM inference use cases. Plano has the feature set that makes it exceptionally well suited as an edge gateway for AI applications.
This includes TLS termination, applying guardrails early in the request flow, and intelligently deciding which agent(s) or LLM(s) should handle each request and in what sequence. In practice, you configure listeners and policies once, and every inbound and outbound call flows through the same hardened gateway.
**Zero-Code Agent Signals™ & Tracing:** Zero-code capture of behavior signals, traces, and metrics consistently across every agent. Plano propagates trace context using the W3C Trace Context standard, specifically through the ``traceparent`` header. This allows each component in the system to record its part of the request flow, enabling end-to-end tracing across the entire application. By using OpenTelemetry, Plano ensures that developers can capture this trace data consistently and in a format compatible with various observability tools.
**Best-In Class Monitoring:** Plano offers several monitoring metrics that help you understand three critical aspects of your application: latency, token usage, and error rates by an upstream LLM provider. Latency measures the speed at which your application is responding to users, which includes metrics like time to first token (TFT), time per output token (TOT) metrics, and the total latency as perceived by users.
**Out-of-process architecture, built on** `Envoy `_:
Plano takes a dependency on Envoy and is a self-contained process that is designed to run alongside your application servers. Plano uses Envoy's HTTP connection management subsystem, HTTP L7 filtering and telemetry capabilities to extend the functionality exclusively for prompts and LLMs.
This gives Plano several advantages:
* Plano builds on Envoy's proven success. Envoy is used at massive scale by the leading technology companies of our time including `AirBnB `_, `Dropbox `_, `Google `_, `Reddit `_, `Stripe `_, etc. Its battle tested and scales linearly with usage and enables developers to focus on what really matters: application features and business logic.
* Plano works with any application language. A single Plano deployment can act as gateway for AI applications written in Python, Java, C++, Go, Php, etc.
* Plano can be deployed and upgraded quickly across your infrastructure transparently without the horrid pain of deploying library upgrades in your applications.
---
### Source/Get Started/Overview
.. _overview:
Overview
========
`Plano `_ is delivery infrastructure for agentic apps. An AI-native proxy server and data plane designed to help you build agents faster, and deliver them reliably to production.
Plano pulls out the rote plumbing work (the “hidden AI middleware”) and decouples you from brittle, ever‑changing framework abstractions. It centralizes what shouldn’t be bespoke in every codebase like agent routing and orchestration, rich agentic signals and traces for continuous improvement, guardrail filters for safety and moderation, and smart LLM routing APIs for UX and DX agility. Use any language or AI framework, and ship agents to production faster with Plano.
Built by core contributors to the widely adopted `Envoy Proxy `_, Plano gives you a production‑grade foundation for agentic applications. It helps **developers** stay focused on the core logic of their agents, helps **product teams** shorten feedback loops for learning, and helps **engineering teams** standardize policy and safety across agents and LLMs. Plano is grounded in open protocols (de facto: OpenAI‑style v1/responses, de jure: MCP) and proven patterns like sidecar deployments, so it plugs in cleanly while remaining robust, scalable, and flexible.
In this documentation, you’ll learn how to set up Plano quickly, trigger API calls via prompts, apply guardrails without tight coupling with application code, simplify model and provider integration, and improve observability — so that you can focus on what matters most: the core product logic of your agents.
.. figure:: /_static/img/plano_network_diagram_high_level.png
:width: 100%
:align: center
High-level network flow of where Plano sits in your agentic stack. Designed for both ingress and egress traffic.
Get Started
-----------
This section introduces you to Plano and helps you get set up quickly:
.. grid:: 3
.. grid-item-card:: :octicon:`apps` Overview
:link: overview.html
Overview of Plano and Doc navigation
.. grid-item-card:: :octicon:`book` Intro to Plano
:link: intro_to_plano.html
Explore Plano's features and developer workflow
.. grid-item-card:: :octicon:`rocket` Quickstart
:link: quickstart.html
Learn how to quickly set up and integrate
Concepts
--------
Deep dive into essential ideas and mechanisms behind Plano:
.. grid:: 3
.. grid-item-card:: :octicon:`package` Agents
:link: ../concepts/agents.html
Learn about how to build and scale agents with Plano
.. grid-item-card:: :octicon:`webhook` Model Providers
:link: ../concepts/llm_providers/llm_providers.html
Explore Plano's LLM integration options
.. grid-item-card:: :octicon:`workflow` Prompt Target (Deprecated)
:link: ../concepts/prompt_target.html
Deprecated — kept for existing users. New apps should use Agents.
Guides
------
Step-by-step tutorials for practical Plano use cases and scenarios:
.. grid:: 3
.. grid-item-card:: :octicon:`shield-check` Guardrails
:link: ../guides/prompt_guard.html
Instructions on securing and validating prompts
.. grid-item-card:: :octicon:`code-square` LLM Routing
:link: ../guides/llm_router.html
A guide to effective model selection strategies
.. grid-item-card:: :octicon:`issue-opened` State Management
:link: ../guides/state.html
Learn to manage conversation and application state
Build with Plano
----------------
End to end examples demonstrating how to build agentic applications using Plano:
.. grid:: 2
.. grid-item-card:: :octicon:`dependabot` Build Agentic Apps
:link: ../get_started/quickstart.html#build-agentic-apps-with-plano
Discover how to create and manage custom agents within Plano
.. grid-item-card:: :octicon:`stack` Build Multi-LLM Apps
:link: ../get_started/quickstart.html#use-plano-as-a-model-proxy-gateway
Learn how to route LLM calls through Plano for enhanced control and observability
---
### Source/Get Started/Quickstart
.. _quickstart:
Quickstart
==========
Follow this guide to learn how to quickly set up Plano and integrate it into your generative AI applications. You can:
- :ref:`Use Plano as a model proxy (Gateway) ` to standardize access to multiple LLM providers.
- :ref:`Build agents ` for multi-step workflows (e.g., travel assistants with flights and hotels).
- :ref:`Call deterministic APIs via prompt targets ` to turn instructions directly into function calls.
.. note::
This quickstart assumes basic familiarity with agents and prompt targets from the Concepts section. For background, see :ref:`Agents ` and :ref:`Prompt Target `.
The full agent and backend API implementations used here are available in the `plano-quickstart repository `_. This guide focuses on wiring and configuring Plano (orchestration, prompt targets, and the model proxy), not application code.
Prerequisites
-------------
Plano runs **natively** by default — no Docker or Rust toolchain required. Pre-compiled binaries are downloaded automatically on first run.
1. `Python `_ (v3.10+)
2. Supported platforms: Linux (x86_64, aarch64), macOS (Apple Silicon)
**Docker mode** (optional):
If you prefer to run inside Docker, add ``--docker`` to ``planoai up`` / ``planoai down``. This requires:
1. `Docker System `_ (v24)
2. `Docker Compose `_ (v2.29)
Plano's CLI allows you to manage and interact with the Plano efficiently. To install the CLI, simply run the following command:
.. tip::
We recommend using **uv** for fast, reliable Python package management. Install uv if you haven't already:
.. code-block:: console
$ curl -LsSf https://astral.sh/uv/install.sh | sh
**Option 1: Install planoai with uv (Recommended)**
.. code-block:: console
$ uv tool install planoai==0.4.33
**Option 2: Install with pip (Traditional)**
.. code-block:: console
$ python -m venv venv
$ source venv/bin/activate # On Windows, use: venv\Scripts\activate
$ pip install planoai==0.4.33
.. _llm_routing_quickstart:
Use Plano as a Model Proxy (Gateway)
------------------------------------
Step 1. Create plano config file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Plano operates based on a configuration file where you can define LLM providers, prompt targets, guardrails, etc. Below is an example configuration that defines OpenAI and Anthropic LLM providers.
Create ``plano_config.yaml`` file with the following content:
.. code-block:: yaml
version: v0.3.0
listeners:
- type: model
name: model_1
address: 0.0.0.0
port: 12000
model_providers:
- access_key: $OPENAI_API_KEY
model: openai/gpt-4o
default: true
- access_key: $ANTHROPIC_API_KEY
model: anthropic/claude-sonnet-4-5
Step 2. Start plano
~~~~~~~~~~~~~~~~~~~
Once the config file is created, ensure that you have environment variables set up for ``ANTHROPIC_API_KEY`` and ``OPENAI_API_KEY`` (or these are defined in a ``.env`` file).
.. code-block:: console
$ planoai up plano_config.yaml
On the first run, Plano automatically downloads Envoy, WASM plugins, and brightstaff and caches them at ``~/.plano/``.
To stop Plano, run ``planoai down``.
**Docker mode** (optional):
.. code-block:: console
$ planoai up plano_config.yaml --docker
$ planoai down --docker
Step 3: Interact with LLM
~~~~~~~~~~~~~~~~~~~~~~~~~
Step 3.1: Using curl command
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
$ curl --header 'Content-Type: application/json' \
--data '{"messages": [{"role": "user","content": "What is the capital of France?"}], "model": "gpt-4o"}' \
http://localhost:12000/v1/chat/completions
{
...
"model": "gpt-4o-2024-08-06",
"choices": [
{
...
"messages": {
"role": "assistant",
"content": "The capital of France is Paris.",
},
}
],
}
.. note::
When the requested model is not found in the configuration, Plano will randomly select an available model from the configured providers. In this example, we use ``"model": "none"`` and Plano selects the default model ``openai/gpt-4o``.
Step 3.2: Using OpenAI Python client
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Make outbound calls via the Plano gateway:
.. code-block:: python
from openai import OpenAI
# Use the OpenAI client as usual
client = OpenAI(
# No need to set a specific openai.api_key since it's configured in Plano's gateway
api_key='--',
# Set the OpenAI API base URL to the Plano gateway endpoint
base_url="http://127.0.0.1:12000/v1"
)
response = client.chat.completions.create(
# we select model from plano_config file
model="--",
messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print("OpenAI Response:", response.choices[0].message.content)
Build Agentic Apps with Plano
-----------------------------
Plano helps you build agentic applications in two complementary ways:
* **Orchestrate agents**: Let Plano decide which agent or LLM should handle each request and in what sequence.
* **Call deterministic backends**: Use prompt targets to turn natural-language prompts into structured, validated API calls.
.. _quickstart_agents:
Building agents with Plano orchestration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Agents are where your business logic lives (the "inner loop"). Plano takes care of the "outer loop"—routing, sequencing, and managing calls across agents and LLMs.
At a high level, building agents with Plano looks like this:
1. **Implement your agent** in your framework of choice (Python, JS/TS, etc.), exposing it as an HTTP service.
2. **Route LLM calls through Plano's Model Proxy**, so all models share a consistent interface and observability.
3. **Configure Plano to orchestrate**: define which agent(s) can handle which kinds of prompts, and let Plano decide when to call an agent vs. an LLM.
This quickstart uses a simplified version of the Travel Booking Assistant; for the full multi-agent walkthrough, see :ref:`Orchestration `.
Step 1. Minimal orchestration config
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here is a minimal configuration that wires Plano-Orchestrator to two HTTP services: one for flights and one for hotels.
.. code-block:: yaml
version: v0.1.0
agents:
- id: flight_agent
url: http://localhost:10520 # your flights service
- id: hotel_agent
url: http://localhost:10530 # your hotels service
model_providers:
- model: openai/gpt-4o
access_key: $OPENAI_API_KEY
listeners:
- type: agent
name: travel_assistant
port: 8001
router: plano_orchestrator_v1
agents:
- id: flight_agent
description: Search for flights and provide flight status.
- id: hotel_agent
description: Find hotels and check availability.
tracing:
random_sampling: 100
Step 2. Start your agents and Plano
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Run your ``flight_agent`` and ``hotel_agent`` services (see :ref:`Orchestration ` for a full Travel Booking example), then start Plano with the config above:
.. code-block:: console
$ planoai up plano_config.yaml
# Or if installed with uv tool:
$ uvx planoai up plano_config.yaml
Plano will start the orchestrator and expose an agent listener on port ``8001``.
Step 3. Send a prompt and let Plano route
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Now send a request to Plano using the OpenAI-compatible chat completions API—the orchestrator will analyze the prompt and route it to the right agent based on intent:
.. code-block:: bash
$ curl --header 'Content-Type: application/json' \
--data '{"messages": [{"role": "user","content": "Find me flights from SFO to JFK tomorrow"}], "model": "openai/gpt-4o"}' \
http://localhost:8001/v1/chat/completions
You can then ask a follow-up like "Also book me a hotel near JFK" and Plano-Orchestrator will route to ``hotel_agent``—your agents stay focused on business logic while Plano handles routing.
.. _quickstart_prompt_targets:
Deterministic API calls with prompt targets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. deprecated:: v0.4.22
:ref:`Prompt Targets ` are deprecated and no longer actively
maintained. The walkthrough below is preserved for users on existing configs;
new applications should use :ref:`Agents ` instead.
Next, we'll show Plano's deterministic API calling using a single prompt target. We'll build a currency exchange backend powered by `https://api.frankfurter.dev/`, assuming USD as the base currency.
Step 1. Create plano config file
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Create ``plano_config.yaml`` file with the following content:
.. code-block:: yaml
version: v0.1.0
listeners:
ingress_traffic:
address: 0.0.0.0
port: 10000
message_format: openai
timeout: 30s
model_providers:
- access_key: $OPENAI_API_KEY
model: openai/gpt-4o
system_prompt: |
You are a helpful assistant.
prompt_targets:
- name: currency_exchange
description: Get currency exchange rate from USD to other currencies
parameters:
- name: currency_symbol
description: the currency that needs conversion
required: true
type: str
in_path: true
endpoint:
name: frankfurther_api
path: /v1/latest?base=USD&symbols={currency_symbol}
system_prompt: |
You are a helpful assistant. Show me the currency symbol you want to convert from USD.
- name: get_supported_currencies
description: Get list of supported currencies for conversion
endpoint:
name: frankfurther_api
path: /v1/currencies
endpoints:
frankfurther_api:
endpoint: api.frankfurter.dev:443
protocol: https
Step 2. Start plano with currency conversion config
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: sh
$ planoai up plano_config.yaml
# Or if installed with uv tool: uvx planoai up plano_config.yaml
2024-12-05 16:56:27,979 - planoai.main - INFO - Starting plano cli version: 0.1.5
...
2024-12-05 16:56:28,485 - planoai.utils - INFO - Schema validation successful!
2024-12-05 16:56:28,485 - planoai.main - INFO - Starting plano model server and plano gateway
...
2024-12-05 16:56:51,647 - planoai.core - INFO - Container is healthy!
Once the gateway is up, you can start interacting with it at port 10000 using the OpenAI chat completion API.
Some sample queries you can ask include: ``what is currency rate for gbp?`` or ``show me list of currencies for conversion``.
Step 3. Interacting with gateway using curl command
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here is a sample curl command you can use to interact:
.. code-block:: bash
$ curl --header 'Content-Type: application/json' \
--data '{"messages": [{"role": "user","content": "what is exchange rate for gbp"}], "model": "gpt-4o"}' \
http://localhost:10000/v1/chat/completions | jq ".choices[0].message.content"
"As of the date provided in your context, December 5, 2024, the exchange rate for GBP (British Pound) from USD (United States Dollar) is 0.78558. This means that 1 USD is equivalent to 0.78558 GBP."
And to get the list of supported currencies:
.. code-block:: bash
$ curl --header 'Content-Type: application/json' \
--data '{"messages": [{"role": "user","content": "show me list of currencies that are supported for conversion"}], "model": "gpt-4o"}' \
http://localhost:10000/v1/chat/completions | jq ".choices[0].message.content"
"Here is a list of the currencies that are supported for conversion from USD, along with their symbols:\n\n1. AUD - Australian Dollar\n2. BGN - Bulgarian Lev\n3. BRL - Brazilian Real\n4. CAD - Canadian Dollar\n5. CHF - Swiss Franc\n6. CNY - Chinese Renminbi Yuan\n7. CZK - Czech Koruna\n8. DKK - Danish Krone\n9. EUR - Euro\n10. GBP - British Pound\n11. HKD - Hong Kong Dollar\n12. HUF - Hungarian Forint\n13. IDR - Indonesian Rupiah\n14. ILS - Israeli New Sheqel\n15. INR - Indian Rupee\n16. ISK - Icelandic Króna\n17. JPY - Japanese Yen\n18. KRW - South Korean Won\n19. MXN - Mexican Peso\n20. MYR - Malaysian Ringgit\n21. NOK - Norwegian Krone\n22. NZD - New Zealand Dollar\n23. PHP - Philippine Peso\n24. PLN - Polish Złoty\n25. RON - Romanian Leu\n26. SEK - Swedish Krona\n27. SGD - Singapore Dollar\n28. THB - Thai Baht\n29. TRY - Turkish Lira\n30. USD - United States Dollar\n31. ZAR - South African Rand\n\nIf you want to convert USD to any of these currencies, you can select the one you are interested in."
Observability
-------------
Plano ships two CLI tools for visibility into LLM traffic. Both consume the same OTLP/gRPC span stream from brightstaff; they just slice it differently — use whichever (or both) fits the question you're answering.
===================== ============================================ =============================================================
Command When to use Shows
===================== ============================================ =============================================================
``planoai obs`` Live view while you drive traffic Per-request rows + aggregates: tokens (prompt / completion / cached / cache-creation / reasoning), TTFT, latency, cost, session id, route name, totals by model
``planoai trace`` Deep-dive into a single request after the fact Full span tree for a trace id: brightstaff → routing → upstream LLM, attributes on every span, status codes, errors
===================== ============================================ =============================================================
Both require brightstaff to be exporting spans. If you're running the zero-config path (``planoai up`` with no config file), tracing is auto-wired to ``http://localhost:4317``. If you have your own ``plano_config.yaml``, add:
.. code-block:: yaml
tracing:
random_sampling: 100
opentracing_grpc_endpoint: http://localhost:4317
Live console — ``planoai obs``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: console
$ planoai obs
# In another terminal:
$ planoai up
Cost is populated automatically from DigitalOcean's public pricing catalog — no signup or token required.
With no API keys set, every provider runs in pass-through mode — supply the ``Authorization`` header yourself on each request:
.. code-block:: console
$ curl localhost:12000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DO_API_KEY" \
-d '{"model":"digitalocean/router:software-engineering",
"messages":[{"role":"user","content":"write code to print prime numbers in python"}],
"stream":false}'
When you export ``OPENAI_API_KEY`` / ``ANTHROPIC_API_KEY`` / ``DO_API_KEY`` / etc. before ``planoai up``, Plano picks them up and clients no longer need to send ``Authorization``.
Press ``Ctrl-C`` in the obs terminal to exit. Data lives in memory only — nothing is persisted to disk.
Single-request traces — ``planoai trace``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When you need to understand what happened on one specific request (which model was picked, how long each hop took, what an upstream returned), use ``trace``:
.. code-block:: console
$ planoai trace listen # start the OTLP listener (daemon)
# drive some traffic through localhost:12000 ...
$ planoai trace # show the most recent trace
$ planoai trace # show a specific trace by id
$ planoai trace --list # list the last 50 trace ids
Use ``obs`` to spot that p95 latency spiked for ``openai-gpt-5.4``; switch to ``trace`` on one of those slow request ids to see which hop burned the time.
Next Steps
==========
Congratulations! You've successfully set up Plano and made your first prompt-based request. To further enhance your GenAI applications, explore the following resources:
- :ref:`Full Documentation `: Comprehensive guides and references.
- `GitHub Repository `_: Access the source code, contribute, and track updates.
- `Support `_: Get help and connect with the Plano community .
With Plano, building scalable, fast, and personalized GenAI applications has never been easier. Dive deeper into Plano's capabilities and start creating innovative AI-driven experiences today!
---
### Source/Guides/Observability/Access Logging
.. _plano_access_logging:
Access Logging
==============
Access logging in Plano refers to the logging of detailed information about each request and response that flows through Plano.
It provides visibility into the traffic passing through Plano, which is crucial for monitoring, debugging, and analyzing the
behavior of AI applications and their interactions.
Key Features
^^^^^^^^^^^^
* **Per-Request Logging**:
Each request that passes through Plano is logged. This includes important metadata such as HTTP method,
path, response status code, request duration, upstream host, and more.
* **Integration with Monitoring Tools**:
Access logs can be exported to centralized logging systems (e.g., ELK stack or Fluentd) or used to feed monitoring and alerting systems.
* **Structured Logging**: where each request is logged as a object, making it easier to parse and analyze using tools like Elasticsearch and Kibana.
How It Works
^^^^^^^^^^^^
Plano exposes access logs for every call it manages on your behalf. By default these access logs can be found under ``~/plano_logs``. For example:
.. code-block:: console
$ tail -F ~/plano_logs/access_*.log
==> /Users/username/plano_logs/access_llm.log <==
[2024-10-10T03:55:49.537Z] "POST /v1/chat/completions HTTP/1.1" 0 DC 0 0 770 - "-" "OpenAI/Python 1.51.0" "469793af-b25f-9b57-b265-f376e8d8c586" "api.openai.com" "162.159.140.245:443"
==> /Users/username/plano_logs/access_internal.log <==
[2024-10-10T03:56:03.906Z] "POST /embeddings HTTP/1.1" 200 - 52 21797 54 53 "-" "-" "604197fe-2a5b-95a2-9367-1d6b30cfc845" "model_server" "192.168.65.254:51000"
[2024-10-10T03:56:03.961Z] "POST /zeroshot HTTP/1.1" 200 - 106 218 87 87 "-" "-" "604197fe-2a5b-95a2-9367-1d6b30cfc845" "model_server" "192.168.65.254:51000"
[2024-10-10T03:56:04.050Z] "POST /v1/chat/completions HTTP/1.1" 200 - 1301 614 441 441 "-" "-" "604197fe-2a5b-95a2-9367-1d6b30cfc845" "model_server" "192.168.65.254:51000"
[2024-10-10T03:56:04.492Z] "POST /hallucination HTTP/1.1" 200 - 556 127 104 104 "-" "-" "604197fe-2a5b-95a2-9367-1d6b30cfc845" "model_server" "192.168.65.254:51000"
[2024-10-10T03:56:04.598Z] "POST /insurance_claim_details HTTP/1.1" 200 - 447 125 17 17 "-" "-" "604197fe-2a5b-95a2-9367-1d6b30cfc845" "api_server" "192.168.65.254:18083"
==> /Users/username/plano_logs/access_ingress.log <==
[2024-10-10T03:56:03.905Z] "POST /v1/chat/completions HTTP/1.1" 200 - 463 1022 1695 984 "-" "OpenAI/Python 1.51.0" "604197fe-2a5b-95a2-9367-1d6b30cfc845" "plano_llm_listener" "0.0.0.0:12000"
Log Format
^^^^^^^^^^
What do these logs mean? Let's break down the log format:
.. code-block:: console
START_TIME METHOD ORIGINAL-PATH PROTOCOL RESPONSE_CODE RESPONSE_FLAGS
BYTES_RECEIVED BYTES_SENT DURATION UPSTREAM-SERVICE-TIME X-FORWARDED-FOR
USER-AGENT X-REQUEST-ID AUTHORITY UPSTREAM_HOST
Most of these fields are self-explanatory, but here are a few key fields to note:
- UPSTREAM-SERVICE-TIME: The time taken by the upstream service to process the request.
- DURATION: The total time taken to process the request.
For example for following request:
.. code-block:: console
[2024-10-10T03:56:03.905Z] "POST /v1/chat/completions HTTP/1.1" 200 - 463 1022 1695 984 "-" "OpenAI/Python 1.51.0" "604197fe-2a5b-95a2-9367-1d6b30cfc845" "plano_llm_listener" "0.0.0.0:12000"
Total duration was 1695ms, and the upstream service took 984ms to process the request. Bytes received and sent were 463 and 1022 respectively.
---
### Source/Guides/Observability/Monitoring
.. _monitoring:
Monitoring
==========
`OpenTelemetry `_ is an open-source observability framework providing APIs
and instrumentation for generating, collecting, processing, and exporting telemetry data, such as traces,
metrics, and logs. Its flexible design supports a wide range of backends and seamlessly integrates with
modern application tools.
Plano acts a *source* for several monitoring metrics related to **agents** and **LLMs** natively integrated
via `OpenTelemetry `_ to help you understand three critical aspects of your application:
latency, token usage, and error rates by an upstream LLM provider. Latency measures the speed at which your application
is responding to users, which includes metrics like time to first token (TFT), time per output token (TOT) metrics, and
the total latency as perceived by users. Below are some screenshots how Plano integrates natively with tools like
`Grafana `_ via `Promethus `_
Metrics Dashboard (via Grafana)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. image:: /_static/img/llm-request-metrics.png
:width: 100%
:align: center
.. image:: /_static/img/input-token-metrics.png
:width: 100%
:align: center
.. image:: /_static/img/output-token-metrics.png
:width: 100%
:align: center
Configure Monitoring
~~~~~~~~~~~~~~~~~~~~
Plano publishes stats endpoint at http://localhost:19901/stats. As noted above, Plano is a source for metrics. To view and manipulate dashbaords, you will
need to configiure `Promethus `_ (as a metrics store) and `Grafana `_ for dashboards. Below
are some sample configuration files for both, respectively.
.. code-block:: yaml
:caption: Sample prometheus.yaml config file
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: []
scheme: http
timeout: 10s
api_version: v2
scrape_configs:
- job_name: plano
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /stats
scheme: http
static_configs:
- targets:
- localhost:19901
params:
format: ["prometheus"]
.. code-block:: yaml
:caption: Sample grafana datasource.yaml config file
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: http://prometheus:9090
isDefault: true
access: proxy
editable: true
Brightstaff metrics
~~~~~~~~~~~~~~~~~~~
In addition to Envoy's stats on ``:9901``, the brightstaff dataplane
process exposes its own Prometheus endpoint on ``0.0.0.0:9092`` (override
with ``METRICS_BIND_ADDRESS``). It publishes:
* HTTP RED — ``brightstaff_http_requests_total``,
``brightstaff_http_request_duration_seconds``,
``brightstaff_http_in_flight_requests`` (labels: ``handler``, ``method``,
``status_class``).
* LLM upstream — ``brightstaff_llm_upstream_requests_total``,
``brightstaff_llm_upstream_duration_seconds``,
``brightstaff_llm_time_to_first_token_seconds``,
``brightstaff_llm_tokens_total`` (labels: ``provider``, ``model``,
``error_class``, ``kind``).
* Routing — ``brightstaff_router_decisions_total``,
``brightstaff_router_decision_duration_seconds``,
``brightstaff_routing_service_requests_total``,
``brightstaff_session_cache_events_total``.
* Process & build — ``process_resident_memory_bytes``,
``process_cpu_seconds_total``, ``brightstaff_build_info``.
A self-contained Prometheus + Grafana stack is shipped under
``config/grafana/``. With Plano already running on the host, bring it up
with one command:
.. code-block:: bash
cd config/grafana
docker compose up -d
open http://localhost:3000 # admin / admin (anonymous viewer also enabled)
Grafana auto-loads the Prometheus datasource and the brightstaff
dashboard (look under the *Plano* folder). Prometheus scrapes the host's
``:9092`` and ``:9901`` via ``host.docker.internal``.
Files:
* ``config/grafana/docker-compose.yaml`` — one-command Prom + Grafana
stack with provisioning.
* ``config/grafana/prometheus_scrape.yaml`` — complete Prometheus config
with ``envoy`` and ``brightstaff`` scrape jobs (mounted by the
compose).
* ``config/grafana/brightstaff_dashboard.json`` — 19-panel dashboard
across HTTP RED, LLM upstream, Routing service, and Process & Envoy
link rows. Auto-provisioned by the compose; can also be imported by
hand via *Dashboards → New → Import*.
* ``config/grafana/provisioning/`` — Grafana provisioning files for the
datasource and dashboard provider.
---
### Source/Guides/Observability/Observability
.. _observability:
Observability
=============
.. toctree::
:maxdepth: 2
tracing
monitoring
access_logging
---
### Source/Guides/Observability/Tracing
.. _plano_overview_tracing:
Tracing
=======
Overview
--------
`OpenTelemetry `_ is an open-source observability framework providing APIs
and instrumentation for generating, collecting, processing, and exporting telemetry data, such as traces,
metrics, and logs. Its flexible design supports a wide range of backends and seamlessly integrates with
modern application tools. A key feature of OpenTelemetry is its commitment to standards like the
`W3C Trace Context `_
**Tracing** is a critical tool that allows developers to visualize and understand the flow of
requests in an AI application. With tracing, you can capture a detailed view of how requests propagate
through various services and components, which is crucial for **debugging**, **performance optimization**,
and understanding complex AI agent architectures like Co-pilots.
**Plano** propagates trace context using the W3C Trace Context standard, specifically through the
``traceparent`` header. This allows each component in the system to record its part of the request
flow, enabling **end-to-end tracing** across the entire application. By using OpenTelemetry, Plano ensures
that developers can capture this trace data consistently and in a format compatible with various observability
tools.
.. image:: /_static/img/tracing.png
:width: 100%
:align: center
Understanding Plano Traces
--------------------------
Plano creates structured traces that capture the complete flow of requests through your AI system. Each trace consists of multiple spans representing different stages of processing.
Inbound Request Handling
~~~~~~~~~~~~~~~~~~~~~~~~~
When a request enters Plano, it creates an **inbound span** (``plano(inbound)``) that represents the initial request reception and processing. This span captures:
- HTTP request details (method, path, headers)
- Request payload size
- Initial validation and authentication
Orchestration & Routing
~~~~~~~~~~~~~~~~~~~~~~~~
For agent systems, Plano performs intelligent routing through orchestration spans:
- **Agent Orchestration** (``plano(orchestrator)``): When multiple agents are available, Plano uses an LLM to analyze the user's intent and select the most appropriate agent. This span captures the orchestration decision-making process.
- **LLM Routing** (``plano(routing)``): For direct LLM requests, Plano determines the optimal endpoint based on your routing strategy (round-robin, least-latency, cost-optimized). This span includes:
- Routing strategy used
- Selected upstream endpoint
- Route determination time
- Fallback indicators (if applicable)
Agent Processing
~~~~~~~~~~~~~~~~
When requests are routed to agents, Plano creates spans for agent execution:
- **Agent Filter Chains** (``plano(filter)``): If filters are configured (guardrails, context enrichment, query rewriting), each filter execution is captured in its own span, showing the transformation pipeline.
- **Agent Execution** (``plano(agent)``): The main agent processing span that captures the agent's work, including any tools invoked and intermediate reasoning steps.
Outbound LLM Calls
~~~~~~~~~~~~~~~~~~
All LLM calls—whether from Plano's routing layer or from agents—are traced with **LLM spans** (``plano(llm)``) that capture:
- Model name and provider (e.g., ``gpt-4``, ``claude-3-sonnet``)
- Request parameters (temperature, max_tokens, top_p)
- Token usage (prompt_tokens, completion_tokens)
- Streaming indicators and time-to-first-token
- Response metadata
**Example Span Attributes**::
# LLM call span
llm.model = "gpt-4"
llm.provider = "openai"
llm.usage.prompt_tokens = 150
llm.usage.completion_tokens = 75
llm.duration_ms = 1250
llm.time_to_first_token = 320
Handoff to Upstream Services
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When Plano forwards requests to upstream services (agents, APIs, or LLM providers), it creates **handoff spans** (``plano(handoff)``) that capture:
- Upstream endpoint URL
- Request/response sizes
- HTTP status codes
- Upstream response times
This creates a complete end-to-end trace showing the full request lifecycle through all system components.
Behavioral Signals in Traces
----------------------------
Plano automatically enriches OpenTelemetry traces with :doc:`../../concepts/signals` — lightweight, model-free behavioral indicators organized into three layers (interaction, execution, environment) per `Chen et al., 2026 `_. Signals are attached as span attributes and per-instance span events, providing immediate visibility into interaction quality.
**What Signals Provide**
Signals act as early warning indicators embedded in your traces:
- **Quality Assessment**: Overall interaction quality (``excellent`` / ``good`` / ``neutral`` / ``poor`` / ``severe``) and numeric score
- **Interaction layer**: misalignment, stagnation, disengagement, satisfaction
- **Execution layer**: tool failures and loop patterns (from ``function_call`` / ``observation`` traces)
- **Environment layer**: exhaustion (API errors, timeouts, rate limits, context overflow)
**Visual Flag Markers**
When concerning signals are detected (disengagement, execution failures / loops, stagnation > 2, or ``poor`` / ``severe`` quality), Plano automatically appends a 🚩 marker to the span's operation name. This makes problematic traces immediately visible in your tracing UI without requiring additional queries.
**Example Span with Signals**::
# Span name: "POST /v1/chat/completions gpt-4 🚩"
# Standard LLM attributes:
llm.model = "gpt-4"
llm.usage.total_tokens = 225
# Top-level signal attributes:
signals.quality = "severe"
signals.quality_score = 0.0
signals.turn_count = 15
signals.efficiency_score = 0.234
# Layered attributes (only non-zero categories are emitted):
signals.interaction.misalignment.count = 4
signals.interaction.misalignment.severity = 2
signals.interaction.disengagement.count = 5
signals.interaction.disengagement.severity = 3
# Per-instance span event:
event: signal.interaction.disengagement.escalation
signal.type = "interaction.disengagement.escalation"
signal.message_index = 14
signal.confidence = 1.0
signal.snippet = "get me a human"
**Querying Signal Data**
In your observability platform (Jaeger, Grafana Tempo, Datadog, etc.), filter traces by signal attributes:
- Find severe interactions: ``signals.quality = "severe"``
- Find disengaged users: ``signals.interaction.disengagement.severity >= 2``
- Find misaligned interactions: ``signals.interaction.misalignment.count > 3``
- Find tool failures: ``signals.execution.failure.count > 0``
- Find external issues: ``signals.environment.exhaustion.count > 0``
- Find inefficient flows: ``signals.efficiency_score < 0.5``
For complete details on all 20 leaf signal types, severity scheme, and best practices, see the :doc:`../../concepts/signals` guide.
Custom Span Attributes
-------------------------------------------
Plano can automatically attach **custom span attributes** derived from request headers and **static** attributes
defined in configuration. This lets you stamp
traces with identifiers like workspace, tenant, or user IDs without changing application code or adding
custom instrumentation.
**Why This Is Useful**
- **Tenant-aware debugging**: Filter traces by ``workspace.id`` or ``tenant.id``.
- **Customer-specific visibility**: Attribute performance or errors to a specific customer.
- **Low overhead**: No code changes in agents or clients—just headers.
How It Works
~~~~~~~~~~~~
You configure one or more header prefixes. Any incoming HTTP header whose name starts with one of these
prefixes is captured as a span attribute. You can also provide static attributes that are always injected.
- The **prefix is only for matching**, not the resulting attribute key.
- The attribute key is the header name **with the prefix removed**, then hyphens converted to dots.
.. note::
Custom span attributes are attached to LLM spans when handling ``/v1/...`` requests via ``llm_chat``. For orchestrator requests to ``/agents/...``,
these attributes are added to both the orchestrator selection span and to each agent span created by ``agent_chat``.
**Example**
Configured prefix::
tracing:
span_attributes:
header_prefixes:
- x-katanemo-
Incoming headers::
X-Katanemo-Workspace-Id: ws_123
X-Katanemo-Tenant-Id: ten_456
Resulting span attributes::
workspace.id = "ws_123"
tenant.id = "ten_456"
Configuration
~~~~~~~~~~~~~
Add the prefix list under ``tracing`` in your config:
.. code-block:: yaml
tracing:
random_sampling: 100
span_attributes:
header_prefixes:
- x-katanemo-
static:
environment: production
service.version: "1.0.0"
Static attributes are always injected alongside any header-derived attributes. If a header-derived
attribute key matches a static key, the header value overrides the static value.
You can provide multiple prefixes:
.. code-block:: yaml
tracing:
span_attributes:
header_prefixes:
- x-katanemo-
- x-tenant-
static:
environment: production
service.version: "1.0.0"
Notes and Examples
~~~~~~~~~~~~~~~~~~
- **Prefix must match exactly**: ``katanemo-`` does not match ``x-katanemo-`` headers.
- **Trailing dash is recommended**: Without it, ``x-katanemo`` would also match ``x-katanemo-foo`` and
``x-katanemofoo``.
- **Keys are always strings**: Values are captured as string attributes.
**Prefix mismatch example**
Config::
tracing:
span_attributes:
header_prefixes:
- x-katanemo-
Request headers::
X-Other-User-Id: usr_999
Result: no attributes are captured from ``X-Other-User-Id``.
Exporting Telemetry Anywhere
----------------------------
Beyond the OTLP/gRPC collector, Plano can stream LLM telemetry directly to
third-party observability backends through ``tracing.exporters``. The list is
provider-agnostic: each entry is tagged by its ``type`` and points at a URL, so
new destinations can be added without changing anything else. Exporters run in
addition to ``opentracing_grpc_endpoint`` — you can use one, the other, or both.
PostHog
~~~~~~~
PostHog is supported as a first-class integration. Every LLM call is captured as
a PostHog `$ai_generation `_
event and POSTed to PostHog's capture API. Setup is intentionally minimal —
point at your PostHog URL and project token::
tracing:
random_sampling: 100
exporters:
- type: posthog
url: https://us.i.posthog.com # /batch/ is appended automatically
api_key: $POSTHOG_API_KEY # PostHog project token (env expansion supported)
distinct_id_header: x-user-id # optional; omit for anonymous capture
capture_messages: false # optional; send user message as $ai_input
That's all that's required. When ``random_sampling`` is greater than ``0`` and at
least one exporter (or ``opentracing_grpc_endpoint``) is configured, tracing is
enabled and ``$ai_generation`` events begin flowing. They appear under PostHog's
**AI Observability** in the Traces and Generations tabs.
**Captured properties**
Plano maps span data onto PostHog ``$ai_*`` properties:
.. list-table::
:header-rows: 1
:widths: 30 70
* - PostHog property
- Source
* - ``$ai_model``
- Resolved upstream model (``llm.model``)
* - ``$ai_provider``
- Provider derived from the resolved model (``llm.provider``)
* - ``$ai_latency``
- Total call duration in seconds (``llm.duration_ms``)
* - ``$ai_time_to_first_token``
- Time to first token in seconds, streaming only
* - ``$ai_input_tokens`` / ``$ai_output_tokens``
- Prompt / completion token usage
* - ``$ai_http_status`` / ``$ai_is_error``
- Upstream HTTP status and error flag
* - ``$ai_trace_id`` / ``$ai_parent_id``
- Trace and parent span identifiers
* - ``distinct_id``
- Value of ``distinct_id_header`` (else anonymous)
**Identifying users**
Set ``distinct_id_header`` to the request header carrying your user identity
(for example ``x-user-id``). When present, Plano stamps the value as the PostHog
``distinct_id``. When the header is missing — or ``distinct_id_header`` is not
configured — the event is captured anonymously (``$process_person_profile`` is
set to ``false``), matching PostHog's anonymous vs. identified semantics.
**Capturing message content**
By default Plano does not send prompt content off-box. Set
``capture_messages: true`` to include the (truncated) user message preview as
``$ai_input``. Leave it ``false`` when prompt content must not leave your data
plane.
**Multiple destinations**
``exporters`` is a list, so you can fan out to several backends (and combine
with an OTLP collector). A common use is shipping to multiple PostHog instances
(for example separate EU and US projects for data-residency).
Benefits of Using ``Traceparent`` Headers
-----------------------------------------
- **Standardization**: The W3C Trace Context standard ensures compatibility across ecosystem tools, allowing
traces to be propagated uniformly through different layers of the system.
- **Ease of Integration**: OpenTelemetry's design allows developers to easily integrate tracing with minimal
changes to their codebase, enabling quick adoption of end-to-end observability.
- **Interoperability**: Works seamlessly with popular tracing tools like AWS X-Ray, Datadog, Jaeger, and many others,
making it easy to visualize traces in the tools you're already usi
How to Initiate A Trace
-----------------------
1. **Enable Tracing Configuration**: Simply add the ``random_sampling`` in ``tracing`` section to 100`` flag to in the :ref:`listener ` config
2. **Trace Context Propagation**: Plano automatically propagates the ``traceparent`` header. When a request is received, Plano will:
- Generate a new ``traceparent`` header if one is not present.
- Extract the trace context from the ``traceparent`` header if it exists.
- Start a new span representing its processing of the request.
- Forward the ``traceparent`` header to downstream services.
3. **Sampling Policy**: The 100 in ``random_sampling: 100`` means that all the requests as sampled for tracing.
You can adjust this value from 0-100.
Tracing with the CLI
--------------------
The Plano CLI ships with a local OTLP/gRPC listener and a trace viewer so you can inspect spans without wiring a full observability backend. This is ideal for development, debugging, and quick QA.
Quick Start
~~~~~~~~~~~
You can enable tracing in either of these ways:
1. Start the local listener explicitly:
.. code-block:: console
$ planoai trace listen
2. Or start Plano with tracing enabled (auto-starts the local OTLP listener):
.. code-block:: console
$ planoai up --with-tracing
# Optional: choose a different listener port
$ planoai up --with-tracing --tracing-port 4318
3. Send requests through Plano as usual. The listener accepts OTLP/gRPC on:
- ``0.0.0.0:4317`` (default)
4. View the most recent trace:
.. code-block:: console
$ planoai trace
Inspect and Filter Traces
~~~~~~~~~~~~~~~~~~~~~~~~~
List available trace IDs:
.. code-block:: console
$ planoai trace --list
Open a specific trace (full or short trace ID):
.. code-block:: console
$ planoai trace 7f4e9a1c
$ planoai trace 7f4e9a1c0d9d4a0bb9bf5a8a7d13f62a
Filter by attributes and time window:
.. code-block:: console
$ planoai trace --where llm.model=gpt-4o-mini --since 30m
$ planoai trace --filter "http.*" --limit 5
Return JSON for automation:
.. code-block:: console
$ planoai trace --json
$ planoai trace --list --json
Show full span attributes (disable default compact view):
.. code-block:: console
$ planoai trace --verbose
$ planoai trace -v
Point the CLI at a different local listener port:
.. code-block:: console
$ export PLANO_TRACE_PORT=50051
$ planoai trace --list
Notes
~~~~~
- ``--where`` accepts repeatable ``key=value`` filters and uses AND semantics.
- ``--filter`` supports wildcards (``*``) to limit displayed attributes.
- ``--no-interactive`` disables prompts when listing traces.
- By default, inbound/outbound spans use a compact attribute view.
Trace Propagation
-----------------
Plano uses the W3C Trace Context standard for trace propagation, which relies on the ``traceparent`` header.
This header carries tracing information in a standardized format, enabling interoperability between different
tracing systems.
Header Format
~~~~~~~~~~~~~
The ``traceparent`` header has the following format::
traceparent: {version}-{trace-id}-{parent-id}-{trace-flags}
- ``{version}``: The version of the Trace Context specification (e.g., ``00``).
- ``{trace-id}``: A 16-byte (32-character hexadecimal) unique identifier for the trace.
- ``{parent-id}``: An 8-byte (16-character hexadecimal) identifier for the parent span.
- ``{trace-flags}``: Flags indicating trace options (e.g., sampling).
Instrumentation
~~~~~~~~~~~~~~~
To integrate AI tracing, your application needs to follow a few simple steps. The steps
below are very common practice, and not unique to Plano, when you reading tracing headers and export
`spans `_ for distributed tracing.
- Read the ``traceparent`` header from incoming requests.
- Start new spans as children of the extracted context.
- Include the ``traceparent`` header in outbound requests to propagate trace context.
- Send tracing data to a collector or tracing backend to export spans
Example with OpenTelemetry in Python
************************************
Install OpenTelemetry packages:
.. code-block:: console
$ pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
$ pip install opentelemetry-instrumentation-requests
Set up the tracer and exporter:
.. code-block:: python
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# Define the service name
resource = Resource(attributes={
"service.name": "customer-support-agent"
})
# Set up the tracer provider and exporter
tracer_provider = TracerProvider(resource=resource)
otlp_exporter = OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
tracer_provider.add_span_processor(span_processor)
trace.set_tracer_provider(tracer_provider)
# Instrument HTTP requests
RequestsInstrumentor().instrument()
Handle incoming requests:
.. code-block:: python
from opentelemetry import trace
from opentelemetry.propagate import extract, inject
import requests
def handle_request(request):
# Extract the trace context
context = extract(request.headers)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("process_customer_request", context=context):
# Example of processing a customer request
print("Processing customer request...")
# Prepare headers for outgoing request to payment service
headers = {}
inject(headers)
# Make outgoing request to external service (e.g., payment gateway)
response = requests.get("http://payment-service/api", headers=headers)
print(f"Payment service response: {response.content}")
Integrating with Tracing Tools
------------------------------
AWS X-Ray
~~~~~~~~~
To send tracing data to `AWS X-Ray `_ :
1. **Configure OpenTelemetry Collector**: Set up the collector to export traces to AWS X-Ray.
Collector configuration (``otel-collector-config.yaml``):
.. code-block:: yaml
receivers:
otlp:
protocols:
grpc:
processors:
batch:
exporters:
awsxray:
region:
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [awsxray]
2. **Deploy the Collector**: Run the collector as a Docker container, Kubernetes pod, or standalone service.
3. **Ensure AWS Credentials**: Provide AWS credentials to the collector, preferably via IAM roles.
4. **Verify Traces**: Access the AWS X-Ray console to view your traces.
Datadog
~~~~~~~
Datadog
To send tracing data to `Datadog `_:
1. **Configure OpenTelemetry Collector**: Set up the collector to export traces to Datadog.
Collector configuration (``otel-collector-config.yaml``):
.. code-block:: yaml
receivers:
otlp:
protocols:
grpc:
processors:
batch:
exporters:
datadog:
api:
key: "${}"
site: "${DD_SITE}"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [datadog]
2. **Set Environment Variables**: Provide your Datadog API key and site.
.. code-block:: console
$ export =
$ export DD_SITE=datadoghq.com # Or datadoghq.eu
3. **Deploy the Collector**: Run the collector in your environment.
4. **Verify Traces**: Access the Datadog APM dashboard to view your traces.
Langtrace
~~~~~~~~~
Langtrace is an observability tool designed specifically for large language models (LLMs). It helps you capture, analyze, and understand how LLMs are used in your applications including those built using Plano.
To send tracing data to `Langtrace `_:
1. **Configure Plano**: Make sure Plano is installed and setup correctly. For more information, refer to the `installation guide `_.
2. **Install Langtrace**: Install the Langtrace SDK.:
.. code-block:: console
$ pip install langtrace-python-sdk
3. **Set Environment Variables**: Provide your Langtrace API key.
.. code-block:: console
$ export LANGTRACE_API_KEY=
4. **Trace Requests**: Once you have Langtrace set up, you can start tracing requests.
Here's an example of how to trace a request using the Langtrace Python SDK:
.. code-block:: python
import os
from langtrace_python_sdk import langtrace # Must precede any llm module imports
from openai import OpenAI
langtrace.init(api_key=os.environ['LANGTRACE_API_KEY'])
client = OpenAI(api_key=os.environ['OPENAI_API_KEY'], base_url="http://localhost:12000/v1")
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Hello"},
]
)
print(chat_completion.choices[0].message.content)
5. **Verify Traces**: Access the Langtrace dashboard to view your traces.
Best Practices
--------------
- **Consistent Instrumentation**: Ensure all services propagate the ``traceparent`` header.
- **Secure Configuration**: Protect sensitive data and secure communication between services.
- **Performance Monitoring**: Be mindful of the performance impact and adjust sampling rates accordingly.
- **Error Handling**: Implement proper error handling to prevent tracing issues from affecting your application.
Summary
----------
By leveraging the ``traceparent`` header for trace context propagation, Plano enables developers to implement
tracing efficiently. This approach simplifies the process of collecting and analyzing tracing data in common
tools like AWS X-Ray and Datadog, enhancing observability and facilitating faster debugging and optimization.
Additional Resources
--------------------
For full command documentation (including ``planoai trace`` and all other CLI commands), see :ref:`cli_reference`.
External References
~~~~~~~~~~~~~~~~~~~
- `OpenTelemetry Documentation `_
- `W3C Trace Context Specification `_
- `AWS X-Ray Exporter `_
- `Datadog Exporter `_
- `Langtrace Documentation `_
.. Note::
Replace placeholders such as ```` and ```` with your actual configurations.
---
### Source/Guides/Function Calling
.. _function_calling:
Function Calling
================
**Function Calling** is a powerful feature in Plano that allows your application to dynamically execute backend functions or services based on user prompts.
This enables seamless integration between natural language interactions and backend operations, turning user inputs into actionable results.
.. deprecated:: v0.4.22
The prompt-target based workflow shown below (see :ref:`Step 2 `)
is deprecated. :ref:`Prompt Targets ` are no longer actively
maintained and may be removed in a future release. For new function-calling
workloads, prefer :ref:`Agents ` with tool definitions.
What is Function Calling?
-------------------------
Function Calling refers to the mechanism where the user's prompt is parsed, relevant parameters are extracted, and a designated backend function (or API) is triggered to execute a particular task.
This feature bridges the gap between generative AI systems and functional business logic, allowing users to interact with the system through natural language while the backend performs the necessary operations.
Function Calling Workflow
-------------------------
#. **Prompt Parsing**
When a user submits a prompt, Plano analyzes it to determine the intent. Based on this intent, the system identifies whether a function needs to be invoked and which parameters should be extracted.
#. **Parameter Extraction**
Plano’s advanced natural language processing capabilities automatically extract parameters from the prompt that are necessary for executing the function. These parameters can include text, numbers, dates, locations, or other relevant data points.
#. **Function Invocation**
Once the necessary parameters have been extracted, Plano invokes the relevant backend function. This function could be an API, a database query, or any other form of backend logic. The function is executed with the extracted parameters to produce the desired output.
#. **Response Handling**
After the function has been called and executed, the result is processed and a response is generated. This response is typically delivered in a user-friendly format, which can include text explanations, data summaries, or even a confirmation message for critical actions.
Arch-Function
-------------
The `Arch-Function `_ collection of large language models (LLMs) is a collection state-of-the-art (SOTA) LLMs specifically designed for **function calling** tasks.
The models are designed to understand complex function signatures, identify required parameters, and produce accurate function call outputs based on natural language prompts.
Achieving performance on par with GPT-4, these models set a new benchmark in the domain of function-oriented tasks, making them suitable for scenarios where automated API interaction and function execution is crucial.
In summary, the Arch-Function collection demonstrates:
- **State-of-the-art performance** in function calling
- **Accurate parameter identification and suggestion**, even in ambiguous or incomplete inputs
- **High generalization** across multiple function calling use cases, from API interactions to automated backend tasks.
- Optimized **low-latency, high-throughput performance**, making it suitable for real-time, production environments.
Key Features
~~~~~~~~~~~~
.. table::
:width: 100%
========================= ===============================================================
**Functionality** **Definition**
========================= ===============================================================
Single Function Calling Call only one function per user prompt
Parallel Function Calling Call the same function multiple times but with parameter values
Multiple Function Calling Call different functions per user prompt
Parallel & Multiple Perform both parallel and multiple function calling
========================= ===============================================================
Implementing Function Calling
-----------------------------
Here’s a step-by-step guide to configuring function calling within your Plano setup:
Step 1: Define the Function
~~~~~~~~~~~~~~~~~~~~~~~~~~~
First, create or identify the backend function you want Plano to call. This could be an API endpoint, a script, or any other executable backend logic.
.. code-block:: python
import requests
def get_weather(location: str, unit: str = "fahrenheit"):
if unit not in ["celsius", "fahrenheit"]:
raise ValueError("Invalid unit. Choose either 'celsius' or 'fahrenheit'.")
api_server = "https://api.yourweatherapp.com"
endpoint = f"{api_server}/weather"
params = {
"location": location,
"unit": unit
}
response = requests.get(endpoint, params=params)
return response.json()
# Example usage
weather_info = get_weather("Seattle, WA", "celsius")
print(weather_info)
Step 2: Configure Prompt Targets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Next, map the function to a prompt target, defining the intent and parameters that Plano will extract from the user’s prompt.
Specify the parameters your function needs and how Plano should interpret these.
.. code-block:: yaml
:caption: Prompt Target Example Configuration
prompt_targets:
- name: get_weather
description: Get the current weather for a location
parameters:
- name: location
description: The city and state, e.g. San Francisco, New York
type: str
required: true
- name: unit
description: The unit of temperature to return
type: str
enum: ["celsius", "fahrenheit"]
endpoint:
name: api_server
path: /weather
.. Note::
For a complete refernce of attributes that you can configure in a prompt target, see :ref:`here `.
Step 3: Plano Takes Over
~~~~~~~~~~~~~~~~~~~~~~~~
Once you have defined the functions and configured the prompt targets, Plano takes care of the remaining work.
It will automatically validate parameters, and ensure that the required parameters (e.g., location) are present in the prompt, and add validation rules if necessary.
.. figure:: /_static/img/plano_network_diagram_high_level.png
:width: 100%
:align: center
High-level network flow of where Plano sits in your agentic stack. Managing incoming and outgoing prompt traffic
Once a downstream function (API) is called, Plano takes the response and sends it an upstream LLM to complete the request (for summarization, Q/A, text generation tasks).
For more details on how Plano enables you to centralize usage of LLMs, please read :ref:`LLM providers `.
By completing these steps, you enable Plano to manage the process from validation to response, ensuring users receive consistent, reliable results - and that you are focused
on the stuff that matters most.
Example Use Cases
-----------------
Here are some common use cases where Function Calling can be highly beneficial:
- **Data Retrieval**: Extracting information from databases or APIs based on user inputs (e.g., checking account balances, retrieving order status).
- **Transactional Operations**: Executing business logic such as placing an order, processing payments, or updating user profiles.
- **Information Aggregation**: Fetching and combining data from multiple sources (e.g., displaying travel itineraries or combining analytics from various dashboards).
- **Task Automation**: Automating routine tasks like setting reminders, scheduling meetings, or sending emails.
- **User Personalization**: Tailoring responses based on user history, preferences, or ongoing interactions.
Best Practices and Tips
-----------------------
When integrating function calling into your generative AI applications, keep these tips in mind to get the most out of our Plano-Function models:
- **Keep it clear and simple**: Your function names and parameters should be straightforward and easy to understand. Think of it like explaining a task to a smart colleague - the clearer you are, the better the results.
- **Context is king**: Don't skimp on the descriptions for your functions and parameters. The more context you provide, the better the LLM can understand when and how to use each function.
- **Be specific with your parameters**: Instead of using generic types, get specific. If you're asking for a date, say it's a date. If you need a number between 1 and 10, spell that out. The more precise you are, the more accurate the LLM's responses will be.
- **Expect the unexpected**: Test your functions thoroughly, including edge cases. LLMs can be creative in their interpretations, so it's crucial to ensure your setup is robust and can handle unexpected inputs.
- **Watch and learn**: Pay attention to how the LLM uses your functions. Which ones does it call often? In what contexts? This information can help you optimize your setup over time.
Remember, working with LLMs is part science, part art. Don't be afraid to experiment and iterate to find what works best for your specific use case.
---
### Source/Guides/Llm Router
.. _llm_router:
LLM Routing
==============================================================
With the rapid proliferation of large language models (LLMs) — each optimized for different strengths, style, or latency/cost profile — routing has become an essential technique to operationalize the use of different models. Plano provides three distinct routing approaches to meet different use cases: :ref:`Model-based routing `, :ref:`Alias-based routing `, and :ref:`Preference-aligned routing `. This enables optimal performance, cost efficiency, and response quality by matching requests with the most suitable model from your available LLM fleet.
.. note::
For details on supported model providers, configuration options, and client libraries, see :ref:`LLM Providers `.
Routing Methods
---------------
.. _model_based_routing:
Model-based routing
~~~~~~~~~~~~~~~~~~~
Direct routing allows you to specify exact provider and model combinations using the format ``provider/model-name``:
- Use provider-specific names like ``openai/gpt-5.2`` or ``anthropic/claude-sonnet-4-5``
- Provides full control and transparency over which model handles each request
- Ideal for production workloads where you want predictable routing behavior
Configuration
^^^^^^^^^^^^^
Configure your LLM providers with specific provider/model names:
.. code-block:: yaml
:caption: Model-based Routing Configuration
listeners:
egress_traffic:
address: 0.0.0.0
port: 12000
message_format: openai
timeout: 30s
llm_providers:
- model: openai/gpt-5.2
access_key: $OPENAI_API_KEY
default: true
- model: openai/gpt-5
access_key: $OPENAI_API_KEY
- model: anthropic/claude-sonnet-4-5
access_key: $ANTHROPIC_API_KEY
Client usage
^^^^^^^^^^^^
Clients specify exact models:
.. code-block:: python
# Direct provider/model specification
response = client.chat.completions.create(
model="openai/gpt-5.2",
messages=[{"role": "user", "content": "Hello!"}]
)
response = client.chat.completions.create(
model="anthropic/claude-sonnet-4-5",
messages=[{"role": "user", "content": "Write a story"}]
)
.. _alias_based_routing:
Alias-based routing
~~~~~~~~~~~~~~~~~~~
Alias-based routing lets you create semantic model names that decouple your application from specific providers:
- Use meaningful names like ``fast-model``, ``reasoning-model``, or ``plano.summarize.v1`` (see :ref:`model_aliases`)
- Maps semantic names to underlying provider models for easier experimentation and provider switching
- Ideal for applications that want abstraction from specific model names while maintaining control
Configuration
^^^^^^^^^^^^^
Configure semantic aliases that map to underlying models:
.. code-block:: yaml
:caption: Alias-based Routing Configuration
listeners:
egress_traffic:
address: 0.0.0.0
port: 12000
message_format: openai
timeout: 30s
llm_providers:
- model: openai/gpt-5.2
access_key: $OPENAI_API_KEY
- model: openai/gpt-5
access_key: $OPENAI_API_KEY
- model: anthropic/claude-sonnet-4-5
access_key: $ANTHROPIC_API_KEY
model_aliases:
# Model aliases - friendly names that map to actual provider names
fast-model:
target: gpt-5.2
reasoning-model:
target: gpt-5
creative-model:
target: claude-sonnet-4-5
Client usage
^^^^^^^^^^^^
Clients use semantic names:
.. code-block:: python
# Using semantic aliases
response = client.chat.completions.create(
model="fast-model", # Routes to best available fast model
messages=[{"role": "user", "content": "Quick summary please"}]
)
response = client.chat.completions.create(
model="reasoning-model", # Routes to best reasoning model
messages=[{"role": "user", "content": "Solve this complex problem"}]
)
.. _preference_aligned_routing:
Preference-aligned routing (Plano-Orchestrator)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Preference-aligned routing uses the `Plano-Orchestrator `_ model to pick the best LLM based on domain, action, and your configured preferences instead of hard-coding a model.
- **Domain**: High-level topic of the request (e.g., legal, healthcare, programming).
- **Action**: What the user wants to do (e.g., summarize, generate code, translate).
- **Routing preferences**: Your mapping from (domain, action) to preferred models.
Plano-Orchestrator analyzes each prompt to infer domain and action, then applies your preferences to select a model. This decouples **routing policy** (how to choose) from **model assignment** (what to run), making routing transparent, controllable, and easy to extend as you add or swap models.
Configuration
^^^^^^^^^^^^^
To configure preference-aligned dynamic routing, declare a top-level ``routing_preferences`` list and attach an ordered ``models`` candidate pool to each route. Starting in ``v0.4.0``, ``routing_preferences`` lives at the root of the config (not inline under ``model_providers``), which lets multiple models serve the same route — the first entry in ``models`` is primary, the rest are fallbacks that the client tries on ``429``/``5xx`` errors.
.. code-block:: yaml
:caption: Preference-Aligned Dynamic Routing Configuration
version: v0.4.0
listeners:
- name: egress_traffic
type: model
address: 0.0.0.0
port: 12000
timeout: 30s
model_providers:
- model: openai/gpt-5.2
access_key: $OPENAI_API_KEY
default: true
- model: openai/gpt-5
access_key: $OPENAI_API_KEY
- model: anthropic/claude-sonnet-4-5
access_key: $ANTHROPIC_API_KEY
routing_preferences:
- name: code understanding
description: understand and explain existing code snippets, functions, or libraries
models:
- openai/gpt-5
- anthropic/claude-sonnet-4-5
- name: complex reasoning
description: deep analysis, mathematical problem solving, and logical reasoning
models:
- openai/gpt-5
- name: creative writing
description: creative content generation, storytelling, and writing assistance
models:
- anthropic/claude-sonnet-4-5
- name: code generation
description: generating new code snippets, functions, or boilerplate based on user prompts
models:
- anthropic/claude-sonnet-4-5
- openai/gpt-5
.. note::
Configs still using the ``v0.3.0`` inline style (``routing_preferences`` nested under each ``model_provider``) are auto-migrated to this top-level shape by the Plano CLI at compile time, with a deprecation warning. Update your config to the form above to silence the warning.
Client usage
^^^^^^^^^^^^
Clients can let the router decide or still specify aliases:
.. code-block:: python
# Let Plano-Orchestrator choose based on content
response = client.chat.completions.create(
messages=[{"role": "user", "content": "Write a creative story about space exploration"}]
# No model specified - router will analyze and choose claude-sonnet-4-5
)
.. _cost_latency_aware_selection:
Cost- and latency-aware selection
---------------------------------
When a route lists more than one candidate model, you can let Plano reorder that
candidate pool using **live cost or latency data** instead of relying solely on the
order you wrote them in. This is controlled per route with ``selection_policy`` and
backed by one or more ``model_metrics_sources``.
This is useful when several models are equally capable for a route and you want Plano
to always reach for the cheapest (or fastest) option first, with the others kept as
fallbacks.
Selection policy
~~~~~~~~~~~~~~~~~
Attach an optional ``selection_policy`` to any entry in ``routing_preferences``:
.. code-block:: yaml
:caption: Per-route selection policy
routing_preferences:
- name: code review
description: reviewing, analyzing, and suggesting improvements to existing code
models:
- anthropic/claude-sonnet-4-5
- groq/llama-3.3-70b-versatile
selection_policy:
prefer: cheapest # cheapest | fastest | none
``prefer`` accepts:
- ``cheapest`` — order candidates by total price (input + output rate) ascending, using a ``cost`` metrics source.
- ``fastest`` — order candidates by observed latency ascending, using a ``latency`` metrics source.
- ``none`` (default) — keep the order you declared; no reordering.
Models that have no data in the selected source are ranked **last**, in their original
order, so routing always degrades gracefully rather than dropping a candidate.
Configuring the pricing source
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``cheapest`` routing needs a price catalog. Plano's **default pricing provider is
DigitalOcean** — its GenAI model catalog is public (no API key, no signup), so cost data
is available out of the box and is what ``planoai obs`` uses if you don't configure
anything. The pricing source is fully swappable: point Plano at `models.dev `_,
or at **any endpoint that exposes a supported pricing structure**.
The ``provider`` field selects which response schema Plano expects (and therefore how it
parses the catalog); the optional ``url`` lets you override the endpoint — for example to
use a mirror, a cached copy, or an internal catalog service that returns the same shape.
.. list-table::
:header-rows: 1
:widths: 18 34 28 20
* - ``provider``
- Default catalog URL
- Key format
- Expected structure
* - ``digitalocean`` *(default)*
- DigitalOcean GenAI model catalog
- ``lowercase(creator)/model_id``
- ``{ data: [ { model_id, pricing: { input_price_per_million, output_price_per_million } } ] }``
* - ``models.dev``
- ``https://models.dev/api.json``
- ``creator/model`` (e.g. ``anthropic/claude-sonnet-4-5``)
- ``{ : { models: { : { cost: { input, output } } } } }``
Because the source is selected per ``provider``, switching is a one-line change. To stay
on the default DigitalOcean catalog you can omit ``model_metrics_sources`` entirely for
``planoai obs``, or declare it explicitly for routing:
.. code-block:: yaml
:caption: Default cost source (DigitalOcean)
model_metrics_sources:
- type: cost
provider: digitalocean # default; uses the public DO GenAI catalog
To switch to models.dev — an open, community-maintained catalog covering a broad range of
providers and models — change the ``provider`` (and optionally ``url``):
.. code-block:: yaml
:caption: Cost source backed by models.dev
model_metrics_sources:
- type: cost
provider: models.dev # models.dev | digitalocean
url: https://models.dev/api.json # optional; defaults per provider
refresh_interval: 3600 # optional, seconds; refetch on this interval
model_aliases: # optional; see below
openai/gpt-oss-120b: openai/gpt-4o
To use your own endpoint, pick the ``provider`` whose structure your endpoint matches and
override ``url`` — Plano parses the response with that provider's schema:
.. code-block:: yaml
:caption: Custom endpoint exposing the DigitalOcean catalog structure
model_metrics_sources:
- type: cost
provider: digitalocean # selects the DO response schema
url: https://catalog.internal.example.com/pricing
.. note::
The cost metric used for ranking is the sum of the input and output per-million-token
rates — a relative signal for ordering candidates, not a per-request bill. For actual
per-request cost, see the observability console below.
Matching catalog keys to your models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The router looks up each candidate model by the exact name you use in
``routing_preferences`` (e.g. ``anthropic/claude-sonnet-4-5``). models.dev keys models as
``creator/model``, which lines up with Plano's ``provider/model`` naming, so most models
match automatically.
When a catalog key does not match your model name — for example a version skew, or an
open-weight model you serve under a different provider — use ``model_aliases`` to map the
**catalog key** to the **Plano model name** used in your routing preferences:
.. code-block:: yaml
model_metrics_sources:
- type: cost
provider: models.dev
model_aliases:
# catalog key : plano model name
openai/gpt-oss-120b: openai/gpt-4o
Latency source
~~~~~~~~~~~~~~~
``fastest`` routing reads observed latency from a Prometheus instance. Provide the query
that returns a per-model latency value (lower is faster), labelled by ``model_name``:
.. code-block:: yaml
:caption: Latency source backed by Prometheus
model_metrics_sources:
- type: latency
provider: prometheus
url: http://prometheus:9090
query: avg by (model_name) (rate(plano_llm_latency_seconds_sum[5m]))
refresh_interval: 60
You can declare both a ``cost`` and a ``latency`` source at the same time; each route
picks whichever it needs based on its ``selection_policy``.
Cost in the observability console
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``planoai obs`` displays a per-request USD cost column derived from the same pricing
catalog. By default it reads the ``cost`` source from your config (the first
``type: cost`` entry under ``model_metrics_sources``); you can also override it on the
command line:
.. code-block:: bash
# Use the cost source from ./config.yaml (default)
planoai obs
# Or override the provider / endpoint explicitly
planoai obs --pricing-provider models.dev
planoai obs --pricing-url https://models.dev/api.json
If no source is configured and no override is given, ``planoai obs`` falls back to the
DigitalOcean catalog so the cost column still populates out of the box.
Plano-Orchestrator
-------------------
Plano-Orchestrator is a **preference-based routing model** specifically designed to address the limitations of traditional LLM routing. It delivers production-ready performance with low latency and high accuracy while solving key routing challenges.
**Addressing Traditional Routing Limitations:**
**Human Preference Alignment**
Unlike benchmark-driven approaches, Plano-Orchestrator learns to match queries with human preferences by using domain-action mappings that capture subjective evaluation criteria, ensuring routing decisions align with real-world user needs.
**Flexible Model Integration**
The system supports seamlessly adding new models for routing without requiring retraining or architectural modifications, enabling dynamic adaptation to evolving model landscapes.
**Preference-Encoded Routing**
Provides a practical mechanism to encode user preferences through domain-action mappings, offering transparent and controllable routing decisions that can be customized for specific use cases.
To support effective routing, Plano-Orchestrator introduces two key concepts:
- **Domain** – the high-level thematic category or subject matter of a request (e.g., legal, healthcare, programming).
- **Action** – the specific type of operation the user wants performed (e.g., summarization, code generation, booking appointment, translation).
Both domain and action configs are associated with preferred models or model variants. At inference time, Plano-Orchestrator analyzes the incoming prompt to infer its domain and action using semantic similarity, task indicators, and contextual cues. It then applies the user-defined routing preferences to select the model best suited to handle the request.
In summary, Plano-Orchestrator demonstrates:
- **Structured Preference Routing**: Aligns prompt request with model strengths using explicit domain–action mappings.
- **Transparent and Controllable**: Makes routing decisions transparent and configurable, empowering users to customize system behavior.
- **Flexible and Adaptive**: Supports evolving user needs, model updates, and new domains/actions without retraining the router.
- **Production-Ready Performance**: Optimized for low-latency, high-throughput applications in multi-model environments.
Self-hosting Plano-Orchestrator
-------------------------------
By default, Plano uses a hosted Plano-Orchestrator endpoint. To run Plano-Orchestrator locally, you can serve the model yourself using either **Ollama** or **vLLM**.
Using Ollama (recommended for local development)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1. **Install Ollama**
Download and install from `ollama.ai `_.
2. **Pull and serve the routing model**
.. code-block:: bash
ollama pull hf.co/katanemo/Arch-Router-1.5B.gguf:Q4_K_M
ollama serve
This downloads the quantized GGUF model from HuggingFace and starts serving on ``http://localhost:11434``.
3. **Configure Plano to use local routing model**
.. code-block:: yaml
version: v0.4.0
overrides:
llm_routing_model: plano/hf.co/katanemo/Arch-Router-1.5B.gguf:Q4_K_M
model_providers:
- model: plano/hf.co/katanemo/Arch-Router-1.5B.gguf:Q4_K_M
base_url: http://localhost:11434
- model: openai/gpt-5.2
access_key: $OPENAI_API_KEY
default: true
- model: anthropic/claude-sonnet-4-5
access_key: $ANTHROPIC_API_KEY
routing_preferences:
- name: creative writing
description: creative content generation, storytelling, and writing assistance
models:
- anthropic/claude-sonnet-4-5
4. **Verify the model is running**
.. code-block:: bash
curl http://localhost:11434/v1/models
You should see ``Arch-Router-1.5B`` listed in the response.
Using vLLM (recommended for production / EC2)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vLLM provides higher throughput and GPU optimizations suitable for production deployments.
1. **Install vLLM**
.. code-block:: bash
pip install vllm
2. **Download the model weights**
The GGUF weights are downloaded automatically from HuggingFace on first use. To pre-download:
.. code-block:: bash
pip install huggingface_hub
huggingface-cli download katanemo/Arch-Router-1.5B.gguf
3. **Start the vLLM server**
After downloading, find the GGUF file and Jinja template in the HuggingFace cache:
.. code-block:: bash
# Find the downloaded files
SNAPSHOT_DIR=$(ls -d ~/.cache/huggingface/hub/models--katanemo--Arch-Router-1.5B.gguf/snapshots/*/ | head -1)
vllm serve ${SNAPSHOT_DIR}Arch-Router-1.5B-Q4_K_M.gguf \
--host 0.0.0.0 \
--port 10000 \
--load-format gguf \
--chat-template ${SNAPSHOT_DIR}template.jinja \
--tokenizer katanemo/Arch-Router-1.5B \
--served-model-name Plano-Orchestrator \
--gpu-memory-utilization 0.3 \
--tensor-parallel-size 1 \
--enable-prefix-caching
4. **Configure Plano to use the vLLM endpoint**
.. code-block:: yaml
version: v0.4.0
overrides:
llm_routing_model: plano/Plano-Orchestrator
model_providers:
- model: plano/Plano-Orchestrator
base_url: http://:10000
- model: openai/gpt-5.2
access_key: $OPENAI_API_KEY
default: true
- model: anthropic/claude-sonnet-4-5
access_key: $ANTHROPIC_API_KEY
routing_preferences:
- name: creative writing
description: creative content generation, storytelling, and writing assistance
models:
- anthropic/claude-sonnet-4-5
5. **Verify the server is running**
.. code-block:: bash
curl http://localhost:10000/health
curl http://localhost:10000/v1/models
Using vLLM on Kubernetes (GPU nodes)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For teams running Kubernetes, Plano-Orchestrator and Plano can be deployed as in-cluster services.
The ``demos/llm_routing/model_routing_service/`` directory includes ready-to-use manifests:
- ``vllm-deployment.yaml`` — Plano-Orchestrator served by vLLM, with an init container to download
the model from HuggingFace
- ``plano-deployment.yaml`` — Plano proxy configured to use the in-cluster Plano-Orchestrator
- ``config_k8s.yaml`` — Plano config with ``llm_routing_model`` pointing at
``http://plano-orchestrator:10000`` instead of the default hosted endpoint
Key things to know before deploying:
- GPU nodes commonly have a ``nvidia.com/gpu:NoSchedule`` taint — the ``vllm-deployment.yaml``
includes a matching toleration. The ``nvidia.com/gpu: "1"`` resource request is sufficient
for scheduling in most clusters; a ``nodeSelector`` is optional and commented out in the
manifest for cases where you need to pin to a specific GPU node pool.
- Model download takes ~1 minute; vLLM loads the model in ~1-2 minutes after that. The
``livenessProbe`` has a 180-second ``initialDelaySeconds`` to avoid premature restarts.
- The Plano config ConfigMap must use ``--from-file=plano_config.yaml=config_k8s.yaml`` with
``subPath`` in the Deployment — omitting ``subPath`` causes Kubernetes to mount a directory
instead of a file.
For the canonical Plano Kubernetes deployment (ConfigMap, Secrets, Deployment YAML), see
:ref:`deployment`. For full step-by-step commands specific to this demo, see the
`demo README `_.
.. _model_affinity:
Model Affinity
--------------
In agentic loops — where a single user request triggers multiple LLM calls through tool use — Plano's router classifies each turn independently. Because successive prompts differ in intent (tool selection looks like code generation, reasoning about results looks like analysis), the router may select different models mid-session. This causes behavioral inconsistency and invalidates provider-side KV caches, increasing both latency and cost.
**Model affinity** pins the routing decision for the duration of a session. Send an ``X-Model-Affinity`` header with any string identifier (typically a UUID). The first request routes normally and caches the result. All subsequent requests with the same affinity ID skip routing and reuse the cached model.
.. code-block:: python
import uuid
from openai import OpenAI
client = OpenAI(base_url="http://localhost:12000/v1", api_key="EMPTY")
affinity_id = str(uuid.uuid4())
# Every call in the loop uses the same header
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=tools,
extra_headers={"X-Model-Affinity": affinity_id},
)
Without the header, routing runs fresh on every request — no behavior change for existing clients.
**Configuration:**
.. code-block:: yaml
routing:
session_ttl_seconds: 600 # How long affinity lasts (default: 10 min)
session_max_entries: 10000 # Max cached sessions (upper limit: 10000)
To start a new routing decision (e.g., when the agent's task changes), generate a new affinity ID.
Session Cache Backends
~~~~~~~~~~~~~~~~~~~~~~
By default, Plano stores session affinity state in an in-process LRU cache. This works well for single-instance deployments, but sessions are not shared across replicas — each instance has its own independent cache.
For deployments with multiple Plano replicas (Kubernetes, Docker Compose with ``scale``, or any load-balanced setup), use Redis as the session cache backend. All replicas connect to the same Redis instance, so an affinity decision made by one replica is honoured by every other replica in the pool.
**In-memory (default)**
No configuration required. Sessions live only for the lifetime of the process and are lost on restart.
.. code-block:: yaml
routing:
session_ttl_seconds: 600 # How long affinity lasts (default: 10 min)
session_max_entries: 10000 # LRU capacity (upper limit: 10000)
**Redis**
Requires a reachable Redis instance. The ``url`` field supports standard Redis URI syntax, including authentication (``redis://:password@host:6379``) and TLS (``rediss://host:6380``). Redis handles TTL expiry natively, so no periodic cleanup is needed.
.. code-block:: yaml
routing:
session_ttl_seconds: 600
session_cache:
type: redis
url: redis://localhost:6379
.. note::
When using Redis in a multi-tenant environment, construct the ``X-Model-Affinity`` header value to include a tenant identifier, for example ``{tenant_id}:{session_id}``. Plano stores each key under the internal namespace ``plano:affinity:{key}``, so tenant-scoped values avoid cross-tenant collisions without any additional configuration.
**Example: Kubernetes multi-replica deployment**
Deploy a Redis instance alongside your Plano pods and point all replicas at it:
.. code-block:: yaml
routing:
session_ttl_seconds: 600
session_cache:
type: redis
url: redis://redis.plano.svc.cluster.local:6379
With this configuration, any replica that first receives a request for affinity ID ``abc-123`` caches the routing decision in Redis. Subsequent requests for ``abc-123`` — regardless of which replica they land on — retrieve the same pinned model.
Combining Routing Methods
-------------------------
You can combine static model selection with dynamic routing preferences for maximum flexibility:
.. code-block:: yaml
:caption: Hybrid Routing Configuration
version: v0.4.0
model_providers:
- model: openai/gpt-5.2
access_key: $OPENAI_API_KEY
default: true
- model: openai/gpt-5
access_key: $OPENAI_API_KEY
- model: anthropic/claude-sonnet-4-5
access_key: $ANTHROPIC_API_KEY
routing_preferences:
- name: complex_reasoning
description: deep analysis and complex problem solving
models:
- openai/gpt-5
- anthropic/claude-sonnet-4-5
- name: creative_tasks
description: creative writing and content generation
models:
- anthropic/claude-sonnet-4-5
- openai/gpt-5
model_aliases:
# Model aliases - friendly names that map to actual provider names
fast-model:
target: gpt-5.2
reasoning-model:
target: gpt-5
# Aliases that can also participate in dynamic routing
creative-model:
target: claude-sonnet-4-5
This configuration allows clients to:
1. **Use direct model selection**: ``model="fast-model"``
2. **Let the router decide**: No model specified, router analyzes content
Example Use Cases
-----------------
Here are common scenarios where Plano-Orchestrator excels:
- **Coding Tasks**: Distinguish between code generation requests ("write a Python function"), debugging needs ("fix this error"), and code optimization ("make this faster"), routing each to appropriately specialized models.
- **Content Processing Workflows**: Classify requests as summarization ("summarize this document"), translation ("translate to Spanish"), or analysis ("what are the key themes"), enabling targeted model selection.
- **Multi-Domain Applications**: Accurately identify whether requests fall into legal, healthcare, technical, or general domains, even when the subject matter isn't explicitly stated in the prompt.
- **Conversational Routing**: Track conversation context to identify when topics shift between domains or when the type of assistance needed changes mid-conversation.
Best practices
--------------
- **💡Consistent Naming:** Route names should align with their descriptions.
- ❌ Bad:
```
{"name": "math", "description": "handle solving quadratic equations"}
```
- ✅ Good:
```
{"name": "quadratic_equation", "description": "solving quadratic equations"}
```
- **💡 Clear Usage Description:** Make your route names and descriptions specific, unambiguous, and minimizing overlap between routes. The Router performs better when it can clearly distinguish between different types of requests.
- ❌ Bad:
```
{"name": "math", "description": "anything closely related to mathematics"}
```
- ✅ Good:
```
{"name": "math", "description": "solving, explaining math problems, concepts"}
```
- **💡Nouns Descriptor:** Preference-based routers perform better with noun-centric descriptors, as they offer more stable and semantically rich signals for matching.
- **💡Domain Inclusion:** for best user experience, you should always include a domain route. This helps the router fall back to domain when action is not confidently inferred.
Unsupported Features
--------------------
The following features are **not supported** by the Plano-Orchestrator routing model:
- **Multi-modality**: The model is not trained to process raw image or audio inputs. It can handle textual queries *about* these modalities (e.g., "generate an image of a cat"), but cannot interpret encoded multimedia data directly.
- **Function calling**: Plano-Orchestrator is designed for **semantic preference matching**, not exact intent classification or tool execution. For structured function invocation, use models in the Plano Function Calling collection instead.
- **System prompt dependency**: Plano-Orchestrator routes based solely on the user’s conversation history. It does not use or rely on system prompts for routing decisions.
---