## File: README.md
**English** | [中文](./.github/README.cn.md) | [日本語](./.github/README.jp.md)
> [!IMPORTANT]
> :rocket: Gateway 2.0 (Pre-Release) Portkey's core enterprise gateway is merging into open-source with our 2.0 release. You can try the pre-release branch [here](https://github.com/portkey-ai/gateway/tree/2.0.0).
> Read more about what's next for Portkey in our [**Series A announcement**](https://portkey.wiki/rohit-a).
The [**AI Gateway**](https://portkey.wiki/gh-10) is designed for fast, reliable & secure routing to 1600+ language, vision, audio, and image models. It is a lightweight, open-source, and enterprise-ready solution that allows you to integrate with any language model in under 2 minutes.
- [x] **Blazing fast** (<1ms latency) with a tiny footprint (122kb)
- [x] **Battle tested**, with over 10B tokens processed everyday
- [x] **Enterprise-ready** with enhanced security, scale, and custom deployments
#### What can you do with the AI Gateway?
- Integrate with any LLM in under 2 minutes - [Quickstart](#quickstart-2-mins)
- Prevent downtimes through **[automatic retries](https://portkey.wiki/gh-11)** and **[fallbacks](https://portkey.wiki/gh-12)**
- Scale AI apps with **[load balancing](https://portkey.wiki/gh-13)** and **[conditional routing](https://portkey.wiki/gh-14)**
- Protect your AI deployments with **[guardrails](https://portkey.wiki/gh-15)**
- Go beyond text with **[multi-modal capabilities](https://portkey.wiki/gh-16)**
- Explore **[agentic workflow](https://portkey.wiki/gh-17)** integrations
- Manage MCP servers with enterprise auth & observability using **[MCP Gateway](https://portkey.ai/docs/product/mcp-gateway)**
> [!TIP]
> Starring this repo helps more developers discover the AI Gateway 🙏🏻
>
>
>
## Quickstart (2 mins)
### 1. Setup your AI Gateway
```bash
# Run the gateway locally (needs Node.js and npm)
npx @portkey-ai/gateway
```
> The Gateway is running on `http://localhost:8787/v1`
>
> The Gateway Console is running on `http://localhost:8787/public/`
Deployment guides:
[ Portkey Cloud (Recommended)](https://portkey.wiki/gh-18)
[ Docker](./docs/installation-deployments.md#docker)
[ Node.js](./docs/installation-deployments.md#nodejs-server)
[ Cloudflare](./docs/installation-deployments.md#cloudflare-workers)
[ Replit](./docs/installation-deployments.md#replit)
[ Others...](./docs/installation-deployments.md)
### 2. Make your first request
```python
# pip install -qU portkey-ai
from portkey_ai import Portkey
# OpenAI compatible client
client = Portkey(
provider="openai", # or 'anthropic', 'bedrock', 'groq', etc
Authorization="sk-***" # the provider API key
)
# Make a request through your AI Gateway
client.chat.completions.create(
messages=[{"role": "user", "content": "What's the weather like?"}],
model="gpt-4o-mini"
)
```
Supported Libraries:
[ JS](https://portkey.wiki/gh-19)
[ Python](https://portkey.wiki/gh-20)
[ REST](https://portkey.sh/gh-84)
[ OpenAI SDKs](https://portkey.wiki/gh-21)
[ Langchain](https://portkey.wiki/gh-22)
[LlamaIndex](https://portkey.wiki/gh-23)
[Autogen](https://portkey.wiki/gh-24)
[CrewAI](https://portkey.wiki/gh-25)
[More..](https://portkey.wiki/gh-26)
On the Gateway Console (`http://localhost:8787/public/`) you can see all of your local logs in one place.
### 3. Routing & Guardrails
`Configs` in the LLM gateway allow you to create routing rules, add reliability and setup guardrails.
```python
config = {
"retry": {"attempts": 5},
"output_guardrails": [{
"default.contains": {"operator": "none", "words": ["Apple"]},
"deny": True
}]
}
# Attach the config to the client
client = client.with_options(config=config)
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}]
)
# This would always response with "Bat" as the guardrail denies all replies containing "Apple". The retry config would retry 5 times before giving up.
```
You can do a lot more stuff with configs in your AI gateway. [Jump to examples →](https://portkey.wiki/gh-27)
### Enterprise Version (Private deployments)
[ AWS](https://portkey.wiki/gh-28)
[ Azure](https://portkey.wiki/gh-29)
[ GCP](https://portkey.wiki/gh-30)
[ OpenShift](https://portkey.wiki/gh-31)
[ Kubernetes](https://portkey.wiki/gh-85)
The LLM Gateway's [enterprise version](https://portkey.wiki/gh-86) offers advanced capabilities for **org management**, **governance**, **security** and [more](https://portkey.wiki/gh-87) out of the box. [View Feature Comparison →](https://portkey.wiki/gh-32)
The enterprise deployment architecture for supported platforms is available here - [**Enterprise Private Cloud Deployments**](https://portkey.ai/docs/self-hosting/hybrid-deployments/architecture)
[](https://portkey.sh/demo-13)
## MCP Gateway
[MCP Gateway](https://portkey.ai/docs/product/mcp-gateway) provides a centralized control plane for managing MCP (Model Context Protocol) servers across your organization.
- **Authentication** — Single auth layer at the gateway. Users authenticate once; your MCP servers receive verified requests
- **Access Control** — Control which teams and users can access which servers and tools. Revoke access instantly
- **Observability** — Every tool call logged with full context: who called what, parameters, response, latency
- **Identity Forwarding** — Forward user identity (email, team, roles) to MCP servers automatically
Works with Claude Desktop, Cursor, VS Code, and any MCP-compatible client. [Get started →](https://portkey.ai/docs/product/mcp-gateway/quickstart)
## Core Features
### Reliable Routing
- [**Fallbacks**](https://portkey.wiki/gh-37): Fallback to another provider or model on failed requests using the LLM gateway. You can specify the errors on which to trigger the fallback. Improves reliability of your application.
- [**Automatic Retries**](https://portkey.wiki/gh-38): Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload.
- [**Load Balancing**](https://portkey.wiki/gh-39): Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance.
- [**Request Timeouts**](https://portkey.wiki/gh-40): Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration.
- [**Multi-modal LLM Gateway**](https://portkey.wiki/gh-41): Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers — all using the familiar OpenAI signature
- [**Realtime APIs**](https://portkey.wiki/gh-42): Call realtime APIs launched by OpenAI through the integrate websockets server.
### Security & Accuracy
- [**Guardrails**](https://portkey.wiki/gh-88): Verify your LLM inputs and outputs to adhere to your specified checks. Choose from the 40+ pre-built guardrails to ensure compliance with security and accuracy standards. You can [bring your own guardrails](https://portkey.wiki/gh-43) or choose from our [many partners](https://portkey.wiki/gh-44).
- [**Secure Key Management**](https://portkey.wiki/gh-45): Use your own keys or generate virtual keys on the fly.
- [**Role-based access control**](https://portkey.wiki/gh-46): Granular access control for your users, workspaces and API keys.
- [**Compliance & Data Privacy**](https://portkey.wiki/gh-47): The AI gateway is SOC2, HIPAA, GDPR, and CCPA compliant.
### Cost Management
- [**Smart caching**](https://portkey.wiki/gh-48): Cache responses from LLMs to reduce costs and improve latency. Supports simple and semantic* caching.
- [**Usage analytics**](https://portkey.wiki/gh-49): Monitor and analyze your AI and LLM usage, including request volume, latency, costs and error rates.
- [**Provider optimization***](https://portkey.wiki/gh-89): Automatically switch to the most cost-effective provider based on usage patterns and pricing models.
### Collaboration & Workflows
- [**Agents Support**](https://portkey.ai/docs/integrations/agents): Seamlessly integrate with popular agent frameworks to build complex AI applications. The gateway seamlessly integrates with [Autogen](https://portkey.wiki/gh-50), [CrewAI](https://portkey.wiki/gh-51), [LangChain](https://portkey.wiki/gh-52), [LlamaIndex](https://portkey.wiki/gh-53), [Phidata](https://portkey.wiki/gh-54), [Control Flow](https://portkey.wiki/gh-55), and even [Custom Agents](https://portkey.wiki/gh-56).
- [**Prompt Template Management***](https://portkey.wiki/gh-57): Create, manage and version your prompt templates collaboratively through a universal prompt playground.
* Available in hosted and enterprise versions
## Portkey Models
Open-source LLM pricing database for 40+ providers - used by the Gateway for cost tracking.
[GitHub](https://github.com/Portkey-AI/models) | [Model Explorer](https://portkey.ai/models)
## Cookbooks
### ☄️ Trending
- Use models from [Nvidia NIM](/cookbook/providers/nvidia.ipynb) with AI Gateway
- Monitor [CrewAI Agents](/cookbook/monitoring-agents/CrewAI_with_Telemetry.ipynb) with Portkey!
- Comparing [Top 10 LMSYS Models](/cookbook/use-cases/LMSYS%20Series/comparing-top10-LMSYS-models-with-Portkey.ipynb) with AI Gateway.
### 🚨 Latest
* [Create Synthetic Datasets using Nemotron](/cookbook/use-cases/Nemotron_GPT_Finetuning_Portkey.ipynb)
* [Use the LLM Gateway with Vercel's AI SDK](/cookbook/integrations/vercel-ai.md)
* [Monitor Llama Agents with Portkey's LLM Gateway](/cookbook/monitoring-agents/Llama_Agents_with_Telemetry.ipynb)
[View all cookbooks →](https://portkey.wiki/gh-58)
## Supported Providers
Explore Gateway integrations with [45+ providers](https://portkey.wiki/gh-59) and [8+ agent frameworks](https://portkey.wiki/gh-90).
| | Provider | Support | Stream |
| -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ------- | ------ |
| | [OpenAI](https://portkey.wiki/gh-60) | ✅ | ✅ |
| | [Azure OpenAI](https://portkey.wiki/gh-61) | ✅ | ✅ |
| | [Anyscale](https://portkey.wiki/gh-62) | ✅ | ✅ |
| | [Google Gemini](https://portkey.wiki/gh-63) | ✅ | ✅ |
| | [Anthropic](https://portkey.wiki/gh-64) | ✅ | ✅ |
| | [Cohere](https://portkey.wiki/gh-65) | ✅ | ✅ |
| | [Together AI](https://portkey.wiki/gh-66) | ✅ | ✅ |
| | [Perplexity](https://portkey.wiki/gh-67) | ✅ | ✅ |
| | [Mistral](https://portkey.wiki/gh-68) | ✅ | ✅ |
| | [Nomic](https://portkey.wiki/gh-69) | ✅ | ✅ |
| | [AI21](https://portkey.wiki/gh-91) | ✅ | ✅ |
| | [Stability AI](https://portkey.wiki/gh-71) | ✅ | ✅ |
| | [DeepInfra](https://portkey.sh/gh-92) | ✅ | ✅ |
| | [Ollama](https://portkey.wiki/gh-72) | ✅ | ✅ |
| | [Novita AI](https://portkey.wiki/gh-73) | ✅ | ✅ | `/chat/completions`, `/completions` |
> [View the complete list of 200+ supported models here](https://portkey.wiki/gh-74)
## Agents
Gateway seamlessly integrates with popular agent frameworks. [Read the documentation here](https://portkey.wiki/gh-75).
| Framework | Call 200+ LLMs | Advanced Routing | Caching | Logging & Tracing* | Observability* | Prompt Management* |
|------------------------------|--------|-------------|---------|------|---------------|-------------------|
| [Autogen](https://portkey.wiki/gh-93) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [CrewAI](https://portkey.wiki/gh-94) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [LangChain](https://portkey.wiki/gh-95) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Phidata](https://portkey.wiki/gh-96) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Llama Index](https://portkey.wiki/gh-97) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Control Flow](https://portkey.wiki/gh-98) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Build Your Own Agents](https://portkey.wiki/gh-99) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| | [IO Intelligence](https://io.net/intelligence) | ✅ | ✅ |
*Available on the [hosted app](https://portkey.wiki/gh-76). For detailed documentation [click here](https://portkey.wiki/gh-100).
## Gateway Enterprise Version
Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.
✅ Secure Key Management - for role-based access control and tracking
✅ Simple & Semantic Caching - to serve repeat queries faster & save costs
✅ Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
✅ PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
✅ SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
✅ Professional Support - along with feature prioritization
[Schedule a call to discuss enterprise deployments](https://portkey.sh/demo-13)
## Contributing
The easiest way to contribute is to pick an issue with the `good first issue` tag 💪. Read the contribution guidelines [here](/.github/CONTRIBUTING.md).
Bug Report? [File here](https://portkey.wiki/gh-78) | Feature Request? [File here](https://portkey.wiki/gh-78)
### Getting Started with the Community
Join our weekly AI Engineering Hours every Friday (8 AM PT) to:
- Meet other contributors and community members
- Learn advanced Gateway features and implementation patterns
- Share your experiences and get help
- Stay updated with the latest development priorities
[Join the next session →](https://portkey.wiki/gh-101) | [Meeting notes](https://portkey.wiki/gh-102)
## Community
Join our growing community around the world, for help, ideas, and discussions on AI.
- View our official [Blog](https://portkey.wiki/gh-78)
- Chat with us on [Discord](https://portkey.wiki/community)
- Follow us on [Twitter](https://portkey.wiki/gh-79)
- Connect with us on [LinkedIn](https://portkey.wiki/gh-80)
- Read the documentation in [Japanese](./.github/README.jp.md)
- Visit us on [YouTube](https://portkey.wiki/gh-103)
- Join our [Dev community](https://portkey.wiki/gh-82)
---
## File: cookbook/integrations/vercel/readme.md
# Portkey + Vercel AI SDK
Portkey natively integrates with the Vercel AI SDK to make your apps production-ready and reliable. Just import Portkey's Vercel package and use it as a provider in your Vercel AI app to enable all of Portkey features:
- Full-stack observability and tracing for all requests
- Interoperability across 250+ LLMS
- Built-in 50+ SOTA guardrails
- Simple & semantic caching to save costs & time
- Route requests conditionally and make them robust with fallbacks, load-balancing, automatic retries, and more
- Continuous improvement based on user feedback
## Getting Started
Getting Started
Follow these steps to get your Vercel app up and running:
Install Dependencies
Run the following command to install all necessary dependencies:
Copynpm install
Configure API Keys
Add your API keys to the appropriate configuration file. This step is crucial for the app to function correctly.
Start the App
Once you've completed the above steps, your app will be running locally. You can access it at:
Copyhttp://localhost:3000
(Note: The port number might be different depending on your specific Vercel app configuration)
### 1. Installation
```sh
npm install @portkey-ai/vercel-provider
npm install
```
### 3. Configure API Keys
Add your API keys to the appropriate configuration file. This step is crucial for the app to function correctly.
Start the App
### 2. Import & Configure Portkey Object
Sign up for Portkey and get your API key, and configure Portkey provider in your Vercel app:
```javascript
import { createPortkey } from '@portkey-ai/vercel-provider';
const portkeyConfig = {
"provider": "openai", // Choose your provider (e.g., 'anthropic')
"api_key": "OPENAI_API_KEY",
"override_params": {
"model": "gpt-4" // Select from 250+ models
}
};
const portkey = createPortkey({
apiKey: 'YOUR_PORTKEY_API_KEY',
config: portkeyConfig,
});
```
### 3. Configure API Keys
Add your API keys to the appropriate configuration file. This step is crucial for the app to function correctly.
### 4. Start the App
Once you've completed the above steps use `npm run dev` to run your app locally. You can access it at:
`http://localhost:3000`
Portkey's configs are a powerful way to manage & govern your app's behaviour. Learn more about Configs [here](https://docs.portkey.ai).
## Using Vercel Functions
Portkey provider works with all of Vercel functions `generateText` & `streamText`.
Here's how to use them with Portkey:
### generateText
```javascript
import { createPortkey } from '@portkey-ai/vercel-provider';
import { generateText } from 'ai';
const portkeyConfig = {
"provider": "openai", // Choose your provider (e.g., 'anthropic')
"api_key": "OPENAI_API_KEY",
"override_params": {
"model": "gpt-4"
}
};
const portkey = createPortkey({
apiKey: 'YOUR_PORTKEY_API_KEY',
config: portkeyConfig,
});
const { text } = await generateText({
model: portkey.chatModel(''), // Provide an empty string, we defined the model in the config
prompt: 'What is Portkey?',
});
console.log(text);
```
### streamText
```js
import { createPortkey } from '@portkey-ai/vercel-provider';
import { streamText } from 'ai';
const portkeyConfig = {
"provider": "openai", // Choose your provider (e.g., 'anthropic')
"api_key": "OPENAI_API_KEY",
"override_params": {
"model": "gpt-4o" // Select from 250+ models
}
};
const portkey = createPortkey({
apiKey: 'YOUR_PORTKEY_API_KEY',
config: portkeyConfig,
});
const result = await streamText({
model: portkey('gpt-4-turbo'), // This gets overwritten by config
prompt: 'Invent a new holiday and describe its traditions.',
});
for await (const chunk of result) {
console.log(chunk);
}
```
Portkey supports `chatModel` and `completionModel` to easily handle chatbots or text completions. In the above examples, we used `portkey.chatModel` for generateText.
## Tool Calling with Portkey
Portkey supports Tool calling with Vercel AI SDK. Here's how:
```javascript
import { z } from 'zod';
import { generateText, tool } from 'ai';
const result = await generateText({
model: portkey.chatModel('gpt-4-turbo'),
tools: {
weather: tool({
parameters: z.object({
location: z.string().describe('The location to get the weather for'),
}),
execute: async ({ location }) => ({
location,
temperature: 72 + Math.floor(Math.random() * 21) - 10,
}),
}),
},
prompt: 'What is the weather in San Francisco?',
});
```
## Portkey Features
Portkey Helps you make your Vercel app more robust and reliable. The portkey config is a modular way to make it work for you in whatever way you want.
### Interoperability
Portkey allows you to easily switch between 250+ AI models by simply changing the model name in your configuration. This flexibility enables you to adapt to the evolving AI landscape without significant code changes.
#### Switch from OpenAI to Anthropic
Here's how you'd use OpenAI with Portkey's Vercel integration:
```javascript
const portkeyConfig = {
"provider": "openai",
"api_key": "OPENAI_API_KEY",
"override_params": {
model: "gpt-4"
}
};
```
Now, to switch to Anthropic, just change your provider slug to anthropic and enter your Anthropic API key along with the model of choice:
```javascript
const portkeyConfig = {
"provider": "anthropic",
"api_key": "Anthropic_API_KEY",
"override_params": {
"model": "claude-3-sonnet-20240229"
}
};
```
### Observability
Portkey's OpenTelemetry-compliant observability suite gives you complete control over all your requests. And Portkey's analytics dashboards provide 40+ key insights you're looking for including cost, tokens, latency, etc. Fast.
### Reliability
Portkey enhances the robustness of your AI applications with built-in features such as Caching, Fallback mechanisms, Load balancing, Conditional routing, Request timeouts, etc.
Here is how you can modify your config to include the following Portkey features:
#### Fallback
```javascript
import { createPortkey } from '@portkey-ai/vercel-provider';
import { generateText } from 'ai';
const portkeyConfig = {
"strategy": {
"mode": "fallback"
},
"targets": [
{
"provider": "anthropic",
"api_key": "Anthropic_API_KEY",
"override_params": {
"model": "claude-3-sonnet-20240229"
} },
{
"provider": "openai",
"api_key": "OPENAI_API_KEY",
"override_params": {
"model": "gpt-4"
} }
]
}
const portkey = createPortkey({
apiKey: 'YOUR_PORTKEY_API_KEY',
config: portkeyConfig,
});
const { text } = await generateText({
model: portkey.chatModel(''),
prompt: 'What is Portkey?',
});
console.log(text);
```
Learn more about Portkey's AI gateway features in detail [here](https://docs.portkey.ai/features/ai-gateway).
### Guardrails
Portkey Guardrails allow you to enforce LLM behavior in real-time, verifying both inputs and outputs against specified checks.
You can create Guardrail checks in UI and then pass them in your Portkey Configs with before request or after request hooks.
Read more about Guardrails [here](https://docs.portkey.ai/features/guardrails).
## Portkey Config
Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
---
## File: cookbook/use-cases/LMSYS Series/README.md
# LMSYS Series
LMSYS is THE defining leaderboard to compare raw capabilities of various LLMs. It has so far collected over **1,000,000** human pairwise comparisons to rank LLMs in Elo-scale.
We are creating cookbooks that help you get the best value of of LMSYS leaderboard for **production use cases**, by leveraging Portkey's Gateway, Observability, and Prompt Management features.
## Cookbooks
1. [Comparing the Top 10 LMSYS models using a single function.](./comparing-top10-LMSYS-models-with-Portkey.ipynb)
2. RAG Benchmark Over LMSYS Models (coming soon!)
---
## File: cookbook/README.md
# Portkey Practitioners' Cookbooks
#### Strategies and examples for tackling the production challenges of LLMs with Portkey Gateway
[](https://portkey.ai/community)
[](https://twitter.com/portkeyai)
## Table of Contents
Please use the below table of contents to navigate through the cookbook.
### Latest Notebooks
- Use models from [Nvidia NIM](/cookbook/providers/nvidia.ipynb) with AI Gateway
- Monitor [Llama Agents](/cookbook/monitoring-agents/Llama_Agents_with_Telemetry.ipynb) with Portkey!
- Create [Synthetic Datasets](/cookbook/use-cases/Nemotron_GPT_Finetuning_Portkey.ipynb) using Nemotron-340B
- Comparing [Top 10 LMSYS Models](./use-cases/LMSYS%20Series/comparing-top10-LMSYS-models-with-Portkey.ipynb) with AI Gateway.
## getting-started
* [Gentle introduction to Portkey Gateway](./getting-started/gentle-introduction-to-portkey-gateway.ipynb)
* [Use Portkey cache to save LLM cost & time](./getting-started/enable-cache.md)
* [Retry automatically on LLM failures](./getting-started/automatic-retries-on-failures.md)
* [Image generation with Gateway](./getting-started/image-generation.ipynb)
* [Writing your first Gateway Config](./getting-started/writing-your-first-gateway-config.md)
* [Automatically Fallback from OpenAI to Azure](./getting-started/fallback-from-openai-to-azure.ipynb)
View the [official docs](https://portkey.ai/docs)
## providers
* [OpenAI](./providers/openai.ipynb)
* [Anthropic](./providers/anthropic.md)
* [Mistral](./providers/mistral.md)
* [Vercel AI](./integrations/vercel-ai.md)
* [Groq](./providers/groq.ipynb)
* [Mistral](./providers/mistral.ipynb)
* [Deepinfra](./providers/deepinfra.ipynb)
* [Segmind](./providers/segmind.ipynb)
* [Nvidia](./providers/nvidia.ipynb)
View the [full list of providers here](https://portkey.ai/docs/welcome/integration-guides).
## integrations
* [Langchain](./integrations/langchain.ipynb)
* [Llama Index](./integrations/llama-index.ipynb)
* [Instructor](./integrations/Instructor_with_Portkey.ipynb)
* [Phidata](./integrations/Phidata_with_Portkey.ipynb)
View the [full list of integrations here](https://portkey.ai/docs/welcome/integration-guides).
## monitoring-agents
* [Autogen](./monitoring-agents/Autogen_with_Telemetry.ipynb)
* [CrewAI](./monitoring-agents/CrewAI_with_Telemetry.ipynb)
* [Llama Agents](./monitoring-agents/Llama_Agents_with_Telemetry.ipynb)
* [ControlFlow](./monitoring-agents/ControlFlow_with_Telemetry.ipynb)
## contributing
This is a community-driven resource! We welcome ideas, improvements, quick fixes, and deeper contributions that help the community.
[Request Cookbooks here](https://github.com/portkey-ai/gateway/issues). Check out our contribution [guidelines here](../.github/CONTRIBUTING.md).
---
## File: docs/deploy-on-replit.md
[Portkey's AI Gateway](https://portkey.ai/features/ai-gateway) is the interface between your app and hosted LLMs. It streamlines API requests to OpenAI, Anthropic, Mistral, LLama2, Anyscale, Google Gemini and more with a unified API.
✅ Blazing **fast** (9.9x faster) with a **tiny footprint** (~45kb installed)
✅ **Load balance** across multiple models, providers, and keys
✅ **Fallbacks** make sure your app stays resilient
✅ **Automatic Retries** with exponential fallbacks come by default
✅ Plug-in middleware as needed
✅ Battle tested over **100B tokens**
### Deploy on Replit
1. Fork the AI gateway published by Portkey with a preferred name into your Replit account.
2. Click [Run] to run the gateway for your apps.
3. Open a new tab to grab the gateway URL. It typically looks like `https://unique-random-numbers.xxx.repl.co/`
4. Click [Deploy] in the top right for a production-ready app gateway (Requires Replit Core).
5. The gateway will be deployed for production use at `https://chosen-subdomain.replit.app`.
**Example usage**
Let's try making a chat completions call to OpenAI through the AI gateway:
```sh
curl 'https://chosen-subdomain.replit.app/v1/chat/completions' \
-H 'x-portkey-provider: openai' \
-H "Authorization: Bearer $OPENAI_KEY" \
-H 'Content-Type: application/json' \
-d '{"messages": [{"role": "user","content": "Say this is test."}], "max_tokens": 20, "model": "gpt-4"}'
```
### Portkey
- Read the [AI gateway documentation](https://portkey.ai/docs/product/ai-gateway-streamline-llm-integrations).
- Explore Oberservability Suite to build reliable gen-AI apps for production. [Try Portkey](https://portkey.ai/).
---
## File: docs/installation-deployments.md
# How to Deploy the Gateway?
1. [Managed Deployment by Portkey](#managed-deployment) for quick setup without infrastructure concerns
2. [Local Deployment](#local-deployment) for complete control & customization
3. [Enterprise Deployment](#enterprise-deployment) for advanced features and dedicated support
## Managed Deployment
Portkey runs this same Gateway on our API and processes **billions of tokens** daily. Portkey's API is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.
Sign up for the free developer plan [here](https://app.portkey.ai/) or [discuss here](https://calendly.com/portkey-ai/quick-meeting?utm_source=github&utm_campaign=install_page) for enterprise deployments.
Check out the [API docs](https://portkey.ai/docs/welcome/make-your-first-request) here.
## Local Deployment
1. Do [NPM](#node) or [Bun](#bun) Install
2. Run a [Node.js Server](#nodejs-server)
3. Deploy on [App Stack](#deploy-to-app-stack)
4. Deploy on [Cloudflare Workers](#cloudflare-workers)
5. Deploy using [Docker](#docker)
6. Deploy using [Docker Compose](#docker-compose)
7. Deploy on [Replit](#replit)
8. Deploy on [Zeabur](#zeabur)
9. Deploy with [Supabase Functions](#supabase-functions)
10. Deploy using [Fastly](#fastly)
---
### Node
```sh
$ npx @portkey-ai/gateway
```
### Bun
```sh
$ bunx @portkey-ai/gateway
```
---
### NodeJS Server
1. Clone the Repository
```sh
git clone https://github.com/portkey-ai/gateway
```
2. Install the NPM Dependencies
```sh
cd gateway
npm i
npm run build
```
3. Run the Server
```sh
node build/start-server.js
```
---
### Deploy to App Stack
F5 Distributed Cloud
1. [Create an App Stack Site](https://docs.cloud.f5.com/docs/how-to/site-management/create-voltstack-site)
2. Retrieve the global kubeconfig
```shell
export DISTRIBUTED_CLOUD_TENANT=mytenantname
# find tenant id in the F5 Distributed Cloud GUI at
# Account -> Account Settings -> Tenant Overview -> Tenant ID
export DISTRIBUTED_CLOUD_TENANT_ID=mytenantnamewithextensionfoundintheconsole
# create an API token in the F5 Distributed Cloud GUI at
# Account -> Account Settings -> Credentials -> Add Credentials
# set Credential Type to API Token, not API Certificate
export DISTRIBUTED_CLOUD_API_TOKEN=myapitoken
export DISTRIBUTED_CLOUD_SITE_NAME=appstacksitename
export DISTRIBUTED_CLOUD_NAMESPACE=mydistributedcloudnamespace
export DISTRIBUTED_CLOUD_APP_STACK_NAMESPACE=portkeyai
export DISTRIBUTED_CLOUD_APP_STACK_SITE=myappstacksite
export DISTRIBUTED_CLOUD_SERVICE_NAME=portkeyai
# adjust the expiry date to a time no more than 90 days in the future
export KUBECONFIG_CERT_EXPIRE_DATE="2021-09-14T09:02:25.547659194Z"
export PORTKEY_GATEWAY_FQDN=the.host.nameof.theservice
export PORTKEY_PROVIDER=openai
export PORTKEY_PROVIDER_AUTH_TOKEN=authorizationtoken
curl --location --request POST 'https://$DISTRIBUTED_CLOUD_TENANT.console.ves.volterra.io/api/web/namespaces/system/sites/$DISTRIBUTED_CLOUD_SITE_NAME/global-kubeconfigs' \
--header 'Authorization: APIToken $DISTRIBUTED_CLOUD_API_TOKEN' \
--header 'Access-Control-Allow-Origin: *' \
--header 'x-volterra-apigw-tenant: $DISTRIBUTED_CLOUD_TENANT'\
--data-raw '{"expirationTimestamp":"$KUBECONFIG_CERT_EXPIRE_DATE"}'
```
Save the response in a YAML file for later use.
[more detailed instructions for retrieving the App Stack kubeconfig file](https://f5cloud.zendesk.com/hc/en-us/articles/4407917988503-How-to-download-kubeconfig-via-API-or-vesctl)
3. Copy the deployment YAML
```shell
wget https://raw.githubusercontent.com/Portkey-AI/gateway/main/deployment.yaml
```
4. Apply the manifest
```shell
export KUBECONFIG=path/to/downloaded/global/kubeconfig/in/step/two
# apply the file downloaded in step 3
kubectl apply -f deployment.yaml
```
5. Create Origin Pool
```shell
# create origin pool
curl --request POST \
--url https://$DISTRIBUTED_CLOUD_TENANT.console.ves.volterra.io/api/config/namespaces/$DISTRIBUTED_CLOUD_NAMESPACE/origin_pools \
--header 'authorization: APIToken $DISTRIBUTED_CLOUD_API_TOKEN' \
--header 'content-type: application/json' \
--data '{"metadata": {"name": "$DISTRIBUTED_CLOUD_SERVICE_NAME","namespace": "$DISTRIBUTED_CLOUD_NAMESPACE","labels": {},"annotations": {},"description": "","disable": false},"spec": {"origin_servers": [{"k8s_service": {"service_name": "$DISTRIBUTED_CLOUD_SERVICE_NAME.$DISTRIBUTED_CLOUD_APP_STACK_NAMESPACE","site_locator": {"site": {"tenant": "$DISTRIBUTED_CLOUD_TENANT_ID","namespace": "system","name": "$DISTRIBUTED_CLOUD_APP_STACK_SITE"}},"inside_network": {}},"labels": {}}],"no_tls": {},"port": 8787,"same_as_endpoint_port": {},"healthcheck": [],"loadbalancer_algorithm": "LB_OVERRIDE","endpoint_selection": "LOCAL_PREFERRED","advanced_options": null}}'
```
or [use the UI](https://docs.cloud.f5.com/docs/how-to/app-networking/origin-pools)
6. Create an HTTP Load Balancer, including header injection of Portkey provider and credentials
```
/* Detailed source-code truncated for AI context efficiency. */
```
or [use the UI](https://docs.cloud.f5.com/docs/how-to/app-networking/http-load-balancer)
7. Test the service
```shell
curl --request POST \
--url https://$PORTKEY_GATEWAY_FQDN/v1/chat/completions \
--header 'content-type: application/json' \
--data '{"messages": [{"role": "user","content": "Say this might be a test."}],"max_tokens": 20,"model": "gpt-4"}'
```
in addition to the response headers, you should get a response body like
```json
{
"id": "chatcmpl-abcde......09876",
"object": "chat.completion",
"created": "0123456789",
"model": "gpt-4-0321",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "This might be a test."
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 14,
"completion_tokens": 6,
"total_tokens": 20
},
"system_fingerprint": null
}
```
---
### Cloudflare Workers
1. Clone the Repository
```sh
git clone https://github.com/portkey-ai/gateway
```
2. Install the NPM Dependencies
```sh
cd gateway
npm install
```
3. Deploy (using [Wrangler CLI](https://developers.cloudflare.com/workers/wrangler/))
```sh
npm run deploy
```
For more details, refer to [Cloudflare Workers official](https://developers.cloudflare.com/workers/).
---
### Docker
**Run through the latest Docker Hub image:**
```sh
docker run --rm -p 8787:8787 portkeyai/gateway:latest
```
For more information on the Docker image, check [here](https://hub.docker.com/r/portkeyai/gateway)
---
### Docker Compose
1. Download Compose File from the Repository:
```sh
wget "https://raw.githubusercontent.com/Portkey-AI/gateway/main/docker-compose.yaml"
```
2. Run:
```sh
docker compose up -d
```
> The service is now running and listening on port 8787
For more details, refer to [Docker Compose official](https://docs.docker.com/compose/).
### AWS EC2
1. Copy the AWS CloudFormation template from below:
```
/* Detailed source-code truncated for AI context efficiency. */
```
2. Create a new stack in the AWS CloudFormation console with the template above(you can upload in your S3 or directly upload the template).
3. Fill the following parameters:
- **VpcId**: The VPC ID of the VPC where the EC2 instance will be launched
- **SubnetId**: The Subnet ID of the Subnet where the EC2 instance will be launched
- **InstanceType**: The instance type of the EC2 instance
4. Create the stack and wait for it to be created.
5. Once the stack is created, you can access the Portkey Gateway URL from the Outputs section.
---
### Replit
[](https://replit.com/@portkey/AI-Gateway?v=1)
---
### Zeabur
[](https://zeabur.com/templates/RU38E3)
---
### Supabase Functions
1. Clone the Repository:
```sh
git clone https://github.com/portkey-ai/gateway
```
2. Set up a Supabase Account and create a new project (in browser).
3. Install the Supabase CLI, login using `supabase login
` and Initialize, follow [guides](https://supabase.com/docs/guides/local-development)
4. Write Your Supabase Function in:
```sh
supabase/functions/your-function-file/index.ts
```
5. Deploy the Function (for windows add `npx` in start):
```sh
supabase functions deploy your-function-file
```
6. Now, its ready to Test, Monitor and Manage Functions (Using url from supabase dashboard):
- Similar to `https://your-project-id.supabase.co/functions/v1/your-function-file`
Refer to [Supabase's official documentation](https://supabase.com/docs) for more details.
---
### Fastly
1. Create / login to your fastly account (in browser).
2. Clone the Gateway Repository:
```sh
git clone https://github.com/portkey-ai/gateway
```
3. Install Fastly’s CLI:
```sh
npm install fastly-cli
```
4. Log in to Fastly:
```sh
fastly login
```
5. Create a New Fastly Service:
```sh
fastly compute init
```
6. Write Your Edge Function Code in `src/main.rs` or `src/index.js`
7. Set up the token locally (token is located on `Account > User(left bottom most) > API Token`)
```sh
fastly profile create
```
Then follow the steps there to add token.
8. Deploy to Fastly
```sh
fastly compute publish
```
9. Now, its ready to Test, Monitor and Manage Functions (Using url from fastly dashboard):
- Similar to `your-new-project.edgecompute.app`
For more details, refer to [Fastly’s official](https://www.fastly.com/products/edge-compute).
---
### Vercel
- Docs to be written, please help!
- Make sure you've deployed and tested the same.
---
### AWS Lambda
- Docs to be written, please help!
- Make sure you've deployed and tested the same.
---
### Lambda edge
- Docs to be written, please help!
- Make sure you've deployed and tested the same.
---
## Enterprise Deployment
Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.
✅ Secure Key Management - for role-based access control and tracking
✅ Simple & Semantic Caching - to serve repeat queries faster & save costs
✅ Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
✅ PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
✅ SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
✅ Professional Support - along with feature prioritization
[Schedule a call to discuss enterprise deployments](https://calendly.com/portkey-ai/quick-meeting?utm_source=github&utm_campaign=install_page)
---
## File: plugins/lasso/README.md
# Lasso Security Plugin
This plugin integrates with Lasso Security's Deputies Public API to provide content classification and security guardrails for AI applications.
## Overview
Lasso Security is a leading genAI security platform that autonomously monitors interactions, detects risks in real-time, and empowers organizations to adopt AI safely. The Deputies Public API provides a flexible interface for accessing various Deputy services for security classification and policy enforcement.
## Features
- **Content Classification**: Analyze messages for various security risks
- **Real-time Protection**: Block harmful content before it reaches your users
- **Detailed Results**: Get comprehensive classification results with confidence scores
## Setup
1. Sign up for a Lasso Security account at [https://app.lasso.security](https://app.lasso.security)
2. Obtain your API key from the Organization Settings page
3. Configure the plugin with your API key
## Usage
### Basic Usage
```javascript
// Example configuration
const config = {
plugins: {
lasso: {
classify: {
credentials: {
apiKey: 'your-lasso-api-key',
},
messages: [
{
role: 'user',
content: 'User message to classify',
},
{
role: 'assistant',
content: 'Assistant response to classify',
},
],
},
},
},
};
```
### Advanced Usage
```javascript
// Example with additional parameters
const config = {
plugins: {
lasso: {
classify: {
credentials: {
apiKey: 'your-lasso-api-key',
},
messages: [
{
role: 'user',
content: 'User message to classify',
},
],
conversationId: 'unique-conversation-id',
userId: 'user-123',
},
},
},
};
```
## Response Format
The plugin returns a response with the following structure:
```json
{
"deputies": {
"jailbreak": false,
"custom-policies": false,
"sexual": false,
"hate": false,
"illegality": false,
"violence": false,
"pattern-detection": true
},
"deputies_predictions": {
"jailbreak": 0.123,
"custom-policies": 0.234,
"sexual": 0.145,
"hate": 0.156,
"illegality": 0.167,
"violence": 0.178,
"pattern-detection": 0.989
},
"violations_detected": true
}
```
## Support
For support or questions, contact Lasso Security at [support@lasso.security](mailto:support@lasso.security) or visit [https://lasso.security](https://lasso.security).
---
## File: plugins/README.md
# Portkey Gateway Plugins
## Table of Contents
- [Portkey Gateway Plugins](#portkey-gateway-plugins)
- [Table of Contents](#table-of-contents)
- [Introduction](#introduction)
- [What are Plugins?](#what-are-plugins)
- [Why are Plugins Important?](#why-are-plugins-important)
- [Concepts](#concepts)
- [Hooks](#hooks)
- [Guardrails](#guardrails)
- [Checks](#checks)
- [Creating a Plugin](#creating-a-plugin)
- [Folder Structure](#folder-structure)
- [Manifest File](#manifest-file)
- [Plugin Function](#plugin-function)
- [Building Plugins into Your Gateway](#building-plugins-into-your-gateway)
- [Using Plugins](#using-plugins)
- [Testing Plugins](#testing-plugins)
## Introduction
This folder contains plugins for the Portkey Gateway. Plugins allow developers to extend the capabilities of the Gateway by adding custom functionality at various stages of the request lifecycle. Currently, the primary type of plugins supported in the AI gateway are guardrails.
## What are Plugins?
Plugins in Portkey enable developers to extend the capabilities of the AI gateway by writing their own functionality. Portkey exposes the concept of hooks, which allows developers to execute custom functions at key points in the request journey. While the plugin system is designed to be extensible, the current focus is on implementing guardrails.
## Why are Plugins Important?
Plugins are crucial for several reasons:
1. **Customization**: They allow you to tailor the Gateway's behavior to your specific needs, especially in terms of implementing guardrails.
2. **Extensibility**: You can add new features and integrations without modifying the core Gateway code.
3. **Modularity**: Plugins can be easily added, removed, or updated without affecting other parts of the system.
4. **Community Contributions**: The plugin system allows the community to contribute and share useful extensions, particularly for guardrail implementations.
## Concepts
### Hooks
Hooks in Portkey's AI gateway allow custom functions to be executed at various stages of the request lifecycle. Currently, four types of hooks are supported:
1. **Start**: Executed at the beginning of the request lifecycle for initial setup and logging.
2. **BeforeRequest**: Executed before the request is sent to the AI model, typically used for implementing guardrails.
3. **AfterRequest**: Executed after the AI model processes the request but before the response is returned, used for post-processing and validations.
4. **End**: Executed at the end of the request lifecycle for final logging and cleanup.
While hooks can be extended for custom logging solutions and other custom functionalities, their primary intended purpose is to set up and enforce guardrails.
### Guardrails
Guardrails in Portkey's AI gateway are a set of checks that are run together within the `beforeRequest` or `afterRequest` hooks to determine a `verdict`. The verdict of a guardrail dictates the actions to be taken on the request or response. For example, if the guardrail fails, the request can be failed, or the response can be returned with a 246 status code indicating that the guardrails failed.
Guardrails can be defined either through the user interface (UI) of Portkey or as a JSON configuration within the Portkey `config`. This flexibility allows for easy management and customization of guardrails according to the specific needs of the application.
### Checks
A check is an individual function that assesses the input prompt or output response against predefined conditions. Each check returns a boolean verdict or may error out if issues are encountered. Checks are the building blocks of guardrails, and Portkey includes a set of predefined checks as well as the ability to add custom checks.
## Creating a Plugin
When creating a plugin for Portkey, keep in mind that it should primarily focus on implementing guardrails. Here's how to structure your plugin:
### Folder Structure
To create a plugin, follow this structure in the `/plugins` folder:
```
/plugins
/your-plugin-name
- manifest.json
- main-function.ts
- test-file.test.ts (recommended)
```
### Manifest File
The `manifest.json` file defines the plugin's properties, required credentials, and available functions. Here's an example structure:
```json
{
"id": "your-plugin-id",
"description": "A brief description of your guardrail plugin",
"credentials": {
"type": "object",
"properties": {
"apiKey": {
"type": "string",
"label": "API Key",
"description": "Your API key description",
"encrypted": true
}
},
"required": ["apiKey"]
},
"functions": [
{
"name": "Your Guardrail Function",
"id": "yourGuardrailId",
"supportedHooks": ["beforeRequestHook", "afterRequestHook"],
"type": "guardrail",
"description": [
{
"type": "subHeading",
"text": "Description of your guardrail function"
}
],
"parameters": {
"type": "object",
"properties": {
"paramName": {
"type": "string",
"label": "Parameter Label",
"description": [
{
"type": "subHeading",
"text": "Description of the parameter"
}
]
}
},
"required": ["paramName"]
}
}
]
}
```
### Plugin Function
Create a TypeScript file (e.g., `main-function.ts`) that exports a handler function. This function will implement your guardrail logic. Here's a basic structure:
```typescript
import { HookEventType, PluginContext, PluginHandler, PluginParameters } from '../types';
export const handler: PluginHandler = async (
context: PluginContext,
parameters: PluginParameters,
eventType: HookEventType
) => {
// Your guardrail logic here
// You can access:
// - context.request for request data
// - context.response for response data (in afterRequestHook)
// - parameters for plugin-specific parameters
// - eventType to determine which hook is being executed
return {
error: null, // or error object if an error occurred
verdict: true, // or false to indicate if the guardrail passed or failed
data: {} // any additional data you want to return
};
};
```
## Building Plugins into Your Gateway
To build plugins into your Gateway, follow these steps:
1. In the root directory of the Gateway repository, locate the `conf.json` file.
2. Edit the `conf.json` file to specify which plugins you want to enable and provide any necessary credentials. Here's an example:
```json
{
"plugins_enabled": ["default", "your-guardrail-plugin"],
"credentials": {
"your-guardrail-plugin": {
"apiKey": "your-api-key-here"
}
},
"cache": false
}
```
3. Run the build command to compile the plugins:
```
npm run build-plugins
```
This command will compile the plugins and prepare them for use with the Gateway.
## Using Plugins
Once you've built the plugins, they will be automatically deployed with the Gateway when you run it. You can start the Gateway using one of the following commands:
- For development:
```
npm run dev
```
- For Node.js:
```
npm run dev:node
```
- For Cloudflare Workers:
```
npm run dev:workerd
```
The guardrail plugins you've enabled in the `conf.json` file will be loaded and ready to use.
## Testing Plugins
To test your guardrail plugins:
1. Ensure you have written test files for your plugins (e.g., `test-file.test.ts`).
2. Run the test command:
```
npx jest
```
This command will execute the test files in the plugins repository, allowing you to verify that your guardrail plugins are functioning correctly.