gateway Documentation

A blazing fast AI Gateway with integrated guardrails. Route to 1,600+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.

11,650 stars TypeScript TypeScript 1 file Β· ~12,490 tokens #ai-gateway#gateway#generative-ai#hacktoberfest#langchain#llm#llm-gateway#llmops
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy gateway on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name gateway -p 8787:8787 --restart always portkeyai/gateway:latest
Configured Services & Ports
βš™οΈ web
image: portkeyai/gateway:latest
πŸ”Œ Ports: 8787:8787
docker-compose.yml
yaml (docker-compose.yml) 
version: '3'
services:
  web:
    ports:
      - "8787:8787"
    image: "portkeyai/gateway:latest"
    restart: always

File: README.md

English | δΈ­ζ–‡ | ζ—₯本θͺž

[!IMPORTANT]
:rocket: Gateway 2.0 (Pre-Release) Portkey's core enterprise gateway is merging into open-source with our 2.0 release. You can try the pre-release branch here.
Read more about what's next for Portkey in our Series A announcement.

The AI Gateway is designed for fast, reliable & secure routing to 1600+ language, vision, audio, and image models. It is a lightweight, open-source, and enterprise-ready solution that allows you to integrate with any language model in under 2 minutes.

  • Blazing fast (<1ms latency) with a tiny footprint (122kb)
  • Battle tested, with over 10B tokens processed everyday
  • Enterprise-ready with enhanced security, scale, and custom deployments

What can you do with the AI Gateway?

[!TIP]
Starring this repo helps more developers discover the AI Gateway πŸ™πŸ»

Quickstart (2 mins)

1. Setup your AI Gateway

bash
# Run the gateway locally (needs Node.js and npm)
npx @portkey-ai/gateway

The Gateway is running on http://localhost:8787/v1

The Gateway Console is running on http://localhost:8787/public/

Deployment guides:   [ Portkey Cloud (Recommended)](https://portkey.wiki/gh-18)   [ Docker](./docs/installation-deployments.md#docker)   [ Node.js](./docs/installation-deployments.md#nodejs-server)   [ Cloudflare](./docs/installation-deployments.md#cloudflare-workers)   [ Replit](./docs/installation-deployments.md#replit)   [ Others...](./docs/installation-deployments.md)

2. Make your first request

python
# pip install -qU portkey-ai

from portkey_ai import Portkey

# OpenAI compatible client
client = Portkey(
    provider="openai", # or 'anthropic', 'bedrock', 'groq', etc
    Authorization="sk-***" # the provider API key
)

# Make a request through your AI Gateway
client.chat.completions.create(
    messages=[{"role": "user", "content": "What's the weather like?"}],
    model="gpt-4o-mini"
)

Supported Libraries:
  JS
  Python
  REST
  OpenAI SDKs
  Langchain
  LlamaIndex
  Autogen
  CrewAI
  More..

On the Gateway Console (http://localhost:8787/public/) you can see all of your local logs in one place.

3. Routing & Guardrails

Configs in the LLM gateway allow you to create routing rules, add reliability and setup guardrails.

python
config = {
  "retry": {"attempts": 5},

  "output_guardrails": [{
    "default.contains": {"operator": "none", "words": ["Apple"]},
    "deny": True
  }]
}

# Attach the config to the client
client = client.with_options(config=config)

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}]
)

# This would always response with "Bat" as the guardrail denies all replies containing "Apple". The retry config would retry 5 times before giving up.

You can do a lot more stuff with configs in your AI gateway. Jump to examples β†’

Enterprise Version (Private deployments)

AWS
  Azure
  GCP
  OpenShift
  Kubernetes

The LLM Gateway's enterprise version offers advanced capabilities for org management, governance, security and more out of the box. View Feature Comparison β†’

The enterprise deployment architecture for supported platforms is available here - Enterprise Private Cloud Deployments

MCP Gateway

MCP Gateway provides a centralized control plane for managing MCP (Model Context Protocol) servers across your organization.

  • Authentication β€” Single auth layer at the gateway. Users authenticate once; your MCP servers receive verified requests
  • Access Control β€” Control which teams and users can access which servers and tools. Revoke access instantly
  • Observability β€” Every tool call logged with full context: who called what, parameters, response, latency
  • Identity Forwarding β€” Forward user identity (email, team, roles) to MCP servers automatically

Works with Claude Desktop, Cursor, VS Code, and any MCP-compatible client. Get started β†’

Core Features

Reliable Routing

  • Fallbacks: Fallback to another provider or model on failed requests using the LLM gateway. You can specify the errors on which to trigger the fallback. Improves reliability of your application.
  • Automatic Retries: Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload.
  • Load Balancing: Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance.
  • Request Timeouts: Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration.
  • Multi-modal LLM Gateway: Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers β€” all using the familiar OpenAI signature
  • Realtime APIs: Call realtime APIs launched by OpenAI through the integrate websockets server.

Security & Accuracy

Cost Management

  • Smart caching: Cache responses from LLMs to reduce costs and improve latency. Supports simple and semantic* caching.
  • Usage analytics: Monitor and analyze your AI and LLM usage, including request volume, latency, costs and error rates.
  • Provider optimization*: Automatically switch to the most cost-effective provider based on usage patterns and pricing models.

Collaboration & Workflows

* Available in hosted and enterprise versions

Portkey Models

Open-source LLM pricing database for 40+ providers - used by the Gateway for cost tracking.

GitHub | Model Explorer

Cookbooks

🚨 Latest

View all cookbooks β†’

Supported Providers

Explore Gateway integrations with 45+ providers and 8+ agent frameworks.

Provider Support Stream
OpenAI βœ… βœ…
Azure OpenAI βœ… βœ…
Anyscale βœ… βœ…
Google Gemini βœ… βœ…
Anthropic βœ… βœ…
Cohere βœ… βœ…
Together AI βœ… βœ…
Perplexity βœ… βœ…
Mistral βœ… βœ…
Nomic βœ… βœ…
AI21 βœ… βœ…
Stability AI βœ… βœ…
DeepInfra βœ… βœ…
Ollama βœ… βœ…
Novita AI βœ… βœ…

View the complete list of 200+ supported models here

Agents

Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.

Framework Call 200+ LLMs Advanced Routing Caching Logging & Tracing* Observability* Prompt Management*
Autogen βœ… βœ… βœ… βœ… βœ… βœ…
CrewAI βœ… βœ… βœ… βœ… βœ… βœ…
LangChain βœ… βœ… βœ… βœ… βœ… βœ…
Phidata βœ… βœ… βœ… βœ… βœ… βœ…
Llama Index βœ… βœ… βœ… βœ… βœ… βœ…
Control Flow βœ… βœ… βœ… βœ… βœ… βœ…
Build Your Own Agents βœ… βœ… βœ… βœ… βœ… βœ…
IO Intelligence βœ… βœ…

*Available on the hosted app. For detailed documentation click here.

Gateway Enterprise Version

Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.

βœ…  Secure Key Management - for role-based access control and tracking
βœ…  Simple & Semantic Caching - to serve repeat queries faster & save costs
βœ…  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
βœ…  PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
βœ…  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
βœ…  Professional Support - along with feature prioritization

Schedule a call to discuss enterprise deployments

Contributing

The easiest way to contribute is to pick an issue with the good first issue tag πŸ’ͺ. Read the contribution guidelines here.

Bug Report? File here | Feature Request? File here

Getting Started with the Community

Join our weekly AI Engineering Hours every Friday (8 AM PT) to:

  • Meet other contributors and community members
  • Learn advanced Gateway features and implementation patterns
  • Share your experiences and get help
  • Stay updated with the latest development priorities

Join the next session β†’ | Meeting notes

Community

Join our growing community around the world, for help, ideas, and discussions on AI.

File: cookbook/integrations/vercel/readme.md

Portkey + Vercel AI SDK

Portkey natively integrates with the Vercel AI SDK to make your apps production-ready and reliable. Just import Portkey's Vercel package and use it as a provider in your Vercel AI app to enable all of Portkey features:

  • Full-stack observability and tracing for all requests
  • Interoperability across 250+ LLMS
  • Built-in 50+ SOTA guardrails
  • Simple & semantic caching to save costs & time
  • Route requests conditionally and make them robust with fallbacks, load-balancing, automatic retries, and more
  • Continuous improvement based on user feedback

Getting Started

Getting Started
Follow these steps to get your Vercel app up and running:

Install Dependencies
Run the following command to install all necessary dependencies:
Copynpm install

Configure API Keys
Add your API keys to the appropriate configuration file. This step is crucial for the app to function correctly.
Start the App
Once you've completed the above steps, your app will be running locally. You can access it at:
Copyhttp://localhost:3000
(Note: The port number might be different depending on your specific Vercel app configuration)

1. Installation

sh
npm install @portkey-ai/vercel-provider
npm install

3. Configure API Keys

Add your API keys to the appropriate configuration file. This step is crucial for the app to function correctly.
Start the App

2. Import & Configure Portkey Object

Sign up for Portkey and get your API key, and configure Portkey provider in your Vercel app:

javascript
import { createPortkey } from '@portkey-ai/vercel-provider';

const portkeyConfig = {
      "provider": "openai", // Choose your provider (e.g., 'anthropic')
      "api_key": "OPENAI_API_KEY",
      "override_params": {
          "model": "gpt-4" // Select from 250+ models
        }
};

const portkey = createPortkey({
  apiKey: 'YOUR_PORTKEY_API_KEY',
  config: portkeyConfig,
});

3. Configure API Keys

Add your API keys to the appropriate configuration file. This step is crucial for the app to function correctly.

4. Start the App

Once you've completed the above steps use npm run dev to run your app locally. You can access it at:
http://localhost:3000

Portkey's configs are a powerful way to manage & govern your app's behaviour. Learn more about Configs here.

Using Vercel Functions

Portkey provider works with all of Vercel functions generateText & streamText.

Here's how to use them with Portkey:

generateText

javascript
import { createPortkey } from '@portkey-ai/vercel-provider';
import { generateText } from 'ai';

const portkeyConfig = {
      "provider": "openai", // Choose your provider (e.g., 'anthropic')
      "api_key": "OPENAI_API_KEY",
      "override_params": {
          "model": "gpt-4"
        }
};

const portkey = createPortkey({
  apiKey: 'YOUR_PORTKEY_API_KEY',
  config: portkeyConfig,
});

const { text } = await generateText({
  model: portkey.chatModel(''), // Provide an empty string, we defined the model in the config
  prompt: 'What is Portkey?',
});

console.log(text);

streamText

js
import { createPortkey } from '@portkey-ai/vercel-provider';
import { streamText } from 'ai';

const portkeyConfig = {
      "provider": "openai", // Choose your provider (e.g., 'anthropic')
      "api_key": "OPENAI_API_KEY",
      "override_params": {
        "model": "gpt-4o" // Select from 250+ models
  } 
};

const portkey = createPortkey({
  apiKey: 'YOUR_PORTKEY_API_KEY',
  config: portkeyConfig,
});

const result = await streamText({
  model: portkey('gpt-4-turbo'), // This gets overwritten by config
  prompt: 'Invent a new holiday and describe its traditions.',
});

for await (const chunk of result) {
  console.log(chunk);
}

Portkey supports chatModel and completionModel to easily handle chatbots or text completions. In the above examples, we used portkey.chatModel for generateText.

Tool Calling with Portkey

Portkey supports Tool calling with Vercel AI SDK. Here's how:

javascript
import { z } from 'zod';
import { generateText, tool } from 'ai';

const result = await generateText({
  model: portkey.chatModel('gpt-4-turbo'),
  tools: {
    weather: tool({
      parameters: z.object({
        location: z.string().describe('The location to get the weather for'),
      }),
      execute: async ({ location }) => ({
        location,
        temperature: 72 + Math.floor(Math.random() * 21) - 10,
      }),
    }),
  },
  prompt: 'What is the weather in San Francisco?',
});

Portkey Features

Portkey Helps you make your Vercel app more robust and reliable. The portkey config is a modular way to make it work for you in whatever way you want.

Interoperability

Portkey allows you to easily switch between 250+ AI models by simply changing the model name in your configuration. This flexibility enables you to adapt to the evolving AI landscape without significant code changes.

Switch from OpenAI to Anthropic

Here's how you'd use OpenAI with Portkey's Vercel integration:

javascript
const portkeyConfig = {
      "provider": "openai",
      "api_key": "OPENAI_API_KEY",
      "override_params": {
          model: "gpt-4"
        }
};

Now, to switch to Anthropic, just change your provider slug to anthropic and enter your Anthropic API key along with the model of choice:

javascript
const portkeyConfig = {
      "provider": "anthropic",
      "api_key": "Anthropic_API_KEY",
      "override_params": {
          "model": "claude-3-sonnet-20240229"
        }
      
};

Observability

Portkey's OpenTelemetry-compliant observability suite gives you complete control over all your requests. And Portkey's analytics dashboards provide 40+ key insights you're looking for including cost, tokens, latency, etc. Fast.

Reliability

Portkey enhances the robustness of your AI applications with built-in features such as Caching, Fallback mechanisms, Load balancing, Conditional routing, Request timeouts, etc.

Here is how you can modify your config to include the following Portkey features:

Fallback

javascript
import { createPortkey } from '@portkey-ai/vercel-provider';
import { generateText } from 'ai';

const portkeyConfig =  {
	"strategy": {
		"mode": "fallback"
	},
	"targets": [
		{ 
		"provider": "anthropic",
	      	"api_key": "Anthropic_API_KEY",
	      	"override_params": {
	          "model": "claude-3-sonnet-20240229"
	        } },
		{
		"provider": "openai",
     	 	"api_key": "OPENAI_API_KEY",
      		"override_params": {
          		"model": "gpt-4"
        } }
	]
}

const portkey = createPortkey({
  apiKey: 'YOUR_PORTKEY_API_KEY',
  config: portkeyConfig,
});

const { text } = await generateText({
  model: portkey.chatModel(''),
  prompt: 'What is Portkey?',
});

console.log(text);

Learn more about Portkey's AI gateway features in detail here.

Guardrails

Portkey Guardrails allow you to enforce LLM behavior in real-time, verifying both inputs and outputs against specified checks.

You can create Guardrail checks in UI and then pass them in your Portkey Configs with before request or after request hooks.

Read more about Guardrails here.

Portkey Config

Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.

For more information on using these features and setting up your Config, please refer to the Portkey documentation.

File: cookbook/use-cases/LMSYS Series/README.md

LMSYS Series

LMSYS is THE defining leaderboard to compare raw capabilities of various LLMs. It has so far collected over 1,000,000 human pairwise comparisons to rank LLMs in Elo-scale.

We are creating cookbooks that help you get the best value of of LMSYS leaderboard for production use cases, by leveraging Portkey's Gateway, Observability, and Prompt Management features.

Cookbooks

  1. Comparing the Top 10 LMSYS models using a single function.
  2. RAG Benchmark Over LMSYS Models (coming soon!)

File: cookbook/README.md

Portkey Practitioners' Cookbooks

Strategies and examples for tackling the production challenges of LLMs with Portkey Gateway


Table of Contents

Please use the below table of contents to navigate through the cookbook.

Latest Notebooks

getting-started

View the official docs

providers

View the full list of providers here.

integrations

View the full list of integrations here.

monitoring-agents

contributing

This is a community-driven resource! We welcome ideas, improvements, quick fixes, and deeper contributions that help the community.

Request Cookbooks here. Check out our contribution guidelines here.

File: docs/deploy-on-replit.md

Portkey's AI Gateway is the interface between your app and hosted LLMs. It streamlines API requests to OpenAI, Anthropic, Mistral, LLama2, Anyscale, Google Gemini and more with a unified API.

βœ…  Blazing fast (9.9x faster) with a tiny footprint (~45kb installed)
βœ…  Load balance across multiple models, providers, and keys
βœ…  Fallbacks make sure your app stays resilient
βœ…  Automatic Retries with exponential fallbacks come by default
βœ…  Plug-in middleware as needed
βœ…  Battle tested over 100B tokens

Deploy on Replit

  1. Fork the AI gateway published by Portkey with a preferred name into your Replit account.
  2. Click [Run] to run the gateway for your apps.
  3. Open a new tab to grab the gateway URL. It typically looks like https://unique-random-numbers.xxx.repl.co/
  4. Click [Deploy] in the top right for a production-ready app gateway (Requires Replit Core).
  5. The gateway will be deployed for production use at https://chosen-subdomain.replit.app.

Example usage

Let's try making a chat completions call to OpenAI through the AI gateway:

sh
curl 'https://chosen-subdomain.replit.app/v1/chat/completions' \
  -H 'x-portkey-provider: openai' \
  -H "Authorization: Bearer $OPENAI_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user","content": "Say this is test."}], "max_tokens": 20, "model": "gpt-4"}'

Portkey

File: docs/installation-deployments.md

How to Deploy the Gateway?

  1. Managed Deployment by Portkey for quick setup without infrastructure concerns
  2. Local Deployment for complete control & customization
  3. Enterprise Deployment for advanced features and dedicated support

Managed Deployment

Portkey runs this same Gateway on our API and processes billions of tokens daily. Portkey's API is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.

Sign up for the free developer plan here or discuss here for enterprise deployments.

Check out the API docs here.

Local Deployment

  1. Do NPM or Bun Install
  2. Run a Node.js Server
  3. Deploy on App Stack
  4. Deploy on Cloudflare Workers
  5. Deploy using Docker
  6. Deploy using Docker Compose
  7. Deploy on Replit
  8. Deploy on Zeabur
  9. Deploy with Supabase Functions
  10. Deploy using Fastly

Node

sh
$ npx @portkey-ai/gateway

Bun

sh
$ bunx @portkey-ai/gateway

NodeJS Server

  1. Clone the Repository
sh
git clone https://github.com/portkey-ai/gateway
  1. Install the NPM Dependencies
sh
cd gateway
npm i
npm run build
  1. Run the Server
sh
node build/start-server.js

Deploy to App Stack

F5 Distributed Cloud

  1. Create an App Stack Site

  2. Retrieve the global kubeconfig

shell
export DISTRIBUTED_CLOUD_TENANT=mytenantname
# find tenant id in the F5 Distributed Cloud GUI at
# Account -> Account Settings -> Tenant Overview -> Tenant ID
export DISTRIBUTED_CLOUD_TENANT_ID=mytenantnamewithextensionfoundintheconsole
# create an API token in the F5 Distributed Cloud GUI at
# Account -> Account Settings -> Credentials -> Add Credentials 
# set Credential Type to API Token, not API Certificate
export DISTRIBUTED_CLOUD_API_TOKEN=myapitoken
export DISTRIBUTED_CLOUD_SITE_NAME=appstacksitename
export DISTRIBUTED_CLOUD_NAMESPACE=mydistributedcloudnamespace
export DISTRIBUTED_CLOUD_APP_STACK_NAMESPACE=portkeyai
export DISTRIBUTED_CLOUD_APP_STACK_SITE=myappstacksite
export DISTRIBUTED_CLOUD_SERVICE_NAME=portkeyai
# adjust the expiry date to a time no more than 90 days in the future
export KUBECONFIG_CERT_EXPIRE_DATE="2021-09-14T09:02:25.547659194Z"
export PORTKEY_GATEWAY_FQDN=the.host.nameof.theservice
export PORTKEY_PROVIDER=openai
export PORTKEY_PROVIDER_AUTH_TOKEN=authorizationtoken

curl --location --request POST 'https://$DISTRIBUTED_CLOUD_TENANT.console.ves.volterra.io/api/web/namespaces/system/sites/$DISTRIBUTED_CLOUD_SITE_NAME/global-kubeconfigs' \
--header 'Authorization: APIToken $DISTRIBUTED_CLOUD_API_TOKEN' \
--header 'Access-Control-Allow-Origin: *' \
--header 'x-volterra-apigw-tenant: $DISTRIBUTED_CLOUD_TENANT'\
--data-raw '{"expirationTimestamp":"$KUBECONFIG_CERT_EXPIRE_DATE"}'

Save the response in a YAML file for later use.
more detailed instructions for retrieving the App Stack kubeconfig file

  1. Copy the deployment YAML
shell
wget https://raw.githubusercontent.com/Portkey-AI/gateway/main/deployment.yaml
  1. Apply the manifest
shell
export KUBECONFIG=path/to/downloaded/global/kubeconfig/in/step/two
# apply the file downloaded in step 3
kubectl apply -f deployment.yaml
  1. Create Origin Pool
shell
# create origin pool
curl --request POST \
  --url https://$DISTRIBUTED_CLOUD_TENANT.console.ves.volterra.io/api/config/namespaces/$DISTRIBUTED_CLOUD_NAMESPACE/origin_pools \
  --header 'authorization: APIToken $DISTRIBUTED_CLOUD_API_TOKEN' \
  --header 'content-type: application/json' \
  --data '{"metadata": {"name": "$DISTRIBUTED_CLOUD_SERVICE_NAME","namespace": "$DISTRIBUTED_CLOUD_NAMESPACE","labels": {},"annotations": {},"description": "","disable": false},"spec": {"origin_servers": [{"k8s_service": {"service_name": "$DISTRIBUTED_CLOUD_SERVICE_NAME.$DISTRIBUTED_CLOUD_APP_STACK_NAMESPACE","site_locator": {"site": {"tenant": "$DISTRIBUTED_CLOUD_TENANT_ID","namespace": "system","name": "$DISTRIBUTED_CLOUD_APP_STACK_SITE"}},"inside_network": {}},"labels": {}}],"no_tls": {},"port": 8787,"same_as_endpoint_port": {},"healthcheck": [],"loadbalancer_algorithm": "LB_OVERRIDE","endpoint_selection": "LOCAL_PREFERRED","advanced_options": null}}'

or use the UI

  1. Create an HTTP Load Balancer, including header injection of Portkey provider and credentials
text
/* Detailed source-code truncated for AI context efficiency. */

or use the UI

  1. Test the service
shell
curl --request POST \
  --url https://$PORTKEY_GATEWAY_FQDN/v1/chat/completions \
  --header 'content-type: application/json' \
  --data '{"messages": [{"role": "user","content": "Say this might be a test."}],"max_tokens": 20,"model": "gpt-4"}'

in addition to the response headers, you should get a response body like

json
{
  "id": "chatcmpl-abcde......09876",
  "object": "chat.completion",
  "created": "0123456789",
  "model": "gpt-4-0321",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This might be a test."
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 6,
    "total_tokens": 20
  },
  "system_fingerprint": null
}

Cloudflare Workers

  1. Clone the Repository
sh
git clone https://github.com/portkey-ai/gateway
  1. Install the NPM Dependencies
sh
cd gateway
npm install
  1. Deploy (using Wrangler CLI)
sh
npm run deploy

For more details, refer to Cloudflare Workers official.

Docker

Run through the latest Docker Hub image:

sh
docker run --rm  -p 8787:8787 portkeyai/gateway:latest

For more information on the Docker image, check here

Docker Compose

  1. Download Compose File from the Repository:
sh
wget "https://raw.githubusercontent.com/Portkey-AI/gateway/main/docker-compose.yaml"
  1. Run:
sh
docker compose up -d

The service is now running and listening on port 8787

For more details, refer to Docker Compose official.

AWS EC2

  1. Copy the AWS CloudFormation template from below:
text
/* Detailed source-code truncated for AI context efficiency. */

  1. Create a new stack in the AWS CloudFormation console with the template above(you can upload in your S3 or directly upload the template).

  2. Fill the following parameters:

  • VpcId: The VPC ID of the VPC where the EC2 instance will be launched
  • SubnetId: The Subnet ID of the Subnet where the EC2 instance will be launched
  • InstanceType: The instance type of the EC2 instance

  1. Create the stack and wait for it to be created.

  2. Once the stack is created, you can access the Portkey Gateway URL from the Outputs section.

Replit

Zeabur

Supabase Functions

  1. Clone the Repository:
sh
git clone https://github.com/portkey-ai/gateway
  1. Set up a Supabase Account and create a new project (in browser).
  2. Install the Supabase CLI, login using supabase login and Initialize, follow guides
  3. Write Your Supabase Function in:
sh
supabase/functions/your-function-file/index.ts
  1. Deploy the Function (for windows add npx in start):
sh
supabase functions deploy your-function-file
  1. Now, its ready to Test, Monitor and Manage Functions (Using url from supabase dashboard):
    • Similar to https://your-project-id.supabase.co/functions/v1/your-function-file

Refer to Supabase's official documentation for more details.

Fastly

  1. Create / login to your fastly account (in browser).

  2. Clone the Gateway Repository:

sh
git clone https://github.com/portkey-ai/gateway
  1. Install Fastly’s CLI:
sh
npm install fastly-cli
  1. Log in to Fastly:
sh
fastly login
  1. Create a New Fastly Service:
sh
fastly compute init

  1. Write Your Edge Function Code in src/main.rs or src/index.js

  2. Set up the token locally (token is located on Account > User(left bottom most) > API Token)

sh
fastly profile create

Then follow the steps there to add token.

  1. Deploy to Fastly
sh
fastly compute publish
  1. Now, its ready to Test, Monitor and Manage Functions (Using url from fastly dashboard):
    • Similar to your-new-project.edgecompute.app

For more details, refer to Fastly’s official.

Vercel

  • Docs to be written, please help!
  • Make sure you've deployed and tested the same.

AWS Lambda

  • Docs to be written, please help!
  • Make sure you've deployed and tested the same.

Lambda edge

  • Docs to be written, please help!
  • Make sure you've deployed and tested the same.

Enterprise Deployment

Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.

βœ…  Secure Key Management - for role-based access control and tracking
βœ…  Simple & Semantic Caching - to serve repeat queries faster & save costs
βœ…  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
βœ…  PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
βœ…  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
βœ…  Professional Support - along with feature prioritization

Schedule a call to discuss enterprise deployments

File: plugins/lasso/README.md

Lasso Security Plugin

This plugin integrates with Lasso Security's Deputies Public API to provide content classification and security guardrails for AI applications.

Overview

Lasso Security is a leading genAI security platform that autonomously monitors interactions, detects risks in real-time, and empowers organizations to adopt AI safely. The Deputies Public API provides a flexible interface for accessing various Deputy services for security classification and policy enforcement.

Features

  • Content Classification: Analyze messages for various security risks
  • Real-time Protection: Block harmful content before it reaches your users
  • Detailed Results: Get comprehensive classification results with confidence scores

Setup

  1. Sign up for a Lasso Security account at https://app.lasso.security
  2. Obtain your API key from the Organization Settings page
  3. Configure the plugin with your API key

Usage

Basic Usage

javascript
// Example configuration
const config = {
  plugins: {
    lasso: {
      classify: {
        credentials: {
          apiKey: 'your-lasso-api-key',
        },
        messages: [
          {
            role: 'user',
            content: 'User message to classify',
          },
          {
            role: 'assistant',
            content: 'Assistant response to classify',
          },
        ],
      },
    },
  },
};

Advanced Usage

javascript
// Example with additional parameters
const config = {
  plugins: {
    lasso: {
      classify: {
        credentials: {
          apiKey: 'your-lasso-api-key',
        },
        messages: [
          {
            role: 'user',
            content: 'User message to classify',
          },
        ],
        conversationId: 'unique-conversation-id',
        userId: 'user-123',
      },
    },
  },
};

Response Format

The plugin returns a response with the following structure:

json
{
  "deputies": {
    "jailbreak": false,
    "custom-policies": false,
    "sexual": false,
    "hate": false,
    "illegality": false,
    "violence": false,
    "pattern-detection": true
  },
  "deputies_predictions": {
    "jailbreak": 0.123,
    "custom-policies": 0.234,
    "sexual": 0.145,
    "hate": 0.156,
    "illegality": 0.167,
    "violence": 0.178,
    "pattern-detection": 0.989
  },
  "violations_detected": true
}

Support

For support or questions, contact Lasso Security at [email protected] or visit https://lasso.security.

File: plugins/README.md

Portkey Gateway Plugins

Table of Contents

Introduction

This folder contains plugins for the Portkey Gateway. Plugins allow developers to extend the capabilities of the Gateway by adding custom functionality at various stages of the request lifecycle. Currently, the primary type of plugins supported in the AI gateway are guardrails.

What are Plugins?

Plugins in Portkey enable developers to extend the capabilities of the AI gateway by writing their own functionality. Portkey exposes the concept of hooks, which allows developers to execute custom functions at key points in the request journey. While the plugin system is designed to be extensible, the current focus is on implementing guardrails.

Why are Plugins Important?

Plugins are crucial for several reasons:

  1. Customization: They allow you to tailor the Gateway's behavior to your specific needs, especially in terms of implementing guardrails.
  2. Extensibility: You can add new features and integrations without modifying the core Gateway code.
  3. Modularity: Plugins can be easily added, removed, or updated without affecting other parts of the system.
  4. Community Contributions: The plugin system allows the community to contribute and share useful extensions, particularly for guardrail implementations.

Concepts

Hooks

Hooks in Portkey's AI gateway allow custom functions to be executed at various stages of the request lifecycle. Currently, four types of hooks are supported:

  1. Start: Executed at the beginning of the request lifecycle for initial setup and logging.
  2. BeforeRequest: Executed before the request is sent to the AI model, typically used for implementing guardrails.
  3. AfterRequest: Executed after the AI model processes the request but before the response is returned, used for post-processing and validations.
  4. End: Executed at the end of the request lifecycle for final logging and cleanup.

While hooks can be extended for custom logging solutions and other custom functionalities, their primary intended purpose is to set up and enforce guardrails.

Guardrails

Guardrails in Portkey's AI gateway are a set of checks that are run together within the beforeRequest or afterRequest hooks to determine a verdict. The verdict of a guardrail dictates the actions to be taken on the request or response. For example, if the guardrail fails, the request can be failed, or the response can be returned with a 246 status code indicating that the guardrails failed.

Guardrails can be defined either through the user interface (UI) of Portkey or as a JSON configuration within the Portkey config. This flexibility allows for easy management and customization of guardrails according to the specific needs of the application.

Checks

A check is an individual function that assesses the input prompt or output response against predefined conditions. Each check returns a boolean verdict or may error out if issues are encountered. Checks are the building blocks of guardrails, and Portkey includes a set of predefined checks as well as the ability to add custom checks.

Creating a Plugin

When creating a plugin for Portkey, keep in mind that it should primarily focus on implementing guardrails. Here's how to structure your plugin:

Folder Structure

To create a plugin, follow this structure in the /plugins folder:

text
/plugins
  /your-plugin-name
    - manifest.json
    - main-function.ts
    - test-file.test.ts (recommended)

Manifest File

The manifest.json file defines the plugin's properties, required credentials, and available functions. Here's an example structure:

json
{
  "id": "your-plugin-id",
  "description": "A brief description of your guardrail plugin",
  "credentials": {
    "type": "object",
    "properties": {
      "apiKey": {
        "type": "string",
        "label": "API Key",
        "description": "Your API key description",
        "encrypted": true
      }
    },
    "required": ["apiKey"]
  },
  "functions": [
    {
      "name": "Your Guardrail Function",
      "id": "yourGuardrailId",
      "supportedHooks": ["beforeRequestHook", "afterRequestHook"],
      "type": "guardrail",
      "description": [
        {
          "type": "subHeading",
          "text": "Description of your guardrail function"
        }
      ],
      "parameters": {
        "type": "object",
        "properties": {
          "paramName": {
            "type": "string",
            "label": "Parameter Label",
            "description": [
              {
                "type": "subHeading",
                "text": "Description of the parameter"
              }
            ]
          }
        },
        "required": ["paramName"]
      }
    }
  ]
}

Plugin Function

Create a TypeScript file (e.g., main-function.ts) that exports a handler function. This function will implement your guardrail logic. Here's a basic structure:

typescript
import { HookEventType, PluginContext, PluginHandler, PluginParameters } from '../types';

export const handler: PluginHandler = async (
  context: PluginContext,
  parameters: PluginParameters,
  eventType: HookEventType
) => {
  // Your guardrail logic here
  // You can access:
  // - context.request for request data
  // - context.response for response data (in afterRequestHook)
  // - parameters for plugin-specific parameters
  // - eventType to determine which hook is being executed

  return {
    error: null, // or error object if an error occurred
    verdict: true, // or false to indicate if the guardrail passed or failed
    data: {} // any additional data you want to return
  };
};

Building Plugins into Your Gateway

To build plugins into your Gateway, follow these steps:

  1. In the root directory of the Gateway repository, locate the conf.json file.
  2. Edit the conf.json file to specify which plugins you want to enable and provide any necessary credentials. Here's an example:
json
{
  "plugins_enabled": ["default", "your-guardrail-plugin"],
  "credentials": {
    "your-guardrail-plugin": {
      "apiKey": "your-api-key-here"
    }
  },
  "cache": false
}
  1. Run the build command to compile the plugins:
text
npm run build-plugins

This command will compile the plugins and prepare them for use with the Gateway.

Using Plugins

Once you've built the plugins, they will be automatically deployed with the Gateway when you run it. You can start the Gateway using one of the following commands:

  • For development:
    text
    npm run dev
  • For Node.js:
    text
    npm run dev:node
  • For Cloudflare Workers:
    text
    npm run dev:workerd

The guardrail plugins you've enabled in the conf.json file will be loaded and ready to use.

Testing Plugins

To test your guardrail plugins:

  1. Ensure you have written test files for your plugins (e.g., test-file.test.ts).
  2. Run the test command:
text
npx jest

This command will execute the test files in the plugins repository, allowing you to verify that your guardrail plugins are functioning correctly.

Explore Related Technologies & Topics

View All 9,111+ Repositories β†’

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#ai-gateway Browse ai-gateway catalog β†’ #gateway Browse gateway catalog β†’ #generative-ai Browse generative-ai catalog β†’ #hacktoberfest Browse hacktoberfest catalog β†’ #langchain Browse langchain catalog β†’ #llm Browse llm catalog β†’ ⚑ MCP Servers Model Context Protocol β†’ 🧠 Agent Skills Prompt & rule files β†’ πŸ”Œ OpenAPI Specs REST & Swagger schemas β†’