1. Project Overview & Quickstart (BerriAI/litellm) Copy Chapter

🚅 LiteLLM

[LiteLLM Proxy Server (AI Gateway)](https://docs.litellm.ai/docs/simple_proxy) | [ Hosted Proxy](https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy) | [Enterprise Tier](https://litellm.ai/enterprise) | [Website](https://www.litellm.ai/ai-gateway)

text Copy

</a>
<a href="https://github.com/BerriAI/litellm" target="_blank">
    
</a>
<a href="https://www.ycombinator.com/companies/berriai">
    
</a>
<a href="https://wa.link/huol9n">
    
</a>
<a href="https://discord.gg/wuPM9dRgDw">
    
</a>
<a href="https://www.litellm.ai/support">
    
</a>
<a href="https://codspeed.io/BerriAI/litellm?utm_source=badge">
    
</a>

Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers

Why LiteLLM Copy Chapter

Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:

• Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
• Drop-in OpenAI compatibility — swap providers without rewriting your code
• Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
• 8ms P95 latency at 1k RPS (benchmarks)

OSS Adopters Copy Section

Features Copy Chapter

LLMs - Call 100+ LLMs (Python SDK + AI Gateway)

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK Copy Section

uv add litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic  
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server) Copy Section

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

uv tool install 'litellm[proxy]'
litellm --model gpt-4o

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Docs: LLM Providers

Agents - Invoke A2A Agents (Python SDK + AI Gateway)

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol Copy Section

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server) Copy Section

Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent

Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0)

import httpx
from a2a.client import A2ACardResolver, ClientConfig, ClientFactory
from a2a.types import Message, Part, Role, SendMessageRequest
from a2a.utils.constants import TransportProtocol
from uuid import uuid4

base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"}    # LiteLLM Virtual Key

async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
    resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    config = ClientConfig(
        httpx_client=http_client,
        streaming=False,
        supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON],
    )
    client = ClientFactory(config).create(agent_card)

    request = SendMessageRequest(
        message=Message(
            message_id=uuid4().hex,
            role=Role.ROLE_USER,
            parts=[Part(text="Hello!")],
        )
    )
    async for event in client.send_message(request):
        populated = event.ListFields()
        if populated and populated[0][0].name in ("message", "msg"):
            print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts))

Docs: A2A Agent Gateway

MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)

Python SDK - MCP Bridge Copy Section

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway Copy Section

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE Copy Section

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer sk-1234"
      }
    }
  }
}

Docs: MCP Gateway

Supported Providers ([Website Supported Models](https://models.litellm.ai/) | [Docs](https://docs.litellm.ai/docs/providers)) Copy Section

Read the Docs

Get Started Copy Chapter

You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:

Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here

Support for more providers. Missing a provider or LLM Platform, raise a feature request.

Deploy on AWS or GCP with Terraform Copy Section

Run the LiteLLM proxy as a production-ready componentized stack (gateway, backend, UI on separate services; managed Postgres + Redis + object store) using the published Terraform modules. Both modules are on the public Terraform Registry — no auth needed.

AWS — ECS Fargate + Aurora + ElastiCache + ALB

— opens an in-browser shell, already authenticated to your AWS account. Once inside, run:

git clone https://github.com/BerriAI/litellm.git
cd litellm/terraform/litellm/aws/examples/default
cp terraform.tfvars.example terraform.tfvars   # edit region/tenant/env
terraform init && terraform apply

Module page →

Or call the module from your own root config:

# main.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.60" }
  }
}

provider "aws" {
  region = "us-west-2"
}

module "litellm" {
  source  = "BerriAI/litellm/aws"
  version = "~> 1.89"

  region = "us-west-2"
  azs    = ["us-west-2a", "us-west-2b"]
  tenant = "acme"
  env    = "prod"

  # Production: provide an ACM cert. Without one, set allow_plaintext_alb = true
  # (dev/trial only).
  # acm_certificate_arn = "arn:aws:acm:us-west-2:111122223333:certificate/..."
  allow_plaintext_alb = true
}

output "litellm_url" {
  value = module.litellm.alb_dns_name
}

terraform init
terraform apply

Provider API keys live in AWS Secrets Manager; reference ARNs via gateway_extra_secrets. Full input list and architecture diagram on the registry page.

GCP — Cloud Run + Cloud SQL + Memorystore + HTTPS LB

Real 1-click. Opens Cloud Shell, clones this repo, and walks you through terraform apply via a built-in DeployStack tutorial — pick the project, the tutorial sets up the Artifact Registry remote repo, writes terraform.tfvars from your answers, and runs apply.

Module page →

To call the module from your own config instead, Cloud Run can't pull from ghcr.io directly, so first set up a one-time Artifact Registry remote repo backed by GHCR:

gcloud artifacts repositories create litellm \
  --location=us-central1 \
  --repository-format=docker \
  --mode=remote-repository \
  --remote-docker-repo=https://ghcr.io \
  --project=my-gcp-project

Then:

# main.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    google      = { source = "hashicorp/google",      version = "~> 6.10" }
    google-beta = { source = "hashicorp/google-beta", version = "~> 6.10" }
  }
}

provider "google"      { project = "my-gcp-project"; region = "us-central1" }
provider "google-beta" { project = "my-gcp-project"; region = "us-central1" }

module "litellm" {
  source  = "BerriAI/litellm/google"
  version = "~> 1.89"

  project_id = "my-gcp-project"
  region     = "us-central1"
  tenant     = "acme"
  env        = "prod"

  # Replace my-gcp-project with your GCP project ID (same value as project_id above).
  image_registry = "us-central1-docker.pkg.dev/my-gcp-project/litellm/berriai"

  # Production: provide DNS already pointing at the LB IP for Google-managed certs.
  # Without one, set allow_plaintext_lb = true (dev/trial only).
  # lb_domains         = ["proxy.example.com"]
  allow_plaintext_lb = true
}

output "litellm_url" {
  value = module.litellm.load_balancer_url
}

terraform init
terraform apply

Provider API keys live in Secret Manager; reference resource IDs (e.g. projects/my-gcp-project/secrets/openai-api-key) via gateway_extra_secrets. Full input list and architecture diagram on the registry page.

Both stacks include

• The full componentized split (gateway / backend / UI as independent services)
• Managed Postgres (writer + reader) and Redis
• Versioned object store for proxy state + file uploads
• An auto-generated LITELLM_MASTER_KEY in your cloud's secret manager
• A one-off migration job that runs prisma migrate deploy before the proxy starts
• The same proxy_config surface as the Helm chart — pass YAML as a typed map

The Terraform modules live at terraform/litellm/aws/ and terraform/litellm/gcp/ in this repo; the registry entries are read-only mirrors updated on each release.

Run in Developer Mode Copy Section

Services

1. Setup .env file in root
2. Run dependent services docker-compose up db prometheus

Backend

1. Run make bootstrap
2. Start proxy backend: uv run python litellm/proxy/proxy_cli.py

Frontend

1. Navigate to ui/litellm-dashboard (dependencies were already installed w/ make bootstrap)
2. Start dashboard: npm run dev

Verify Docker Image Signatures Copy Section

All LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).

Enterprise

For companies that need better security, user management and professional support

Get an Enterprise License
Talk to founders

This covers:

• ✅ Features under the LiteLLM Commercial License:
• ✅ Feature Prioritization
• ✅ Custom Integrations
• ✅ Professional Support - Dedicated discord + slack
• ✅ Custom SLAs
• ✅ Secure access with Single Sign-On

Contributing

We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.

Quick Start for Contributors Copy Chapter

This requires uv to be installed.

git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev    # Install development dependencies
make format         # Format your code
make lint           # Run all linting checks
make test-unit      # Run unit tests
make format-check   # Check formatting only

For detailed contributing guidelines, see CONTRIBUTING.md.

📖 Contributing to documentation? The LiteLLM docs have moved to a separate repository: BerriAI/litellm-docs. Please open doc PRs there. Docs are served at docs.litellm.ai.

Code Quality / Linting Copy Chapter

LiteLLM follows the Google Python Style Guide.

Our automated checks include:

• Black for code formatting
• Ruff for linting and code quality
• MyPy for type checking
• Circular import detection
• Import safety checks

All these checks must pass before your PR can be merged.

Support / talk with founders

• Schedule Demo 👋
• Community Discord 💭
• Community Slack 💭
• Our emails ✉️ [email protected] / [email protected]

Contributors

2. Official Technical Reference & Guides (BerriAI/litellm-docs) Copy Chapter

File: README.md Copy Chapter

litellm-docs

Source for docs.litellm.ai — the documentation site for LiteLLM.

Built with Docusaurus 3.

Local development Copy Chapter

npm install
npm start

Open http://localhost:3000 to see the site with live reload.

Build Copy Chapter

npm run build

Static output goes to build/.

Deploy Copy Chapter

Deploys are handled automatically by Vercel on push to main.

Contributing Copy Chapter

Edits are welcome via pull request. For substantive content changes, please open an issue first to discuss.

The main LiteLLM repository is at https://github.com/BerriAI/litellm.

File: docs/index.md Copy Chapter

id: index title: Getting Started sidebar_label: Quickstart Copy Chapter

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import NavigationCards from '@site/src/components/NavigationCards';
import Image from '@theme/IdealImage';

LiteLLM is an open-source library that gives you a single, unified interface to call 100+ LLMs — OpenAI, Anthropic, Vertex AI, Bedrock, and more — using the OpenAI format.

• Call any provider using the same completion() interface — no re-learning the API for each one
• Consistent output format regardless of which provider or model you use
• Built-in retry / fallback logic across multiple deployments via the Router
• Self-hosted LLM Gateway (Proxy) with virtual keys, cost tracking, and an admin UI

Installation Copy Chapter

uv add litellm

To deploy the full AI Gateway (Proxy) with the Admin UI, follow the Quickstart; it runs as a container and needs no Python setup. To run it from the CLI instead, see the Gateway Quickstart.

Quick Start Copy Chapter

Make your first LLM call using the provider of your choice:

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-api-key"

response = completion(
  model="openai/gpt-4o",
  messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)

from litellm import completion
import os

os.environ["ANTHROPIC_API_KEY"] = "your-api-key"

response = completion(
  model="anthropic/claude-3-5-sonnet-20241022",
  messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)

from litellm import completion
import os

# auth: run 'gcloud auth application-default login'
os.environ["VERTEXAI_PROJECT"] = "your-project-id"
os.environ["VERTEXAI_LOCATION"] = "us-central1"

response = completion(
  model="vertex_ai/gemini-1.5-pro",
  messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)

from litellm import completion
import os

os.environ["AWS_ACCESS_KEY_ID"] = "your-key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your-secret"
os.environ["AWS_REGION_NAME"] = "us-east-1"

response = completion(
  model="bedrock/anthropic.claude-haiku-4-5-20251001:0",
  messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)

from litellm import completion

response = completion(
  model="ollama/llama3",
  messages=[{"role": "user", "content": "Hello, how are you?"}],
  api_base="http://localhost:11434"
)
print(response.choices[0].message.content)

from litellm import completion
import os

os.environ["AZURE_API_KEY"] = "your-key"
os.environ["AZURE_API_BASE"] = "https://your-resource.openai.azure.com"
os.environ["AZURE_API_VERSION"] = "2024-02-01"

response = completion(
  model="azure/your-deployment-name",
  messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.choices[0].message.content)

Every response follows the OpenAI Chat Completions format, regardless of provider. ✅

Response Format Copy Section

Non-streaming responses return a ModelResponse object:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1677858242,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I'm doing well, thanks for asking."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 12,
    "total_tokens": 25
  }
}

Streaming responses (stream=True) yield ModelResponseStream chunks:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion.chunk",
  "created": 1677858242,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "delta": {
        "role": "assistant",
        "content": "Hello"
      },
      "finish_reason": null
    }
  ]
}

📖 Full output format reference →

:::tip Open in Colab

New to LiteLLM? Copy Chapter

Want to get started fast? Head to Tutorials for step-by-step walkthroughs — AI coding tools, agent SDKs, proxy setup, and more.

Need to understand a specific feature? Check Guides for streaming, function calling, prompt caching, and other how-tos.

Choose Your Path Copy Chapter

<NavigationCards
columns={2}
items={[
{
icon: "🐍",
title: "Python SDK",
listDescription: [
"completion(), embedding(), image_generation() and more",
"Router with retry, fallback, and load balancing",
"OpenAI-compatible exceptions across all providers",
"Observability callbacks (Langfuse, MLflow, Helicone…)",
],
to: "#litellm-python-sdk",
},
{
icon: "🖥️",
title: "Proxy Server (LLM Gateway)",
listDescription: [
"Virtual keys with per-key/team/user budgets",
"Centralized logging, guardrails, and caching",
"Admin UI for monitoring and management",
"Drop-in replacement for any OpenAI-compatible client",
],
to: "#litellm-proxy-server-llm-gateway",
},
]}
/>

LiteLLM Python SDK Copy Chapter

Streaming Copy Section

Add stream=True to receive chunks as they are generated:

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-api-key"

for chunk in completion(
  model="openai/gpt-4o",
  messages=[{"role": "user", "content": "Write a short poem"}],
  stream=True,
):
    print(chunk.choices[0].delta.content or "", end="")

Exception Handling Copy Section

LiteLLM maps every provider's errors to the OpenAI exception types — your existing error handling works out of the box:

import litellm

try:
    litellm.completion(
      model="anthropic/claude-instant-1",
      messages=[{"role": "user", "content": "Hey!"}]
    )
except litellm.AuthenticationError as e:
    print(f"Bad API key: {e}")
except litellm.RateLimitError as e:
    print(f"Rate limited: {e}")
except litellm.APIError as e:
    print(f"API error: {e}")

Logging & Observability Copy Section

Send input/output to Langfuse, MLflow, Helicone, Lunary, and more with a single line:

import litellm

litellm.success_callback = ["langfuse", "mlflow", "helicone"]

response = litellm.completion(
  model="gpt-4o",
  messages=[{"role": "user", "content": "Hi!"}]
)

📖 See all observability integrations →

Track Costs & Usage Copy Section

Use a callback to capture cost per response:

import litellm

def track_cost(kwargs, completion_response, start_time, end_time):
    print("Cost:", kwargs.get("response_cost", 0))

litellm.success_callback = [track_cost]

litellm.completion(
  model="gpt-4o",
  messages=[{"role": "user", "content": "Hello!"}],
  stream=True
)

📖 Custom callback docs →

LiteLLM Proxy Server (LLM Gateway) Copy Chapter

The proxy is a self-hosted OpenAI-compatible gateway. Any client that works with OpenAI works with the proxy — no code changes needed.

Step 1 — Start the proxy

litellm --model huggingface/bigcode/starcoder
# Proxy running on http://0.0.0.0:4000

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/your-deployment
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2023-07-01-preview"

docker run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e AZURE_API_KEY=your-key \
  -e AZURE_API_BASE=https://your-resource.openai.azure.com/ \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:latest \
  --config /app/config.yaml --detailed_debug

Step 2 — Call it with the OpenAI client

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")

response = client.chat.completions.create(
  model="gpt-3.5-turbo",
  messages=[{"role": "user", "content": "Write a short poem"}]
)
print(response.choices[0].message.content)

👉 Full proxy quickstart →

:::tip Debugging tool
Use /utils/transform_request to inspect exactly what LiteLLM sends to any provider — useful for debugging prompt formatting, header issues, and provider-specific parameters.
:::

🔗 Interactive API explorer (Swagger) →

Agent & MCP Gateway Copy Chapter

LiteLLM is a unified gateway for LLMs, agents, and MCP — you don't need a separate agent or MCP gateway. One endpoint for 100+ models, A2A agents, and MCP tools.

<NavigationCards
columns={3}
items={[
{
icon: "🔗",
title: "A2A Agents",
to: "/docs/a2a",
},
{
icon: "🛠️",
title: "MCP Gateway",
to: "/docs/mcp",
},
{
icon: "✨",
title: "✨ Enterprise Quickstart",
to: "/docs/learn/enterprise_quickstart",
},
]}
/>

What to Explore Next Copy Chapter

<NavigationCards
columns={3}
items={[
{
icon: "🔀",
title: "Routing & Load Balancing",
to: "/docs/routing-load-balancing",
},
{
icon: "🔑",
title: "Virtual Keys",
to: "/docs/proxy/virtual_keys",
},
{
icon: "📊",
title: "Spend Tracking",
to: "/docs/proxy/cost_tracking",
},
{
icon: "🛡️",
title: "Guardrails",
to: "/docs/proxy/guardrails/quick_start",
},
{
icon: "📡",
title: "Observability",
to: "/docs/observability/opentelemetry_v2",
},
{
icon: "🏭",
title: "Enterprise",
to: "/docs/enterprise",
},
]}
/>