Repository: BerriAI/litellm

Stars: 43673

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Development Commands

Installation

- make install-dev - Install core development dependencies
- make install-proxy-dev - Install proxy development dependencies with full feature set
- make install-test-deps - Install the full local test environment and generate the Prisma client

Testing

- make test - Run all tests
- make test-unit - Run unit tests (tests/test_litellm) with 4 parallel workers
- make test-integration - Run integration tests (excludes unit tests)
- pytest tests/ - Direct pytest execution

Code Quality

- make lint - Run all linting (Ruff, MyPy, Black, circular imports, import safety)
- make format - Apply Black code formatting
- make lint-ruff - Run Ruff linting only
- make lint-mypy - Run MyPy type checking only
- Before committing, always run uv run black . to format your code. Black formatting is enforced in CI.

Single Test Files

- uv run pytest tests/path/to/test_file.py -v - Run specific test file
- uv run pytest tests/path/to/test_file.py::test_function -v - Run specific test

Running Scripts

- uv run python script.py - Run Python scripts (use for non-test files)

GitHub Issue & PR Templates

When contributing to the project, use the appropriate templates:

Bug Reports (.github/ISSUE_TEMPLATE/bug_report.yml):
- Describe what happened vs. what you expected
- Include relevant log output
- Specify your LiteLLM version

Feature Requests (.github/ISSUE_TEMPLATE/feature_request.yml):
- Describe the feature clearly
- Explain the motivation and use case

Pull Requests (.github/pull_request_template.md):
- Add at least 1 test in tests/litellm/
- Ensure make test-unit passes

Architecture Overview

LiteLLM is a unified interface for 100+ LLM providers with two main components:

Core Library (`litellm/`)

- Main entry point: litellm/main.py - Contains core completion() function
- Provider implementations: litellm/llms/ - Each provider has its own subdirectory
- Router system: litellm/router.py + litellm/router_utils/ - Load balancing and fallback logic
- Type definitions: litellm/types/ - Pydantic models and type hints
- Integrations: litellm/integrations/ - Third-party observability, caching, logging
- Caching: litellm/caching/ - Multiple cache backends (Redis, in-memory, S3, etc.)

Proxy Server (`litellm/proxy/`)

- Main server: proxy_server.py - FastAPI application
- Authentication: auth/ - API key management, JWT, OAuth2
- Database: db/ - Prisma ORM with PostgreSQL/SQLite support
- Management endpoints: management_endpoints/ - Admin APIs for keys, teams, models
- Pass-through endpoints: pass_through_endpoints/ - Provider-specific API forwarding
- Guardrails: guardrails/ - Safety and content filtering hooks
- UI Dashboard: Served from _experimental/out/ (Next.js build)

Key Patterns

Provider Implementation

- Providers inherit from base classes in litellm/llms/base.py
- Each provider has transformation functions for input/output formatting
- Support both sync and async operations
- Handle streaming responses and function calling

Error Handling

- Provider-specific exceptions mapped to OpenAI-compatible errors
- Fallback logic handled by Router system
- Comprehensive logging through litellm/_logging.py

Configuration

- YAML config files for proxy server (see proxy/example_config_yaml/)
- Environment variables for API keys and settings
- Database schema managed via Prisma (proxy/schema.prisma)

Development Notes

Code Style

- Uses Black formatter, Ruff linter, MyPy type checker
- Pydantic v2 for data validation
- Async/await patterns throughout
- Type hints required for all public APIs
- Avoid imports within methods — place all imports at the top of the file (module-level). Inline imports inside functions/methods make dependencies harder to trace and hurt readability. The only exception is avoiding circular imports where absolutely necessary.
- Use dict spread for immutable copies — prefer {original, "key": new_value} over dict(obj) + mutation. The spread produces the final dict in one step and makes intent clear.
- Guard at resolution time — when resolving an optional value through a fallback chain (a or b or ""), raise immediately if the resolved result being empty is an error. Don't pass empty strings or sentinel values downstream for the callee to deal with.
- Extract complex comprehensions to named helpers — a set/dict comprehension that calls into the DB or manager (e.g. "which of these server IDs are OAuth2?") belongs in a named helper function, not inline in the caller.
- FastAPI parameter declarations — mark required query/form params with = Query(...) / = Form(...) explicitly when other params in the same handler are optional. Mixing str (required) with Optional[str] = None in the same signature causes silent 422s when the required param is missing.

Testing Strategy

- Unit tests in tests/test_litellm/
- Integration tests for each provider in tests/llm_translation/
- Proxy tests in tests/proxy_unit_tests/
- Load tests in tests/load_tests/
- Always add tests when adding new entity types or features — if the existing test file covers other entity types, add corresponding tests for the new one
- Keep monkeypatch stubs in sync with real signatures — when a function gains a new optional parameter, update every fake_ / stub_ in tests that patch it to also accept that kwarg (even as kwargs). Stale stubs fail with unexpected keyword argument and mask real bugs.
- Test all branches of name→ID resolution — when adding server/resource lookup that resolves names to UUIDs, test: (1) name resolves and UUID is allowed, (2) name resolves but UUID is not allowed, (3) name does not resolve at all. The silent-fallback path is where access-control bugs hide.

UI / Backend Consistency

- When wiring a new UI entity type to an existing backend endpoint, verify the backend API contract (single value vs. array, required vs. optional params) and ensure the UI controls match — e.g., use a single-select dropdown when the backend accepts a single value, not a multi-select

UI Component Library

- Always use antd for new UI components — we are migrating off of @tremor/react. Do not introduce new Badge, Text, Card, Grid, Title, or other imports from @tremor/react in any new or modified file. Use antd equivalents: Tag for labels, plain <span>/<div> with Tailwind classes (or Typography.Text) for text, Card from antd, etc. Note that antd has no "yellow" Tag color — use "gold" for amber/yellow.

MCP OAuth / OpenAPI Transport Mapping

- TRANSPORT.OPENAPI is a UI-only concept. The backend only accepts "http", "sse", or "stdio". Always map it to "http" before any API call (including pre-OAuth temp-session calls).
- FastAPI validation errors return detail as an array of {loc, msg, type} objects. Error extractors must handle: array (map .msg), string, nested {error: string}, and fallback.
- When an MCP server already has authorization_url stored, skip OAuth discovery (_discovery_metadata) — the server URL for OpenAPI MCPs is the spec file, not the API base, and fetching it causes timeouts.
- client_id should be optional in the /authorize endpoint — if the server has a stored client_id in credentials, use that. Never require callers to re-supply it.

MCP Credential Storage

- OAuth credentials and BYOK credentials share the litellm_mcpusercredentials table, distinguished by a "type" field in the JSON payload ("oauth2" vs plain string).
- When deleting OAuth credentials, check type before deleting to avoid accidentally deleting a BYOK credential for the same (user_id, server_id) pair.
- Always pass the raw expires_at timestamp to the client — never set it to None for expired credentials. Let the frontend compute the "Expired" display state from the timestamp.
- Use RecordNotFoundError (not bare except Exception) when catching "already deleted" in credential delete endpoints.

Browser Storage Safety (UI)

- Never write LiteLLM access tokens or API keys to localStorage — use sessionStorage only. localStorage survives browser close and is readable by any injected script (XSS).
- Shared utility functions (e.g. extractErrorMessage) belong in src/utils/ — never define them inline in hooks or duplicate them across files.

Database Migrations

- Prisma handles schema migrations
- Migration files auto-generated with prisma migrate dev
- Always test migrations against both PostgreSQL and SQLite

Proxy database access

- Do not write raw SQL for proxy DB operations. Use Prisma model methods instead of execute_raw / query_raw.
- Use the generated client: prisma_client.db.<model> (e.g. litellm_tooltable, litellm_usertable) with .upsert(), .find_many(), .find_unique(), .update(), .update_many() as appropriate. This avoids schema/client drift, keeps code testable with simple mocks, and matches patterns used in spend logs and other proxy code.
- No N+1 queries. Never query the DB inside a loop. Batch-fetch with {"in": ids} and distribute in-memory.
- Batch writes. Use create_many/update_many/delete_many instead of individual calls (these return counts only; update_many/delete_many no-op silently on missing rows). When multiple separate writes target the same table (e.g. in batch_()), order by primary key to avoid deadlocks.
- Push work to the DB. Filter, sort, group, and aggregate in SQL, not Python. Verify Prisma generates the expected SQL — e.g. prefer group_by over find_many(distinct=...) which does client-side processing.
- Bound large result sets. Prisma materializes full results in memory. For results over ~10 MB, paginate with take/skip or cursor/take, always with an explicit order. Prefer cursor-based pagination (skip is O(n)). Don't paginate naturally small result sets.
- Limit fetched columns on wide tables. Use select to fetch only needed fields — returns a partial object, so downstream code must not access unselected fields.
- Check index coverage. For new or modified queries, check schema.prisma for a supporting index. Prefer extending an existing index (e.g. @@index([a]) → @@index([a, b])) over adding a new one, unless it's a @@unique. Only add indexes for large/frequent queries.
- Keep schema files in sync. Apply schema changes to all schema.prisma copies (schema.prisma, litellm/proxy/, litellm-proxy-extras/, litellm-js/spend-logs/ for SpendLogs) with a migration under litellm-proxy-extras/litellm_proxy_extras/migrations/.

Setup Wizard (`litellm/setup_wizard.py`)

- The wizard is implemented as a single SetupWizard class with @staticmethod methods — keep it that way. No module-level functions except run_setup_wizard() (the public entrypoint) and pure helpers (color, ANSI).
- Use litellm.utils.check_valid_key(model, api_key) for credential validation — never roll a custom completion call.
- Do not hardcode provider env-key names or model lists that already exist in the codebase. Add a test_model field to each provider entry to drive check_valid_key; set it to None for providers that can't be validated with a single API key (Azure, Bedrock, Ollama).

Enterprise Features

- Enterprise-specific code in enterprise/ directory
- Optional features enabled via environment variables
- Separate licensing and authentication for enterprise features

CI Supply-Chain Safety

- Never pipe a remote script into a shell (curl ... | bash, wget ... | sh). Download the artifact to a file, verify its SHA-256 checksum, then install.
- Pin every external tool to a specific version with a full URL (not latest or stable). Unversioned downloads silently change under you.
- Verify checksums for all downloaded binaries. Use the provider's official .sha256 / .sha256sum sidecar file when available; otherwise compute and hardcode the digest.
- Prefer reusable CircleCI commands (commands: section) so a tool is installed and verified in exactly one place, then referenced everywhere with - install_<tool> or - wait_for_service.
- Don't add tools just because they were there before. Audit whether an external dependency is still needed. If it can be replaced with a shell one-liner or a tool already in the image, remove it.
- These rules apply to every download in CI: binaries, install scripts, language version managers, package repos. No exceptions.

HTTP Client Cache Safety

- Never close HTTP/SDK clients on cache eviction. LLMClientCache._remove_key() must not call close()/aclose() on evicted clients — they may still be used by in-flight requests. Doing so causes RuntimeError: Cannot send a request, as the client has been closed. after the 1-hour TTL expires. Cleanup happens at shutdown via close_litellm_async_clients().

Troubleshooting: DB schema out of sync after proxy restart

litellm-proxy-extras runs prisma migrate deploy on startup using its own bundled migration files, which may lag behind schema changes in the current worktree. Symptoms: Unknown column, Invalid prisma invocation, or missing data on new fields.

Diagnose: Run \d "TableName" in psql and compare against schema.prisma — missing columns confirm the issue.

Fix options:
1. Create a Prisma migration (permanent) — run prisma migrate dev --name <description> in the worktree. The generated file will be picked up by prisma migrate deploy on next startup.
2. Apply manually for local dev — psql -d litellm -c "ALTER TABLE ... ADD COLUMN IF NOT EXISTS ..." after each proxy start. Fine for dev, not for production.
3. Update litellm-proxy-extras — if the package is installed from PyPI, its migration directory must include the new file. Either update the package or run the migration manually until the next release ships it.

README.md

<h1 align="center">
🚅 LiteLLM
</h1>
<p align="center">
<p align="center">LiteLLM AI Gateway
</p>
<p align="center">Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.</p>
<p align="center">
<a href="https://render.com/deploy?repo=https://github.com/BerriAI/litellm" target="_blank" rel="nofollow"><img src="https://render.com/images/deploy-to-render-button.svg" alt="Deploy to Render"></a>
<a href="https://railway.com/deploy/RhvhdC?referralCode=7mRv9K&utm_medium=integration&utm_source=template&utm_campaign=generic">
<img src="https://railway.com/button.svg" alt="Deploy on Railway">
</a>
</p>
</p>
<h4 align="center"><a href="https://docs.litellm.ai/docs/simple_proxy" target="_blank">LiteLLM Proxy Server (AI Gateway)</a> | <a href="https://docs.litellm.ai/docs/enterprise#hosted-litellm-proxy" target="_blank"> Hosted Proxy</a> | <a href="https://litellm.ai/enterprise"target="_blank">Enterprise Tier</a> | <a href="https://www.litellm.ai/ai-gateway" target="_blank">Website</a></h4>
<h4 align="center">
<a href="https://pypi.org/project/litellm/" target="_blank">
<img src="https://img.shields.io/pypi/v/litellm.svg" alt="PyPI Version">
</a>
<a href="https://github.com/BerriAI/litellm" target="_blank">
<img src="https://img.shields.io/github/stars/BerriAI/litellm.svg?style=social" alt="GitHub Stars">
</a>
<a href="https://www.ycombinator.com/companies/berriai">
<img src="https://img.shields.io/badge/Y%20Combinator-W23-orange?style=flat-square" alt="Y Combinator W23">
</a>
<a href="https://wa.link/huol9n">
<img src="https://img.shields.io/static/v1?label=Chat%20on&message=WhatsApp&color=success&logo=WhatsApp&style=flat-square" alt="Whatsapp">
</a>
<a href="https://discord.gg/wuPM9dRgDw">
<img src="https://img.shields.io/static/v1?label=Chat%20on&message=Discord&color=blue&logo=Discord&style=flat-square" alt="Discord">
</a>
<a href="https://www.litellm.ai/support">
<img src="https://img.shields.io/static/v1?label=Chat%20on&message=Slack&color=black&logo=Slack&style=flat-square" alt="Slack">
</a>
<a href="https://codspeed.io/BerriAI/litellm?utm_source=badge">
<img src="https://img.shields.io/endpoint?url=https://codspeed.io/badge.json" alt="CodSpeed"/>
</a>
</h4>

---

What is LiteLLM

LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.

Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.

Jump to LiteLLM Proxy (LLM Gateway) Docs <br>
Jump to Supported LLM Providers

---

Why LiteLLM

Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:

- Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
- Drop-in OpenAI compatibility — swap providers without rewriting your code
- Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
- 8ms P95 latency at 1k RPS (benchmarks)

OSS Adopters

<table>
<tr>
<td><img height="60" alt="Stripe" src="https://github.com/user-attachments/assets/f7296d4f-9fbd-460d-9d05-e4df31697c4b" /></td>
<td><img height="60" alt="image" src="https://github.com/user-attachments/assets/436fca71-988b-40bb-b5fe-8450c80fdbd0" /></td>
<td><img height="60" alt="Google ADK" src="https://github.com/user-attachments/assets/caf270a2-5aee-45c4-8222-41a2070c4f19" /></td>
<td><img height="60" alt="Greptile" src="https://github.com/user-attachments/assets/0be4bd8a-7cfa-48d3-9090-f415fe948280" /></td>
<td><img height="60" alt="OpenHands" src="https://github.com/user-attachments/assets/a6150c4c-149e-4cae-888b-8b92be6e003f" /></td>
<td><h2>Netflix</h2></td>
<td><img height="60" alt="OpenAI Agents SDK" src="https://github.com/user-attachments/assets/c02f7be0-8c2e-4d27-aea7-7c024bfaebc0" /></td>
</tr>
</table>

---

Features

<details open>
<summary><b>LLMs</b> - Call 100+ LLMs (Python SDK + AI Gateway)</summary>

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK

shell

uv add litellm

python

from litellm import completion
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
OpenAI

response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])
Anthropic  

response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server)

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

shell

uv tool install 'litellm[proxy]'
litellm --model gpt-4o

python

import openaiclient = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Docs: LLM Providers

</details>

<details>
<summary><b>Agents</b> - Invoke A2A Agents (Python SDK + AI Gateway)</summary>

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol

python

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4
client = A2AClient(base_url="http://localhost:10001")request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server)

Step 1. Add your Agent to the AI Gateway

Step 2. Call Agent via A2A SDK

python

from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
from uuid import uuid4
import httpx
base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"}    # LiteLLM Virtual Key
async with httpx.AsyncClient(headers=headers) as httpx_client:
    resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)    request = SendMessageRequest(
        id=str(uuid4()),
        params=MessageSendParams(
            message={
                "role": "user",
                "parts": [{"kind": "text", "text": "Hello!"}],
                "messageId": uuid4().hex,
            }
        )
    )
    response = await client.send_message(request)

Docs: A2A Agent Gateway

</details>

<details>
<summary><b>MCP Tools</b> - Connect MCP servers to any LLM (Python SDK + AI Gateway)</summary>

Python SDK - MCP Bridge

python

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm
server_params = StdioServerParameters(command="python", args=["mcp_server.py"])
async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()
        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

bash

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE

json

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer sk-1234"
      }
    }
  }
}

Docs: MCP Gateway

</details>

Supported Providers (Website Supported Models | Docs)

| Provider | /chat/completions | /messages | /responses | /embeddings | /image/generations | /audio/transcriptions | /audio/speech | /moderations | /batches | /rerank |
|-------------------------------------------------------------------------------------|---------------------|-------------|--------------|---------------|----------------------|-------------------------|-----------------|----------------|-----------|-----------|
| Abliteration (abliteration) | ✅ | | | | | | | | | |
| AI/ML API (aiml) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
| AI21 (ai21) | ✅ | ✅ | ✅ | | | | | | | |
| AI21 Chat (ai21_chat) | ✅ | ✅ | ✅ | | | | | | | |
| Aleph Alpha | ✅ | ✅ | ✅ | | | | | | | |
| Amazon Nova | ✅ | ✅ | ✅ | | | | | | | |
| Anthropic (anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | |
| Anthropic Text (anthropic_text) | ✅ | ✅ | ✅ | | | | | | ✅ | |
| Anyscale | ✅ | ✅ | ✅ | | | | | | | |
| AssemblyAI (assemblyai) | ✅ | ✅ | ✅ | | | ✅ | | | | |
| Auto Router (auto_router) | ✅ | ✅ | ✅ | | | | | | | |
| AWS - Bedrock (bedrock) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ |
| AWS - Sagemaker (sagemaker) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| Azure (azure) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Azure AI (azure_ai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| Azure Text (azure_text) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| Baseten (baseten) | ✅ | ✅ | ✅ | | | | | | | |
| Bytez (bytez) | ✅ | ✅ | ✅ | | | | | | | |
| Cerebras (cerebras) | ✅ | ✅ | ✅ | | | | | | | |
| Clarifai (clarifai) | ✅ | ✅ | ✅ | | | | | | | |
| Cloudflare AI Workers (cloudflare) | ✅ | ✅ | ✅ | | | | | | | |
| Codestral (codestral) | ✅ | ✅ | ✅ | | | | | | | |
| Cohere (cohere) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ |
| Cohere Chat (cohere_chat) | ✅ | ✅ | ✅ | | | | | | | |
| CometAPI (cometapi) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| CompactifAI (compactifai) | ✅ | ✅ | ✅ | | | | | | | |
| Custom (custom) | ✅ | ✅ | ✅ | | | | | | | |
| Custom OpenAI (custom_openai) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| Dashscope (dashscope) | ✅ | ✅ | ✅ | | | | | | | |
| Databricks (databricks) | ✅ | ✅ | ✅ | | | | | | | |
| DataRobot (datarobot) | ✅ | ✅ | ✅ | | | | | | | |
| Deepgram (deepgram) | ✅ | ✅ | ✅ | | | ✅ | | | | |
| DeepInfra (deepinfra) | ✅ | ✅ | ✅ | | | | | | | |
| Deepseek (deepseek) | ✅ | ✅ | ✅ | | | | | | | |
| ElevenLabs (elevenlabs) | ✅ | ✅ | ✅ | | | ✅ | ✅ | | | |
| Empower (empower) | ✅ | ✅ | ✅ | | | | | | | |
| Fal AI (fal_ai) | ✅ | ✅ | ✅ | | ✅ | | | | | |
| Featherless AI (featherless_ai) | ✅ | ✅ | ✅ | | | | | | | |
| Fireworks AI (fireworks_ai) | ✅ | ✅ | ✅ | | | | | | | |
| FriendliAI (friendliai) | ✅ | ✅ | ✅ | | | | | | | |
| Galadriel (galadriel) | ✅ | ✅ | ✅ | | | | | | | |
| GitHub Copilot (github_copilot) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| GitHub Models (github) | ✅ | ✅ | ✅ | | | | | | | |
| Google - PaLM | ✅ | ✅ | ✅ | | | | | | | |
| Google - Vertex AI (vertex_ai) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
| Google AI Studio - Gemini (gemini) | ✅ | ✅ | ✅ | | | | | | | |
| GradientAI (gradient_ai) | ✅ | ✅ | ✅ | | | | | | | |
| Groq AI (groq) | ✅ | ✅ | ✅ | | | | | | | |
| Heroku (heroku) | ✅ | ✅ | ✅ | | | | | | | |
| Hosted VLLM (hosted_vllm) | ✅ | ✅ | ✅ | | | | | | | |
| Huggingface (huggingface) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ |
| Hyperbolic (hyperbolic) | ✅ | ✅ | ✅ | | | | | | | |
| IBM - Watsonx.ai (watsonx) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| Infinity (infinity) | | | | ✅ | | | | | | |
| Jina AI (jina_ai) | | | | ✅ | | | | | | |
| Lambda AI (lambda_ai) | ✅ | ✅ | ✅ | | | | | | | |
| Lemonade (lemonade) | ✅ | ✅ | ✅ | | | | | | | |
| LiteLLM Proxy (litellm_proxy) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | |
| Llamafile (llamafile) | ✅ | ✅ | ✅ | | | | | | | |
| LM Studio (lm_studio) | ✅ | ✅ | ✅ | | | | | | | |
| Maritalk (maritalk) | ✅ | ✅ | ✅ | | | | | | | |
| Meta - Llama API (meta_llama) | ✅ | ✅ | ✅ | | | | | | | |
| Mistral AI API (mistral) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| Moonshot (moonshot) | ✅ | ✅ | ✅ | | | | | | | |
| Morph (morph) | ✅ | ✅ | ✅ | | | | | | | |
| Nebius AI Studio (nebius) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| NLP Cloud (nlp_cloud) | ✅ | ✅ | ✅ | | | | | | | |
| Novita AI (novita) | ✅ | ✅ | ✅ | | | | | | | |
| Nscale (nscale) | ✅ | ✅ | ✅ | | | | | | | |
| Nvidia NIM (nvidia_nim) | ✅ | ✅ | ✅ | | | | | | | |
| OCI (oci) | ✅ | ✅ | ✅ | | | | | | | |
| Ollama (ollama) | ✅ | ✅ | ✅ | ✅ | | | | | | |
| Ollama Chat (ollama_chat) | ✅ | ✅ | ✅ | | | | | | | |
| Oobabooga (oobabooga) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| OpenAI (openai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| OpenAI-like (openai_like) | | | | ✅ | | | | | | |
| OpenRouter (openrouter) | ✅ | ✅ | ✅ | | | | | | | |
| OVHCloud AI Endpoints (ovhcloud) | ✅ | ✅ | ✅ | | | | | | | |
| Perplexity AI (perplexity) | ✅ | ✅ | ✅ | | | | | | | |
| Petals (petals) | ✅ | ✅ | ✅ | | | | | | | |
| Predibase (predibase) | ✅ | ✅ | ✅ | | | | | | | |
| Recraft (recraft) | | | | | ✅ | | | | | |
| Replicate (replicate) | ✅ | ✅ | ✅ | | | | | | | |
| Sagemaker Chat (sagemaker_chat) | ✅ | ✅ | ✅ | | | | | | | |
| Sambanova (sambanova) | ✅ | ✅ | ✅ | | | | | | | |
| Snowflake (snowflake) | ✅ | ✅ | ✅ | | | | | | | |
| Text Completion Codestral (text-completion-codestral) | ✅ | ✅ | ✅ | | | | | | | |
| Text Completion OpenAI (text-completion-openai) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | |
| Together AI (together_ai) | ✅ | ✅ | ✅ | | | | | | | |
| Topaz (topaz) | ✅ | ✅ | ✅ | | | | | | | |
| Triton (triton) | ✅ | ✅ | ✅ | | | | | | | |
| V0 (v0) | ✅ | ✅ | ✅ | | | | | | | |
| Vercel AI Gateway (vercel_ai_gateway) | ✅ | ✅ | ✅ | | | | | | | |
| VLLM (vllm) | ✅ | ✅ | ✅ | | | | | | | |
| Volcengine (volcengine) | ✅ | ✅ | ✅ | | | | | | | |
| Voyage AI (voyage) | | | | ✅ | | | | | | |
| WandB Inference (wandb) | ✅ | ✅ | ✅ | | | | | | | |
| Watsonx Text (watsonx_text) | ✅ | ✅ | ✅ | | | | | | | |
| xAI (xai) | ✅ | ✅ | ✅ | | | | | | | |
| Xinference (xinference) | | | | ✅ | | | | | | |

Read the Docs

---

Get Started

You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:

<table style={{width: '100%', tableLayout: 'fixed'}}>
<thead>
<tr>
<th style={{width: '14%'}}></th>
<th style={{width: '43%'}}><strong><a href="https://docs.litellm.ai/docs/simple_proxy">LiteLLM AI Gateway</a></strong></th>
<th style={{width: '43%'}}><strong><a href="https://docs.litellm.ai/docs/">LiteLLM Python SDK</a></strong></th>
</tr>
</thead>
<tbody>
<tr>
<td style={{width: '14%'}}><strong>Use Case</strong></td>
<td style={{width: '43%'}}>Central service (LLM Gateway) to access multiple LLMs</td>
<td style={{width: '43%'}}>Use LiteLLM directly in your Python code</td>
</tr>
<tr>
<td style={{width: '14%'}}><strong>Who Uses It?</strong></td>
<td style={{width: '43%'}}>Gen AI Enablement / ML Platform Teams</td>
<td style={{width: '43%'}}>Developers building LLM projects</td>
</tr>
<tr>
<td style={{width: '14%'}}><strong>Key Features</strong></td>
<td style={{width: '43%'}}>Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management</td>
<td style={{width: '43%'}}>Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - <a href="https://docs.litellm.ai/docs/routing">Router</a>, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)</td>
</tr>
</tbody>
</table>

Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here

Support for more providers. Missing a provider or LLM Platform, raise a feature request.

Run in Developer Mode

#### Services
1. Setup .env file in root
2. Run dependant services docker-compose up db prometheus

#### Backend
1. (In root) create virtual environment python -m venv .venv
2. Activate virtual environment source .venv/bin/activate
3. Install dependencies uv sync --all-extras --group proxy-dev
4. uv run prisma generate
5. prisma generate
6. Start proxy backend python litellm/proxy/proxy_cli.py

#### Frontend
1. Navigate to ui/litellm-dashboard
2. Install dependencies npm install
3. Run npm run dev to start the dashboard

Verify Docker Image Signatures

All LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

bash

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

bash

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).

---

Enterprise

For companies that need better security, user management and professional support

Get an Enterprise License
Talk to founders

This covers:
- ✅ Features under the LiteLLM Commercial License:
- ✅ Feature Prioritization
- ✅ Custom Integrations
- ✅ Professional Support - Dedicated discord + slack
- ✅ Custom SLAs
- ✅ Secure access with Single Sign-On

Contributing

We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.

Quick Start for Contributors

This requires uv to be installed.

bash

git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev    # Install development dependencies
make format         # Format your code
make lint           # Run all linting checks
make test-unit      # Run unit tests
make format-check   # Check formatting only

For detailed contributing guidelines, see CONTRIBUTING.md.

Code Quality / Linting

LiteLLM follows the Google Python Style Guide.

Our automated checks include:
- Black for code formatting
- Ruff for linting and code quality
- MyPy for type checking
- Circular import detection
- Import safety checks

All these checks must pass before your PR can be merged.

Support / talk with founders

- Schedule Demo 👋
- Community Discord 💭
- Community Slack 💭
- Our emails ✉️ [email protected] / [email protected]

Contributors

Repository: BerriAI/litellm

Stars: 43673

CLAUDE.md Copy Chapter

CLAUDE.md

Development Commands Copy Chapter

Installation Copy Section

Testing Copy Section

Code Quality Copy Section

Single Test Files Copy Section

Running Scripts Copy Section

GitHub Issue & PR Templates Copy Section

Architecture Overview Copy Chapter

Core Library (litellm/) litellm/)')" title="Copy section prompt for LLMs"> Copy Section

Proxy Server (litellm/proxy/) litellm/proxy/)')" title="Copy section prompt for LLMs"> Copy Section

Key Patterns Copy Chapter

Provider Implementation Copy Section

Error Handling Copy Section

Configuration Copy Section

Development Notes Copy Chapter

Code Style Copy Section

Testing Strategy Copy Section

UI / Backend Consistency Copy Section

UI Component Library Copy Section

MCP OAuth / OpenAPI Transport Mapping Copy Section

MCP Credential Storage Copy Section

Browser Storage Safety (UI) Copy Section

Database Migrations Copy Section

Proxy database access Copy Section

Setup Wizard (litellm/setup_wizard.py) litellm/setup_wizard.py)')" title="Copy section prompt for LLMs"> Copy Section

Enterprise Features Copy Section

CI Supply-Chain Safety Copy Section

HTTP Client Cache Safety Copy Section

Troubleshooting: DB schema out of sync after proxy restart Copy Section

README.md Copy Chapter

What is LiteLLM Copy Chapter

Why LiteLLM Copy Chapter

OSS Adopters Copy Section

Features Copy Chapter

Python SDK Copy Section

OpenAI

Anthropic

AI Gateway (Proxy Server) Copy Section

Python SDK - A2A Protocol Copy Section

AI Gateway (Proxy Server) Copy Section

Python SDK - MCP Bridge Copy Section

AI Gateway - MCP Gateway Copy Section

Use with Cursor IDE Copy Section

Supported Providers (Website Supported Models | Docs) Website Supported Models | Docs)')" title="Copy section prompt for LLMs"> Copy Section

Get Started Copy Chapter

Run in Developer Mode Copy Section

Verify Docker Image Signatures Copy Section

Enterprise

Contributing

Quick Start for Contributors Copy Chapter

Code Quality / Linting Copy Chapter

Support / talk with founders

Contributors

CLAUDE.md

Development Commands

Installation

Testing

Code Quality

Single Test Files

Running Scripts

GitHub Issue & PR Templates

Architecture Overview

Core Library (`litellm/`)

Proxy Server (`litellm/proxy/`)

Key Patterns

Provider Implementation

Error Handling

Configuration

Development Notes

Code Style

Testing Strategy

UI / Backend Consistency

UI Component Library

MCP OAuth / OpenAPI Transport Mapping

MCP Credential Storage

Browser Storage Safety (UI)

Database Migrations

Proxy database access

Setup Wizard (`litellm/setup_wizard.py`)

Enterprise Features

CI Supply-Chain Safety

HTTP Client Cache Safety

Troubleshooting: DB schema out of sync after proxy restart

README.md

What is LiteLLM

Why LiteLLM

OSS Adopters

Features

Python SDK

AI Gateway (Proxy Server)

Python SDK - A2A Protocol

AI Gateway (Proxy Server)

Python SDK - MCP Bridge

AI Gateway - MCP Gateway

Use with Cursor IDE

Supported Providers (Website Supported Models | Docs)

Get Started

Run in Developer Mode

Verify Docker Image Signatures

Quick Start for Contributors

Code Quality / Linting