# Technical Documentation: GoogleCloudPlatform/kubectl-ai

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/GoogleCloudPlatform/kubectl-ai)

### Overview (`README.md`)

# kubectl-ai

[](https://goreportcard.com/report/github.com/GoogleCloudPlatform/kubectl-ai)

[](https://deepwiki.com/GoogleCloudPlatform/kubectl-ai)
[](https://github.com/GoogleCloudPlatform/kubectl-ai/stargazers)

`kubectl-ai` acts as an intelligent interface, translating user intent into
precise Kubernetes operations, making Kubernetes management more accessible and
efficient.



## Table of Contents

- [Quick Start](#quick-start)
  - [Installation](#installation)
  - [Usage](#usage)
- [Configuration](#configuration)
- [Tools](#tools)
- [Docker Quick Start](#docker-quick-start)
- [MCP Client Mode](#mcp-client-mode)
- [Extras](#extras)
- [MCP Server Mode](#mcp-server-mode)
- [Start Contributing](#start-contributing)
- [Learning Resources](#learning-resources)

## Quick Start

First, ensure that kubectl is installed and configured.

### Installation

#### Quick Install (Linux & MacOS only)

```shell
curl -sSL https://raw.githubusercontent.com/GoogleCloudPlatform/kubectl-ai/main/install.sh | bash
```

<details>
<summary>Other Installation Methods</summary>

#### Manual Installation (Linux, MacOS and Windows)

1. Download the latest release from the [releases page](https://github.com/GoogleCloudPlatform/kubectl-ai/releases/latest) for your target machine.

2. Untar the release, make the binary executable and move it to a directory in your $PATH (as shown below).

```shell
tar -zxvf kubectl-ai_Darwin_arm64.tar.gz
chmod a+x kubectl-ai
sudo mv kubectl-ai /usr/local/bin/
```

#### Install with Krew (Linux/macOS/Windows)

First of all, you need to have krew installed, refer to [krew document](https://krew.sigs.k8s.io/docs/user-guide/setup/install/) for more details
Then you can install with krew

```shell
kubectl krew install ai
```

Now you can invoke `kubectl-ai` as a kubectl plugin like this: `kubectl ai`.

#### Install on NixOS

There are multiple ways to install `kubectl-ai` on NixOS. For a permanent installation add the following to your NixOS-Configuration:

```nix
  environment.systemPackages = with pkgs; [
    kubectl-ai
  ];
```

For a temporary installation, you can use the following command:

```shell
nix-shell -p kubectl-ai
```

</details>

### Usage

`kubectl-ai` supports AI models from `gemini`, `vertexai`, `azopenai`, `openai`, `grok`, `bedrock` and local LLM providers such as `ollama` and `llama.cpp`.

#### Using Gemini (Default)

Set your Gemini API key as an environment variable. If you don't have a key, get one from [Google AI Studio](https://aistudio.google.com).

```bash
export GEMINI_API_KEY=your_api_key_here
kubectl-ai

# Use different gemini model
kubectl-ai --model gemini-2.5-pro-exp-03-25

# Use 2.5 flash (faster) model
kubectl-ai --quiet --model gemini-2.5-flash-preview-04-17 "check logs for nginx app in hello namespace"
```

<details>
<summary>Use other AI models</summary>

#### Using AI models running locally (ollama or llama.cpp)

You can use `kubectl-ai` with AI models running locally. `kubectl-ai` supports [ollama](https://ollama.com/) and [llama.cpp](https://github.com/ggml-org/llama.cpp) to use the AI models running locally.

Additionally, the [`modelserving`](modelserving) directory provides tools and instructions for deploying your own `llama.cpp`-based LLM serving endpoints locally or on a Kubernetes cluster. This allows you to host models like Gemma directly in your environment.

An example of using Google's `gemma3` model with `ollama`:

```shell
# assuming ollama is already running and you have pulled one of the gemma models
# ollama pull gemma3:12b-it-qat

# if your ollama server is at remote, use OLLAMA_HOST variable to specify the host
# export OLLAMA_HOST=http://192.168.1.3:11434/

# enable-tool-use-shim because models require special prompting to enable tool calling
kubectl-ai --llm-provider ollama --model gemma3:12b-it-qat --enable-tool-use-shim

# you can use `models` command to discover the locally available models
>> models
```

#### Using Grok

You can use X.AI's Grok model by setting your X.AI API key:

```bash
export GROK_API_KEY=your_xai_api_key_here
kubectl-ai --llm-provider=grok --model=grok-3-beta
```

#### Using AWS Bedrock

You can use AWS Bedrock Claude models with your AWS credentials:

```bash
# Configure AWS credentials using AWS SSO
aws sso login --profile your-profile-name
# Or use other AWS credential methods (IAM roles, environment variables, etc.)

# Use Claude 4 Sonnet (default)
kubectl-ai --llm-provider=bedrock --model=us.anthropic.claude-sonnet-4-20250514-v1:0

# Use Claude 3.7 Sonnet
kubectl-ai --llm-provider=bedrock --model=us.anthropic.claude-3-7-sonnet-20250219-v1:0

# Override model via environment variable
export BEDROCK_MODEL=us.anthropic.claude-sonnet-4-20250514-v1:0
kubectl-ai --llm-provider=bedrock
```

AWS Bedrock uses the standard AWS SDK credential chain, supporting:

- AWS SSO profiles
- IAM roles (for EC2/ECS/Lambda)
- Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
- AWS CLI configuration files

#### Using Azure OpenAI

You can also use Azure OpenAI deployment by setting your OpenAI API key and specifying the provider:

```bash
export AZURE_OPENAI_API_KEY=your_azure_openai_api_key_here
export AZURE_OPENAI_ENDPOINT=https://your_azure_openai_endpoint_here
kubectl-ai --llm-provider=azopenai --model=your_azure_openai_deployment_name_here
# or
az login
kubectl-ai --llm-provider=openai://your_azure_openai_endpoint_here --model=your_azure_openai_deployment_name_here
```

#### Using OpenAI

You can also use OpenAI models by setting your OpenAI API key and specifying the provider:

```bash
export OPENAI_API_KEY=your_openai_api_key_here
kubectl-ai --llm-provider=openai --model=gpt-4.1
```

#### Using OpenAI Compatible API

For example, you can use aliyun qwen-xxx models as follows.

```bash
export OPENAI_API_KEY=your_openai_api_key_here
export OPENAI_ENDPOINT=https://dashscope.aliyuncs.com/compatible-mode/v1
kubectl-ai --llm-provider=openai --model=qwen-plus
```

</details>

Run interactively:

```shell
kubectl-ai
```

The interactive mode allows you to have a chat with `kubectl-ai`, asking multiple questions in sequence while maintaining context from previous interactions. Simply type your queries and press Enter to receive responses. To exit the interactive shell, type `exit` or press Ctrl+C.

Or, run with a task as input:

```shell
kubectl-ai --quiet "fetch logs for nginx app in hello namespace"
```

Combine it with other unix commands:

```shell
kubectl-ai < query.txt
# OR
echo "list pods in the default namespace" | kubectl-ai
```

You can even combine a positional argument with stdin input. The positional argument will be used as a prefix to the stdin content:

```shell
cat error.log | kubectl-ai "explain the error"
```

We also support persistence between runs with an opt-in. This lets you save a session to the local filesystem, and resume it to maintain previous context. It even works between different interfaces!

```shell
kubectl-ai --new-session # start a new session
kubectl-ai --list-sessions # list all saved sessions
kubectl-ai --resume-session 20250807-510872 # resume session 20250807-510872
kubectl-ai --delete-session 20250807-510872 # delete session 20250807-510872
```

## Configuration

You can also configure `kubectl-ai` using a YAML configuration file at `~/.config/kubectl-ai/config.yaml`:

```shell
mkdir -p ~/.config/kubectl-ai/
cat <<EOF > ~/.config/kubectl-ai/config.yaml
model: gemini-2.5-flash-preview-04-17
llmProvider: gemini
toolConfigPaths: ~/.config/kubectl-ai/tools.yaml
EOF
```

Verify your configuration:

```shell
kubectl-ai --quiet model
```

<details>
<summary>More configuration Options</summary>

Here's a complete configuration file with all available options and their default values:

```yaml
# LLM provider configuration
llmProvider: "gemini"               # Default LLM provider
model: "gemini-2.5-pro-preview-06-05" # Default model
skipVerifySSL: false              # Skip SSL verification for LLM API calls

# Tool and permission settings
toolConfigPaths: ["~/.config/kubectl-ai/tools.yaml"]  # Custom tools configuration paths
skipPermissions: false             # Skip confirmation for resource-modifying commands
enableToolUseShim: false        # Enable tool use shim for certain models

# MCP configuration
mcpServer: false                  # Run in MCP server mode
mcpClient: false                  # Enable MCP client mode
externalTools: false             # Discover external MCP tools (requires mcp-server)

# Runtime settings
maxIterations: 20                 # Maximum iterations for the agent
quiet: false                       # Run in non-interactive mode
removeWorkdir: false             # Remove temporary working directory after execution

# Kubernetes configuration
kubeconfig: "~/.kube/config"      # Path to kubeconfig file

# UI configuration
uiType: "terminal"                # UI mode: "terminal" or "web"
uiListenAddress: "localhost:8888" # Address for HTML UI server

# Prompt configuration
promptTemplateFilePath: ""      # Custom prompt template file
extraPromptPaths: []            # Additional prompt template paths

# Debug and trace settings
tracePath: "/tmp/kubectl-ai-trace.txt" # Path to trace file
```

</details>

All these settings can be configured through either:

1. Command line flags (e.g., `--model=gemini-2.5-pro`)
2. Configuration file (`~/.config/kubectl-ai/config.yaml`)
3. Environment variables (e.g., `GEMINI_API_KEY`)

Command line flags take precedence over configuration file settings.

## Tools

`kubectl-ai` leverages LLMs to suggest and execute Kubernetes operations using a set of powerful tools. It comes with built-in tools like `kubectl` and `bash`.

You can also extend its capabilities by defining your own custom tools. By default, `kubectl-ai` looks for your tool configurations in `~/.config/kubectl-ai/tools.yaml`.

To specify tools configuration files or directories containing tools configuration files, use:

```sh
./kubectl-ai --custom-tools-config=<path-to-tools-directory> "your prompt here"
```

For further details on how to configure your own tools, [go here](docs/tools.md).

## Docker Quick Start

This project provides a Docker image that gives you a standalone environment for running kubectl-ai, including against a GKE cluster.

### Running the container against GKE

#### Step 1: Build the Image

Clone the repository and build the image with the following command

```bash
git clone https://github.com/GoogleCloudPlatform/kubectl-ai.git
cd kubectl-ai
docker build -t kubectl-ai:latest -f images/kubectl-ai/Dockerfile .
```

#### Step 2: Connect to Your GKE Cluster

Set up application default credentials and connect to your GKE cluster.

```bash
gcloud auth application-default login # If in a gcloud shell this is not necessary
gcloud container clusters get-credentials <cluster-name> --zone <zone>
```

#### Step 3: Run the kubectl-ai container

Below is a sample command that can be used to launch the container with a locally hosted web-ui. Be sure to replace the placeholder values with your specific Google Cloud project ID and location. Note you do not need to mount the gcloud config directory if you're on a cloudshell machine.

```bash
docker run --rm -it -p 8080:8080 -v ~/.kube:/root/.kube -v ~/.config/gcloud:/root/.config/gcloud -e GOOGLE_CLOUD_LOCATION=us-central1 -e GOOGLE_CLOUD_PROJECT=my-gcp-project kubectl-ai:latest --llm-provider vertexai --ui-listen-address 0.0.0.0:8080 --ui-type web
```

For more info about running from the container image see [CONTAINER.md](CONTAINER.md)

## MCP Client Mode

> **Note:** MCP Client Mode is available in `kubectl-ai` version v0.0.12 and onwards.

`kubectl-ai` can connect to external [MCP](https://modelcontextprotocol.io/examples) Servers to access additional tools in addition to built-in tools.

### Quick Start with MCP Client

Enable MCP client mode:

```bash
kubectl-ai --mcp-client
```

### MCP Client Configuration

Create or edit `~/.config/kubectl-ai/mcp.yaml` to customize MCP servers:

```yaml
servers:
  # Local MCP server (stdio-based)
  # sequential-thinking: Advanced reasoning and step-by-step analysis
  - name: sequential-thinking
    command: npx
    args:
      - -y
      - "@modelcontextprotocol/server-sequential-thinking"
  
  # Remote MCP server (HTTP-based)
  - name: cloudflare-documentation
    url: https://docs.mcp.cloudflare.com/mcp
    
  # Optional: Remote MCP server with authentication
  - name: custom-api
    url: https://api.example.com/mcp
    auth:
      type: "bearer"
      token: "${MCP_TOKEN}"
```

The system automatically:

- Converts parameter names (snake_case → camelCase)
- Handles type conversion (strings → numbers/booleans when appropriate)
- Provides fallback behavior for unknown servers

No additional setup required - just use the `--mcp-client` flag and the AI will have access to all configured MCP tools.

📖 **For detailed configuration options, troubleshooting, and advanced features for MCP Client mode, see the [MCP Client Documentation](docs/mcp-client.md).**

📖 **For multi-server orchestration and security automation examples, see the [MCP Client Integration Guide](docs/mcp-client.md).**

## Extras

You can use the following special keywords for specific actions:

- `model`: Display the currently selected model.
- `models`: List all available models.
- `tools`: List all available tools.
- `version`: Display the `kubectl-ai` version.
- `reset`: Clear the conversational context.
- `clear`: Clear the terminal screen.
- `exit` or `quit`: Terminate the interactive shell (Ctrl+C also works).

### Invoking as kubectl plugin

You can also run `kubectl ai`. `kubectl` finds any executable file in your `PATH` whose name begins with `kubectl-` as a [plugin](https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/).

## MCP Server Mode

`kubectl-ai` can act as an MCP server that exposes kubectl tools to other MCP clients (like Claude, Cursor, or VS Code). The server can run in two modes:

### Basic MCP Server (Built-in tools only)

Expose only kubectl-ai's native Kubernetes tools:

```bash
kubectl-ai --mcp-server
```

### Enhanced MCP Server (With external tool discovery)

Additionally discover and expose tools from other MCP servers as a unified interface:

```bash
kubectl-ai --mcp-server --external-tools
```

This creates a powerful **tool aggregation hub** where kubectl-ai acts as both:

- **MCP Server**: Exposing kubectl tools to clients
- **MCP Client**: Consuming tools from other MCP servers

To serve clients over HTTP using the streamable transport, run:

```bash
kubectl-ai --mcp-server --mcp-server-mode streamable-http --http-port 9080
```

This starts an MCP endpoint at `http://localhost:9080/mcp`.

The enhanced mode provides AI clients with access to both Kubernetes operations and general-purpose tools (filesystem, web search, databases, etc.) through a single MCP endpoint.

### Authenticating the HTTP Endpoint (OAuth 2.1)

The `streamable-http` endpoint is unauthenticated by default — anyone who can reach the port can run `kubectl` and `bash`. For non-localhost use, turn the server into an OAuth 2.1 Resource Server that requires a Bearer access token (verified locally against an authorization server's JWKS, e.g. [AuthGate](https://github.com/go-authgate/authgate)):

```bash
kubectl-ai --mcp-server --mcp-server-mode streamable-http --http-port 9080 \
  --mcp-auth-issuer https://authgate.corp \
  --mcp-auth-audience https://kubectl-ai.corp/mcp
```

`--mcp-auth-audience` is this server's resource identifier; the JWT's `aud` claim must match it, which binds each token to *this* MCP server and prevents tokens issued for other services (same authorization server) from being replayed here. It is required once `--mcp-auth-issuer` is set — see [Why the audience (`aud`) matters](docs/mcp-server.md#why-the-audience-aud-matters).

Authentication is opt-in: without `--mcp-auth-issuer` the endpoint behaves exactly as before. See the [MCP Server Documentation](docs/mcp-server.md#securing-the-http-endpoint-with-oauth-21-authgate) for details.

📖 **For detailed configuration, examples, and troubleshooting, see the [MCP Server Documentation](docs/mcp-server.md).**

## Start Contributing

We welcome contributions to `kubectl-ai` from the community. Take a look at our
[contribution guide](contributing.md) to get started.

## Learning Resources

### Talks and Presentations

- [From Natural Language to K8s Operations: The MCP Architecture and Practice of kubectl-ai](https://blog.wu-boy.com/2025/10/from-natural-language-to-k8s-operations-the-mcp-architecture-and-practice-of-kubectl-ai-en) - A comprehensive presentation covering the architecture and practical usage of kubectl-ai with MCP (Model Context Protocol).

---

*Note: This is not an officially supported Google product. This project is not
eligible for the [Google Open Source Software Vulnerability Rewards
Program](https://bughunters.google.com/open-source-security).*

---

### Overview (`gollm/README.md`)

# gollm

A Go library for calling into multiple Large Language Model (LLM) providers with a unified interface.

This library is intended for use by kubectl-ai, but may prove useful for other similar go tools in future.

Note that the library is still evolving and will likely make incompatible changes often.  We are focusing on kubectl-ai's use-case,
but will consider changes to support additional use-cases.

## Overview

gollm provides a consistent API for interacting with various LLM providers, making it easy to switch between different models and services without changing your application code. The library supports both chat-based conversations and single completions, with features like function calling, streaming responses, and retry logic.

## Features

- **Multi-provider support**: OpenAI, Azure OpenAI, Google Gemini, Ollama, LlamaCPP, Grok, and more
- **Unified interface**: Consistent API across all providers
- **Chat conversations**: Multi-turn conversations with conversation history
- **Function calling**: Define and use custom functions with LLMs
- **Streaming support**: Real-time streaming responses
- **Retry logic**: Built-in retry mechanisms with configurable backoff
- **Response schemas**: Constrain LLM responses to specific JSON schemas
- **SSL configuration**: Optional SSL certificate verification skipping
- **Environment-based configuration**: Easy setup via environment variables

## Providers

| Provider | ID | Description |
|----------|----|-------------|
| OpenAI | `openai://` | OpenAI's GPT models |
| Azure OpenAI | `azopenai://` | Microsoft Azure's OpenAI service |
| Google Gemini | `gemini://` | Google's Gemini models |
| Vertex AI | `vertexai://` | Google Cloud Vertex AI (via Gemini) |
| Ollama | `ollama://` | Local Ollama models |
| LlamaCPP | `llamacpp://` | Local LlamaCPP models |
| Grok | `grok://` | xAI's Grok models |
| Anthropic | `anthropic://` | Claude models with native tool use, prompt caching, and extended thinking |

## Quick Start

### Installation

```bash
go get github.com/GoogleCloudPlatform/kubectl-ai/gollm
```

### Basic Usage

```go
package main

import (
    "context"
    "fmt"
    "log"
    
    "github.com/GoogleCloudPlatform/kubectl-ai/gollm"
)

func main() {
    ctx := context.Background()
    
    // Create a client using environment variable
    client, err := gollm.NewClient(ctx, "")
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()
    
    // Start a chat conversation
    chat := client.StartChat("You are a helpful assistant.", "gpt-3.5-turbo")
    
    // Send a message
    response, err := chat.Send(ctx, "Hello, how are you?")
    if err != nil {
        log.Fatal(err)
    }
    
    // Print the response
    for _, candidate := range response.Candidates() {
        fmt.Println(candidate.String())
    }
}
```

### Environment Configuration

Set the `LLM_CLIENT` environment variable to specify your preferred provider:

```bash
# OpenAI
export LLM_CLIENT="openai://api.openai.com"
export OPENAI_API_KEY="your-api-key"

# Azure OpenAI
export LLM_CLIENT="azopenai://your-resource.openai.azure.com"
export AZURE_OPENAI_API_KEY="your-api-key"

# Google Gemini
export LLM_CLIENT="gemini://generativelanguage.googleapis.com"
export GOOGLE_API_KEY="your-api-key"

# Ollama (local)
export LLM_CLIENT="ollama://localhost:11434"
```


## Examples

### Single Completion

```go
ctx := context.Background()
client, err := gollm.NewClient(ctx, "openai://api.openai.com")
if err != nil {
    log.Fatal(err)
}
defer client.Close()

req := &gollm.CompletionRequest{
    Model:  "gpt-3.5-turbo",
    Prompt: "Write a short poem about programming",
}

response, err := client.GenerateCompletion(ctx, req)
if err != nil {
    log.Fatal(err)
}

fmt.Println(response.Response())
```

### Streaming Chat

```go
ctx := context.Background()
client, err := gollm.NewClient(ctx, "openai://api.openai.com")
if err != nil {
    log.Fatal(err)
}
defer client.Close()

chat := client.StartChat("You are a helpful assistant.", "gpt-3.5-turbo")

// Send a streaming message
iterator, err := chat.SendStreaming(ctx, "Tell me a story about a robot")
if err != nil {
    log.Fatal(err)
}

// Process streaming response
for response := range iterator {
    if response.V1 != nil {
        for _, candidate := range response.V1.Candidates() {
            for _, part := range candidate.Parts() {
                if text, ok := part.AsText(); ok {
                    fmt.Print(text)
                }
            }
        }
    }
    if response.V2 != nil {
        // Handle error
        log.Printf("Error: %v", response.V2)
        break
    }
}
```

### Function Calling

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Response Schema Constraints

```go
// Define a schema for structured responses
schema := &gollm.Schema{
    Type: gollm.TypeObject,
    Properties: map[string]*gollm.Schema{
        "name": {
            Type:        gollm.TypeString,
            Description: "The person's name",
        },
        "age": {
            Type:        gollm.TypeInteger,
            Description: "The person's age",
        },
        "interests": {
            Type: gollm.TypeArray,
            Items: &gollm.Schema{
                Type: gollm.TypeString,
            },
            Description: "List of interests",
        },
    },
    Required: []string{"name", "age"},
}

client.SetResponseSchema(schema)

// Now all responses will be constrained to match this schema
response, err := chat.Send(ctx, "Tell me about a person named Alice who is 30 years old")
```

### Retry Logic

```go
// Configure retry behavior
retryConfig := gollm.RetryConfig{
    MaxAttempts:    3,
    InitialBackoff: time.Second,
    MaxBackoff:     30 * time.Second,
    BackoffFactor:  2.0,
    Jitter:         true,
}

// Create a chat with retry logic
chat := client.StartChat("You are a helpful assistant.", "gpt-3.5-turbo")
retryChat := gollm.NewRetryChat(chat, retryConfig)

// Use the retry chat - it will automatically retry on retryable errors
response, err := retryChat.Send(ctx, "Hello!")
```

### Building Schemas from Go Types

```go
type Person struct {
    Name     string   `json:"name"`
    Age      int      `json:"age"`
    Interests []string `json:"interests,omitempty"`
}

// Automatically build a schema from a Go struct
schema := gollm.BuildSchemaFor(reflect.TypeOf(Person{}))

// Use the schema to constrain responses
client.SetResponseSchema(schema)
```

## Configuration Options

### Client Options

```go
// Create a client with custom options
client, err := gollm.NewClient(ctx, "openai://api.openai.com",
    gollm.WithSkipVerifySSL(), // Skip SSL verification (for development)
)
```

### Environment Variables

- `LLM_CLIENT`: The provider URL to use (e.g., "openai://api.openai.com")
- `LLM_SKIP_VERIFY_SSL`: Set to "1" or "true" to skip SSL certificate verification
- Provider-specific API keys (e.g., `OPENAI_API_KEY`, `GOOGLE_API_KEY`)

<details>
<summary><h3>Anthropic-specific environment variables and provider features</h3></summary>

#### Anthropic-specific environment variables

| Variable | Description | Default |
|----------|-------------|---------|
| `ANTHROPIC_API_KEY` | Anthropic API key (required) | — |
| `ANTHROPIC_MODEL` | Default Claude model to use | `claude-sonnet-4-6` |
| `ANTHROPIC_PROMPT_CACHING` | Enable prompt caching (`"false"` to disable) | `true` |
| `ANTHROPIC_EXTENDED_THINKING` | Enable extended thinking(`"true"` to enable) | `false` |
| `ANTHROPIC_MAX_TOKENS` | Max output tokens per request | `4096` |

### Anthropic provider features

#### Prompt caching

Enabled by default. Anthropic's [prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)
attaches a `cache_control` breakpoint to the system prompt and the last tool
definition so that both are reused from cache on every subsequent turn. Because
kubectl-ai's system prompt is large and repeated verbatim each turn, this
typically cuts input token costs significantly after the first request.

Set `ANTHROPIC_PROMPT_CACHING=false` to disable.

#### Extended thinking

Disabled by default. When enabled, Claude produces a `thinking` content block
containing its internal reasoning before giving a final answer. This can improve
accuracy on complex multi-step queries (e.g. root-cause analysis across multiple
Kubernetes resources). Requires a model that supports extended thinking
(`claude-3-7-sonnet-20250219` or later) and reserves 8 000 tokens for the
thinking budget.

The thinking block is kept in the conversation history (required by the API for
multi-turn consistency) but is **not** shown in the terminal output.

```bash
ANTHROPIC_EXTENDED_THINKING=true \
    kubectl-ai --llm-provider=anthropic --model claude-3-7-sonnet-20250219 \
    "why is my pod crashlooping"
```

Set `ANTHROPIC_EXTENDED_THINKING=true` to enable.

#### Native streaming

The Anthropic provider uses the official SSE event stream directly, bypassing
the OpenAI compatibility shim. Tool input JSON is accumulated across
`content_block_delta` events and only emitted as a complete `FunctionCall` once
the block closes, so partial JSON is never forwarded to the agent loop.

#### Retryable errors

The provider maps Anthropic-native HTTP status codes to retry decisions:

| Status | Meaning | Retried? |
|--------|---------|----------|
| 429 | Rate limit | Yes |
| 529 | Overloaded | Yes |
| 5xx | Server error | Yes |
| 4xx (other) | Client error | No |

</details>

## Error Handling

The library provides structured error handling with retryable error detection:

```go
var apiErr *gollm.APIError
if errors.As(err, &apiErr) {
    fmt.Printf("API Error: Status=%d, Message=%s\n", apiErr.StatusCode, apiErr.Message)
}

// Check if an error is retryable
if chat.IsRetryableError(err) {
    // Implement retry logic
}
```

## Adding a provider

To add a new provider:

1. Create a new file (e.g., `myprovider.go`)
2. Implement the `Client` interface
3. Register the provider in an `init()` function:

```go
func init() {
    if err := gollm.RegisterProvider("myprovider", myProviderFactory); err != nil {
        panic(err)
    }
}
```

## License

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for details.

---

### Overview (`kubectl-utils/README.md`)

kubectl-utils contains some experimental kubectl extensions that should help us write simpler evals for kubectl-ai

They may one day be useful in their own right, but that is not the current goal.

# kubectl expect

kubectl expect polls an object, waiting for a CEL expression to be true.

Example usage: `kubectl expect StatefulSet/mysql 'self.status.replicas >= 1'`

---

### Overview (`modelserving/README.md`)

# Model Serving

This directory provides components to build and deploy Large Language Model (LLM) serving endpoints.

- [`k8s/`](k8s/): Kubernetes manifests for model serving components.
- [`images/`](images/): Dockerfiles for building model serving container images.
- [`dev/tasks`](dev/tasks): Development-related scripts for model serving.
  - `download-model`: fetch the required model weights (e.g., Gemma 3 12B IT).
  - `build-images`: runs `download-model`, and then build the Docker image using the provided Dockerfile in `images/`.
  - `deploy-to-gke` or `dev/tasks/deploy-to-kind`: runs `build-images`, and then deploy the model serving Kubernetes manifests to Google Kubernetes Engine (GKE) or a local KinD cluster. Once deployed, the model server will be accessible via a Kubernetes Service defined in the manifest. You can use `kubectl get svc` to find the service details and access its endpoint.
  - `run-local`: run the model server locally for testing purposes, bypassing Kubernetes.

---

### Overview (`pkg/mcp/README.md`)

# MCP (Model Context Protocol) Client

This package provides functionality to interact with MCP (Model Context Protocol) servers from `kubectl-ai`.

## Overview

The MCP client allows `kubectl-ai` to connect to MCP servers, discover available tools, and execute them. This enables integration with various services and systems that expose their functionality through the MCP protocol.

## Features

- Connect to multiple MCP servers simultaneously
- Support for both local (stdio-based) and remote (HTTP-based) MCP servers
- Authentication support for HTTP-based servers (Basic, Bearer Token, API Key)
- Automatic discovery of available tools from connected servers
- Execute tools on MCP servers with parameter conversion
- Configuration-based server management
- Generic parameter name and type conversion (snake_case → camelCase, intelligent type inference)
- Synchronous initialization ensuring tools are available before conversation starts

## Configuration

MCP server configurations are stored in `~/.config/kubectl-ai/mcp.yaml`. If this file doesn't exist, a default configuration will be created automatically.

### Default Configuration

By default, the MCP client is configured with sequential thinking MCP server:

```yaml
servers:
  - name: sequential-thinking
    command: npx
    args:
      - -y
      - "@modelcontextprotocol/server-sequential-thinking"
```

### Configuration Format

The configuration file uses YAML format and supports both local (stdio-based) and remote (HTTP-based) MCP servers:

#### Local (stdio-based) Server Configuration

```yaml
servers:
  - name: server-name
    command: path-to-server-binary
    args:
      - --flag1
      - value1
    env:
      ENV_VAR: value
```

#### Remote (HTTP-based) Server Configuration

```yaml
servers:
  - name: remote-server
    url: "https://mcp-server.example.com/"
    timeout: 30  # Optional: Timeout in seconds
    use_streaming: true  # Optional: Use streaming HTTP client
    # Optional authentication
    auth:
      type: "bearer"  # Options: "basic", "bearer", "api-key"
      token: "${YOUR_ENV_VAR}"  # Will be read from YOUR_ENV_VAR environment variable
    # Optional custom headers
    headers:
      X-Custom-Header: "custom-value"
      X-API-Version: "v1"
```

### Authentication Options

Remote MCP servers support different authentication methods:

1. **Bearer Token**:

   ```yaml
   auth:
     type: "bearer"
     token: "your-bearer-token"
   ```

2. **Basic Authentication**:

   ```yaml
   auth:
     type: "basic"
     username: "username"
     password: "password"
   ```

3. **API Key**:

   ```yaml
   auth:
     type: "api-key"
     api_key: "your-api-key"
     header_name: "X-Api-Key"  # Optional: Defaults to X-Api-Key
   ```

### Custom Headers

Remote MCP servers support custom HTTP headers for additional configuration or authentication requirements:

```yaml
servers:
  - name: remote-server
    url: "https://mcp-server.example.com/"
    headers:
      X-Custom-Header: "custom-value"
      X-API-Version: "v1"
      User-Agent: "kubectl-ai/1.0"
      Accept-Language: "en-US"
```

**Key Points:**

- Custom headers are applied to all HTTP requests sent to the MCP server
- Headers can be combined with authentication methods
- Authentication headers (e.g., `Authorization`) may override custom headers if both are specified
- All header values are strings
- Headers are case-sensitive as per HTTP specification

**Common Use Cases:**

- API versioning headers (e.g., `X-API-Version`)
- Custom user agent strings
- Request tracking headers (e.g., `X-Request-ID`)
- Content negotiation headers (e.g., `Accept`, `Accept-Language`)

### Environment Variable Support

Sensitive information like tokens and passwords can be read from environment variables using the `${VAR_NAME}` syntax in the configuration file. You can also set environment variables with the prefix `MCP_SERVER_NAME_` to override configuration values.

## Usage

Enable MCP client functionality with the `--mcp-client` flag:

```bash
kubectl-ai --mcp-client
```

### Checking Server Status

When you run kubectl-ai with the MCP client enabled, you'll see information about connected servers:

```txt
MCP Server Status:

Successfully connected to 2 MCP server(s) (2 tools discovered)

• sequential-thinking (npx) - Connected, Tools: sequentialthinking
• fetch (remote) - Connected, Tools: fetch  
```

MCP servers are automatically discovered and their tools made available to the AI. The system handles:

- **Parameter conversion**: Automatically converts snake_case parameters to camelCase
- **Type inference**: Intelligently converts string parameters to numbers/booleans based on naming patterns
- **Error handling**: Graceful fallbacks for connection issues

### Custom Server Examples

To add custom MCP servers, edit the configuration file at `~/.config/kubectl-ai/mcp.yaml`:
You can combine both local and remote servers in your configuration:

```yaml
servers:
  - name: sequential-thinking
    command: npx
    args:
      - -y
      - '@modelcontextprotocol/server-sequential-thinking'
  - name: cloudflare-documentation
    url: https://docs.mcp.cloudflare.com/mcp
```

### Environment Variables

You can configure the following environment variables to customize MCP client behavior:

- `KUBECTL_AI_MCP_CONFIG`: Override the default configuration file path
- `MCP_<SERVER_NAME>_<ENV_VAR>`: Set environment variables for specific servers

## Parameter Conversion

The MCP client automatically handles parameter name and type conversion to ensure compatibility with different MCP servers:

### Name Conversion

- Converts snake_case parameter names to camelCase
- Example: `thought_number` → `thoughtNumber`

### Type Conversion

Parameters are intelligently converted based on naming patterns:

**Numbers:** Parameters containing `number`, `count`, `total`, `max`, `min`, `limit`
**Booleans:** Parameters starting with `is`, `has`, `needs`, `enable` or containing `required`, `enabled`

### Fallback Behavior

- If type conversion fails, the original value is preserved
- Unknown servers use generic conversion rules
- No configuration required - works automatically with any MCP server

## Implementation Details

### Client

The `Client` struct represents a connection to an MCP server. It provides methods to:

- Connect to the server
- List available tools
- Execute tools
- Close the connection

### Manager

The `Manager` struct manages multiple MCP client connections. It provides:

- Connection management for multiple servers
- Tool discovery across all connected servers
- Thread-safe operations

### Configuration

The `Config` struct handles loading and saving MCP server configurations from disk. The configuration is automatically loaded from `~/.config/kubectl-ai/mcp.yaml` when needed.

## Integration with kubectl-ai

The MCP client is integrated with `kubectl-ai` to automatically discover and use tools from configured MCP servers. The system:

1. **Loads configuration** from `~/.config/kubectl-ai/mcp.yaml` on startup
2. **Connects synchronously** to all configured MCP servers (when `--mcp-client` flag is used)
3. **Registers tools** before the conversation starts, ensuring they're immediately available
4. **Converts parameters** automatically using generic snake_case → camelCase conversion
5. **Handles execution** with proper error handling and result formatting
6. **Displays status** showing connected servers and available tool counts

📖 **For practical multi-server orchestration examples and security automation workflows, see the [MCP Client Integration Guide](../../docs/mcp-client.md).**

## Security Considerations

- MCP servers can execute arbitrary commands with the same permissions as the `kubectl-ai` process
- Only connect to trusted MCP servers
- The configuration file has strict permissions (0600) by default
- Be cautious when adding environment variables with sensitive information

## Troubleshooting

### Common Issues

**MCP tools are not available:**

- Ensure you're using the `--mcp-client` flag
- Check that `~/.config/kubectl-ai/mcp.yaml` exists and is valid (created by default)
- Verify MCP servers are installed (e.g., `npx` commands work)

**Connection failures:**

- Check network connectivity
- Ensure server commands and paths are correct in configuration
- Verify environment variables are properly set

**Parameter conversion issues:**

- The system automatically converts snake_case → camelCase
- String parameters are converted to numbers/booleans based on naming patterns
- Fallback behavior preserves original values if conversion fails

### Debug Information

- Use `-v=1` for basic MCP operation logging
- Use `-v=2` for detailed connection and tool discovery info  
- Check server status in the startup message
- Tool counts are displayed for each connected server

---

### Bedrock (`docs/bedrock.md`)

# AWS Bedrock Provider

kubectl-ai supports AWS Bedrock models including Claude Sonnet 4 and Claude 3.7.

## Setup

### AWS Credentials

Configure AWS credentials using standard AWS SDK methods:

```bash
# Option 1: Environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_REGION="us-east-1"

# Option 2: AWS Profile (recommended)
export AWS_PROFILE="your-profile-name"
export AWS_REGION="us-east-1"

# Option 3: Use IAM roles (on EC2/ECS/Lambda)
export AWS_REGION="us-east-1"
```

### Model Configuration

```bash
# Optional: Set default model
export BEDROCK_MODEL="us.anthropic.claude-3-7-sonnet-20250219-v1:0"
```

## Supported Models

See [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html) for current model availability and regional support.

Currently supported:

- Claude Sonnet 4: `us.anthropic.claude-sonnet-4-20250514-v1:0` (default)
- Claude 3.7 Sonnet: `us.anthropic.claude-3-7-sonnet-20250219-v1:0`

## Usage

```bash
# Use default model (Claude Sonnet 4)
kubectl-ai --provider bedrock "explain this deployment"

# Specify model explicitly
kubectl-ai --provider bedrock --model us.anthropic.claude-3-7-sonnet-20250219-v1:0 "help me debug this pod"
```

## Authentication

kubectl-ai uses the standard AWS SDK credential provider chain:

1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
2. AWS credentials file (~/.aws/credentials)
3. AWS config file (~/.aws/config)
4. IAM roles for EC2 instances
5. IAM roles for ECS tasks
6. IAM roles for Lambda functions

For more details, see [AWS SDK Go Configuration](https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/).

## Region Configuration

Bedrock is available in specific AWS regions. Set your region using:

```bash
export AWS_REGION="us-east-1"  # Primary Bedrock region
```

Alternatively, configure region in `~/.aws/config`:

```ini
[default]
region = us-east-1
```

---

### Gke Deployment (`docs/gke-deployment.md`)

# Deploying k8-kate to Google Kubernetes Engine

## Prerequisites

- A GKE cluster (Standard or Autopilot).
- `gcloud` CLI authenticated with `gcloud auth login`.
- Local Docker environment (or Cloud Build) capable of building and pushing container images.
- [`kubectl` configured](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) to talk to your target cluster.
- A Gemini API key with access to the model you plan to demo.

## 1. Set your Google Cloud context

```bash
export PROJECT_ID="my-gcp-project"
export REGION="us-central1"
export CLUSTER_NAME="kubectl-ai-demo"

gcloud config set project "${PROJECT_ID}"
gcloud container clusters get-credentials "${CLUSTER_NAME}" --region "${REGION}"
```

These commands configure both `gcloud` and `kubectl` to operate on the cluster that will host `kubectl-ai`.

## 2. Build and push the kubectl-ai image

Pick an Artifact Registry repository or another container registry that your cluster can pull from. The snippet below creates an Artifact Registry repo (if needed), builds the image locally, and pushes it.

```bash
# Create an Artifact Registry (skip if you already have one)
gcloud artifacts repositories create kubectl-ai \
  --location="${REGION}" \
  --repository-format=DOCKER \
  --description="kubectl-ai demo images"

# Configure Docker to authenticate to Artifact Registry
gcloud auth configure-docker "${REGION}"-docker.pkg.dev

# Build and push the container
IMAGE="${REGION}-docker.pkg.dev/${PROJECT_ID}/kubectl-ai/kubectl-ai:latest"
docker build -t "${IMAGE}" -f images/kubectl-ai/Dockerfile .
docker push "${IMAGE}"
```
## 3. Prepare cluster namespaces and RBAC

Create the namespaces and RBAC that the hosted agent requires:

```bash
# Sandbox namespace + RBAC (creates `computer` namespace, service account, and reader roles)
kubectl apply -f k8s/sandbox/all-in-one.yaml
```

The sandbox manifest provisions the `computer` namespace and the `normal-user` service account used for sandbox pods. The `kubectl-ai-gke.yaml` manifest will create the `kubectl-ai` namespace automatically.

## 4. Configure the deployment manifest

Copy `k8s/kubectl-ai-gke.yaml` to a working file and edit the following sections:

1. **Container image** – replace the `REPLACE_WITH_YOUR_IMAGE` 
2. **Gemini API key** – change `REPLACE_WITH_YOUR_GEMINI_API_KEY` to the key you obtained from Google AI Studio

Review the RBAC objects in the manifest and adjust them if your security posture requires tighter permissions.

## 5. Deploy kubectl-ai

Apply the updated manifest to your cluster:

```bash
kubectl apply -f kubectl-ai-gke.yaml
```

Kubernetes creates the Deployment, ServiceAccount, RBAC bindings, and Service for the hosted agent. You can watch the rollout with:

```bash
kubectl get pods -n kubectl-ai
kubectl describe pod -n kubectl-ai -l app=kubectl-ai | grep -i image
```

## 6. Access the hosted web UI

Port-forward the Service locally to interact with the hosted UI:

```bash
kubectl port-forward svc/kubectl-ai -n kubectl-ai 8080:80
```

Then open [http://localhost:8080](http://localhost:8080) in your browser. Each browser session can create, rename, and delete conversations, and messages stream in real time via Server-Sent Events.

If you prefer to expose the UI via an external Load Balancer, replace the Service type in the manifest with `LoadBalancer` and configure the appropriate firewall rules.

## 7. Verify sandboxed tool execution

When the UI creates a conversation, the agent launches a sandbox pod in the `computer` namespace. You can confirm sandbox activity with:

```bash
kubectl get pods -n computer
```

Pods named `kubectl-ai-sandbox-*` indicate that commands are running inside isolated helper containers. Asking the agent to execute commands such as `uname -a` should produce output that matches the sandbox image (for example `bitnami/kubectl`).

## 8. Cleanup

Remove the deployment and sandbox resources when you are done:

```bash
kubectl delete -f kubectl-ai-gke.yaml
kubectl delete namespace kubectl-ai
kubectl delete -f k8s/sandbox/all-in-one.yaml
```

If you no longer need the Artifact Registry repository or pushed image, delete them using `gcloud artifacts repositories delete` and `gcloud artifacts docker images delete`.

---

### Mcp Client (`docs/mcp-client.md`)

# kubectl-ai MCP Client Integration

## Multi-Server Orchestration for Security Automation

The MCP (Model Context Protocol) Client feature enables kubectl-ai to coordinate multiple specialized tools through natural language commands. This integration demonstrates automated security workflows that combine RBAC scanning with email reporting.

**Problem**: Traditional security audits require manual execution of multiple tools, data correlation, and report distribution—a time-consuming process prone to human error.

**Solution**: Single command orchestration across multiple MCP servers:

```bash
kubectl-ai --mcp-client --quiet "scan rbac and send urgent report to incident-team@company.com from sender@company.com"
```

**Architecture Components:**

- **kubectl-ai**: Central orchestrator interpreting natural language commands
- **Permiflow**: RBAC security scanning and analysis
- **Resend**: Automated email delivery service
- **Additional servers**: Documentation, reasoning, and extensible integrations

## Workflow Sequence Diagram

```mermaid
sequenceDiagram
    participant User
    participant kubectl-ai as kubectl-ai<br/>(MCP Client)
    participant Permiflow as Permiflow<br/>(MCP Server)
    participant K8s as Kubernetes<br/>Cluster
    participant Resend as Resend<br/>(MCP Server)
    participant Email as Email<br/>Recipient

    User->>kubectl-ai: "scan rbac and send report to admin@company.com"
    kubectl-ai->>Permiflow: scan_rbac()
    Permiflow->>K8s: Query RBAC policies
    K8s-->>Permiflow: Return roles, bindings, permissions
    Permiflow->>Permiflow: Analyze security risks
    Permiflow-->>kubectl-ai: Security findings report
    kubectl-ai->>kubectl-ai: Format report for email
    kubectl-ai->>Resend: send_email(to, from, subject, content)
    Resend->>Email: Deliver formatted security report
    Email-->>User: Email confirmation
    kubectl-ai-->>User: "✅ RBAC scan completed and report sent"
```

## Execution Flow

The command execution follows this sequence:

1. **kubectl-ai** parses the natural language request
2. [**Permiflow**](https://github.com/tutran-se/permiflow) performs comprehensive RBAC analysis across cluster resources
3. [**Resend**](https://github.com/resend/mcp-send-email) formats and delivers the security report via email

**Extensibility**: The architecture supports additional MCP servers for Slack notifications, Jira ticket creation, compliance databases, and custom integrations.

## Configuration and Setup

### MCP Server Configuration

Configure the MCP servers in `~/.config/kubectl-ai/mcp.yaml`:

```yaml
servers:
  - name: resend
    command: node
    args:
      - "~/mcp-send-email/build/index.js"
    env:
      RESEND_API_KEY: "api-key-here"
  - name: permiflow
    url: http://localhost:8080/mcp
```

### Quick Start

```bash
# 1. Start the Permiflow MCP server
permiflow mcp --transport http --http-port 8080

# 2. Execute kubectl-ai with MCP client enabled
kubectl-ai --mcp-client --quiet "scan rbac and send report to admin@company.com from sec@company.com"
```

## Automation Use Cases

### Scheduled Security Monitoring

Implement automated daily security scans using cron:

```bash
# Daily RBAC audit at 9 AM
0 9 * * * kubectl-ai --mcp-client --quiet "scan rbac and send daily report to admin@company.com from sec@company.com"
```

### Incident Response

Execute immediate security assessments during incidents:

```bash
kubectl-ai --mcp-client --quiet "scan rbac for production namespace and send urgent report to incident-team@company.com from sec@company.com"
```

## Usage Examples

### Interactive Mode

Launch kubectl-ai in interactive mode for exploratory analysis:

```bash
kubectl-ai --mcp-client
>>> "scan rbac and send report to admin@company.com"
>>> "analyze RBAC for kubeflow namespace"
>>> "show me the most dangerous permissions in production"
>>> "which service accounts can access secrets across namespaces?"
```

### Direct Commands

Execute specific security queries directly:

```bash
kubectl-ai --mcp-client "show wildcard permissions and suggest fixes"
```

## Extended Integration

### Additional MCP Servers

Expand the automation capabilities by adding specialized servers:

```yaml
servers:
  - name: slack-notifier
    url: "https://slack-mcp.company.com/mcp"
  - name: jira-tickets
    url: "https://jira-mcp.company.com/mcp"
  - name: trivy-scanner
    command: npx
    args: ["-y", "@aquasecurity/trivy-mcp"]
```

### Advanced Workflows

**Multi-Channel Incident Response:**

```bash
"scan rbac, create jira ticket, email security team, post to slack"
```

**Compliance Automation:**

```bash
"scan vulnerabilities, update compliance database, email leadership"
```

## Benefits

- **Unified Interface**: Single natural language interface for multiple tools
- **Automation**: Reduces manual security audit processes
- **Consistency**: Standardized security scanning and reporting
- **Extensibility**: Modular architecture supports additional integrations
- **Efficiency**: Rapid security assessment and stakeholder notification

---

### Mcp Server (`docs/mcp-server.md`)

# kubectl-ai MCP Server

kubectl-ai can run as an MCP (Model Context Protocol) server, exposing kubectl-ai tools to other MCP clients. The server can run in two modes:

1. **Built-in tools only**: Exposes only kubectl-ai's native tools
2. **External tool discovery**: Additionally discovers and exposes tools from other MCP servers

## Quick Start

### Basic MCP Server (Built-in tools only)

Start the MCP server with only kubectl-ai's built-in tools:

```bash
kubectl-ai --mcp-server
```

### Enhanced MCP Server (With external tool discovery)

Start the MCP server with external MCP tool discovery enabled:

```bash
kubectl-ai --mcp-server --external-tools
```

### Expose an HTTP Endpoint for MCP Clients

Run the server with the streamable HTTP transport to serve compatible MCP clients (including kubectl-ai MCP client mode) over HTTP:

```bash
kubectl-ai --mcp-server --mcp-server-mode streamable-http --http-port 9080
```

This listens on `http://localhost:9080/mcp` by default.

> **Warning:** Without authentication, anyone who can reach this port can execute
> arbitrary `kubectl` and `bash` commands through the MCP protocol. When exposing
> the `streamable-http` endpoint beyond localhost, enable authentication (see
> below).

### Securing the HTTP Endpoint with OAuth 2.1 (AuthGate)

In `streamable-http` mode the MCP server can act as an OAuth 2.1 **Resource
Server**, requiring callers to present a valid `Authorization: Bearer <JWT>`
access token. Tokens are verified locally against the Authorization Server's
JWKS (signature + `iss` + `aud` + `exp`, and `type == access`). This integrates
with [go-authgate/authgate](https://github.com/go-authgate/authgate) as the
Authorization Server, but works with any OAuth 2.1 / OIDC server that publishes
a JWKS.

Authentication is **opt-in**: if `--mcp-auth-issuer` is not set, the endpoint
behaves exactly as before (no authentication, no metadata endpoint).

```bash
kubectl-ai --mcp-server --mcp-server-mode streamable-http --http-port 9080 \
  --mcp-auth-issuer https://authgate.corp \
  --mcp-auth-audience https://kubectl-ai.corp/mcp
```

What this enables:

- `POST /mcp` requires a valid Bearer access token. Missing or invalid tokens
  receive `401 Unauthorized` with a `WWW-Authenticate` header pointing at the
  Protected Resource Metadata document.
- `GET /.well-known/oauth-protected-resource` serves the Protected Resource
  Metadata (RFC 9728) — unauthenticated — so MCP clients can discover the
  Authorization Server automatically:

  ```json
  {
    "resource": "https://kubectl-ai.corp/mcp",
    "authorization_servers": ["https://authgate.corp"],
    "bearer_methods_supported": ["header"]
  }
  ```

Authentication flags:

| Flag                  | Description                                                                                                                               |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `--mcp-auth-issuer`   | Authorization Server base URL (= JWT `iss`). **Non-empty enables authentication.** Empty keeps the previous, unauthenticated behavior.    |
| `--mcp-auth-audience` | This server's resource identifier (expected JWT `aud`, e.g. `https://kubectl-ai.corp/mcp`). **Required** when `--mcp-auth-issuer` is set. |
| `--mcp-auth-jwks-url` | Optional override for the JWKS URL. Empty uses OIDC discovery from `<issuer>/.well-known/openid-configuration`.                           |

#### Why the audience (`aud`) matters

The `--mcp-auth-audience` value is this server's **resource identifier**. During
verification kubectl-ai requires the JWT's `aud` claim to match it exactly (in
addition to checking the signature, `iss`, and `exp`). This is what binds a token
to *this specific* MCP server, and it is why the flag is mandatory once
authentication is enabled — there is no safe default to guess.

The signature and `iss` checks only prove the token is *genuine* (really issued
by your Authorization Server). The `aud` check proves the token was *meant for
you*. Without it, **any** valid token signed by the same Authorization Server
would be accepted, which opens up:

- **Cross-service token reuse / confused deputy:** one Authorization Server
  typically issues tokens for many resource servers (other internal APIs,
  dashboards, microservices). A user holding a legitimate token for, say, a
  reporting API could replay it against the MCP server and gain the ability to
  run `kubectl` and `bash`. With `aud` enforced, that token's audience does not
  match this server's resource identifier, so it is rejected.
- **Token theft by a downstream service:** if a token is not bound to an
  audience, any service that receives it (or a compromised/man-in-the-middle
  hop) can turn around and impersonate the user against the MCP server. Binding
  the audience makes a token issued for someone else useless here.

The audience also closes the discovery loop: the value is published verbatim as
the `resource` field of the Protected Resource Metadata document (see below), so
compliant MCP clients request a token scoped to exactly this resource (per RFC
8707 Resource Indicators) and the server then enforces that same scope.

> **Pick a stable, unique value.** Use a URI that uniquely identifies this MCP
> server (commonly its public `/mcp` URL, e.g. `https://kubectl-ai.corp/mcp`).
> It must be the same string the Authorization Server stamps into the `aud`
> claim — a mismatch results in every request being rejected with `401`.

Behavior notes:

- **Fail-fast on configuration errors:** an invalid issuer URL or a missing
  audience aborts startup with a clear error.
- **Tolerant of a temporarily unreachable Authorization Server:** if the JWKS
  cannot be fetched at startup, kubectl-ai logs a warning and keeps retrying in
  the background instead of crashing.
- **Fail-closed while keys are unavailable:** until signing keys are loaded,
  `/mcp` returns `503 Service Unavailable` rather than letting any request
  through.
- **Reverse proxies:** the `resource_metadata` URL in the `WWW-Authenticate`
  header is built from `X-Forwarded-Proto` (falling back to TLS detection) and
  the request `Host`. Make sure your proxy forwards `X-Forwarded-Proto`.

> **Tip:** [go-authgate/authgate](https://github.com/go-authgate/authgate) can
> be used as the Authorization Server that issues and signs these access tokens.
> See its documentation for installation and client setup.

## Configuration

When `--external-tools` is enabled, the enhanced MCP server will automatically discover and expose tools from configured MCP servers. You can configure MCP servers using the standard MCP client configuration file.

### Example MCP Configuration

Create `~/.config/kubectl-ai/mcp.yaml`:

```yaml
servers:
  filesystem:
    command: "npx"
    args:
      [
        "-y",
        "@modelcontextprotocol/server-filesystem",
        "/path/to/allowed/files",
      ]

  brave-search:
    command: "npx"
    args: ["-y", "@modelcontextprotocol/server-brave-search"]
    env:
      BRAVE_API_KEY: "your-api-key"
```

## Features

### Tool Aggregation

When external tool discovery is enabled with `--external-tools`, the kubectl-ai MCP server acts as a **tool aggregator**, providing:

- All kubectl-ai built-in tools (kubectl, cluster analysis, etc.)
- Tools from external MCP servers (filesystem, web search, etc.)
- Unified interface for all tools through a single MCP endpoint

### Graceful Degradation

The server handles external MCP connection failures gracefully:

- If external MCP servers are unavailable, the server continues with built-in tools only
- Individual tool failures don't affect the overall server operation
- Clear logging for troubleshooting connection issues

### Example Usage in Claude Desktop

Configure Claude Desktop to use kubectl-ai as an MCP server:

**Basic usage (built-in tools only):**

```json
{
  "mcpServers": {
    "kubectl-ai": {
      "command": "kubectl-ai",
      "args": ["--mcp-server"]
    }
  }
}
```

**Enhanced usage (with external tools):**

```json
{
  "mcpServers": {
    "kubectl-ai": {
      "command": "kubectl-ai",
      "args": ["--mcp-server", "--external-tools"]
    }
  }
}
```

## Available Tools

### Built-in Tools

kubectl-ai provides the following native tools:

- `bash`: Executes a bash command. Use this tool only when you need to execute a shell command.
- `kubectl`: Executes a kubectl command against the user's Kubernetes cluster. Use this tool only when you need to query or modify the state of the user's Kubernetes cluster.

### External Tools (when `--external-tools` is enabled)

Additional tools are available depending on the configured MCP servers:

- **Filesystem tools**: Read/write files, list directories
- **Web search tools**: Search the internet for information
- **Database tools**: Query databases
- **API tools**: Interact with external APIs
- **Custom tools**: Any MCP-compatible tools

## Command Line Options

| Flag                  | Default          | Description                                                                                                           |
| --------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------- |
| `--mcp-server`        | `false`          | Run in MCP server mode                                                                                                |
| `--external-tools`    | `false`          | Discover and expose external MCP tools (requires --mcp-server)                                                        |
| `--kubeconfig`        | `~/.kube/config` | Path to kubeconfig file                                                                                               |
| `--mcp-server-mode`   | `stdio`          | Transport for the MCP server (`stdio` or `streamable-http`)                                                           |
| `--http-port`         | `9080`           | Port for the HTTP endpoint when using `streamable-http` modes                                                         |
| `--mcp-auth-issuer`   | `""`             | OAuth 2.1 authorization server base URL (JWT issuer). Non-empty enables Bearer auth on the `streamable-http` endpoint |
| `--mcp-auth-audience` | `""`             | Expected JWT audience (this server's resource identifier). Required when `--mcp-auth-issuer` is set                   |
| `--mcp-auth-jwks-url` | `""`             | Optional override for the JWKS URL; empty uses OIDC discovery from the issuer                                         |

## Architecture

```txt
┌─────────────────┐    ┌───────────────────┐    ┌─────────────────┐
│   MCP Client    │───▶│ kubectl-ai Server │───▶│ External Tools  │
│  (Claude, etc.) │    │                   │    │ (filesystem,    │
│                 │    │ ┌───────────────┐ │    │  web search,    │
│                 │    │ │ Built-in      │ │    │  etc.)          │
│                 │    │ │ kubectl tools │ │    │                 │
│                 │    │ └───────────────┘ │    │                 │
└─────────────────┘    └───────────────────┘    └─────────────────┘
```

The kubectl-ai MCP server acts as both:

- An **MCP Server** (exposing tools to clients)
- An **MCP Client** (consuming tools from other servers, when `--external-tools` is enabled)

This creates a powerful tool aggregation pattern where kubectl-ai becomes a central hub for both Kubernetes operations and general-purpose tools.

## Troubleshooting

### External Tools Not Available

If external tools aren't appearing:

1. Ensure you're using both `--mcp-server` and `--external-tools` flags
2. Check MCP configuration file exists and is valid
3. Verify external MCP servers are working independently
4. Check kubectl-ai logs for connection errors
5. Try running with external tools disabled to isolate issues

### Performance Considerations

- Tool discovery adds startup time (usually 2-3 seconds) when `--external-tools` is enabled
- Each external tool call has network overhead
- Consider running without `--external-tools` for faster startup if external tools aren't needed

### Debugging

Enable verbose logging to troubleshoot:

```bash
kubectl-ai --mcp-server --external-tools -v=2
```

This will show:

- MCP server connection attempts
- Tool discovery results
- Tool call routing decisions

---

### Mocking (`docs/mocking.md`)

# Mocking in kubectl-ai

## Gomock developer workflow

We use [gomock](https://github.com/uber-go/mock) to mock external dependencies. All mocks and generated files live under `internal/mocks/`.

- **Everyday commands**
  - Regenerate mocks after changing interfaces or adding new ones: `make generate`
  - Verify nothing is stale (locally or in CI): `make verify-mocks`
  - Run tests: `go test ./...`
- **Generator install** (if you don’t have it yet):  
  `go install go.uber.org/mock/mockgen@latest`
- **What `make generate` does**  
  Runs `go generate ./internal/mocks`. Note: `go generate` is **not** part of `go build/test`; commit generated mocks.
- **Add a new mock**
  1. Add a `go:generate` line in `internal/mocks/generate.go`, e.g.:  
     ```go
     //go:generate mockgen -destination=tools_mock.go -package=mocks      //  github.com/GoogleCloudPlatform/kubectl-ai/pkg/tools Tool
     ```
  2. Run `make generate` and import the mocks in tests:
     ```go
     ctrl := gomock.NewController(t)
     defer ctrl.Finish()

     llm := mocks.NewMockClient(ctrl) // example
     llm.EXPECT().NewChat(gomock.Any()).AnyTimes()
     ```

## When and when not to use gomock

**Use gomock for:**
- **External boundaries / side effects**: `gollm.Client`, `gollm.Chat`, `pkg/tools.Tool`, network/IO, anything slow or flaky.
- **Behavioral checks**: asserting specific calls/arguments or injecting failures/timeouts.

**Prefer fakes/in‑memory over mocks for:**
- **Stateful components with an in‑memory impl** (e.g., session/message store). Don’t mock storage if an in‑memory version exists.
- **Pure functions / simple value types**—call them directly.

**Good practices:**
- Keep expectations minimal—assert only what matters. Use `gomock.Any()` and `AnyTimes()`/`MinTimes(1)` where exact call counts don’t matter.
- Centralize `mockgen` directives in `internal/mocks/generate.go`.
- **If an interface changes**: run `make generate`, fix compile errors in tests (signatures/matchers), update/remove `go:generate` lines if package paths or names changed, and commit the regenerated mocks.

---

### Tools (`docs/tools.md`)

# Custom Tools for kubectl-ai

`kubectl-ai` leverages LLMs to suggest and execute Kubernetes operations using a set of powerful tools. It comes with built-in tools like `kubectl` and `bash`.

The `kubectl-ai` assistant can be extended with custom tools to interact with various command-line interfaces (CLIs) beyond `kubectl`. This allows the AI to perform a wider range of tasks related to infrastructure management, CI/CD, and more.

This document outlines how you can add custom tools by detailing the steps and providing samples.

This document also outlines the available tools, their locations, and how to use them.

## Adding Custom Tools

Custom tools can be added by following these two steps:

- describing or templating the tool through YAML file
- enabling the tool in the configuration file by pointing the **--custom-tools-config** to this file / directory

## Describing the Tool in YAML file

A custom tool can be described by providing the following four pieces of information:

- **name**: name of the tool
- **description**: "A clear description that helps the LLM understand when to use this tool."
- **command** : "your_command" # For example: 'gcloud' or 'gcloud container clusters'
- **command_desc**: "Detailed information for the LLM, including command syntax and usage examples."

Samples are provided in the `pkg/tools/samples` directory. Below is a sample for the `kustomize` tool:

```yaml
- name: kustomize
  description: "A tool to customize Kubernetes resource configurations. Use it to render and apply declarative configurations from a directory containing a kustomization.yaml file."
  command: "kustomize"
  command_desc: |
    The kustomize command-line interface.
    
    Core subcommands and usage patterns:
    - `kustomize build <kustomization_dir>`: Prints the customized resources to standard output. This is useful for inspecting the final configuration before applying it.
    - `kustomize build <kustomization_dir> | kubectl apply -f -`: A common pattern to apply the output directly to the cluster.
    
    Note: `kubectl apply -k <dir>` is a shorthand for the pipe command above and is often preferred.
```

## Enabling the Custom Tool

To enable the custom tools, you must point `kubectl-ai` to the directory containing the tool configuration YAML files using the `--custom-tools-config` flag. `kubectl-ai` can pick up a single YAML file (e.g., `tools.yaml`) containing all the tool descriptions or multiple individual YAML files when pointed to a directory containing them. This example uses multiple YAML files located in a single directory.

In case, you don't want to use a tool (that is provided in samples), just move the file out of the directory into some other location & restart `kubectl-ai`.

### Running from a Local Binary

When running the `kubectl-ai` binary directly, provide the path to your local tools directory.

```sh
./kubectl-ai --custom-tools-config=<path-to-tools-directory> "your prompt here"
```

### Running with Docker Image

When using the Docker image, you can either use the tools baked into the image or mount your own custom directory.

#### Using Built-in Tools

The official Docker image includes the default tool configurations. You can enable them by pointing to the internal path.

```sh
docker run --rm -it your-kubectl-ai-image:latest \
  --custom-tools-config=/etc/kubectl-ai/tools \
  "list all pull requests on GitHub"
```

#### Using a Local Tools Directory

To use a custom set of tools from your local machine, mount the directory into the container and point the flag to the mounted path. This is useful for developing and testing new tools.

```sh
docker run --rm -it \
  -v /path/to/your/local/tools:/my-custom-tools \
  your-kubectl-ai-image:latest \
  --custom-tools-config=/my-custom-tools \
  "your prompt here"
```

## Sample Custom Tools

The following sample custom tools are configured by default.

| Tool                                                       | Description                                                     | YAML File                                           |
| :--------------------------------------------------------- | :-------------------------------------------------------------- | :------------------------------------------------------ |
| Argo CD (`argocd`)      | A declarative, GitOps continuous delivery tool for Kubernetes.  | [argocd.yaml](./tool-samples/argocd.yaml)         |
| GitHub CLI (`gh`)               | The official command-line tool to interact with GitHub.         | [gh.yaml](./tool-samples/gh.yaml)                 |
| Google Cloud CLI (`gcloud`) | The primary CLI for managing Google Cloud resources.            | [gcloud.yaml](./tool-samples/gcloud.yaml)               |
| Kustomize (`kustomize`)           | A tool to customize Kubernetes resource configurations.         | [kustomize.yaml](./tool-samples/kustomize.yaml)   |

---

### Tool Samples Argocd.Yaml (`docs/tool-samples/argocd.yaml`)

- name: argocd
  description: "A declarative, GitOps continuous delivery tool for Kubernetes. Use it to manage application deployments from Git repositories."
  command: "argocd"
  command_desc: |
    The argocd command-line interface.

    Core subcommands and usage patterns:
    - `argocd login <server> [flags]`: Log in to an Argo CD server. This is required before most other commands.
    - `argocd app list [flags]`: List all applications managed by Argo CD.
    - `argocd app get <app-name> [flags]`: Get detailed information about a specific application.
    - `argocd app sync <app-name> [flags]`: Sync an application to its target state defined in the Git repository.
    - `argocd app history <app-name> [flags]`: View the deployment history of an application.
    - `argocd app rollback <app-name> <history-id>`: Roll back an application to a previous deployed state.
    - `argocd app create <app-name> [flags]`: Create a new application.
    - `argocd app delete <app-name>`: Delete an application.

    Use `argocd --help` or `argocd <command> --help` for full syntax and available flags.

---

### Tool Samples Gcloud.Yaml (`docs/tool-samples/gcloud.yaml`)

- name: gcloud
  description: "The gcloud command-line tool is the primary CLI for Google Cloud. Use it to manage Google Cloud resources, including Google Kubernetes Engine (GKE) clusters, virtual machines, and networking."
  command: "gcloud"
  command_desc: |
    The gcloud CLI manages authentication, local configuration, developer workflow, and interactions with Google Cloud APIs.

    For Kubernetes-related tasks, the `gcloud container clusters` command group is the most relevant.

    Core subcommands and usage patterns:
    - `gcloud container clusters list [flags]`: List all GKE clusters in the configured project and zone/region.
    - `gcloud container clusters describe <cluster-name> [flags]`: Get detailed information about a specific GKE cluster.
    - `gcloud container clusters get-credentials <cluster-name> [flags]`: Fetch cluster endpoint and auth data and configure kubectl to use the cluster. This is essential for connecting to a GKE cluster.
    - `gcloud container clusters create <cluster-name> [flags]`: Create a new GKE cluster.
    - `gcloud container clusters delete <cluster-name> [flags]`: Delete an existing GKE cluster.

    You can set the default project, region, and zone using:
    - `gcloud config set project <project-id>`
    - `gcloud config set compute/region <region>`
    - `gcloud config set compute/zone <zone>`

    Use `gcloud --help` or `gcloud <command> <subcommand> --help` for full syntax and available flags.

---

### Tool Samples Gh.Yaml (`docs/tool-samples/gh.yaml`)

- name: gh
  description: "The official GitHub command-line tool. Use it to interact with GitHub repositories, pull requests, issues, actions, and more, directly from the terminal."
  command: "gh"
  command_desc: |
    The gh command-line interface for GitHub.

    Core subcommands and usage patterns:
    - `gh auth login`: Authenticate with a GitHub host. This is required before most other commands.
    - `gh repo view [repo]`: View a repository.
    - `gh pr list`: List pull requests in the current repository.
    - `gh pr view <number>`: View a specific pull request.
    - `gh pr checkout <number>`: Check out a pull request locally.
    - `gh pr create [flags]`: Create a new pull request.
    - `gh pr merge [flags]`: Merge a pull request.
    - `gh issue list`: List issues in the current repository.
    - `gh issue view <number>`: View a specific issue.
    - `gh workflow list`: List workflows in the current repository.
    - `gh workflow run <workflow-name> [flags]`: Trigger a workflow run.

    Use `gh --help` or `gh <subcommand> --help` for full syntax and available flags.

---

### Tool Samples Kustomize.Yaml (`docs/tool-samples/kustomize.yaml`)

- name: kustomize
  description: "A tool to customize Kubernetes resource configurations. Use it to render and apply declarative configurations from a directory containing a kustomization.yaml file."
  command: "kustomize"
  command_desc: |
    The kustomize command-line interface.
    
    Core subcommands and usage patterns:
    - `kustomize build <kustomization_dir>`: Prints the customized resources to standard output. This is useful for inspecting the final configuration before applying it.
    - `kustomize build <kustomization_dir> | kubectl apply -f -`: A common pattern to apply the output directly to the cluster.
    
    Note: `kubectl apply -k <dir>` is a shorthand for the pipe command above and is often preferred.

---

--- METRICS ---
- Files Extracted: 15
- Estimated Token Budget: ~18336 tokens
- Canonical Reference: https://codewiki.google/github.com/GoogleCloudPlatform/kubectl-ai