# Technical Documentation: vxcontrol/pentagi

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/vxcontrol/pentagi)

### Overview (README.md)

# PentAGI

<div align="center" style="font-size: 1.5em; margin: 20px 0;">
    <strong>P</strong>enetration testing <strong>A</strong>rtificial <strong>G</strong>eneral <strong>I</strong>ntelligence
</div>
<br>
<div align="center">

> **Join the Community!** Connect with security researchers, AI enthusiasts, and fellow ethical hackers. Get support, share insights, and stay updated with the latest PentAGI developments.

[](https://discord.gg/2xrMh7qX6m)⠀[](https://t.me/+Ka9i6CNwe71hMWQy)

<a href="https://trendshift.io/repositories/15161?utm_source=repository-badge&amp;utm_medium=badge&amp;utm_campaign=badge-repository-15161" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/15161" alt="vxcontrol%2Fpentagi | Trendshift" width="250" height="55"/></a> <a href="https://trendshift.io/repositories/15161?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-15161" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/15161/daily" alt="vxcontrol%2Fpentagi | Trendshift" width="250" height="55"/></a>

<a href="https://trendshift.io/repositories/15161?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-15161" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/15161/daily?language=Go" alt="vxcontrol%2Fpentagi | Trendshift" width="250" height="55"/></a> <a href="https://trendshift.io/repositories/15161?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-15161" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/15161/weekly?language=Go" alt="vxcontrol%2Fpentagi | Trendshift" width="250" height="55"/></a> <a href="https://trendshift.io/repositories/15161?utm_source=trendshift-badge&amp;utm_medium=badge&amp;utm_campaign=badge-trendshift-15161" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/15161/monthly?language=Go" alt="vxcontrol%2Fpentagi | Trendshift" width="250" height="55"/></a>

</div>

## Table of Contents

- [Overview](#overview)
- [Features](#features)
- [Architecture](#architecture)
  - [Agent Supervision](#advanced-agent-supervision)
- [Quick Start](#quick-start)
  - [Agent Docker Access](#giving-agents-docker-without-giving-away-the-host)
  - [Running Several Instances](#running-several-instances-tenant_id)
- [How to Use PentAGI After Login](#how-to-use-pentagi-after-login)
- [API Access](#api-access)
  - [LLM Provider Configuration](#custom-llm-provider-configuration)
    - [Azure OpenAI](#using-azure-openai)
    - [Ollama](#ollama-provider-configuration)
    - [OpenAI](#openai-provider-configuration)
    - [Anthropic](#anthropic-provider-configuration)
    - [Google AI (Gemini)](#google-ai-gemini-provider-configuration)
    - [AWS Bedrock](#aws-bedrock-provider-configuration)
    - [DeepSeek](#deepseek-provider-configuration)
    - [GLM](#glm-provider-configuration)
    - [Kimi](#kimi-provider-configuration)
    - [Qwen](#qwen-provider-configuration)
    - [MiniMax](#minimax-provider-configuration)
    - [Mistral](#mistral-provider-configuration)
    - [xAI (Grok)](#xai-grok-provider-configuration)
- [Advanced Setup](#advanced-setup)
  - [Langfuse Integration](#langfuse-integration)
  - [Monitoring and Observability](#monitoring-and-observability)
  - [Knowledge Graph (Graphiti)](#knowledge-graph-integration-graphiti)
  - [OAuth Integration](#github-and-google-oauth-integration)
  - [Docker Image Configuration](#docker-image-configuration)
- [Development](#development)
- [Testing LLM Agents](#testing-llm-agents)
- [Embedding Configuration and Testing](#embedding-configuration-and-testing)
- [Function Testing with ftester](#function-testing-with-ftester)
- [Building](#building)
- [Credits](#credits)
- [License](#license)

## Overview

PentAGI is an open-source, self-hosted multi-agent system for autonomous penetration testing. You describe a target and a goal in plain language; a team of specialized AI agents plans the work, runs 20+ professional tools such as nmap, Metasploit and sqlmap inside an isolated Docker sandbox, remembers what it learned and hands you a report.

- **Who it is for.** Security engineers, penetration testers, researchers and red teams who want an autonomous assistant on their own infrastructure, for authorized testing only.
- **What sets it apart.** It runs on your own servers with the model of your choice: 10+ cloud LLM providers, your own OpenAI-compatible gateway, or local models through Ollama or vLLM. Results and successful approaches are kept in PostgreSQL with pgvector for later flows, an optional Neo4j knowledge graph tracks hosts, services and findings, execution monitoring and task planning help smaller open models, and REST and GraphQL APIs automate it. PentAGI Enterprise adds agent profiles, plugins and MCP servers, customer-ready reports and a real-time attack-chain view.
- **How to install.** An interactive installer for Linux, macOS and Windows sets PentAGI up with Docker Compose — see [Quick Start](#quick-start).

PentAGI is developed by [VXControl](https://vxcontrol.com), a security engineering company based in Dubai.

You can watch the video **PentAGI overview**:
[](https://youtu.be/R70x5Ddzs1o)

## Features

- Secure & Isolated. All operations are performed in a sandboxed Docker environment with complete isolation.
- Fully Autonomous. AI-powered agent that automatically determines and executes penetration testing steps with optional execution monitoring and intelligent task planning for enhanced reliability.
- Professional Pentesting Tools. Built-in suite of 20+ professional security tools including nmap, metasploit, sqlmap, and more.
- Smart Memory System. Long-term storage of research results and successful approaches for future use.
- Optional Knowledge Graph Integration. Graphiti-powered knowledge graph using Neo4j for semantic relationship tracking and advanced context understanding.
- Web Intelligence. Built-in browser via [scraper](https://hub.docker.com/r/vxcontrol/scraper) for gathering latest information from web sources.
- External Search Systems. Integration with advanced search APIs including [Tavily](https://tavily.com), [Firecrawl](https://www.firecrawl.dev), [Traversaal](https://traversaal.ai), [Perplexity](https://www.perplexity.ai), [DuckDuckGo](https://duckduckgo.com/), [Google Custom Search](https://programmablesearchengine.google.com/), [Sploitus Search](https://sploitus.com) and [SearXNG](https://searxng.org) for comprehensive information gathering.
- Team of Specialists. Delegation system with specialized AI agents for research, development, and infrastructure tasks, enhanced with optional execution monitoring and intelligent task planning for optimal performance with smaller models.
- Comprehensive Monitoring. Detailed logging and integration with Grafana/Prometheus for real-time system observation.
- Detailed Reporting. Generation of thorough vulnerability reports with exploitation guides.
- Smart Container Management. Automatic Docker image selection based on specific task requirements.
- Modern Interface. Clean and intuitive web UI for system management and monitoring.
- Comprehensive APIs. Full-featured REST and GraphQL APIs with Bearer token authentication for automation and integration.
- Persistent Storage. All commands and outputs are stored in PostgreSQL with [pgvector](https://hub.docker.com/r/vxcontrol/pgvector) extension.
- Scalable Architecture. Microservices-based design supporting horizontal scaling.
- Self-Hosted Solution. Complete control over your deployment and data.
- Flexible Authentication. Support for 10+ LLM providers ([OpenAI](https://platform.openai.com/), [Anthropic](https://www.anthropic.com/), [Google AI/Gemini](https://ai.google.dev/), [AWS Bedrock](https://aws.amazon.com/bedrock/), [Ollama](https://ollama.com/), [DeepSeek](https://www.deepseek.com/en/), [GLM](https://z.ai/), [Kimi](https://platform.moonshot.ai/), [Qwen](https://www.alibabacloud.com/en/), [MiniMax](https://www.minimax.io/), [Mistral](https://mistral.ai/), [xAI](https://x.ai/), Custom for any OpenAI-compatible endpoint including [Azure OpenAI](#using-azure-openai)) plus aggregators ([OpenRouter](https://openrouter.ai/), [DeepInfra](https://deepinfra.com/), [Atlas Cloud](https://www.atlascloud.ai/), [OpenCode Go plan](https://opencode.ai/en/go)). For production local deployments, see our [vLLM + Qwen3.5-27B-FP8 guide](examples/guides/vllm-qwen35-27b-fp8.md).
- API Token Authentication. Secure Bearer token system for programmatic access to REST and GraphQL APIs.
- Quick Deployment. Easy setup through [Docker Compose](https://docs.docker.com/compose/) with comprehensive environment configuration.

### Current Capability Boundaries

- PentAGI today is an autonomous and assistant-guided penetration testing platform, not a CALDERA-style Breach and Attack Simulation (BAS) or adversary emulation product with predefined campaigns or attack plans.
- BAS-like agent-authored attack scripts should be treated as conceptual or future work, not as a feature that is implemented today.
- The current flow report UI supports web view, copy to clipboard, Markdown download, and PDF download. JSON flow-report export is not documented as a supported output format today.
- Provider flexibility is available today through built-in providers and custom/OpenAI-compatible endpoints. See [Custom LLM Provider Configuration](#custom-llm-provider-configuration) and the [vLLM + Qwen3.5-27B-FP8 guide](examples/guides/vllm-qwen35-27b-fp8.md).

## Architecture

### System Context

```mermaid
flowchart TB
    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef external fill:#666666,stroke:#0B4884,color:#fff

    pentester["👤 Security Engineer
    (User of the system)"]

    pentagi["✨ PentAGI
    (Autonomous penetration testing system)"]

    target["🎯 target-system
    (System under test)"]
    llm["🧠 llm-provider
    (OpenAI/Anthropic/Ollama/Bedrock/Gemini/Custom)"]
    search["🔍 search-systems
    (Google/DuckDuckGo/Tavily/Firecrawl/Traversaal/Perplexity/Sploitus/SearXNG)"]
    langfuse["📊 langfuse-ui
    (LLM Observability Dashboard)"]
    grafana["📈 grafana
    (System Monitoring Dashboard)"]

    pentester --> |Uses HTTPS| pentagi
    pentester --> |Monitors AI HTTPS| langfuse
    pentester --> |Monitors System HTTPS| grafana
    pentagi --> |Tests Various protocols| target
    pentagi --> |Queries HTTPS| llm
    pentagi --> |Searches HTTPS| search
    pentagi --> |Reports HTTPS| langfuse
    pentagi --> |Reports HTTPS| grafana

    class pentester person
    class pentagi system
    class target,llm,search,langfuse,grafana external

    linkStyle default stroke:#ffffff,color:#ffffff
```

<details>
<summary><b>Container Architecture</b> (click to expand)</summary>

```
/* Detailed source-code truncated for AI context efficiency. */
```

</details>

<details>
<summary><b>Entity Relationship</b> (click to expand)</summary>

```
/* Detailed source-code truncated for AI context efficiency. */
```

</details>

<details>
<summary><b>Agent Interaction</b> (click to expand)</summary>

```mermaid
sequenceDiagram
    participant O as Orchestrator
    participant R as Researcher
    participant D as Developer
    participant E as Executor
    participant VS as Vector Store
    participant KB as Knowledge Base

    Note over O,KB: Flow Initialization
    O->>VS: Query similar tasks
    VS-->>O: Return experiences
    O->>KB: Load relevant knowledge
    KB-->>O: Return context

    Note over O,R: Research Phase
    O->>R: Analyze target
    R->>VS: Search similar cases
    VS-->>R: Return patterns
    R->>KB: Query vulnerabilities
    KB-->>R: Return known issues
    R->>VS: Store findings
    R-->>O: Research results

    Note over O,D: Planning Phase
    O->>D: Plan attack
    D->>VS: Query exploits
    VS-->>D: Return techniques
    D->>KB: Load tools info
    KB-->>D: Return capabilities
    D-->>O: Attack plan

    Note over O,E: Execution Phase
    O->>E: Execute plan
    E->>KB: Load tool guides
    KB-->>E: Return procedures
    E->>VS: Store results
    E-->>O: Execution status
```

</details>

<details>
<summary><b>Memory System</b> (click to expand)</summary>

```mermaid
graph TB
    subgraph "Long-term Memory"
        VS[(Vector Store<br/>Embeddings DB)]
        KB[Knowledge Base<br/>Domain Expertise]
        Tools[Tools Knowledge<br/>Usage Patterns]
    end

    subgraph "Working Memory"
        Context[Current Context<br/>Task State]
        Goals[Active Goals<br/>Objectives]
        State[System State<br/>Resources]
    end

    subgraph "Episodic Memory"
        Actions[Past Actions<br/>Commands History]
        Results[Action Results<br/>Outcomes]
        Patterns[Success Patterns<br/>Best Practices]
    end

    Context --> |Query| VS
    VS --> |Retrieve| Context

    Goals --> |Consult| KB
    KB --> |Guide| Goals

    State --> |Record| Actions
    Actions --> |Learn| Patterns
    Patterns --> |Store| VS

    Tools --> |Inform| State
    Results --> |Update| Tools

    VS --> |Enhance| KB
    KB --> |Index| VS

    classDef ltm fill:#f9f,stroke:#333,stroke-width:2px,color:#000
    classDef wm fill:#bbf,stroke:#333,stroke-width:2px,color:#000
    classDef em fill:#bfb,stroke:#333,stroke-width:2px,color:#000

    class VS,KB,Tools ltm
    class Context,Goals,State wm
    class Actions,Results,Patterns em
```

</details>

<details>
<summary><b>Chain Summarization</b> (click to expand)</summary>

The chain summarization system manages conversation context growth by selectively summarizing older messages. This is critical for preventing token limits from being exceeded while maintaining conversation coherence.

```mermaid
flowchart TD
    A[Input Chain] --> B{Needs Summarization?}
    B -->|No| C[Return Original Chain]
    B -->|Yes| D[Convert to ChainAST]
    D --> E[Apply Section Summarization]
    E --> F[Process Oversized Pairs]
    F --> G[Manage Last Section Size]
    G --> H[Apply QA Summarization]
    H --> I[Rebuild Chain with Summaries]
    I --> J{Is New Chain Smaller?}
    J -->|Yes| K[Return Optimized Chain]
    J -->|No| C

    classDef process fill:#bbf,stroke:#333,stroke-width:2px,color:#000
    classDef decision fill:#bfb,stroke:#333,stroke-width:2px,color:#000
    classDef output fill:#fbb,stroke:#333,stroke-width:2px,color:#000

    class A,D,E,F,G,H,I process
    class B,J decision
    class C,K output
```

The algorithm operates on a structured representation of conversation chains (ChainAST) that preserves message types including tool calls and their responses. All summarization operations maintain critical conversation flow while reducing context size.

### Global Summarizer Configuration Options

| Parameter             | Environment Variable             | Default | Description                                                |
| --------------------- | -------------------------------- | ------- | ---------------------------------------------------------- |
| Preserve Last         | `SUMMARIZER_PRESERVE_LAST`       | `true`  | Whether to keep all messages in the last section intact    |
| Use QA Pairs          | `SUMMARIZER_USE_QA`              | `true`  | Whether to use QA pair summarization strategy              |
| Summarize Human in QA | `SUMMARIZER_SUM_MSG_HUMAN_IN_QA` | `false` | Whether to summarize human messages in QA pairs            |
| Last Section Size     | `SUMMARIZER_LAST_SEC_BYTES`      | `51200` | Maximum byte size for last section (50KB)                  |
| Max Body Pair Size    | `SUMMARIZER_MAX_BP_BYTES`        | `16384` | Maximum byte size for a single body pair (16KB)            |
| Max QA Sections       | `SUMMARIZER_MAX_QA_SECTIONS`     | `10`    | Maximum QA pair sections to preserve                       |
| Max QA Size           | `SUMMARIZER_MAX_QA_BYTES`        | `65536` | Maximum byte size for QA pair sections (64KB)              |
| Keep QA Sections      | `SUMMARIZER_KEEP_QA_SECTIONS`    | `1`     | Number of recent QA sections to keep without summarization |

### Assistant Summarizer Configuration Options

Assistant instances can use customized summarization settings to fine-tune context management behavior:

| Parameter          | Environment Variable                    | Default | Description                                                          |
| ------------------ | --------------------------------------- | ------- | -------------------------------------------------------------------- |
| Preserve Last      | `ASSISTANT_SUMMARIZER_PRESERVE_LAST`    | `true`  | Whether to preserve all messages in the assistant's last section     |
| Last Section Size  | `ASSISTANT_SUMMARIZER_LAST_SEC_BYTES`   | `76800` | Maximum byte size for assistant's last section (75KB)                |
| Max Body Pair Size | `ASSISTANT_SUMMARIZER_MAX_BP_BYTES`     | `16384` | Maximum byte size for a single body pair in assistant context (16KB) |
| Max QA Sections    | `ASSISTANT_SUMMARIZER_MAX_QA_SECTIONS`  | `7`     | Maximum QA sections to preserve in assistant context                 |
| Max QA Size        | `ASSISTANT_SUMMARIZER_MAX_QA_BYTES`     | `76800` | Maximum byte size for assistant's QA sections (75KB)                 |
| Keep QA Sections   | `ASSISTANT_SUMMARIZER_KEEP_QA_SECTIONS` | `3`     | Number of recent QA sections to preserve without summarization       |

The assistant summarizer configuration provides more memory for context retention compared to the global settings, preserving more recent conversation history while still ensuring efficient token usage.

### Summarizer Environment Configuration

```bash
# Default values for global summarizer logic
SUMMARIZER_PRESERVE_LAST=true
SUMMARIZER_USE_QA=true
SUMMARIZER_SUM_MSG_HUMAN_IN_QA=false
SUMMARIZER_LAST_SEC_BYTES=51200
SUMMARIZER_MAX_BP_BYTES=16384
SUMMARIZER_MAX_QA_SECTIONS=10
SUMMARIZER_MAX_QA_BYTES=65536
SUMMARIZER_KEEP_QA_SECTIONS=1

# Default values for assistant summarizer logic
ASSISTANT_SUMMARIZER_PRESERVE_LAST=true
ASSISTANT_SUMMARIZER_LAST_SEC_BYTES=76800
ASSISTANT_SUMMARIZER_MAX_BP_BYTES=16384
ASSISTANT_SUMMARIZER_MAX_QA_SECTIONS=7
ASSISTANT_SUMMARIZER_MAX_QA_BYTES=76800
ASSISTANT_SUMMARIZER_KEEP_QA_SECTIONS=3
```

</details>

<a id="advanced-agent-supervision"></a>
<details>
<summary><b>Advanced Agent Supervision</b> (click to expand)</summary>

PentAGI includes sophisticated multi-layered agent supervision mechanisms to ensure efficient task execution, prevent infinite loops, and provide intelligent recovery from stuck states:

### Execution Monitoring (Beta)
- **Automatic Mentor Intervention**: Adviser agent (mentor) is automatically invoked when execution patterns indicate potential issues
- **Pattern Detection**: Monitors identical tool calls (threshold: 5, configurable) and total tool calls (threshold: 10, configurable)
- **Progress Analysis**: Evaluates whether agent advances toward subtask objective, detects loops and inefficiencies
- **Alternative Strategies**: Recommends different approaches when current strategy fails
- **Information Retrieval Guidance**: Suggests searching for established solutions instead of reinventing
- **Enhanced Response Format**: Tool responses include both `<original_result>` and `<mentor_analysis>` sections
- **Configurable**: Enable via `EXECUTION_MONITOR_ENABLED` (default: false), customize thresholds with `EXECUTION_MONITOR_SAME_TOOL_LIMIT` and `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT`

**Best for**: Smaller models (< 32B parameters), complex attack scenarios requiring continuous guidance, preventing agents from getting stuck on single approach

**Performance Impact**: 2-3x increase in execution time and token usage, but delivers **2x improvement in result quality** based on testing with Qwen3.5-27B-FP8

### Intelligent Task Planning (Beta)
- **Automated Decomposition**: Planner (adviser in planning mode) generates 3-7 specific, actionable steps before specialist agents begin work
- **Context-Aware Plans**: Analyzes full execution context via enricher agent to create informed plans
- **Structured Assignment**: Original request wrapped in `<task_assignment>` structure with execution plan and instructions
- **Scope Management**: Prevents scope creep by keeping agents focused on current subtask only
- **Enriched Instructions**: Plans highlight critical actions, potential pitfalls, and verification points
- **Configurable**: Enable via `AGENT_PLANNING_STEP_ENABLED` (default: false)

**Best for**: Models < 32B parameters, complex penetration testing workflows, improving success rates on sophisticated tasks

**Enhanced Adviser Configuration**: Works exceptionally well when adviser agent uses stronger model or enhanced settings. Example: using same base model with maximum reasoning mode for adviser (see [`vllm-qwen3.5-27b-fp8.provider.yml`](examples/configs/vllm-qwen3.5-27b-fp8.provider.yml)) enables comprehensive task analysis and strategic planning from identical model architecture.

**Performance Impact**: Adds planning overhead but significantly improves completion rates and reduces redundant work

### Tool Call Limits (Always Active)
- **Hard Limits**: Prevent runaway executions regardless of supervision mode status
- **Differentiated by Agent Type**:
  - General agents (Assistant, Primary Agent, Pentester, Coder, Installer): `MAX_GENERAL_AGENT_TOOL_CALLS` (default: 100)
  - Limited agents (Searcher, Enricher, Memorist, Generator, Reporter, Adviser, Reflector, Planner): `MAX_LIMITED_AGENT_TOOL_CALLS` (default: 20)
- **Graceful Termination**: Reflector guides agents to proper completion when approaching limits
- **Resource Protection**: Ensures system stability and prevents resource exhaustion

### Reflector Integration (Always Active)
- **Automatic Correction**: Invoked when LLM fails to generate tool calls after 3 attempts
- **Strategic Guidance**: Analyzes failures and guides agents toward proper tool usage or barrier tools (`done`, `ask`)
- **Recovery Mechanism**: Provides contextual guidance based on specific failure patterns
- **Limit Enforcement**: Coordinates graceful termination when tool call limits are reached

### Recommendations for Open Source Models

**Must-Have for Models < 32B Parameters**:
Testing with Qwen3.5-27B-FP8 demonstrates that enabling both Execution Monitoring and Task Planning is **essential** for smaller open source models:
- **Quality Improvement**: 2x better results compared to baseline execution without supervision
- **Loop Prevention**: Significantly reduces infinite loops and redundant work
- **Attack Diversity**: Encourages exploration of multiple attack vectors instead of fixating on single approach
- **Air-Gapped Deployments**: Enables production-grade autonomous pentesting in closed network environments with local LLM inference

**Trade-offs**:
- Token consumption: 2-3x increase due to mentor/planner invocations
- Execution time: 2-3x longer due to analysis and planning steps
- Result quality: 2x improvement in completeness, accuracy, and attack coverage
- Model requirements: Works best when adviser uses enhanced configuration (higher reasoning parameters, stronger model variant, or different model)

**Configuration Strategy**:
For optimal performance with smaller models, configure adviser agent with enhanced settings:
- Use same model with maximum reasoning mode (example: [`vllm-qwen3.5-27b-fp8.provider.yml`](examples/configs/vllm-qwen3.5-27b-fp8.provider.yml))
- Or use stronger model for adviser while keeping base model for other agents
- Adjust monitoring thresholds based on task complexity and model capabilities



</details>

The architecture of PentAGI is designed to be modular, scalable, and secure. Here are the key components:

1. **Core Services**
   - Frontend UI: React-based web interface with TypeScript for type safety
   - Backend API: Go-based REST and GraphQL APIs with Bearer token authentication for programmatic access
   - Vector Store: PostgreSQL with pgvector for semantic search and memory storage
   - Task Queue: Async task processing system for reliable operation
   - AI Agent: Multi-agent system with specialized roles for efficient testing

2. **Optional Knowledge Graph**
   - Graphiti: Knowledge graph API for semantic relationship tracking and contextual understanding
   - Neo4j: Graph database for storing and querying relationships between entities, actions, and outcomes
   - When enabled, automatically captures agent responses and tool executions for a flow-scoped knowledge base

3. **Monitoring Stack**
   - OpenTelemetry: Unified observability data collection and correlation
   - Grafana: Real-time visualization and alerting dashboards
   - VictoriaMetrics: High-performance time-series metrics storage
   - Jaeger: End-to-end distributed tracing for debugging
   - Loki: Scalable log aggregation and analysis

4. **Analytics Platform**
   - Langfuse: Advanced LLM observability and performance analytics
   - ClickHouse: Column-oriented analytics data warehouse
   - Redis: High-speed caching and rate limiting
   - MinIO: S3-compatible object storage for artifacts

5. **Security Tools**
   - Web Scraper: Isolated browser environment for safe web interaction
   - Pentesting Tools: Comprehensive suite of 20+ professional security tools
   - Sandboxed Execution: All operations run in isolated containers

6. **Memory Systems**
   - Long-term Memory: Persistent storage of knowledge and experiences
   - Working Memory: Active context and goals for current operations
   - Episodic Memory: Historical actions and success patterns
   - Knowledge Base: Structured domain expertise and tool capabilities
   - Context Management: Intelligently manages growing LLM context windows using chain summarization

The system uses Docker containers for isolation and easy deployment, with separate networks for core services, monitoring, and analytics to ensure proper security boundaries. Each component is designed to scale horizontally and can be configured for high availability in production environments.

## Quick Start

For a step-by-step walkthrough that connects installation, configuration, LLM and embedding provider testing, and your first login, see the [Installing and Configuring PentAGI](examples/guides/installation_configuration.md) guide. The sections below remain the detailed reference for each step.

### System Requirements

- Docker and Docker Compose (or Podman - see [Podman configuration](#running-pentagi-with-podman))
- Minimum 2 vCPU
- Minimum 4GB RAM
- 20GB free disk space
- Internet access for downloading images and updates

### Using Installer (Recommended)

PentAGI provides an interactive installer with a terminal-based UI for streamlined configuration and deployment. The installer guides you through system checks, LLM provider setup, search engine configuration, and security hardening.

**Supported Platforms:**
- **Linux**: amd64 [download](https://pentagi.com/downloads/linux/amd64/installer-latest.zip) | arm64 [download](https://pentagi.com/downloads/linux/arm64/installer-latest.zip)
- **Windows**: amd64 [download](https://pentagi.com/downloads/windows/amd64/installer-latest.zip)
- **macOS**: amd64 (Intel) [download](https://pentagi.com/downloads/darwin/amd64/installer-latest.zip) | arm64 (M-series) [download](https://pentagi.com/downloads/darwin/arm64/installer-latest.zip)

> **macOS security warning:** If macOS flags a downloaded installer, use only the official PentAGI links above, choose the archive that matches your CPU architecture, verify the source before continuing, and follow the [installer troubleshooting guide](backend/docs/installer/installer-troubleshooting.md#macos-reports-the-installer-as-malware) before allowing the app to run.

**Quick Installation (Linux amd64):**

```bash
# Create installation directory
mkdir -p pentagi && cd pentagi

# Download installer
wget -O installer.zip https://pentagi.com/downloads/linux/amd64/installer-latest.zip

# Extract
unzip installer.zip

# Run interactive installer
./installer
```

**Prerequisites & Permissions:**

The installer requires appropriate privileges to interact with the Docker API for proper operation. By default, it uses the Docker socket (`/var/run/docker.sock`) which requires either:

- **Option 1 (Recommended for production):** Run the installer as root:
  ```bash
  sudo ./installer
  ```

- **Option 2 (Development environments):** Grant your user access to the Docker socket by adding them to the `docker` group:
  ```bash
  # Add your user to the docker group
  sudo usermod -aG docker $USER
  
  # Log out and log back in, or activate the group immediately
  newgrp docker
  
  # Verify Docker access (should run without sudo)
  docker ps
  ```

  ⚠️ **Security Note:** Adding a user to the `docker` group grants root-equivalent privileges. Only do this for trusted users in controlled environments. For production deployments, consider using rootless Docker mode or running the installer with sudo.

The installer will:
1. **System Checks**: Verify Docker, network connectivity, and system requirements
2. **Environment Setup**: Create and configure `.env` file with optimal defaults
3. **Provider Configuration**: Set up LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Ollama, DeepSeek, GLM, Kimi, Qwen, MiniMax, Mistral, xAI, Custom)
4. **Search Engines**: Configure DuckDuckGo, Google, Tavily, Firecrawl, Traversaal, Perplexity, Sploitus, SearXNG, and the optional internal browser-analytics fallback engine
5. **Security Hardening**: Generate secure credentials and configure SSL certificates
6. **Deployment**: Start PentAGI with docker-compose

### Current Web Settings Coverage

The PentAGI web console already manages several settings areas after the server is up and running:

- **Settings -> Providers**: Create, edit, delete, and test user-defined provider profiles for supported provider types. These profiles control per-agent model selection, runtime parameters, reasoning options, and pricing metadata.
- **Settings -> Prompts**: Manage system, human, and tool prompt templates.
- **Settings -> PentAGI API**: Create and manage PentAGI Bearer tokens for REST and GraphQL access.
- **Other UI-managed preferences**: Favorite flows are stored as user preferences, and theme selection is handled from the main sidebar/profile controls rather than the Settings pages.

### Still Server-Managed

The following configuration areas still need to be set on the server through environment variables, compose files, or mounted config files:

- **LLM credentials and connection details**: API keys, endpoints, auth modes, and provider-specific connection settings for OpenAI, Anthropic, Bedrock, Ollama, custom providers, and similar backends; config-path settings apply only where supported, such as `OLLAMA_SERVER_CONFIG_PATH`, `LLM_SERVER_CONFIG_PATH`, and `BEDROCK_CONFIG_PATH`.
- **Search provider credentials and options**: Settings such as `DUCKDUCKGO_*`, `GOOGLE_*`, `TAVILY_API_KEY`, `FIRECRAWL_API_*`, `TRAVERSAAL_API_KEY`, `PERPLEXITY_*`, `SEARXNG_*`, `SPLOITUS_ENABLED`, and the optional `WEB_SEARCH_INTERNAL_*` browser-analytics fallback settings.
- **Third-party integrations**: Langfuse, Graphiti, and similar external services remain server-side configuration.

**For Production & Enhanced Security:**

For production deployments or security-sensitive environments, we **strongly recommend** using a distributed two-node architecture where worker operations are isolated on a separate server. This prevents untrusted code execution and network access issues on your main system.

**See detailed guide**: [Worker Node Setup](examples/guides/worker_node.md)

The two-node setup provides:
- **Isolated Execution**: Worker containers run on dedicated hardware
- **Network Isolation**: Separate network boundaries for penetration testing
- **Security Boundaries**: Docker-in-Docker with TLS authentication
- **OOB Attack Support**: Dedicated port ranges for out-of-band techniques

#### Giving Agents Docker Without Giving Away the Host

Many pentest workflows need `docker` inside the agent's sandbox. There are two ways to provide it, and they differ sharply in risk.

**Recommended — point sandboxes at a hardened dind daemon over TLS.** Set `DOCKER_INSIDE=true`, leave `DOCKER_SOCKET` empty, and configure the daemon the sandbox may talk to:

```bash
DOCKER_INSIDE=true
DOCKER_SOCKET=                                          # mount no socket
DOCKER_INSIDE_HOST=tcp://10.0.0.5:3376                  # hardened dind endpoint
DOCKER_INSIDE_TLS_VERIFY=1
DOCKER_INSIDE_CERT_PATH=/etc/docker/dind/certs/client   # path on the worker node
DOCKER_INSIDE_POLICY_TESTS=true                         # prove it on every worker; off by default
```

PentAGI injects these into every worker container as `DOCKER_HOST`, `DOCKER_TLS_VERIFY` and `DOCKER_CERT_PATH` (the `_INSIDE_` segment is dropped) and bind-mounts the certificate directory read-only at the same path, so `docker` works inside the sandbox with no further setup.

**Not recommended — bind-mounting a Docker socket (`DOCKER_SOCKET`).** This has two failure modes:

- **Boot-order race**: a bind-mount source that does not exist yet is created by Docker as a *directory*. After a worker-node reboot, a worker container can start before dind has recreated its socket — Docker then puts a directory where the socket belongs, and dind cannot start until it is removed by hand.
- **Blast radius**: the race is only reliably avoided when the mounted socket is the **host** daemon's, since that one always exists first. But that grants an autonomous agent the host Docker API: it can start a privileged container, mount `/`, and compromise the entire node — PentAGI included.

Use `DOCKER_SOCKET` only on single-node development setups where the host daemon is already trusted.

**See**: [Worker Node Setup](examples/guides/worker_node.md) for the full dind hardening and TLS configuration, and [Worker Docker Access](backend/docs/docker.md#worker-docker-access) for the exact resolution algorithm.

#### Running Several Instances (`TENANT_ID`)

A single PentAGI installation needs none of this — leave `TENANT_ID` empty (the default) and nothing changes.

Set it when several PentAGI installations share external resources: one PostgreSQL server, one worker node, one Neo4j/Graphiti, one Langfuse. The typical case is a management backend per server with a common worker node and database. Because every instance numbers its flows from `1`, they would otherwise collide on container names, database rows, knowledge-graph namespaces and session cookies. `TENANT_ID` namespaces all of it:

| Area | Effect when `TENANT_ID=acme` |
| ---- | ---------------------------- |
| PostgreSQL | The instance creates and works inside schema `acme` instead of `public`; extensions stay shared in `DATABASE_EXTENSIONS_SCHEMA` (default `public`, `extensions` on Supabase) |
| Worker containers | `acme-pentagi-terminal-<flow>` instead of `pentagi-terminal-<flow>`; volumes and hostnames follow, and both carry a `pentagi.tenant` label |
| Knowledge graph | Graphiti/Neo4j group ids become `acme-flow-<id>` |
| Auth | Cookie and API token keys are derived from `COOKIE_SIGNING_SALT` **plus** the tenant, and the session cookie is renamed |
| Telemetry | Langfuse traces carry the tenant as their `environment` and a `tenant:acme` tag; OTel resources gain `tenant_id` |

The value must match `^[a-z][a-z0-9_]{0,31}$` — an invalid one aborts startup rather than being silently normalised.

Some things stay yours to set per instance, because they are host resources rather than names: `DATA_DIR` (two instances sharing it **will** overwrite each other's flow data), `DOCKER_PORTS_BASE`, the published ports, and `INSTALLATION_ID`. The effective values are printed at startup under `Instance identity`.

The installer provisions one instance per server. Running several on one server is possible — for example behind a shared nginx — but the stock `docker-compose.yml` uses fixed container and network names, so it has to be adapted to your own network layout first.

**See**: [Multi-Instance Deployment](backend/docs/config.md#multi-instance-deployment-tenant_id) for validation rules, upgrade notes and the full list of operator responsibilities.

### Manual Installation

1. Create a working directory or clone the repository:

```bash
mkdir pentagi && cd pentagi
```

2. Copy `.env.example` to `.env` or download it:

```bash
curl -o .env https://raw.githubusercontent.com/vxcontrol/pentagi/master/.env.example
```

3. Touch examples files (`example.custom.provider.yml`, `example.ollama.provider.yml`, `example.bedrock.provider.yml`) or download it. `docker-compose.yml` mounts each of them into the container, and Docker creates a directory in place of a mount source that does not exist:

```bash
curl -o example.custom.provider.yml https://raw.githubusercontent.com/vxcontrol/pentagi/master/examples/configs/custom-openai.provider.yml
curl -o example.ollama.provider.yml https://raw.githubusercontent.com/vxcontrol/pentagi/master/examples/configs/ollama-llama318b.provider.yml
curl -o example.bedrock.provider.yml https://raw.githubusercontent.com/vxcontrol/pentagi/master/examples/configs/bedrock.provider.yml
```

4. Fill in the required API keys in `.env` file.

```
/* Detailed source-code truncated for AI context efficiency. */
```

5. Change all security related environment variables in `.env` file to improve security.

<details>
    <summary>Security related environment variables</summary>

### Main Security Settings
- `COOKIE_SIGNING_SALT` - Salt for cookie signing, change to random value
- `PUBLIC_URL` - Public URL of your server (eg. `https://pentagi.example.com`)
- `SERVER_SSL_CRT` and `SERVER_SSL_KEY` - Custom paths to your existing SSL certificate and key for HTTPS (these paths should be used in the docker-compose.yml file to mount as volumes)
- `TENANT_ID` - Leave empty unless this instance shares external resources with another PentAGI installation. When set, it is mixed into the cookie and API token signing keys and renames the session cookie, so a session minted by one instance is rejected by the others even though they share the same `COOKIE_SIGNING_SALT`. See [Running Several Instances](#running-several-instances-tenant_id)

### Scraper Access
- `SCRAPER_PUBLIC_URL` - Public URL for scraper if you want to use different scraper server for public URLs
- `SCRAPER_PRIVATE_URL` - Private URL for scraper (local scraper server in docker-compose.yml file to access it to local URLs)

### Access Credentials
- `PENTAGI_POSTGRES_USER` and `PENTAGI_POSTGRES_PASSWORD` - PostgreSQL credentials
- `NEO4J_USER` and `NEO4J_PASSWORD` - Neo4j credentials (for Graphiti knowledge graph)

</details>

6. Remove all inline comments from `.env` file if you want to use it in VSCode or other IDEs as a envFile option:

```bash
perl -i -pe 's/\s+#.*$//' .env
```

7. Run the PentAGI stack:

```bash
curl -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose.yml
docker compose up -d
```

Visit [localhost:8443](https://localhost:8443) to access PentAGI Web UI (default is `admin@pentagi.com` / `admin`)

#### Web UI Accounts

PentAGI does not expose public self-service sign-up from the login page. A fresh installation creates the default local administrator account:

- **Email**: `admin@pentagi.com`
- **Password**: `admin`

On first login, change the default password before using the instance for real work. If the administrator password is lost later, use the installer maintenance menu to reset the default `admin@pentagi.com` account password.

For multi-user setups, an authenticated administrator can manage local users through the Users REST API (`/api/v1/users/`). The OpenAPI UI is available at `https://localhost:8443/api/v1/swagger/index.html` after the instance is running.

> [!NOTE]
> If you caught an error about `pentagi-network` or `observability-network` or `langfuse-network` you need to run `docker-compose.yml` firstly to create these networks and after that run `docker-compose-langfuse.yml`, `docker-compose-graphiti.yml`, and `docker-compose-observability.yml` to use Langfuse, Graphiti, and Observability services.
>
> You have to set at least one Language Model provider (OpenAI, Anthropic, Gemini, AWS Bedrock, or Ollama) to use PentAGI. AWS Bedrock provides enterprise-grade access to multiple foundation models from leading AI companies, while Ollama provides zero-cost local inference if you have sufficient computational resources. Additional API keys for search engines are optional but recommended for better results.
>
> **For fully local deployment with advanced models**: See our comprehensive guide on [Running PentAGI with vLLM and Qwen3.5-27B-FP8](examples/guides/vllm-qwen35-27b-fp8.md) for a production-grade local LLM setup. This configuration achieves ~13,000 TPS for prompt processing and ~650 TPS for completion on 4× RTX 5090 GPUs, supporting 12+ concurrent flows with complete independence from cloud providers.
>
> `LLM_SERVER_*` environment variables are experimental feature and will be changed in the future. Right now you can use them to specify custom LLM server URL and one model for all agent types.
>
> `PROXY_URL` routes the backend's own HTTP calls through a proxy: every request to LLM and embedding providers and to search engines — self-hosted ones such as Ollama or SearXNG included, as there is no `NO_PROXY` exemption — and the update check. Sandbox containers (where agents run their commands), the scraper, the Graphiti service, OAuth sign-in, and Langfuse/OpenTelemetry export do not use it, so if you need isolation from external networks, restrict their outbound traffic at the network level.
>
> The `docker-compose.yml` file runs the PentAGI service as root user because it needs access to docker.sock for container management. If you're using TCP/IP network connection to Docker instead of socket file, you can remove root privileges and use the default `pentagi` user for better security.

### Accessing PentAGI from External Networks

By default, the PentAGI web interface and the other compose services bind to `127.0.0.1` (localhost only) for security. To access PentAGI from other machines on your network, you need to configure external access.

> [!IMPORTANT]
> Flow sandboxes are not bound to localhost. Each flow publishes two TCP ports for out-of-band callbacks such as reverse shells, taken from the 2000-port window that starts at `DOCKER_PORTS_BASE` (`28000`–`29999` by default), on `DOCKER_PUBLIC_IP` — `0.0.0.0` by default, that is every interface of the Docker host. Docker routes published ports around `ufw` rules and lets them through `firewalld` with its own `docker` zone, so restrict this range with a firewall in front of the host or, with Docker's default iptables backend, with rules in the `DOCKER-USER` chain. With `DOCKER_NETWORK=host`, sandboxes share the host's network stack, so their listeners are exposed without any port mapping.

#### Configuration Steps

1. **Update `.env` file** with your server's IP address:

```bash
# Network binding - allow external connections
PENTAGI_LISTEN_IP=0.0.0.0
PENTAGI_LISTEN_PORT=8443

# Public URL - use your actual server IP or hostname
# Replace 192.168.1.100 with your server's IP address
PUBLIC_URL=https://192.168.1.100:8443

# CORS origins - list all URLs that will access PentAGI
# Include localhost for local access AND your server IP for external access
CORS_ORIGINS=https://localhost:8443,https://192.168.1.100:8443
```

> [!IMPORTANT]
> - Replace `192.168.1.100` with your actual server's IP address
> - Do NOT use `0.0.0.0` in `PUBLIC_URL` or `CORS_ORIGINS` - use the actual IP address
> - Include both localhost and your server IP in `CORS_ORIGINS` for flexibility

2. **Recreate containers** to apply the changes:

```bash
docker compose down
docker compose up -d --force-recreate
```

3. **Verify port binding:**

```bash
docker ps | grep pentagi
```

You should see `0.0.0.0:8443->8443/tcp` or `:::8443->8443/tcp`.

If you see `127.0.0.1:8443->8443/tcp`, the environment variable wasn't picked up. In this case, directly edit `docker-compose.yml` line 31:

```yaml
ports:
  - "0.0.0.0:8443:8443"
```

Then recreate containers again.

4. **Configure firewall** to allow incoming connections on port 8443:

```bash
# Ubuntu/Debian with UFW
sudo ufw allow 8443/tcp
sudo ufw reload

# CentOS/RHEL with firewalld
sudo firewall-cmd --permanent --add-port=8443/tcp
sudo firewall-cmd --reload
```

5. **Access PentAGI:**

- **Local access:** `https://localhost:8443`
- **Network access:** `https://your-server-ip:8443`

> [!NOTE]
> You'll need to accept the self-signed SSL certificate warning in your browser when accessing via IP address.

---

### Running PentAGI with Podman

PentAGI fully supports Podman as a Docker alternative. However, when using **Podman in rootless mode**, the scraper service requires special configuration because rootless containers cannot bind privileged ports (ports below 1024).

#### Podman Rootless Configuration

The default scraper configuration uses port 443 (HTTPS), which is a privileged port. For Podman rootless, reconfigure the scraper to use a non-privileged port:

**1. Edit `docker-compose.yml`** - modify the `scraper` service (around line 199):

```yaml
scraper:
  image: vxcontrol/scraper:latest
  restart: unless-stopped
  container_name: scraper
  hostname: scraper
  expose:
    - 3000/tcp  # Changed from 443 to 3000
  ports:
    - "${SCRAPER_LISTEN_IP:-127.0.0.1}:${SCRAPER_LISTEN_PORT:-9443}:3000"  # Map to port 3000
  environment:
    - MAX_CONCURRENT_SESSIONS=${LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS:-10}
    - USERNAME=${LOCAL_SCRAPER_USERNAME:-someuser}
    - PASSWORD=${LOCAL_SCRAPER_PASSWORD:-somepass}
  logging:
    options:
      max-size: 50m
      max-file: "7"
  volumes:
    - scraper-ssl:/usr/src/app/ssl
  networks:
    - pentagi-network
  shm_size: 2g
```

**2. Update `.env` file** - change the scraper URL to use HTTP and port 3000:

```bash
# Scraper configuration for Podman rootless
SCRAPER_PRIVATE_URL=http://someuser:somepass@scraper:3000/
LOCAL_SCRAPER_USERNAME=someuser
LOCAL_SCRAPER_PASSWORD=somepass
```

> [!IMPORTANT]
> Key changes for Podman:
> - Use **HTTP** instead of HTTPS for `SCRAPER_PRIVATE_URL`
> - Use port **3000** instead of 443
> - Change internal `expose` to `3000/tcp`
> - Update port mapping to target `3000` instead of `443`

**3. Recreate containers:**

```bash
podman-compose down
podman-compose up -d --force-recreate
```

**4. Test scraper connectivity:**

```bash
# Test from within the pentagi container
podman exec -it pentagi wget -O- "http://someuser:somepass@scraper:3000/html?url=http://example.com"
```

If you see HTML output, the scraper is working correctly.

#### Podman Rootful Mode

If you're running Podman in rootful mode (with sudo), you can use the default configuration without modifications. The scraper will work on port 443 as intended.

#### Docker Compatibility

All Podman configurations remain fully compatible with Docker. The non-privileged port approach works identically on both container runtimes.

### Assistant Configuration

PentAGI allows you to configure default behavior for assistants:

| Variable               | Default | Description                                                             |
| ---------------------- | ------- | ----------------------------------------------------------------------- |
| `ASSISTANT_USE_AGENTS` | `false` | Controls the default value for agent usage when creating new assistants |

The `ASSISTANT_USE_AGENTS` setting affects the initial state of the "Use Agents" toggle when creating a new assistant in the UI:
- `false` (default): New assistants are created with agent delegation disabled by default
- `true`: New assistants are created with agent delegation enabled by default

Note that users can always override this setting by toggling the "Use Agents" button in the UI when creating or editing an assistant. This environment variable only controls the initial default state.

## How to Use PentAGI After Login

Once the stack is running and you can sign in to the web UI, the fastest way to start is through the Flows workflow.

### 1. Create your first flow

1. Open **Flows** in the sidebar.
2. Click **New Flow**.
3. Choose the mode that fits your goal:
   - **Automation**: fully autonomous execution for a testing goal you want PentAGI to carry out end-to-end
   - **Assistant**: interactive back-and-forth help when you want to steer the investigation step by step. In this mode you can also enable the **Use Agents** toggle to let PentAGI delegate subtasks to specialized sub-agents for more complex investigations.
4. Select the LLM provider you want to use for this flow.
5. Describe the target and the objective in natural language in the message box.

Good first prompts usually include:

- the target system or URL
- the type of assessment you want
- any scope limitations or rules of engagement
- the result you expect, such as a vulnerability report or validation of a hypothesis

Example:

```text
Assess https://target.example for common web application vulnerabilities. Focus on authentication, file handling, and injection issues. Stay within the provided target only and summarize confirmed findings with reproduction steps.
```

Only test systems you own or are explicitly authorized to assess. See [EULA.md](EULA.md) for the acceptable use requirements.

### 2. Use templates for repeatable workflows

The new flow form includes a template picker, which can prefill the message box with a saved flow template. This is useful when you run similar assessments repeatedly.

- Use an existing template if you already have one saved in **Templates**
- Start from the example prompt in [`examples/prompts/base_web_pentest.md`](examples/prompts/base_web_pentest.md) if you need a practical baseline for web testing
- Adjust the target, scope, and constraints before starting the flow

Templates are starting points. You do not need special syntax to use PentAGI: plain natural-language instructions work well as long as the target and goal are clear.

### 3. Monitor execution and review output

After submitting the flow, PentAGI opens the flow page automatically.

- Use the main flow view to follow messages, agent activity, and task progress
- Inspect tool activity and terminal output as the flow runs
- Review generated tasks and subtasks to understand what PentAGI is doing

Once the flow has enough results, use the **Report** menu on the flow page to:

- open the report in a web view
- copy the generated report to the clipboard
- download the report as Markdown
- download the report as PDF

### 4. Use the Assistant view to steer an active flow

Each flow also includes an **Assistant** view for interactive guidance. This is useful when the autonomous run uncovers something that needs human direction instead of a hard restart.

- Open the **Assistant** view for the same flow when you want to inspect the current state before changing anything.
- Use the assistant to check flow status, stop the current task, submit follow-up instructions, or patch the remaining planned subtasks before the next step runs.
- Treat this as an explicit control path for the current flow, not as an invisible background queue. If you want to change direction, say so clearly and keep the new instruction tied to the current engagement scope.
- This works best for clarifying scope, redirecting priorities after intermediate findings, or answering an automation checkpoint without losing the rest of the flow context.

### 5. Manage flow-scoped files

Each flow has its own **Files** tab in the flow page. Files are scoped to the parent flow: they live in `{dataDir}/flow-{id}-data/` on the host and never leak into other flows.

The tab exposes three sources of files:

- **Uploads** (`uploads/`): files you provide from the web UI. Use the **Upload files** action, or drag and drop directly onto the Files tab. While the agent container is running, uploaded files are also pushed into it at `/work/uploads/` so the agent can read them with normal shell tools.
- **Resources** (`resources/`): files attached from your saved user resources library via **Attach resources from library**. Attached resources are copied into the flow and pushed into the running container at `/work/resources/`.
- **Container** (`container/`): snapshots pulled from the running agent container via **Pull file or directory from container**. These are read-only on the flow side and are never sent back to the container.

Per-file actions in the Files tab include **Download**, **Copy path**, **Save as resource** (promote a flow file into your reusable resources library), and **Delete**. The Pull action is disabled when the container is not running, with the tooltip "Container is not running".

Uploaded files and attached resources are listed automatically in the agent's system prompts via the `{{.UserFiles}}` template variable, which renders a compact `<task_files>` XML block (with nested `<uploads>` and `<resources>` sections), so the assistant and automation agents can reference them by path without you pasting the contents into chat. Container snapshots are visible in the UI only and are not auto-injected back into the prompt.

Current limits and limitations to be aware of:

- Maximum upload file size is 300 MB; per upload request up to 1000 files and 2 GB total. File names are capped at 255 bytes (roughly 255 ASCII characters; non-ASCII names use multiple bytes per character).
- Uploads and resources are mirrored into the running container at the fixed paths `/work/uploads/` and `/work/resources/`; files written to other container paths are not auto-mirrored back into the flow file model. Container snapshots can originate from any container path you pull (for example `/etc/...`) and are cached on the flow side under `container/`; they are not pushed back into the container.
- Container snapshots are point-in-time pulls. Editing a snapshot in the UI does not write back into the running container.
- Deleting a flow today removes the flow record and its long-term memory entries, but does not yet archive or remove the flow's `flow-{id}-data/` directory on disk. Operators are still expected to clean up the data directory manually if they want to reclaim the space.

For early testing, start with a narrow target and a single clear objective. This makes the output easier to review and helps you refine your prompts before running larger assessments.

## API Access

PentAGI provides comprehensive programmatic access through both REST and GraphQL APIs, allowing you to integrate penetration testing workflows into your automation pipelines, CI/CD processes, and custom applications.

### Generating API Tokens

API tokens are managed through the PentAGI web interface:

1. Navigate to **Settings** → **API Tokens** in the web UI
2. Click **Create Token** to generate a new API token
3. Configure token properties:
   - **Name** (optional): A descriptive name for the token
   - **Expiration Date**: When the token will expire (minimum 1 minute, maximum 3 years)
4. Click **Create** and **copy the token immediately** - it will only be shown once for security reasons
5. Use the token as a Bearer token in your API requests

Each token is associated with your user account and inherits your role's permissions.

### Using API Tokens

Include the API token in the `Authorization` header of your HTTP requests:

```bash
# GraphQL API example
curl -X POST https://your-pentagi-instance:8443/api/v1/graphql \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "{ flows { id title status } }"}'

# REST API example
curl https://your-pentagi-instance:8443/api/v1/flows \
  -H "Authorization: Bearer YOUR_API_TOKEN"
```

### API Exploration and Testing

PentAGI provides interactive documentation for exploring and testing API endpoints:

#### GraphQL Playground

Access the GraphQL Playground at `https://your-pentagi-instance:8443/api/v1/graphql/playground`

1. Click the **HTTP Headers** tab at the bottom
2. Add your authorization header:
   ```json
   {
     "Authorization": "Bearer YOUR_API_TOKEN"
   }
   ```
3. Explore the schema, run queries, and test mutations interactively

#### Swagger UI

Access the REST API documentation at `https://your-pentagi-instance:8443/api/v1/swagger/index.html`

1. Click the **Authorize** button
2. Enter your token in the format: `Bearer YOUR_API_TOKEN`
3. Click **Authorize** to apply
4. Test endpoints directly from the Swagger UI

### Generating API Clients

You can generate type-safe API clients for your preferred programming language using the schema files included with PentAGI:

#### GraphQL Clients

The GraphQL schema is available at:
- **Web UI**: Navigate to Settings to download `schema.graphqls`
- **Direct file**: `backend/pkg/graph/schema.graphqls` in the repository

Generate clients using tools like:
- **GraphQL Code Generator** (JavaScript/TypeScript): [https://the-guild.dev/graphql/codegen](https://the-guild.dev/graphql/codegen)
- **genqlient** (Go): [https://github.com/Khan/genqlient](https://github.com/Khan/genqlient)
- **Apollo iOS** (Swift): [https://www.apollographql.com/docs/ios](https://www.apollographql.com/docs/ios)

#### REST API Clients

The OpenAPI specification is available at:
- **Swagger JSON**: `https://your-pentagi-instance:8443/api/v1/swagger/doc.json`
- **Swagger YAML**: Available in `backend/pkg/server/docs/swagger.yaml`

Generate clients using:
- **OpenAPI Generator**: [https://openapi-generator.tech](https://openapi-generator.tech)
  ```bash
  openapi-generator-cli generate \
    -i https://your-pentagi-instance:8443/api/v1/swagger/doc.json \
    -g python \
    -o ./pentagi-client
  ```

- **Swagger Codegen**: [https://github.com/swagger-api/swagger-codegen](https://github.com/swagger-api/swagger-codegen)
  ```bash
  swagger-codegen generate \
    -i https://your-pentagi-instance:8443/api/v1/swagger/doc.json \
    -l typescript-axios \
    -o ./pentagi-client
  ```

- **swagger-typescript-api** (TypeScript): [https://github.com/acacode/swagger-typescript-api](https://github.com/acacode/swagger-typescript-api)
  ```bash
  npx swagger-typescript-api \
    -p https://your-pentagi-instance:8443/api/v1/swagger/doc.json \
    -o ./src/api \
    -n pentagi-api.ts
  ```

### API Usage Examples

<details>
<summary><b>Creating a New Flow (GraphQL)</b></summary>

```graphql
mutation CreateFlow {
  createFlow(
    modelProvider: "openai"
    input: "Test the security of https://example.com"
  ) {
    id
    title
    status
    createdAt
  }
}
```

</details>

<details>
<summary><b>Listing Flows (REST API)</b></summary>

```bash
curl https://your-pentagi-instance:8443/api/v1/flows \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  | jq '.flows[] | {id, title, status}'
```

</details>

<details>
<summary><b>Python Client Example</b></summary>

```python
import requests

class PentAGIClient:
    def __init__(self, base_url, api_token):
        self.base_url = base_url
        self.headers = {
            "Authorization": f"Bearer {api_token}",
            "Content-Type": "application/json"
        }
    
    def create_flow(self, provider, target):
        query = """
        mutation CreateFlow($provider: String!, $input: String!) {
          createFlow(modelProvider: $provider, input: $input) {
            id
            title
            status
          }
        }
        """
        response = requests.post(
            f"{self.base_url}/api/v1/graphql",
            json={
                "query": query,
                "variables": {
                    "provider": provider,
                    "input": target
                }
            },
            headers=self.headers
        )
        return response.json()
    
    def get_flows(self):
        response = requests.get(
            f"{self.base_url}/api/v1/flows",
            headers=self.headers
        )
        return response.json()

# Usage
client = PentAGIClient(
    "https://your-pentagi-instance:8443",
    "your_api_token_here"
)

# Create a new flow
flow = client.create_flow("openai", "Scan https://example.com for vulnerabilities")
print(f"Created flow: {flow}")

# List all flows
flows = client.get_flows()
print(f"Total flows: {len(flows['flows'])}")
```

</details>

<details>
<summary><b>TypeScript Client Example</b></summary>

```
/* Detailed source-code truncated for AI context efficiency. */
```

</details>

### Security Best Practices

When working with API tokens:

- **Never commit tokens to version control** - use environment variables or secrets management
- **Rotate tokens regularly** - set appropriate expiration dates and create new tokens periodically
- **Use separate tokens for different applications** - makes it easier to revoke access if needed
- **Monitor token usage** - review API token activity in the Settings page
- **Revoke unused tokens** - disable or delete tokens that are no longer needed
- **Use HTTPS only** - never send API tokens over unencrypted connections

### Token Management

- **View tokens**: See all your active tokens in Settings → API Tokens
- **Edit tokens**: Update token names or revoke tokens
- **Delete tokens**: Permanently remove tokens (this action cannot be undone)
- **Token ID**: Each token has a unique ID that can be copied for reference

The token list shows:
- Token name (if provided)
- Token ID (unique identifier)
- Status (active/revoked/expired)
- Creation date
- Expiration date

### Custom LLM Provider Configuration

When using custom LLM providers with the `LLM_SERVER_*` variables, you can fine-tune the reasoning format used in requests.

> [!TIP]
> For production-grade local deployments, consider using **vLLM** with **Qwen3.5-27B-FP8** for optimal performance. See our [comprehensive deployment guide](examples/guides/vllm-qwen35-27b-fp8.md) which includes hardware requirements, configuration templates ([thinking mode](examples/configs/vllm-qwen3.5-27b-fp8.provider.yml) and [non-thinking mode](examples/configs/vllm-qwen3.5-27b-fp8-no-think.provider.yml)), and performance benchmarks showing 13K TPS prompt processing on 4× RTX 5090 GPUs.

| Variable                        | Default      | Description                                                                                                        |
| ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------ |
| `LLM_SERVER_URL`                |              | Base URL for the custom LLM API endpoint                                                                           |
| `LLM_SERVER_KEY`                |              | API key for the custom LLM provider                                                                                |
| `LLM_SERVER_MODEL`              |              | Default model to use (can be overridden in provider config)                                                        |
| `LLM_SERVER_CONFIG_PATH`        |              | Path to the YAML configuration file for agent-specific models                                                      |
| `LLM_SERVER_PROVIDER`           |              | Provider name prefix for model names (e.g., `openrouter`, `deepseek` for LiteLLM proxy)                            |
| `LLM_SERVER_PRESERVE_REASONING` | `false`      | Preserve reasoning content in multi-turn conversations (required by some providers)                                |
| `LLM_SERVER_API_TYPE`           |              | `azure` or `azure_ad` for an Azure OpenAI deployment, empty for a plain endpoint; see [Using Azure OpenAI](#using-azure-openai) |
| `LLM_SERVER_API_VERSION`        | `2024-10-21` | The `api-version` Azure requires; ignored by a plain endpoint                                                      |

The `LLM_SERVER_PROVIDER` setting is particularly useful when using **LiteLLM proxy**, which adds a provider prefix to model names. For example, when connecting to Moonshot API through LiteLLM, models like `kimi-2.5` become `moonshot/kimi-2.5`. By setting `LLM_SERVER_PROVIDER=moonshot`, you can use the same provider configuration file for both direct API access and LiteLLM proxy access without modifications.

The `LLM_SERVER_PRESERVE_REASONING` setting controls whether reasoning content is preserved in multi-turn conversations:
- `false` (default): Reasoning content is not preserved in conversation history
- `true`: Reasoning content is preserved and sent in subsequent API calls

This setting is required by some LLM providers (e.g., Moonshot) that return errors like "thinking is enabled but reasoning_content is missing in assistant tool call message" when reasoning content is not included in multi-turn conversations. Enable this setting if your provider requires reasoning content to be preserved.

#### Using Azure OpenAI

The custom provider calls Azure OpenAI deployments directly:

```bash
LLM_SERVER_URL=https://<resource>.openai.azure.com  # the Endpoint from the resource's Keys and Endpoint page, without an /openai path
LLM_SERVER_KEY=your_azure_openai_api_key            # KEY 1 or KEY 2 from the same page
LLM_SERVER_API_TYPE=azure                           # the api-key header and deployment URLs
LLM_SERVER_API_VERSION=2024-10-21                   # the default; a newer version works as well
LLM_SERVER_MODEL=                                   # Leave empty, models are specified in the config
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/azure-openai.provider.yml
```

Azure addresses a **deployment**, not a model: PentAGI calls `<LLM_SERVER_URL>/openai/deployments/<model>/chat/completions?api-version=<version>`, so every `model:` in the provider config must be the name of a deployment in your resource, and `LLM_SERVER_PROVIDER` must stay empty, because its prefix would become part of the deployment name. The bundled [`azure-openai.provider.yml`](examples/configs/azure-openai.provider.yml) expects deployments named after their models: `gpt-4.1`, `gpt-4.1-mini` and `o4-mini`. If yours are named differently, copy the file, change the `model:` values, and mount your copy:

```bash
PENTAGI_LLM_SERVER_CONFIG_PATH=/path/on/host/my-azure.provider.yml  # mounted at /opt/pentagi/conf/custom.provider.yml
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom.provider.yml
```

After `docker compose up -d`, check the deployments before running a flow. Without `-config`, `ctester` tests the file `LLM_SERVER_CONFIG_PATH` names, the bundled one or your copy; the bundled configuration's own run is in [`examples/tests/azure-openai-report.md`](examples/tests/azure-openai-report.md):

```bash
docker exec -it pentagi /opt/pentagi/bin/ctester
```

- PentAGI does not read a model catalogue from Azure, so it knows no context window for a deployment and does not compact agent chains to fit one.
- `LLM_SERVER_API_TYPE=azure_ad` sends `LLM_SERVER_KEY` as a Microsoft Entra ID bearer token instead of an API key. PentAGI does not refresh it, and Entra tokens expire after about an hour, so use an API key for anything longer.
- Azure's OpenAI-compatible [v1 API](https://learn.microsoft.com/en-us/azure/foundry/openai/api-version-lifecycle) works as a plain endpoint too: set `LLM_SERVER_URL=https://<resource>.openai.azure.com/openai/v1` and leave `LLM_SERVER_API_TYPE` empty; `model:` is still the deployment name.
- The installer's custom provider form has **API Type** and **API Version** fields for the same two settings.
- Embeddings are configured separately through `EMBEDDING_*` and have no Azure mode; `LLM_SERVER_API_TYPE` does not apply to them.

#### Troubleshooting: tool-call (function-call) parser errors

PentAGI drives its agents with tool calls (also called function calls), so any custom OpenAI-compatible backend configured through `LLM_SERVER_*` must return valid tool-call JSON in the format the OpenAI Chat Completions API defines. When the backend emits malformed, truncated, or non-conforming tool-call arguments, the agent chain cannot continue.

Self-hosted engines such as llama.cpp, SGLang, and vLLM usually require a specific tool-call parser and a matching chat template to produce correct tool-call output. If the parser is missing or mismatched for the model you are serving, tool-call arguments can come back corrupted. Compatibility therefore depends on the backend's tool-call/function-call behavior and configuration, not on PentAGI alone; not every llama.cpp or SGLang setup produces valid tool calls out of the box.

Typical symptoms:

- Backend or proxy errors such as `Failed to parse tool call arguments as JSON` (often surfaced through a LiteLLM proxy as an HTTP 500), or other unexpected 5xx/4xx responses from the LLM endpoint.
- A flow that runs for a few steps and then stops responding to new input in the UI.
- Repeated or looping tool calls that never converge.
- A flow that fails right at the start with `failed to select primary docker image via llm call`, because the first action in a flow is an LLM tool call to choose the container image; a backend that cannot return a valid tool call fails at this step too.

How to investigate:

1. Check both sides of the connection: the PentAGI logs (`docker compose logs -f pentagi`) and the inference backend or proxy logs (llama.cpp, SGLang, vLLM, or LiteLLM). The backend log usually shows the same parse error when it produced the malformed tool call.
2. Validate the provider before running a full flow with the `ctester` utility, which exercises tool-calling agent types directly. See [Testing LLM Agents](https://github.com/vxcontrol/pentagi#testing-llm-agents).
3. Confirm the backend's tool-call parser and chat template are the ones recommended for the model you are serving, and that the model itself supports tool calling.
4. Update PentAGI to the latest build. Recent versions sanitize malformed function-call arguments returned by the model so a single bad response no longer stalls the whole flow; older builds forwarded the corrupted arguments and could get stuck.

### Ollama Provider Configuration

PentAGI supports Ollama for both local LLM inference (zero-cost, enhanced privacy) and Ollama Cloud (managed service with free tier).

#### Configuration Variables

| Variable                            | Default     | Description                               |
| ----------------------------------- | ----------- | ----------------------------------------- |
| `OLLAMA_SERVER_URL`                 |             | URL of your Ollama server or Ollama Cloud |
| `OLLAMA_SERVER_API_KEY`             |             | API key for Ollama Cloud authentication   |
| `OLLAMA_SERVER_MODEL`               |             | Default model for inference               |
| `OLLAMA_SERVER_CONFIG_PATH`         |             | Path to custom agent configuration file   |
| `OLLAMA_SERVER_PULL_MODELS_TIMEOUT` | `600`       | Timeout for model downloads (seconds)     |
| `OLLAMA_SERVER_PULL_MODELS_ENABLED` | `false`     | Auto-download models on startup           |
| `OLLAMA_SERVER_LOAD_MODELS_ENABLED` | `false`     | Query server for available models         |

#### Ollama Cloud Configuration

Ollama Cloud provides managed inference with a free tier and paid plans. Paid usage is billed from included or purchased credits at each model's published per-token rate; off-peak discounts apply to selected DeepSeek models.

**Free Tier Setup (Single Model)**

```bash
# Free tier allows one model at a time
OLLAMA_SERVER_URL=https://ollama.com
OLLAMA_SERVER_API_KEY=your_ollama_cloud_api_key
OLLAMA_SERVER_MODEL=gpt-oss:120b  # Example: OpenAI OSS 120B model
```

**Paid Tier Setup (Multi-Model with Pre-built Configuration)**

For paid tiers supporting concurrent requests and pay-as-you-go credits, use the pre-built Ollama Cloud configuration:

```bash
# Using pre-built Ollama Cloud configuration (included in Docker image)
OLLAMA_SERVER_URL=https://ollama.com
OLLAMA_SERVER_API_KEY=your_ollama_cloud_api_key
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-cloud.provider.yml
```

The pre-built `ollama-cloud.provider.yml` configuration includes optimized model assignments for all agent types:
- **Simple/Reflector/Enricher**: `minimax-m2.7:cloud` - Reliable low-latency utility model
- **Simple JSON/Primary Agent/Assistant/Pentester/Searcher**: `deepseek-v4.1-flash:cloud` - Efficient long-context model at the same input/output rate as MiniMax M2.7
- **Generator/Adviser**: `kimi-k3:cloud` - Flagship long-horizon planning and knowledge-work model
- **Refiner**: `glm-5.3:cloud` - Flagship coding and agentic model
- **Coder/Installer**: `kimi-k2.7-code:cloud` - Coding-specialized long-context model

**Custom Configuration (Advanced)**

To create your own agent configuration, mount a custom file from your host filesystem:

```bash
# Using custom provider configuration
OLLAMA_SERVER_URL=https://ollama.com
OLLAMA_SERVER_API_KEY=your_ollama_cloud_api_key
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama.provider.yml

# Mount custom configuration from host filesystem (in .env or docker-compose override)
PENTAGI_OLLAMA_SERVER_CONFIG_PATH=/path/on/host/my-ollama-config.yml
```

The `PENTAGI_OLLAMA_SERVER_CONFIG_PATH` environment variable maps your host configuration file to `/opt/pentagi/conf/ollama.provider.yml` inside the container.

**Example custom configuration** (`my-ollama-config.yml`):

```yaml
primary_agent:
  model: "deepseek-v4-flash:cloud"
  temperature: 1.0
  top_p: 0.95
  max_tokens: 32768

coder:
  model: "kimi-k2.7-code:cloud"
  temperature: 1.0
  max_tokens: 20480
```

#### Local Ollama Configuration

For self-hosted Ollama instances:

```bash
# Basic local Ollama setup
OLLAMA_SERVER_URL=http://localhost:11434
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0

# Production setup with auto-pull and model discovery
OLLAMA_SERVER_URL=http://ollama-server:11434
OLLAMA_SERVER_PULL_MODELS_ENABLED=true
OLLAMA_SERVER_PULL_MODELS_TIMEOUT=900
OLLAMA_SERVER_LOAD_MODELS_ENABLED=true

# Using pre-built configurations from Docker image
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-llama318b.provider.yml
# or
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-qwen332b-fp16-tc.provider.yml
# or
OLLAMA_SERVER_CONFIG_PATH=/opt/pentagi/conf/ollama-qwq32b-fp16-tc.provider.yml
```

**Performance Considerations:**

- **Model Discovery** (`OLLAMA_SERVER_LOAD_MODELS_ENABLED=true`): Adds 1-2s startup latency querying Ollama API
- **Auto-pull** (`OLLAMA_SERVER_PULL_MODELS_ENABLED=true`): First startup may take several minutes downloading models
- **Pull timeout** (`OLLAMA_SERVER_PULL_MODELS_TIMEOUT=900`): 15 minutes in seconds
- **Static Config**: Disable both flags and specify models in config file for fastest startup

#### Creating Custom Ollama Models with Extended Context

PentAGI requires models with larger context windows than the default Ollama configurations. You need to create custom models with increased `num_ctx` parameter through Modelfiles. While typical agent workflows consume around 64K tokens, PentAGI uses 110K context size for safety margin and handling complex penetration testing scenarios.

**Important**: The `num_ctx` parameter can only be set during model creation via Modelfile - it cannot be changed after model creation or overridden at runtime.

##### Example: Qwen3 32B FP16 with Extended Context

Create a Modelfile named `Modelfile_qwen3_32b_fp16_tc`:

```dockerfile
FROM qwen3:32b-fp16
PARAMETER num_ctx 110000
PARAMETER temperature 0.3
PARAMETER top_p 0.8
PARAMETER min_p 0.0
PARAMETER top_k 20
PARAMETER repeat_penalty 1.1
```

Build the custom model:

```bash
ollama create qwen3:32b-fp16-tc -f Modelfile_qwen3_32b_fp16_tc
```

##### Example: QwQ 32B FP16 with Extended Context

Create a Modelfile named `Modelfile_qwq_32b_fp16_tc`:

```dockerfile
FROM qwq:32b-fp16
PARAMETER num_ctx 110000
PARAMETER temperature 0.2
PARAMETER top_p 0.7
PARAMETER min_p 0.0
PARAMETER top_k 40
PARAMETER repeat_penalty 1.2
```

Build the custom model:

```bash
ollama create qwq:32b-fp16-tc -f Modelfile_qwq_32b_fp16_tc
```

> **Note**: The QwQ 32B FP16 model requires approximately **71.3 GB VRAM** for inference. Ensure your system has sufficient GPU memory before attempting to use this model.

These custom models are referenced in the pre-built provider configuration files (`ollama-qwen332b-fp16-tc.provider.yml` and `ollama-qwq32b-fp16-tc.provider.yml`) that are included in the Docker image at `/opt/pentagi/conf/`.

### OpenAI Provider Configuration

PentAGI integrates with OpenAI's comprehensive model lineup, featuring advanced reasoning capabilities with extended chain-of-thought, agentic models with enhanced tool integration, and specialized code models for security engineering.

#### Configuration Variables

| Variable             | Default                     | Description                 |
| -------------------- | --------------------------- | --------------------------- |
| `OPEN_AI_KEY`        |                             | API key for OpenAI services |
| `OPEN_AI_SERVER_URL` | `https://api.openai.com/v1` | OpenAI API endpoint         |

#### Configuration Examples

```bash
# Basic OpenAI setup
OPEN_AI_KEY=your_openai_api_key
OPEN_AI_SERVER_URL=https://api.openai.com/v1

# Using with proxy for enhanced security
OPEN_AI_KEY=your_openai_api_key
PROXY_URL=http://your-proxy:8080
```

#### Supported Models

PentAGI supports 32 OpenAI models with tool calling, streaming, reasoning modes, and prompt caching. Models marked with `*` are used in default configuration. Models marked `⚠️` are deprecated by OpenAI and kept only for backward compatibility with agent configs already pinned to those names — avoid them for new assignments.

**GPT-5.6 Series - Latest Frontier (Feb 2026 knowledge cutoff, 1.05M context, 128K max output)**

| Model ID              | Thinking | Reasoning Effort            | Price (Input/Output/Cache) | Use Case                                        |
| --------------------- | -------- | ---------------------------- | -------------------------- | ----------------------------------------------- |
| `gpt-5.6-sol`         | ✅        | low/medium/high/xhigh        | $5.00/$30.00/$0.50         | Frontier model for complex professional work, most demanding autonomous pentesting, sophisticated exploit chain development, deep multi-stage attack simulation |
| `gpt-5.6-terra`*      | ✅        | low/medium/high/xhigh        | $2.50/$15.00/$0.25         | Balances intelligence and cost; multi-phase security assessments, coordinated multi-tool pentesting (generator/refiner/adviser/coder default) |
| `gpt-5.6-luna`        | ✅        | low/medium/high/xhigh        | $1.00/$6.00/$0.10          | Optimized for cost-sensitive, high-volume workloads; rapid reconnaissance, bulk vulnerability scanning, real-time monitoring |

**GPT-5.5 Series - Frontier (Dec 2025 knowledge cutoff, 1.05M context, 128K max output)**

| Model ID              | Thinking | Reasoning Effort              | Price (Input/Output/Cache) | Use Case                                        |
| --------------------- | -------- | ------------------------------ | -------------------------- | ----------------------------------------------- |
| `gpt-5.5`             | ✅        | none/low/medium/high/xhigh     | $5.00/$30.00/$0.50         | New class of intelligence for coding and professional work; complex security research, advanced autonomous pentesting |
| `gpt-5.5-pro`         | ✅        | medium/high/xhigh              | $30.00/$180.00/$0.00       | Uses more compute for smarter, more precise responses; no cached-input discount; mission-critical security research, zero-day discovery |

**GPT-5.4 Series - Advanced Reasoning at Scale (1M context)**

| Model ID              | Thinking | Reasoning Effort         | Price (Input/Output/Cache) | Use Case                                        |
| --------------------- | -------- | -------------------------- | -------------------------- | ----------------------------------------------- |
| `gpt-5.4`             | ✅        | low/medium/high/xhigh      | $2.50/$15.00/$0.25          | Best intelligence at scale for agentic, coding, and professional workflows; maximum cognitive depth for pentesting |
| `gpt-5.4-mini`*       | ✅        | low/medium/high/xhigh      | $0.75/$4.50/$0.075          | Strongest mini model for coding, computer use, subagents (primary_agent/assistant/reflector/installer/pentester default) |
| `gpt-5.4-nano`*       | ✅        | low/medium/high/xhigh      | $0.20/$1.25/$0.02           | Cheapest GPT-5.4-class model for simple, high-volume tasks (simple/simple_json/searcher/enricher default) |

**GPT-5.2 Series - Previous Flagship Agentic**

| Model ID              | Thinking | Reasoning Effort       | Price (Input/Output/Cache) | Use Case                                        |
| --------------------- | -------- | ------------------------ | -------------------------- | ----------------------------------------------- |
| `gpt-5.2`             | ✅        | low/medium/high/xhigh    | $1.75/$14.00/$0.175        | Superseded by 5.4/5.6; autonomous security research, complex exploit chain development |
| `gpt-5.2-pro`         | ✅        | medium/high/xhigh        | $21.00/$168.00/$0.00       | Superior agentic coding and long-context performance, mission-critical security research, zero-day discovery |

**GPT-5/5.1 Series - Advanced Agentic Models**

| Model ID              | Thinking | Price (Input/Output/Cache) | Use Case                                        |
| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
| `gpt-5`               | ✅        | $1.25/$10.00/$0.125        | Autonomous security research, exploit chain development, coordinating multi-tool pentesting workflows |
| `gpt-5.1`             | ✅        | $1.25/$10.00/$0.125        | Bridges GPT-5 and GPT-5.2 with faster responses; balanced penetration testing with strong tool coordination |
| `gpt-5-pro`           | ✅ (high) | $15.00/$120.00/$0.00       | Reduced hallucinations, exceptional accuracy, critical security operations |
| `gpt-5-mini`          | ✅        | $0.25/$2.00/$0.025         | Automated vulnerability analysis, exploit generation with strong function calling |
| `gpt-5-nano`          | ✅        | $0.05/$0.40/$0.005         | High-throughput security scanning, reconnaissance, real-time monitoring |

**GPT-4.1 Series - Enhanced Intelligence (Non-Reasoning)**

| Model ID              | Thinking | Price (Input/Output/Cache) | Use Case                                        |
| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
| `gpt-4.1`             | ❌        | $2.00/$8.00/$0.50          | Superior function calling, complex threat analysis, sophisticated exploit development |
| `gpt-4.1-mini`        | ❌        | $0.40/$1.60/$0.10          | Routine security assessments, automated code analysis (no longer used in default configuration) |

**GPT-4o Series - Multimodal (Non-Reasoning)**

| Model ID              | Thinking | Price (Input/Output/Cache) | Use Case                                        |
| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
| `gpt-4o-mini`         | ❌        | $0.15/$0.60/$0.075         | Compact multimodal with strong function calling, high-frequency scanning, cost-effective bulk operations |

**o-Series - Advanced Reasoning Models (Current)**

| Model ID              | Thinking | Price (Input/Output/Cache) | Use Case                                        |
| --------------------- | -------- | -------------------------- | ----------------------------------------------- |
| `o3`                  | ✅        | $2.00/$8.00/$0.50          | Succeeded by GPT-5; multi-stage attack chains, deep vulnerability analysis |
| `o3-pro`              | ✅        | $20.00/$80.00/$0.00        | More compute for better responses; zero-day research, critical security investigations |

**Deprecated Models - Kept for Backward Compatibility ⚠️**

These models were marked deprecated by OpenAI. PentAGI keeps them defined only so that pre-existing agent configs pinned to these names keep working; do not assign them to new agents.

| Model ID              | Thinking | Price (Input/Output/Cache) | Notes                                        |
| --------------------- | -------- | -------------------------- | --------------------------------------------- |
| `gpt-5.2-codex`       | ✅        | $1.75/$14.00/$0.175        | Superseded code-specialized model; use `gpt-5.6-terra`/`gpt-5.4-mini` instead |
| `gpt-5.1-codex-max`   | ✅        | $1.25/$10.00/$0.125        | Superseded; enhanced reasoning for coding workflows |
| `gpt-5.1-codex`       | ✅        | $1.25/$10.00/$0.125        | Superseded standard code-optimized model |
| `gpt-5-codex`         | ✅        | $1.25/$10.00/$0.125        | Superseded foundational code-specialized model |
| `gpt-5.1-codex-mini`  | ✅        | $0.25/$2.00/$0.025         | Superseded compact code model |
| `codex-mini-latest`   | ✅        | $1.50/$6.00/$0.375         | Superseded compact code model |
| `gpt-4o`              | ❌        | $2.50/$10.00/$1.25         | Superseded by GPT-5.x/5.6 series multimodal flagship |
| `gpt-4.1-nano`        | ❌        | $0.10/$0.40/$0.025         | Superseded ultra-fast lightweight model |
| `o3-mini`             | ✅        | $1.10/$4.40/$0.55          | Superseded compact reasoning model |
| `o4-mini`             | ✅        | $1.10/$4.40/$0.275         | Succeeded by `gpt-5-mini` |
| `o1`                  | ✅        | $15.00/$60.00/$7.50        | Superseded premier reasoning model |
| `o1-pro`              | ✅        | $150.00/$600.00/$0.00      | Superseded, highest cost point of the o-series |

**Prices**: Per 1M tokens. Reasoning models include thinking tokens in output pricing.

> [!WARNING]
> **GPT-5/5.1/5.2 Models - Trusted Access Required**
>
> The original GPT-5, GPT-5.1, and GPT-5.2 models (`gpt-5`, `gpt-5.1`, `gpt-5.2`, `gpt-5-pro`, `gpt-5.2-pro`, and all deprecated Codex variants) work **unstably with PentAGI** and may trigger OpenAI's cybersecurity safety mechanisms without verified access. This does not affect the newer GPT-5.4/5.5/5.6 series used in PentAGI's default configuration below.
>
> **To use these models reliably:**
> 1. **Individual users**: Verify your identity at [chatgpt.com/cyber](https://chatgpt.com/cyber)
> 2. **Enterprise teams**: Request trusted access through your OpenAI representative
> 3. **Security researchers**: Apply for the [Cybersecurity Grant Program](https://openai.com/form/cybersecurity-grant-program/) (includes $10M in API credits)
>
> **Recommended alternatives without verification:**
> - Use PentAGI's defaults — `gpt-5.4-mini`/`gpt-5.4-nano`/`gpt-5.6-terra` — which work out of the box
> - Use `o3`/`o3-pro` for reasoning tasks
> - Use `gpt-4.1` series for general intelligence and function calling without reasoning

**Reasoning Configuration**:
- **Reasoning forced off by default**: every default agent assigned `gpt-5.4-mini` or `gpt-5.6-terra` (primary_agent, assistant, generator, refiner, adviser, reflector, coder, installer, pentester) sets `reasoning: {mode: off}` — this genuinely disables reasoning, it is not simply "low effort". PentAGI calls OpenAI exclusively through `/v1/chat/completions` (never `/v1/responses`), and this endpoint rejects requests that combine function tools with these models' default-on thinking; forcing thinking off is required for tool calls to work reliably (see the investigation notes in [`backend/pkg/providers/openai/config.yml`](backend/pkg/providers/openai/config.yml)).
- **No override needed for `gpt-5.4-nano`**: used for simple, simple_json, searcher, and enricher, this tier does not default to thinking on, so tools attach without conflict and no `reasoning` override is required.
- **Manual tuning available**: outside the default assignments, GPT-5.6/5.5/5.4/5.2 series models expose explicit reasoning effort levels (`low`/`medium`/`high`/`xhigh`, plus `none` on GPT-5.5) for custom agent configs that need variable reasoning depth with tool calling disabled or via `/v1/responses`.

**Key Features**:
- **Extended Reasoning**: GPT-5.4/5.5/5.6 and o-series models with chain-of-thought for complex security analysis
- **Agentic Intelligence**: GPT-5.4/5.5/5.6 series with enhanced tool integration, million-token context windows, and autonomous capabilities
- **Prompt Caching**: Cost reduction on repeated context (10-50% of input price)
- **Code Specialization**: Legacy Codex models remain available (deprecated) for vulnerability discovery and exploit development in pinned configs
- **Multimodal Support**: `gpt-4o-mini` for vision-based security assessments
- **Tool Calling**: Robust function calling across all models for pentesting tool orchestration
- **Streaming**: Real-time response streaming for interactive workflows
- **Proven Track Record**: Industry-leading models with CVE discoveries and real-world security applications

### Anthropic Provider Configuration

PentAGI integrates with Anthropic's Claude models, featuring advanced extended thinking capabilities, exceptional safety mechanisms, and sophisticated understanding of complex security contexts with prompt caching.

#### Configuration Variables

| Variable               | Default                        | Description                    |
| ---------------------- | ------------------------------ | ------------------------------ |
| `ANTHROPIC_API_KEY`    |                                | API key for Anthropic services |
| `ANTHROPIC_SERVER_URL` | `https://api.anthropic.com/v1` | Anthropic API endpoint         |

#### Configuration Examples

```bash
# Basic Anthropic setup
ANTHROPIC_API_KEY=your_anthropic_api_key
ANTHROPIC_SERVER_URL=https://api.anthropic.com/v1

# Using with proxy for secure environments
ANTHROPIC_API_KEY=your_anthropic_api_key
PROXY_URL=http://your-proxy:8080

# Workload Identity Federation instead of a long-lived key (leave ANTHROPIC_API_KEY empty)
ANTHROPIC_FEDERATION_RULE_ID=fdrl_...
ANTHROPIC_ORGANIZATION_ID=00000000-0000-0000-0000-000000000000
ANTHROPIC_SERVICE_ACCOUNT_ID=svac_...
ANTHROPIC_WORKSPACE_ID=wrkspc_...
ANTHROPIC_IDENTITY_TOKEN_FILE=/var/run/secrets/anthropic.com/token
```

With [Workload Identity Federation](https://platform.claude.com/docs/en/manage-claude/workload-identity-federation), PentAGI exchanges the identity token your platform issues (Kubernetes, GitHub Actions, cloud IAM, any OIDC issuer) for a short-lived Claude API token and refreshes it on its own. The token file is re-read on every exchange and must be mounted into the pentagi container. A token carrying a `jti` claim, as Kubernetes and GitHub Actions tokens do, can be exchanged only once, so the file must hold a new token before each refresh: rotate it well within the lifetime of the minted token. `ANTHROPIC_IDENTITY_TOKEN` takes the token itself where the platform injects it as a variable, but it is read once and cannot be rotated, so it suits only runs shorter than the identity token's lifetime. A set `ANTHROPIC_API_KEY` wins over federation. See [backend/docs/config.md](backend/docs/config.md#workload-identity-federation) for every variable.

> [!NOTE]
> **Google Vertex AI for Claude models**
>
> PentAGI does not currently expose a dedicated Google Vertex AI configuration path for Anthropic Claude in `.env`. There is no separate Vertex AI API key field at this time, and the existing Anthropic variables (`ANTHROPIC_API_KEY`, `ANTHROPIC_SERVER_URL`) target the direct Anthropic API. Supported routes for Claude are:
>
> - **Direct Anthropic API**: `ANTHROPIC_API_KEY` and `ANTHROPIC_SERVER_URL` (see above).
> - **AWS Bedrock**: `BEDROCK_*` variables (see [AWS Bedrock Provider Configuration](#aws-bedrock-provider-configuration)).
>
> If you need to use Vertex AI today, the safest supported workaround is to expose Vertex AI through an OpenAI-compatible proxy or gateway that translates Vertex AI calls into the Chat Completions format while preserving the chat and tool-call behavior PentAGI relies on, then point the Custom LLM provider at that gateway via `LLM_SERVER_URL`, `LLM_SERVER_KEY`, and `LLM_SERVER_MODEL`. This path is only as reliable as the gateway you choose.

#### Supported Models

PentAGI supports 9 Claude models with tool calling, streaming, extended thinking, adaptive thinking, and prompt caching. Models marked with `*` are used in default configuration.

**Claude 5 Series - Newest Models (2026)**

| Model ID                 | Thinking | Release Date | Price (Input/Output/Cache R/W) | Use Case                                        |
| ------------------------ | -------- | ------------ | ------------------------------ | ----------------------------------------------- |
| `claude-sonnet-5`*       | ✅        | Jul 2026     | $3.00/$15.00/$0.30/$3.75       | Best combination of speed and intelligence for coding, agents, and professional work at scale. Adaptive thinking only (manual budget thinking rejected); sampling parameters not supported. Default model for primary agent, assistant, coder, adviser, installer, pentester |
| `claude-fable-5`         | ✅        | Jun 2026     | $10.00/$50.00/$1.00/$12.50     | Anthropic's most capable widely released model for long-running agents and the most demanding reasoning workloads. Adaptive thinking always on (budget thinking and an explicit disable are rejected); sampling parameters not supported |

**Claude 4 Series**

| Model ID                 | Thinking | Release Date | Price (Input/Output/Cache R/W) | Use Case                                        |
| ------------------------ | -------- | ------------ | ------------------------------ | ----------------------------------------------- |
| `claude-opus-4-8`*       | ✅        | May 2026     | $5.00/$25.00/$0.50/$6.25       | Flagship for coding, agents, and deep reasoning in enterprise security workflows. Adaptive thinking only — budget thinking and sampling params (temperature/top_p/top_k) are rejected. Default model for generator and refiner; most demanding exploit development and multi-stage attack simulation |
| `claude-opus-4-7`        | ✅        | Apr 2026     | $5.00/$25.00/$0.50/$6.25       | Advanced software engineering and long-running agentic security analysis. Adaptive thinking only (manual budget thinking rejected) |
| `claude-sonnet-4-6`      | ✅        | Feb 2026     | $3.00/$15.00/$0.30/$3.75       | Best speed/intelligence balance with adaptive thinking. Multi-phase security assessments, intelligent vulnerability analysis, real-time threat hunting |
| `claude-opus-4-6`        | ✅        | Feb 2026     | $5.00/$25.00/$0.50/$6.25       | Most intelligent model for autonomous agents and coding. Extended + adaptive thinking for complex exploit development, multi-stage attack simulation |
| `claude-haiku-4-5`*      | ❌        | Oct 2025     | $1.00/$5.00/$0.10/$1.25        | Fast and efficient model with exceptional function calling and low latency, no thinking support. Default model for simple, simple_json, reflector, searcher, enricher; high-frequency scanning, real-time monitoring, bulk automated testing |

**Legacy Models - Still Supported**

| Model ID                 | Thinking | Release Date | Price (Input/Output/Cache R/W) | Use Case                                        |
| ------------------------ | -------- | ------------ | ------------------------------ | ----------------------------------------------- |
| `claude-sonnet-4-5`      | ✅        | Sep 2025     | $3.00/$15.00/$0.30/$3.75       | State-of-the-art reasoning (superseded by sonnet-4-6/sonnet-5). Sophisticated penetration testing, advanced threat analysis |
| `claude-opus-4-5`        | ✅        | Nov 2025     | $5.00/$25.00/$0.50/$6.25       | Ultimate reasoning (superseded by opus-4-6/4-7/4-8). Critical security research, zero-day discovery, red team operations |

**Prices**: Per 1M tokens. Cache pricing includes both Read and Write costs.

**Extended Thinking Configuration** (default agent config, see [`backend/pkg/providers/anthropic/config.yml`](backend/pkg/providers/anthropic/config.yml)):
- **Generator / Refiner** (`claude-opus-4-8`): adaptive reasoning at `xhigh`/`high` effort for maximum reasoning depth on complex exploit development
- **Primary agent, assistant, coder, adviser, installer, pentester** (`claude-sonnet-5`): adaptive reasoning (adviser at `xhigh` effort) for balanced code analysis and vulnerability research
- **Reflector, searcher** (`claude-haiku-4-5`): fixed reasoning budget of 1024 tokens for focused reasoning on specific tasks
- **Simple, simple_json, enricher** (`claude-haiku-4-5`): no thinking, optimized for speed

**Key Features**:
- **Extended Thinking**: All Claude 4.5+ models support configurable chain-of-thought reasoning depths for complex security analysis
- **Adaptive Thinking**: Claude 4.6 series (Opus/Sonnet) dynamically adjusts reasoning depth based on task complexity; Claude Opus 4.7/4.8 and the Claude 5 series (Sonnet/Fable) are adaptive-thinking-only (manual budget thinking and sampling parameters are rejected with HTTP 400)
- **Prompt Caching**: Significant cost reduction with separate read/write pricing (10% read, 125% write of input)
- **Extended Context Window**: 200K tokens standard, up to 1M tokens (beta) for Claude Opus/Sonnet 4.6 for comprehensive codebase analysis
- **Tool Calling**: Robust function calling with exceptional accuracy for security tool orchestration
- **Streaming**: Real-time response streaming for interactive penetration testing workflows
- **Safety-First Design**: Built-in safety mechanisms ensuring responsible security testing practices
- **Multimodal Support**: Vision capabilities in latest models for screenshot analysis and UI security assessment
- **Constitutional AI**: Advanced safety training providing reliable and ethical security guidance

### Google AI (Gemini) Provider Configuration

PentAGI integrates with Google's Gemini models through the Google AI API, offering state-of-the-art multimodal reasoning capabilities with extended thinking and context caching.

#### Configuration Variables

| Variable            | Default                                     | Description                    |
| ------------------- | ------------------------------------------- | ------------------------------ |
| `GEMINI_API_KEY`    |                                             | API key for Google AI services |
| `GEMINI_SERVER_URL` | `https://generativelanguage.googleapis.com` | Google AI API endpoint         |

#### Configuration Examples

```bash
# Basic Gemini setup
GEMINI_API_KEY=your_gemini_api_key
GEMINI_SERVER_URL=https://generativelanguage.googleapis.com

# Using with proxy
GEMINI_API_KEY=your_gemini_api_key
PROXY_URL=http://your-proxy:8080
```

#### Supported Models

PentAGI lists 14 Gemini model IDs that responded through the tested API endpoint. Availability in the catalogue does not mean a model is suitable for every agent role: the saved-chain results below determine the bundled defaults. Models marked with `*` are used in `config.yml`; the [full replay matrix](MODEL_REPLAY_RESULTS.md) describes role and outcome limits.

| Model ID | Thinking | Context | Price (Input/Output/Cache) | Saved-chain result |
| --- | --- | --- | --- | --- |
| `gemini-3.8-flash` | ✅ | 1M | $0.75/$3.75/$0.075 | API responds; 10/10 saved generator calls were content-filtered |
| `gemini-3.7-flash` | ✅ | 1M | $0.75/$3.75/$0.075 | API responds; 10/10 saved generator calls were content-filtered; former default |
| `gemini-3.6-flash` | ✅ | 1M | $0.75/$3.75/$0.075 | API responds; 10/10 saved generator calls returned text without a tool |
| `gemini-3.5-flash` | ✅ | 1M | $1.5/$9/$0.15 | API responds; 10/10 saved generator calls returned text without a tool |
| `gemini-3.5-flash-lite`* | ✅ | 1M | $0.3/$2.5/$0.03 | Complex-role default; eight direct plans and two after a safe `memorist` continuation |
| `gemini-3.1-pro-preview` | ✅ | 1M | $2/$12/$0.2 | API responds; 10/10 saved generator calls returned text without a tool |
| `gemini-3.1-pro-preview-customtools` | ✅ | 1M | $2/$12/$0.2 | API responds; 10/10 saved generator calls returned text without a tool |
| `gemini-3.1-flash-lite`* | ✅ | 1M | $0.25/$1.5/$0.025 | Simple-role default; direct plan on all ten saved generator chains |
| `gemini-3-flash-preview` | ✅ | 1M | $0.5/$3/$0.05 | API responds; three direct plans and five generator timeouts |
| `gemini-2.5-pro` | ✅ | 1M | $1.25/$10/$0.125 | API responds; ten direct plans, but four role-smoke provider errors |
| `gemini-2.5-flash` | ✅ | 1M | $0.3/$2.5/$0.03 | Ten direct plans; access is limited for new API projects |
| `gemini-2.5-flash-lite` | ✅ | 1M | $0.1/$0.4/$0.01 | API responds; five of ten generator answers were empty |
| `gemma-4-31b-it` | ✅ | 256K | Free/Free/Free | API responds; seven direct plans and three generator timeouts |
| `gemma-4-26b-a4b-it` | ✅ | 256K | Free/Free/Free | Direct plan on all ten saved generator chains |

**Prices**: Per 1M tokens. Google [limits Gemini 2.5 access for new projects](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-pro), so PentAGI uses the tested 3.1 Flash-Lite as its fallback and simple-role default. Existing custom 2.5 configurations remain usable where the project has access.

**Default Model Assignments (config.yml)**:
- **`gemini-3.1-flash-lite`** - `simple`, `simple_json`, `reflector`, `searcher`, `enricher`
- **`gemini-3.5-flash-lite`** - `primary_agent`, `assistant`, `generator`, `refiner`, `adviser`, `coder`, `installer`, `pentester`

Both default models passed a basic API/configuration smoke for every role. [Google reports stronger agentic and coding performance for 3.5 Flash-Lite than 3.1 Flash-Lite](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-6-flash-3-5-flash-lite-3-5-flash-cyber/); PentAGI's saved production chains cover generator and reflector only, so the smoke does not establish semantic quality or multi-turn reliability for other roles. On two saved generator chains, 3.5 Flash-Lite first chose the allowed `memorist` tool and called `subtask_list` after a synthetic tool response; no real tool was executed in that continuation.

**Reasoning Effort Levels**:
- **High**: Deeper reasoning for planning and review (`primary_agent`, `generator`, `refiner`, `adviser`)
- **Medium**: Balanced reasoning for execution roles (`assistant`, `coder`, `installer`, `pentester`)
- **Not set**: No thinking setting is sent and the model's own default applies (`simple`, `simple_json`, `reflector`, `searcher`, `enricher`)

### AWS Bedrock Provider Configuration

PentAGI integrates with Amazon Bedrock, offering access to 20+ foundation models from leading AI companies including Anthropic, Amazon, Cohere, DeepSeek, OpenAI, Qwen, Mistral, and Moonshot.

#### Configuration Variables

| Variable                    | Default     | Description                                                                                         |
| --------------------------- | ----------- | --------------------------------------------------------------------------------------------------- |
| `BEDROCK_REGION`            | `us-east-1` | AWS region for Bedrock service                                                                      |
| `BEDROCK_DEFAULT_AUTH`      | `false`     | Use AWS SDK default credential chain (environment, EC2 role, ~/.aws/credentials) - highest priority |
| `BEDROCK_BEARER_TOKEN`      |             | Bearer token authentication - priority over static credentials                                      |
| `BEDROCK_ACCESS_KEY_ID`     |             | AWS access key ID for static credentials                                                            |
| `BEDROCK_SECRET_ACCESS_KEY` |             | AWS secret access key for static credentials                                                        |
| `BEDROCK_SESSION_TOKEN`     |             | AWS session token for temporary credentials (optional, used with static credentials)                |
| `BEDROCK_SERVER_URL`        |             | Custom Bedrock endpoint (VPC endpoints, local testing)                                              |
| `BEDROCK_CONFIG_PATH`       |             | Path to a custom YAML provider config file (overrides the built-in default config for model/pricing definitions) |

**Authentication Priority**: `BEDROCK_DEFAULT_AUTH` → `BEDROCK_BEARER_TOKEN` → `BEDROCK_ACCESS_KEY_ID`+`BEDROCK_SECRET_ACCESS_KEY`

#### Configuration Examples

```bash
# Recommended: Default AWS SDK authentication (EC2/ECS/Lambda roles)
BEDROCK_REGION=us-east-1
BEDROCK_DEFAULT_AUTH=true

# Bearer token authentication (AWS STS, custom auth)
BEDROCK_REGION=us-east-1
BEDROCK_BEARER_TOKEN=your_bearer_token

# Static credentials (development, testing)
BEDROCK_REGION=us-east-1
BEDROCK_ACCESS_KEY_ID=your_aws_access_key
BEDROCK_SECRET_ACCESS_KEY=your_aws_secret_key

# With proxy and custom endpoint
BEDROCK_REGION=us-east-1
BEDROCK_DEFAULT_AUTH=true
BEDROCK_SERVER_URL=https://bedrock-runtime.us-east-1.vpce-xxx.amazonaws.com
PROXY_URL=http://your-proxy:8080
```

#### Custom Provider Config and Models (advanced)

By default the Bedrock provider uses a per-agent config and model catalog compiled into the binary. One optional path overrides them without rebuilding:

- `BEDROCK_CONFIG_PATH` — a YAML file (same shape as the other provider configs) that **replaces** the built-in per-agent model/price assignments. See [`examples/configs/bedrock-glm-flash.provider.yml`](examples/configs/bedrock-glm-flash.provider.yml).

This is useful to expose a Bedrock model newer than the compiled-in catalog — for example Z.AI's `zai.glm-4.7-flash`. Use the exact Model ID from the model's AWS Bedrock detail page; add a `us.`/`eu.`/`apac.` inference-profile prefix only when that page marks the model as requiring cross-region inference (`zai.glm-4.7-flash` is In-Region, so it is used as-is, with no prefix).

With Docker Compose the file is mounted from the host at `/opt/pentagi/conf/bedrock.provider.yml`. The default source is `./example.bedrock.provider.yml` beside `docker-compose.yml`, a copy of the built-in config ([`examples/configs/bedrock.provider.yml`](examples/configs/bedrock.provider.yml)) that the installer extracts, so editing it and pointing the backend at the mounted path is enough:

```bash
# tell the backend to read the mounted file
BEDROCK_CONFIG_PATH=/opt/pentagi/conf/bedrock.provider.yml

# optional: mount another file from the host instead of ./example.bedrock.provider.yml
PENTAGI_BEDROCK_CONFIG_PATH=/path/on/host/my-bedrock.provider.yml
```

#### Supported Models

PentAGI supports 24 AWS Bedrock models with tool calling, streaming, and multimodal capabilities. Models marked with `*` are used in default configuration.

| Model ID                                         | Provider        | Thinking | Multimodal | Price (Input/Output) | Use Case                                |
| ------------------------------------------------ | --------------- | -------- | ---------- | -------------------- | --------------------------------------- |
| `us.amazon.nova-2-lite-v1:0`                     | Amazon Nova     | ❌        | ✅          | $0.33/$2.75          | Adaptive reasoning, efficient thinking  |
| `us.amazon.nova-premier-v1:0`                    | Amazon Nova     | ❌        | ✅          | $2.50/$12.50         | Complex reasoning, advanced analysis    |
| `us.amazon.nova-pro-v1:0`                        | Amazon Nova     | ❌        | ✅          | $0.80/$3.20          | Balanced accuracy, speed, cost          |
| `us.amazon.nova-lite-v1:0`                       | Amazon Nova     | ❌        | ✅          | $0.06/$0.24          | Fast processing, high-volume operations |
| `us.amazon.nova-micro-v1:0`                      | Amazon Nova     | ❌        | ❌          | $0.035/$0.14         | Ultra-low latency, real-time monitoring |
| `us.anthropic.claude-opus-4-8`                   | Anthropic       | ✅        | ✅          | $5.00/$25.00         | Flagship coding/agents/deep reasoning; adaptive thinking only (sampling params rejected) |
| `us.anthropic.claude-opus-4-7`                   | Anthropic       | ✅        | ✅          | $5.00/$25.00         | Advanced engineering, long-running agents; adaptive thinking only |
| `us.anthropic.claude-opus-4-6-v1`*               | Anthropic       | ✅        | ✅          | $5.00/$25.00         | World-class coding, enterprise agents   |
| `us.anthropic.claude-sonnet-4-6`                 | Anthropic       | ✅        | ✅          | $3.00/$15.00         | Frontier intelligence, enterprise scale |
| `us.anthropic.claude-opus-4-5-20251101-v1:0`     | Anthropic       | ✅        | ✅          | $5.00/$25.00         | Multi-day software development          |
| `us.anthropic.claude-haiku-4-5-20251001-v1:0`*   | Anthropic       | ✅        | ✅          | $1.00/$5.00          | Near-frontier performance, high speed   |
| `us.anthropic.claude-sonnet-4-5-20250929-v1:0`*  | Anthropic       | ✅        | ✅          | $3.00/$15.00         | Real-world agents, coding excellence    |
| `us.anthropic.claude-sonnet-4-20250514-v1:0`     | Anthropic       | ✅        | ✅          | $3.00/$15.00         | Balanced performance, production-ready  |
| `us.anthropic.claude-3-5-haiku-20241022-v1:0`    | Anthropic       | ❌        | ❌          | $0.80/$4.00          | Fastest model, cost-effective scanning  |
| `cohere.command-r-plus-v1:0`                     | Cohere          | ❌        | ❌          | $3.00/$15.00         | Large-scale operations, superior RAG    |
| `deepseek.v3.2`                                  | DeepSeek        | ❌        | ❌          | $0.58/$1.68          | Long-context reasoning, efficiency      |
| `openai.gpt-oss-120b-1:0`*                       | OpenAI (OSS)    | ✅        | ❌          | $0.15/$0.60          | Strong reasoning, scientific analysis   |
| `openai.gpt-oss-20b-1:0`                         | OpenAI (OSS)    | ✅        | ❌          | $0.07/$0.30          | Efficient coding, software development  |
| `qwen.qwen3-next-80b-a3b`                        | Qwen            | ❌        | ❌          | $0.15/$1.20          | Ultra-long context, flagship reasoning  |
| `qwen.qwen3-32b-v1:0`                            | Qwen            | ❌        | ❌          | $0.15/$0.60          | Balanced reasoning, research use cases  |
| `qwen.qwen3-coder-30b-a3b-v1:0`                  | Qwen            | ❌        | ❌          | $0.15/$0.60          | Vibe coding, natural-language first     |
| `qwen.qwen3-coder-next`                          | Qwen            | ❌        | ❌          | $0.45/$1.80          | Tool use, function calling optimized    |
| `mistral.mistral-large-3-675b-instruct`          | Mistral         | ❌        | ✅          | $4.00/$12.00         | Advanced multimodal, long-context       |
| `moonshotai.kimi-k2.5`                           | Moonshot        | ❌        | ✅          | $0.60/$3.00          | Vision, language, code in one model     |

**Prices**: Per 1M tokens. Models with thinking/reasoning support additional compute costs during reasoning phase.

#### Tested but Incompatible Models

Some AWS Bedrock models were tested but are **not supported** due to technical limitations:

| Model Family              | Reason for Incompatibility                                                                |
| ------------------------- | ----------------------------------------------------------------------------------------- |
| **GLM (Z.AI)**            | Tool calling format incompatible with Converse API (expects string instead of JSON)       |
| **AI21 Jamba**            | Severe rate limits (1-2 req/min) prevent reliable testing and production use              |
| **Meta Llama 3.3/3.1**    | Unstable tool call result processing, causes unexpected failures in multi-turn workflows  |
| **Mistral Magistral**     | Tool calling not supported by the model                                                   |
| **Moonshot K2-Thinking**  | Unstable streaming behavior with tool calls, unreliable in production                     |
| **Qwen3-VL**              | Unstable streaming with tool calling, multimodal + tools combination fails intermittently |

> [!IMPORTANT]
> **Rate Limits & Quota Management**
>
> Default AWS Bedrock quotas for Claude models are **extremely restrictive** (2-20 requests/minute for new accounts). For production penetration testing:
>
> 1. **Request quota increases** through AWS Service Quotas console for models you plan to use
> 2. **Use Amazon Nova models** - higher default quotas and excellent performance
> 3. **Enable provisioned throughput** for consistent high-volume testing
> 4. **Monitor usage** - AWS throttles aggressively at quota limits
>
> Without quota increases, expect frequent delays and workflow interruptions.

> [!WARNING]
> **Converse API Requirements**
>
> PentAGI uses Amazon Bedrock **Converse API** for unified model access. All supported models require:
>
> - ✅ Converse/ConverseStream API support
> - ✅ Tool use (function calling) for penetration testing workflows
> - ✅ Streaming tool use for real-time feedback
>
> Verify model capabilities at: [AWS Bedrock Model Features](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-supported-models-features.html)

**Key Features**:
- **Automatic Prompt Caching**: 40-70% cost reduction on repeated context (Claude 4.x models)
- **Extended Thinking**: Step-by-step reasoning for complex security analysis (Claude, DeepSeek R1, OpenAI GPT)
- **Multimodal Analysis**: Process screenshots, diagrams, video for comprehensive testing (Nova, Claude, Mistral, Kimi)
- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
- **Streaming**: Real-time response streaming for interactive security assessment workflows

### DeepSeek Provider Configuration

PentAGI integrates with DeepSeek, providing access to advanced AI models with strong reasoning, coding capabilities, and context caching at competitive prices.

#### Configuration Variables

| Variable              | Default Value              | Description                                         |
| --------------------- | -------------------------- | --------------------------------------------------- |
| `DEEPSEEK_API_KEY`    |                            | DeepSeek API key for authentication                 |
| `DEEPSEEK_SERVER_URL` | `https://api.deepseek.com` | DeepSeek API endpoint URL                           |
| `DEEPSEEK_PROVIDER`   |                            | Provider prefix for LiteLLM integration (optional)  |

#### Configuration Examples

```bash
# Direct API usage
DEEPSEEK_API_KEY=your_deepseek_api_key
DEEPSEEK_SERVER_URL=https://api.deepseek.com

# With LiteLLM proxy
DEEPSEEK_API_KEY=your_litellm_key
DEEPSEEK_SERVER_URL=http://litellm-proxy:4000
DEEPSEEK_PROVIDER=deepseek  # Adds prefix to model names (deepseek/deepseek-v4-flash) for LiteLLM
```

#### Supported Models

PentAGI supports 2 DeepSeek V4 models with tool calling, streaming, hybrid thinking/non-thinking modes, and context caching. Both models think by default and switch to non-thinking mode with `reasoning.mode: off`. Models marked with `*` are used in default configuration.

| Model ID              | Thinking | Max Output | Context | Price (Input/Output/Cache) | Use Case                                             |
| --------------------- | -------- | ---------- | ------- | -------------------------- | ---------------------------------------------------- |
| `deepseek-v4-flash`*  | ✅ hybrid | 384K       | 1M      | $0.14/$0.28/$0.0028        | Utility agents, general dialogue, fast tool calling  |
| `deepseek-v4-pro`*    | ✅ hybrid | 384K       | 1M      | $1.74/$3.48/$0.0145        | Advanced reasoning, complex logic, security analysis |

**Prices**: Per 1M tokens. Cache pricing applies to prompt tokens served from cache (input cache hit, reduced to 1/10 of launch price since 2026-04-26). Both models support hybrid thinking — `thinking` mode is enabled by default; set `reasoning.mode: off` (Reasoning Mode Off in the UI) to switch to non-thinking mode for faster/cheaper responses.

> **Pricing Note (deepseek-v4-pro)**: The 75% promotional discount on `deepseek-v4-pro` officially ended on 2026-05-31 15:59 UTC. The prices above reflect the standard post-promotional pricing. If you have legacy configurations using the discounted prices ($0.435/$0.87/$0.003625), update them to the current rates for accurate cost tracking.

> The legacy model names `deepseek-chat` and `deepseek-reasoner` are scheduled
> for deprecation by DeepSeek on 2026-07-24. Existing user configurations
> referencing the legacy names continue to work until then; the defaults above
> use the current V4 names. `deepseek-chat` maps to `deepseek-v4-flash`
> non-thinking mode; `deepseek-reasoner` maps to `deepseek-v4-flash` thinking mode.

**Default Agent Configuration**:

Strategy: prefer `deepseek-v4-flash` (12x cheaper input, 12x cheaper output) as the workhorse for utility/lightweight agents; reserve `deepseek-v4-pro` for complex multi-step reasoning. The `installer` agent runs on Flash with thinking enabled because environment setup tasks (shell commands, config edits) rarely require pro-level reasoning. Run A/B tests on your own workloads before promoting more agents to Pro.

| Agent Role                                  | Default Model        | Thinking | Reasoning Effort | Max Output | Temperature | Top P |
| ------------------------------------------- | -------------------- | -------- | ---------------- | ---------- | ----------- | ----- |
| Generator / Refiner                         | `deepseek-v4-pro`    | Enabled  | High             | 32768      | (auto)      | (auto) |
| Coder                                       | `deepseek-v4-pro`    | Enabled  | High             | 20480      | (auto)      | (auto) |
| Primary Agent / Assistant / Pentester       | `deepseek-v4-pro`    | Enabled  | High             | 16384      | (auto)      | (auto) |
| Adviser (mentor/planner)                    | `deepseek-v4-pro`    | Enabled  | High             | 8192       | (auto)      | (auto) |
| Installer                                   | `deepseek-v4-flash`  | Enabled  | High             | 12288      | (auto)      | (auto) |
| Reflector / Searcher / Enricher             | `deepseek-v4-flash`  | Disabled | —                | 4096       | 0.5         | 0.9   |
| Simple / Simple JSON                        | `deepseek-v4-flash`  | Disabled | —                | 2048       | 0.3         | 0.9   |

> **Note**: DeepSeek ignores `presence_penalty` and `frequency_penalty` on both models. While thinking runs, `temperature` has no effect and langchaingo drops it, but `top_p` is sent and applied (values below 0.95 are raised to 0.95); in non-thinking mode `top_p` is fixed at 1.0, so langchaingo drops it and sends `temperature` instead. The thinking agents leave both unset, shown as "(auto)" in the table above.

**Key Features**:
- **Hybrid Thinking Modes**: Switch between thinking (deep reasoning) and non-thinking (fast) modes via `reasoning.mode`
- **Automatic Prompt Caching**: Significant cost reduction on repeated context via cache-hit pricing (1/10 of launch price)
- **Extended Thinking**: Reinforcement learning CoT for complex security analysis (both V4 models)
- **Strong Coding**: Optimized for code generation and exploit development
- **Long Context**: 1M token context window with up to 384K output tokens
- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
- **Streaming**: Real-time response streaming for interactive workflows
- **Multilingual**: Strong Chinese and English support
- **Additional Features**: JSON Output, Chat Prefix Completion (beta), FIM/Fill-in-the-Middle Completion (non-thinking mode only)

**Concurrency Limits**: `deepseek-v4-flash`: 2500 concurrent requests; `deepseek-v4-pro`: 500 concurrent requests.

**LiteLLM Integration**: Set `DEEPSEEK_PROVIDER=deepseek` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

### GLM Provider Configuration

PentAGI integrates with GLM from Zhipu AI (Z.AI), providing advanced language models with MoE architecture, strong reasoning, and agentic capabilities developed by Tsinghua University.

#### Configuration Variables

| Variable          | Default Value                   | Description                                                |
| ----------------- | ------------------------------- | ---------------------------------------------------------- |
| `GLM_API_KEY`     |                                 | GLM API key for authentication                             |
| `GLM_SERVER_URL`  | `https://api.z.ai/api/paas/v4`  | GLM API endpoint URL (international)                       |
| `GLM_PROVIDER`    |                                 | Provider prefix for LiteLLM integration (optional)         |

#### Configuration Examples

```bash
# Direct API usage (international endpoint)
GLM_API_KEY=your_glm_api_key
GLM_SERVER_URL=https://api.z.ai/api/paas/v4

# Alternative endpoints
GLM_SERVER_URL=https://open.bigmodel.cn/api/paas/v4  # China
GLM_SERVER_URL=https://api.z.ai/api/coding/paas/v4   # Coding-specific

# With LiteLLM proxy
GLM_API_KEY=your_litellm_key
GLM_SERVER_URL=http://litellm-proxy:4000
GLM_PROVIDER=zai  # Adds prefix to model names (zai/glm-4) for LiteLLM
```

#### Supported Models

PentAGI supports 16 GLM models with tool calling, streaming, reasoning controls, and prompt caching. Models marked with `*` are used in the default configuration. GLM-5.3, GLM-5.3-Flash, and GLM-5.3-FlashX always reason and accept `low`, `high`, or `max` effort.

**GLM-5.x Series - Latest Generation**

| Model ID         | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case                                                            |
| ---------------- | -------- | ------- | ---------- | -------------------------- | ------------------------------------------------------------------- |
| `glm-5.3`*       | ✅ Always | 1M      | 128K       | $1.40/$4.40/$0.26          | Current flagship for complex software engineering, agent tasks, and vulnerability research |
| `glm-5.3-flash`* | ✅ Always | 1M      | 128K       | $0.15/$0.50/$0.03          | Cost-efficient native multimodal model for utility and latency-sensitive agents |
| `glm-5.3-flashx` | ✅ Always | 1M      | 128K       | $0.37/$1.25/$0.075         | Accelerated GLM-5.3-Flash endpoint delivering up to 200 output tokens per second |
| `glm-5.2`        | ✅ Hybrid | 1M      | 128K       | $1.40/$4.40/$0.26          | Previous flagship for long-horizon engineering tasks |
| `glm-5.1`        | ✅ Hybrid | 200K    | 128K       | $1.40/$4.40/$0.26          | Long-horizon tasks: 8h sustained autonomous execution, Claude Opus 4.6-aligned coding |
| `glm-5`          | ✅ Hybrid | 200K    | 128K       | $1.00/$3.20/$0.20          | Foundation for Agentic Engineering, MoE 744B/40B active, Claude Opus 4.5-level coding |

**GLM-4.7 Series - Premium with Interleaved Thinking**

| Model ID          | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case                                            |
| ----------------- | -------- | ------- | ---------- | -------------------------- | --------------------------------------------------- |
| `glm-4.7`         | ✅ Hybrid | 200K    | 128K       | $0.60/$2.20/$0.11          | Enhanced programming, stable multi-step reasoning   |
| `glm-4.7-flashx`  | ✅ Hybrid | 200K    | 128K       | $0.07/$0.40/$0.01          | Ultra-cheap with priority GPU, but lower RPM limits (avoid for high-frequency use) |
| `glm-4.7-flash`   | ✅ Hybrid | 200K    | 128K       | Free/Free/Free             | Free ~30B SOTA model, 1 concurrent request          |

**GLM-4.6 Series - Balanced with Auto-Thinking**

| Model ID  | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case                                          |
| --------- | -------- | ------- | ---------- | -------------------------- | ------------------------------------------------- |
| `glm-4.6` | ✅ Auto   | 200K    | 128K       | $0.60/$2.20/$0.11          | Balanced, streaming tool calls, token-efficient   |

**GLM-4.5 Series - Unified Reasoning/Coding/Agents**

| Model ID         | Thinking | Context | Max Output | Price (Input/Output/Cache) | Use Case                                          |
| ---------------- | -------- | ------- | ---------- | -------------------------- | ------------------------------------------------- |
| `glm-4.5`        | ✅ Auto   | 128K    | 96K        | $0.60/$2.20/$0.11          | Unified, MoE 355B/32B active                      |
| `glm-4.5-x`      | ✅ Auto   | 128K    | 96K        | $2.20/$8.90/$0.45          | Ultra-fast premium, lowest latency                |
| `glm-4.5-air`    | ✅ Auto   | 128K    | 96K        | $0.20/$1.10/$0.03          | Cost-effective MoE 106B/12B |
| `glm-4.5-airx`   | ✅ Auto   | 128K    | 96K        | $1.10/$4.50/$0.22          | Accelerated Air with priority GPU                 |
| `glm-4.5-flash`  | ✅ Auto   | 128K    | 96K        | Free/Free/Free             | Free with reasoning/coding/agents support         |

**GLM-4 Legacy - Dense Architecture**

| Model ID              | Thinking | Context | Max Output | Price (Input/Output) | Use Case                                      |
| --------------------- | -------- | ------- | ---------- | -------------------- | --------------------------------------------- |
| `glm-4-32b-0414-128k` | ❌        | 128K    | 16K        | $0.10/$0.10          | Ultra-budget dense 32B, parsing without reasoning |

**Prices**: Per 1M tokens. Cache pricing is for prompt cache hit; cache storage is currently free per Z.AI promotion. GLM-4-32B has no cache support.

**Default Agent Configuration**:

Strategy: `glm-5.3` for critical reasoning and `glm-5.3-flash` for inexpensive utility and latency-sensitive work. `glm-5.3-flashx` remains selectable but is not a default because not every compatible gateway deploys it.

| Agent Role                          | Default Model | Thinking | Temperature | Top P | Max Output |
| ----------------------------------- | ------------- | -------- | ----------- | ----- | ---------- |
| Generator / Refiner                 | `glm-5.3`       | Max      | 1.0         | 0.95  | 32768      |
| Coder                               | `glm-5.3`       | Max      | 1.0         | 0.95  | 20480      |
| Adviser / Pentester                 | `glm-5.3`       | Max      | 1.0         | 0.95  | 16384      |
| Primary Agent                       | `glm-5.3`       | High     | 1.0         | 0.95  | 16384      |
| Assistant / Installer               | `glm-5.3-flash` | High     | 1.0         | 0.95  | 16384      |
| Simple / Reflector                  | `glm-5.3-flash` | Low      | 1.0         | 0.95  | 8192       |
| Searcher / Enricher / Simple JSON   | `glm-5.3-flash` | Low      | 1.0         | 0.95  | 4096       |

All default roles use Z.AI's recommended `temperature: 1.0` and `top_p: 0.95`. The full provider replay passed 288 of 289 cases; the sole failure was a transient gateway timeout and its focused retry passed.

**Thinking Modes**:
- **Always on** (GLM-5.3, GLM-5.3-Flash, GLM-5.3-FlashX): Cannot be disabled; `reasoning.effort` accepts `low`, `high`, or `max`
- **Hybrid** (GLM-5.2 and earlier GLM-5.x, GLM-4.7): Explicit toggle via `reasoning.mode`
- **Auto** (GLM-4.6, GLM-4.5 series): Model automatically determines when reasoning is needed
- **Preserved Thinking** (Z.AI Coding capability): all thinking-enabled agents in PentAGI also pass `extra_body.thinking.clear_thinking: false` so that `reasoning_content` from previous assistant turns is retained across the conversation. This is required on the standard API endpoint (`/api/paas/v4`) — on the Coding Plan endpoint it would be enabled by default. Improves reasoning continuity and cache hit rates in multi-turn tool call chains.
- All thinking-enabled agents also pass `extra_body.tool_choice: auto` defensively

**Key Features**:
- **Long-Horizon Tasks**: GLM-5.1 supports 8-hour sustained autonomous execution, ideal for complex multi-stage agentic workflows
- **OpenClaw-Native Orchestration**: GLM-5-Turbo is specifically optimized for tool invocation, instruction following, and long-chain execution
- **Prompt Caching**: Significant cost reduction on repeated context (cached input pricing shown)
- **Ultra-Long Context**: 200K tokens for GLM-5.x/4.7/4.6 series
- **MoE Architecture**: Efficient 744B/40B active (GLM-5/5.1), 355B/32B (GLM-4.5), 106B/12B (GLM-4.5-Air)
- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
- **Streaming**: Real-time streaming with streaming tool calls support (GLM-4.6+)
- **Multilingual**: Exceptional Chinese and English NLP capabilities
- **Free Options**: GLM-4.7-Flash and GLM-4.5-Flash for prototyping and experimentation

**LiteLLM Integration**: Set `GLM_PROVIDER=zai` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

### Kimi Provider Configuration

PentAGI integrates with Kimi from Moonshot AI, providing ultra-long context models with multimodal capabilities perfect for analyzing extensive codebases and documentation.

#### Configuration Variables

| Variable           | Default Value                | Description                                         |
| ------------------ | -----------------------------| --------------------------------------------------- |
| `KIMI_API_KEY`     |                              | Kimi API key for authentication                     |
| `KIMI_SERVER_URL`  | `https://api.moonshot.ai/v1` | Kimi API endpoint URL (international)               |
| `KIMI_PROVIDER`    |                              | Provider prefix for LiteLLM integration (optional)  |

#### Configuration Examples

```bash
# Direct API usage (international endpoint)
KIMI_API_KEY=your_kimi_api_key
KIMI_SERVER_URL=https://api.moonshot.ai/v1

# Alternative endpoint
KIMI_SERVER_URL=https://api.moonshot.cn/v1  # China

# With LiteLLM proxy
KIMI_API_KEY=your_litellm_key
KIMI_SERVER_URL=http://litellm-proxy:4000
KIMI_PROVIDER=moonshot  # Adds prefix to model names (moonshot/kimi-k3) for LiteLLM
```

#### Supported Models

PentAGI supports the four current Kimi models with tool calling, streaming, context caching, and multimodal input. Kimi K2.5, the older K2 series, `kimi-latest`, `kimi-thinking-preview`, and all Moonshot V1 models are retired and not included. Models marked with `*` are used in the default configuration.

**Kimi K3 - Flagship (Always Reasoning)**

| Model ID   | Thinking | Multimodal | Context | Price (Input Miss / Output / Cache Hit) | Use Case                                                |
| ---------- | -------- | ---------- | ------- | --------------------------------------- | ------------------------------------------------------- |
| `kimi-k3`* | ✅ always | ✅          | 1M      | $3.00 / $15.00 / $0.30                  | Flagship for long-horizon coding and knowledge work; used by adviser with `reasoning_effort: max` |

**Kimi K2.7 Code Series - Coding-Focused**

| Model ID                    | Thinking | Multimodal | Context | Price (Input Miss / Output / Cache Hit) | Use Case                                                |
| ---------------------------- | -------- | ---------- | ------- | --------------------------------------- | ------------------------------------------------------- |
| `kimi-k2.7-code`*            | ✅ always | ✅          | 256K    | $0.95 / $4.00 / $0.19                   | Coding-focused model used by generator, refiner, coder, and pentester |
| `kimi-k2.7-code-highspeed`*  | ✅ always | ✅          | 256K    | $1.90 / $8.00 / $0.38                   | Same model as `kimi-k2.7-code` with higher output throughput (~180-260 tokens/s) (primary_agent/assistant default) |

**Kimi K2.6 - Multimodal Model**

| Model ID         | Thinking | Multimodal | Context | Price (Input Miss / Output / Cache Hit) | Use Case                                                |
| ---------------- | -------- | ---------- | ------- | --------------------------------------- | ------------------------------------------------------- |
| `kimi-k2.6`*     | ✅ hybrid | ✅          | 256K    | $0.95 / $4.00 / $0.16                   | Utility and installer model with optional thinking |

**Prices**: Per 1M tokens. K3 cache writes cost $3.00 with the default five-minute TTL or $6.00 with a one-hour TTL; cache hits cost $0.30. K2.7 and K2.6 table prices distinguish cache misses from cache hits.

> **CRITICAL — Kimi parameter constraints per model family**: API returns `invalid_request_error` for any deviation:
> - `kimi-k3`: always reasons, no `thinking` param at all; reasoning depth is set via the top-level `reasoning_effort` field (`low`/`high`/`max`, default `max`) — PentAGI pins it to `max` for all agents using this model. `temperature` MUST be `1.0`, `top_p` MUST be `0.95`, `n` MUST be `1`, `presence_penalty`/`frequency_penalty` MUST be `0`. Do not switch effort per call — it invalidates the prefix cache.
> - `kimi-k2.7-code` / `kimi-k2.7-code-highspeed`: `thinking` may be omitted; if set explicitly, only `{"type":"enabled","keep":"all"}` is accepted (`type: disabled` is rejected). `reasoning_effort` is not supported. `temperature` MUST be `1.0`, `top_p` MUST be `0.95`, `n` MUST be `1`; `tool_choice: required` is not supported (use `auto`).
> - `kimi-k2.6`: thinking mode needs `temperature=1.0`, `top_p=0.95`, `n=1`, `thinking.keep="all"`; non-thinking mode needs `temperature=0.6`, `top_p=0.95`, `n=1`.
> - All Kimi models: `presence_penalty=0`, `frequency_penalty=0`, `tool_choice` in `{auto, none}`.

**Default Agent Configuration**:

Strategy: `kimi-k2.6` handles utility and installer work, `kimi-k2.7-code-highspeed` serves the primary/assistant loop, `kimi-k2.7-code` serves tool-heavy generator/refiner/coder/pentester roles, and `kimi-k3` handles adviser reasoning. This configuration passed all 294 conformance cases.

| Agent Role                                   | Default Model               | Thinking              | Temperature | Top P | Max Output |
| -------------------------------------------- | --------------------------- | --------------------- | ----------- | ----- | ---------- |
| Adviser (mentor/planner)                     | `kimi-k3`                   | Always (effort=max)   | 1.0         | 0.95  | 8192       |
| Generator / Refiner                          | `kimi-k2.7-code`            | Always (keep=all)     | 1.0         | 0.95  | 32768      |
| Coder                                        | `kimi-k2.7-code`            | Enabled (keep=all)    | 1.0         | 0.95  | 20480      |
| Pentester                                    | `kimi-k2.7-code`            | Enabled (keep=all)    | 1.0         | 0.95  | 16384      |
| Primary Agent / Assistant                    | `kimi-k2.7-code-highspeed`  | Enabled (keep=all)    | 1.0         | 0.95  | 16384      |
| Installer                                    | `kimi-k2.6`                 | Enabled               | 1.0         | 0.95  | 16384      |
| Reflector / Searcher / Enricher              | `kimi-k2.6`                 | Disabled              | 0.6         | 0.95  | 4096       |
| Simple                                       | `kimi-k2.6`                 | Disabled              | 0.6         | 0.95  | 8192       |
| Simple JSON                                  | `kimi-k2.6`                 | Disabled              | 0.6         | 0.95  | 4096       |

**Key Features**:
- **Always-On Reasoning Flagship**: `kimi-k3` never disables thinking and offers a 1M token context for the most demanding long-horizon coding and knowledge work
- **Ultra-Long Context**: Up to 256K tokens (K2.7/K2.x) or 1M tokens (K3) for comprehensive codebase/documentation analysis
- **Native Multimodal**: K3, K2.7, and K2.6 support text, image, and video input
- **Hybrid Thinking**: K2.6 can toggle between thinking and non-thinking via `reasoning.mode`; K3 and K2.7 always reason
- **Preserved Thinking** (K2.7, K2.6): `thinking.keep: "all"` preserves historical `reasoning_content` across turns — required for multi-turn tool call chains
- **Automatic Context Caching**: K3/K2.7/K2.x models cache repeated prefixes
- **Tool Calling**: Full function-calling support for K3, K2.7, K2.x, and Moonshot V1
- **Coding-Optimized Variants**: `kimi-k2.7-code`/`kimi-k2.7-code-highspeed` target higher success rates on long-context programming tasks, with the highspeed variant tuned for throughput
- **Multilingual**: Strong Chinese, English, and multi-language support

**Multi-turn with thinking + tool calls**: PentAGI's universal reasoning preservation pattern (`TextPartWithReasoning` + `WithPreserveReasoningContent`) automatically ensures `reasoning_content` is sent back in the required TextContent → ToolCall order, satisfying Moonshot's "thinking is enabled but reasoning_content is missing in assistant tool call message" requirement.

**LiteLLM Integration**: Set `KIMI_PROVIDER=moonshot` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

### Qwen Provider Configuration

PentAGI uses one OpenAI-compatible door for both Qwen Cloud and Alibaba Cloud Model Studio (DashScope). The bundled catalogue is the union of their agent-capable Chat Completions models, while the default role configuration uses only model IDs available on both platforms.

#### Configuration Variables

| Variable           | Default Value                                          | Description                                         |
| ------------------ | ------------------------------------------------------ | --------------------------------------------------- |
| `QWEN_API_KEY`     |                                                        | Qwen Cloud, DashScope, or gateway API key           |
| `QWEN_SERVER_URL`  | `https://dashscope-us.aliyuncs.com/compatible-mode/v1` | Direct API or OpenAI-compatible gateway base URL    |
| `QWEN_PROVIDER`    |                                                        | Gateway route prefix: `qwen_cloud` or `dashscope`   |

#### Configuration Examples

```bash
# Direct Alibaba Cloud Model Studio usage (Global/US endpoint)
QWEN_API_KEY=your_qwen_api_key
QWEN_SERVER_URL=https://dashscope-us.aliyuncs.com/compatible-mode/v1
QWEN_PROVIDER=

# Alternative endpoints
QWEN_SERVER_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1  # International (Singapore)
QWEN_SERVER_URL=https://dashscope.aliyuncs.com/compatible-mode/v1       # Chinese Mainland (Beijing)

# Qwen Cloud through LiteLLM
QWEN_API_KEY=your_litellm_key
QWEN_SERVER_URL=http://litellm-proxy:4000
QWEN_PROVIDER=qwen_cloud

# Alibaba Cloud through LiteLLM
QWEN_PROVIDER=dashscope
```

#### Supported Models

PentAGI ships 50 text, reasoning, coding, and vision-language catalogue entries available from at least one of the two platforms. Image generation, video, speech, realtime, translation-only, embedding, reranking, and decision APIs use different protocols or billing units and are not exposed as PentAGI chat models. Prices below are Alibaba Cloud international list rates per 1M tokens where Alibaba offers the model; Qwen Cloud can bill differently.

**Cross-platform model IDs used for portable configuration**

| Availability | Models |
| ------------ | ------ |
| Qwen Cloud and DashScope | `qwen3.8-max`, `qwen3.8-flash`, `qwen3.7-plus`, `qwen3.7-max`, `qwen3.6-flash`, `deepseek-v4.1-flash`, `deepseek-v4-pro` |
| Qwen Cloud only | `deepseek-v4-pro-0813` |
| DashScope only | All other entries in the bundled catalogue |

**Flagship Models (Top-tier Reasoning)**

| Model ID                     | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3.8-max`                | ✅        | —    | —         | —     | $2.00/$6.00/$0.25          | Newest flagship succeeding Qwen3.7-Max                  |
| `qwen3.8-omni-flash`         | ✅        | ✅    | ✅         | ✅     | $0.15/$0.47/—              | Current multimodal Omni tier                            |
| `qwen3.7-max`                | ✅        | ✅    | ✅         | ✅     | $2.50/$7.50/$0.50          | Next-gen flagship for agent-centric era                 |
| `qwen3.6-max-preview`        | ✅        | ✅    | ✅         | ✅     | $1.30/$7.80/$0.13          | Preview Max with enhanced vibe coding & front-end skills |
| `qwen3-max`                  | ✅        | ✅    | ✅         | ✅     | $1.20/$6.00/$0.24          | Previous-gen flagship with agent programming upgrades   |
| `qwen-max`                   | ❌        | ✅    | ✅         | ✅     | $1.60/$6.40/—              | Legacy 32K text flagship                                |
| `qwen-plus`                  | ✅        | ✅    | ✅         | ✅     | $0.40/$4.00/$0.08          | Qwen3-backbone Plus with switchable thinking modes      |

**Balanced Plus Models (Mid-tier)**

| Model ID                     | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3.7-plus`               | ✅        | —    | —         | —     | $0.40/$1.60/$0.08          | Cost-efficient tier of the Qwen3.7 generation           |
| `qwen3.6-plus`               | ✅        | ✅    | ✅         | ✅     | $0.50/$3.00/$0.05          | Native VL Plus with agentic coding                      |
| `qwen3.5-plus`               | ✅        | ✅    | ✅         | ✅     | $0.40/$2.40/$0.04          | Previous-gen native VL with strong multimodal capabilities |

**Fast Flash Models (Cost-optimized)**

| Model ID                     | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3.8-flash`              | ✅        | —    | —         | —     | $0.15/$0.47/$0.016         | Cost-efficient tier of the Qwen3.8 generation           |
| `qwen3.7-flash`              | ✅        | —    | —         | —     | $0.03/$0.13/$0.02          | Low-latency tier of the Qwen3.7 generation              |
| `qwen3.6-flash`*             | ✅        | ✅    | ✅         | ✅     | $0.25/$1.50/$0.05          | Shared utility model                                    |
| `qwen3.5-flash`              | ✅        | ✅    | ✅         | ✅     | $0.10/$0.40/$0.01          | Ultra-fast lightweight                                  |
| `qwen-flash`                 | ✅        | ✅    | ✅         | ✅     | $0.05/$0.40/$0.01          | Qwen3-series Flash with 1M context, tiered pricing      |

**Code-Specialized Models**

| Model ID                     | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3-coder-plus`           | ❌        | ✅    | ✅         | ✅     | $1.00/$5.00/$0.20          | Strong coding agent with autonomous programming         |
| `qwen3-coder-flash`          | ❌        | ✅    | ✅         | ✅     | $0.30/$1.50/$0.06          | Fast code-gen with multi-turn tool stability            |
| `qwen3-coder-next`           | ❌        | ✅    | ✅         | ✅     | $0.30/$1.50/—              | Open-source code generation, SOTA at same scale         |

**Vision-Language Models (Browser & Screenshot Analysis)**

| Model ID                     | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3-vl-plus`              | ✅        | ✅    | ✅         | ✅     | $0.20/$1.60/$0.04          | VL with visual agent capabilities, ultra-long video understanding |
| `qwen3-vl-flash`             | ✅        | ✅    | ✅         | ✅     | $0.05/$0.40/$0.01          | Small VL with 2D/3D localization for browser triage     |
| `qvq-max`                    | ✅        | ✅    | ✅         | ✅     | $1.20/$4.80/—              | Visual reasoning with chain-of-thought                  |
| `qwen3-vl-235b-a22b-thinking` | ✅        | —    | —         | —     | $0.40/$4.00/—              | Open-source 235B MoE VL (~22B active) with reasoning    |
| `qwen3-vl-235b-a22b-instruct` | ❌        | —    | —         | —     | $0.40/$1.60/—              | Open-source 235B MoE VL (~22B active), instruction-following |
| `qwen3-vl-32b-instruct`      | ❌        | —    | —         | —     | $0.16/$0.64/—              | Open-source 32B dense VL, instruction-following         |
| `qwen3-vl-30b-a3b-thinking`  | ✅        | —    | —         | —     | $0.20/$2.40/—              | Open-source 30B MoE VL (~3B active) with reasoning      |
| `qwen3-vl-30b-a3b-instruct`  | ❌        | —    | —         | —     | $0.20/$0.80/—              | Open-source 30B MoE VL (~3B active), instruction-following |

**Open-Source Qwen3.8 Series**

| Model ID                     | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3.8-2.4t-a95b`          | ✅        | —    | —         | —     | $2.00/$6.00/—              | Open-source 3.8 flagship, always reasons                |
| `qwen3.8-27b`                | ✅        | —    | —         | —     | $0.50/$3.00/$0.10          | Dense open-weight 3.8 tier, hybrid thinking, 1M context |

**Open-Source Qwen3.6 Series**

| Model ID                     | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3.6-27b`                | ✅        | ✅    | ✅         | ✅     | $0.60/$3.60/—              | Native VL on hybrid architecture, on-premises ready     |
| `qwen3.6-35b-a3b`            | ✅        | ✅    | ✅         | ✅     | $0.25/$1.49/—              | Efficient 35B MoE (~3B active) for continuous monitoring |

**Open-Source Qwen3.5 Series**

| Model ID                     | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ---------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3.5-397b-a17b`          | ✅        | ✅    | ✅         | ✅     | $0.60/$3.60/—              | Largest 397B params (~17B active), exceptional reasoning |
| `qwen3.5-122b-a10b`          | ✅        | ✅    | ✅         | ✅     | $0.40/$3.20/—              | Large 122B params (~10B active), strong balance         |
| `qwen3.5-35b-a3b`            | ✅        | ✅    | ✅         | ✅     | $0.25/$2.00/—              | Efficient 35B MoE (~3B active), cost-effective          |
| `qwen3.5-27b`                | ✅        | ✅    | ✅         | ✅     | $0.30/$2.40/—              | Medium 27B with hybrid linear attention + sparse MoE    |

**Open-Source Qwen3 Coder Series**

| Model ID                              | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ------------------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3-coder-480b-a35b-instruct`      | ❌        | ✅    | ✅         | ✅     | $1.50/$7.50/—              | Largest open coder MoE (480B/~35B active)               |
| `qwen3-coder-30b-a3b-instruct`        | ❌        | ✅    | ✅         | ✅     | $0.45/$2.25/—              | Efficient 30B MoE (~3B active), repository-scale        |

**Open-Source Qwen3 Dense & MoE Series**

| Model ID                              | Thinking | Intl | Global/US | China | Price (Input/Output/Cache) | Use Case                                                |
| ------------------------------------- | -------- | ---- | --------- | ----- | -------------------------- | ------------------------------------------------------- |
| `qwen3-next-80b-a3b-thinking`         | ✅        | ✅    | ✅         | ✅     | $0.15/$1.20/—              | Next-gen 80B MoE (~3B active) thinking-only             |
| `qwen3-next-80b-a3b-instruct`         | ❌        | ✅    | ✅         | ✅     | $0.15/$1.20/—              | Next-gen 80B MoE instruction-following                  |
| `qwen3-235b-a22b-thinking-2507`       | ✅        | —    | —         | —     | $0.23/$2.30/—              | 235B MoE (~22B active) thinking variant                 |
| `qwen3-30b-a3b-thinking-2507`         | ✅        | —    | —         | —     | $0.20/$2.40/—              | 30B MoE (~3B active) thinking variant                   |
| `qwen3-30b-a3b-instruct-2507`         | ❌        | —    | —         | —     | $0.20/$0.80/—              | 30B MoE (~3B active) non-thinking variant               |

**Third-Party Models**

These models share the OpenAI-compatible Qwen door. Availability differs by platform; the cross-platform table above is authoritative for portable role configuration.

| Model ID                              | Thinking | Context | Price (Input/Output/Cache) | Use Case                                                |
| ------------------------------------- | -------- | ------- | -------------------------- | ------------------------------------------------------- |
| `glm-5.3-prime`                       | ✅        | 1M      | $2.80/$8.80/—              | DashScope priority GLM route |
| `glm-5.2-fast-preview`                | ✅        | 1M      | $2.80/$8.80/—              | DashScope high-throughput GLM 5.2 route |
| `glm-5.2`                             | ✅        | 1M      | $1.40/$4.40/$0.28          | Zhipu AI GLM flagship route |
| `deepseek-v4.1-flash`*                | ✅        | 1M      | $0.15/$0.60/—              | Current shared low-latency DeepSeek model |
| `deepseek-v4-pro`*                    | ✅        | 1M      | $2.40/$4.80/$0.20          | Shared model for security and tool-heavy roles |
| `deepseek-v4-pro-0813`                | ✅        | 1M      | $2.40/$4.80/$0.20          | Qwen Cloud-only V4 Pro snapshot |
| `deepseek-v4-flash-0731`              | ✅        | 1M      | $0.44/$1.32/$0.088         | Legacy DashScope Flash snapshot |
| `kimi-k3`                             | ✅        | 1M      | $3.00/$15.00/$0.30         | Current Kimi flagship on DashScope |
| `kimi-k2.7-code`                      | ✅        | 262K    | $0.95/$4.00/$0.19          | Moonshot coding model, thinking always on, 16K output ceiling |

**Prices**: Per 1M tokens. Cache pricing reflects implicit cache hit (when available); MoE/dense open-source models do not expose cache pricing. Tiered models (Max/Plus) show lowest-tier pricing (typically ≤32k or ≤256k input); larger contexts incur higher rates per Alibaba Cloud pricing. `deepseek-v4-flash-0731` shows the busy-hours rate; idle hours bill half.

**Region Availability**:
- **Intl** (International): Singapore region (`dashscope-intl.aliyuncs.com`)
- **Global/US**: US Virginia region (`dashscope-us.aliyuncs.com`)
- **China**: Chinese Mainland Beijing region (`dashscope.aliyuncs.com`)
- **—**: availability in that region is not recorded here

**Default Agent Configuration**:
| Agent Role                                                                   | Default Model       | Tier      |
| ---------------------------------------------------------------------------- | ------------------- | --------- |
| `simple`, `simple_json`, `reflector`, `searcher`, `enricher`                 | `qwen3.6-flash`     | Utility   |
| `primary_agent`, `assistant`, `coder`, `installer`, `pentester`              | `deepseek-v4.1-flash` | Workhorse |
| `generator`, `refiner`, `adviser`                                            | `deepseek-v4-pro`   | Planning  |

The role configuration must run unchanged behind `qwen_cloud/` and `dashscope/`. Qwen 3.6 Flash handles utility calls, DeepSeek V4.1 Flash provides the lower-cost workhorse tier, and DeepSeek V4 Pro remains on planning and advice. Thinking controls are left at provider defaults because Qwen Cloud does not accept the same disable operation that DashScope supports.

**Key Features**:
- **Agent-Centric Design**: Qwen3.7-Max is purpose-built for long-horizon autonomous execution and tool invocation
- **Automatic Context Caching**: 30-50% cost reduction on repeated context with implicit cache
- **Extended Thinking**: Chain-of-thought reasoning for complex security analysis (Qwen3.7/3.6/3.5/3-Max, QVQ-Max)
- **Code Specialization**: Qwen3-Coder series with multi-turn tool interaction and repository-level understanding
- **Vision-Language**: Qwen3-VL series for browser screenshot triage, 2D/3D localization, OCR-level analysis
- **Tool Calling**: Seamless integration with 20+ pentesting tools via function calling
- **Streaming**: Real-time response streaming for interactive workflows
- **Multilingual**: Strong Chinese, English, and multi-language support
- **Open-Source Variants**: Dense and MoE models from 27B to 2.4T for on-premises/air-gapped deployments

**LiteLLM Integration**: Set `QWEN_PROVIDER=qwen_cloud` for Qwen Cloud routes or `QWEN_PROVIDER=dashscope` for Alibaba Cloud routes. Leave it empty for direct DashScope API usage.

#### Alternative Integrations

DashScope is fully OpenAI-compatible, so Qwen can also power two other PentAGI subsystems through the standard OpenAI client.

**As embedding provider** (`text-embedding-v4`, see [Alibaba Cloud Model Studio pricing](https://modelstudio.console.alibabacloud.com/ap-southeast-1?tab=doc#/doc/?type=model&url=prices)):

```bash
EMBEDDING_PROVIDER=openai
EMBEDDING_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1  # International (Singapore)
# EMBEDDING_URL=https://dashscope.aliyuncs.com/compatible-mode/v1     # Chinese Mainland
EMBEDDING_KEY=sk-*******
EMBEDDING_MODEL=text-embedding-v4
EMBEDDING_BATCH_SIZE=         # optional, default applies
EMBEDDING_STRIP_NEW_LINES=    # optional, default applies
```

> Note: the Global/US DashScope endpoint (`dashscope-us.aliyuncs.com`) does **not** expose embedding APIs — use the International or China endpoints for `text-embedding-v4`.

**As OpenAI-typed custom LLM provider**: instead of the dedicated `QWEN_*` variables, you can wire any Qwen chat model through PentAGI's custom OpenAI-compatible provider by pointing `OPENAI_SERVER_URL` (or a custom provider entry) to the DashScope `/compatible-mode/v1` endpoint and selecting the desired Qwen model name. Useful when you already manage all model traffic through a single OpenAI-shaped client (e.g. shared with LiteLLM/OneAPI proxies).

### MiniMax Provider Configuration

PentAGI integrates with MiniMax's M-series through the OpenAI-compatible `https://api.minimax.io/v1` endpoint: large-context agentic models with tool calling, JSON output, and streaming.

#### Configuration Variables

| Variable             | Default Value               | Description                                        |
| -------------------- | --------------------------- | -------------------------------------------------- |
| `MINIMAX_API_KEY`    |                             | MiniMax API key for authentication                 |
| `MINIMAX_SERVER_URL` | `https://api.minimax.io/v1` | MiniMax API endpoint URL                           |
| `MINIMAX_PROVIDER`   |                             | Provider prefix for LiteLLM integration (optional) |

#### Configuration Examples

```bash
# Direct API usage
MINIMAX_API_KEY=your_minimax_api_key
MINIMAX_SERVER_URL=https://api.minimax.io/v1

# With LiteLLM proxy
MINIMAX_API_KEY=your_litellm_key
MINIMAX_SERVER_URL=http://litellm-proxy:4000
MINIMAX_PROVIDER=minimax  # Adds prefix to model names (minimax/MiniMax-M3) for LiteLLM
```

#### Supported Models

PentAGI ships 3 MiniMax models with tool calling, JSON output, and streaming. `MiniMax-M3` is the default catalogue model; the bundled agent configuration uses `MiniMax-M2.7` for roles where conformance testing showed more reliable context retention and unified-diff generation.

| Model ID                 | Context | Price (Input/Output, ≤512K context) | Use Case                                                                                             |
| ------------------------ | ------- | ------------------------------------ | --------------------------------------------------------------------------------------------------- |
| `MiniMax-M3`*            | ~1M     | $0.30/$1.20 (2x above 512K tokens)   | Latest flagship for agentic reasoning, tool use, code generation, and long-context tasks            |
| `MiniMax-M2.7`           | 204K    | $0.30/$1.20                          | Reliable reasoning and coding model used by stateful and diff-producing agents                       |
| `MiniMax-M2.7-highspeed` | 204K    | $0.60/$2.40                          | Low-latency variant of M2.7 for fast-response scenarios                                              |

**LiteLLM Integration**: Set `MINIMAX_PROVIDER=minimax` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

### Mistral Provider Configuration

PentAGI talks to Mistral through the OpenAI-compatible `https://api.mistral.ai/v1` endpoint: the
Medium/Large/Small line, the Ministral small models, Codestral for code, and one Z.ai model Mistral
serves itself.

#### Configuration Variables

| Variable             | Default Value               | Description                                        |
| -------------------- | --------------------------- | -------------------------------------------------- |
| `MISTRAL_API_KEY`    |                             | Mistral API key for authentication                 |
| `MISTRAL_SERVER_URL` | `https://api.mistral.ai/v1` | Mistral API endpoint URL                           |
| `MISTRAL_PROVIDER`   |                             | Provider prefix for LiteLLM integration (optional) |

#### Configuration Examples

```bash
# Direct API usage
MISTRAL_API_KEY=your_mistral_api_key
MISTRAL_SERVER_URL=https://api.mistral.ai/v1

# With LiteLLM proxy
MISTRAL_API_KEY=your_litellm_key
MISTRAL_SERVER_URL=http://litellm-proxy:4000
MISTRAL_PROVIDER=mistral  # Adds prefix to model names (mistral/mistral-small-latest) for LiteLLM
```

#### Supported Models

PentAGI ships 6 Mistral models. The default configuration uses `mistral-small-latest` for utility and read-heavy agents, `mistral-large-latest` for primary and execution agents, and `mistral-medium-latest` for planning and advice. Small, Medium, and Large each have a 256K-token context window.

| Model ID                | Price (Input/Output) | Reasons | Use Case                                                                 |
| ----------------------- | -------------------- | ------- | ------------------------------------------------------------------------ |
| `mistral-small-latest`* | $0.15/$0.60          | yes     | Hybrid instruct/reasoning/coding model for utility and read-heavy agents |
| `mistral-medium-latest` | $1.50/$7.50          | yes     | Frontier-class multimodal model for planning and advice                  |
| `mistral-large-latest`  | $0.50/$1.50          | no      | General-purpose model for primary, assistant, coding, and pentesting     |
| `codestral-latest`      | $0.30/$0.90          | no      | Code completion and fill-in-the-middle                                   |
| `zai-glm-5-2`           | $1.40/$4.40          | yes     | Z.ai GLM 5.2 served by Mistral, public preview                           |
| `zai-glm-5-3`           | $1.40/$4.40          | yes     | Z.ai GLM 5.3 served by Mistral, public preview; thinking cannot be turned off |

**LiteLLM Integration**: Set `MISTRAL_PROVIDER=mistral` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.

### xAI (Grok) Provider Configuration

PentAGI talks to xAI through the OpenAI-compatible `https://api.x.ai/v1` endpoint: the Grok 4.x line with large context windows, plus a coding model.

#### Configuration Variables

| Variable         | Default Value           | Description                                        |
| ---------------- | ----------------------- | -------------------------------------------------- |
| `XAI_API_KEY`    |                         | xAI API key for authentication                     |
| `XAI_SERVER_URL` | `https://api.x.ai/v1`   | xAI API endpoint URL                               |
| `XAI_PROVIDER`   |                         | Provider prefix for LiteLLM integration (optional) |

#### Configuration Examples

```bash
# Direct API usage
XAI_API_KEY=your_xai_api_key
XAI_SERVER_URL=https://api.x.ai/v1

# With LiteLLM proxy
XAI_API_KEY=your_litellm_key
XAI_SERVER_URL=http://litellm-proxy:4000
XAI_PROVIDER=xai  # Adds prefix to model names (xai/grok-4.3) for LiteLLM
```

#### Supported Models

PentAGI lists seven xAI Chat Completions model IDs that responded through the tested API. Catalogue availability does not imply suitability as an agent default. The bundled configuration uses `grok-4.3` for tool-heavy roles, including `searcher`, and `grok-4.20-0309-non-reasoning` for `simple`, `simple_json`, `reflector`, and `enricher`; the exact bindings are in `backend/pkg/providers/xai/config.yml`. In saved generator chains, `grok-4.3` called `subtask_list` directly on all nine, whereas `grok-4.6` returned text without a tool call on all nine. The [full replay matrix](MODEL_REPLAY_RESULTS.md) records outcomes and limitations. The Responses-only multi-agent model is not listed because PentAGI agents supply their own client-side tools.

| Model ID | Context | Price (Input/Output) | Reasons | Saved-chain result |
| --- | --- | --- | --- | --- |
| `grok-4.7` | 500K | $2.00/$6.00 | yes | API responds; generator/reflector refusals make it unsuitable as a default |
| `grok-4.6` | 500K | $2.00/$6.00 | yes | API responds; all nine saved generator requests returned text without a tool |
| `grok-4.5` | 500K | $2.00/$6.00 | yes | API responds; eight of nine generator requests explicitly refused |
| `grok-4.3`* | 1M | $1.25/$2.50 | yes | Direct plan on all nine saved generator chains; tool-role default and fallback |
| `grok-4.20-0309-reasoning` | 1M | $1.25/$2.50 | yes | Four direct plans, three more after safe tool continuation, two pending |
| `grok-4.20-0309-non-reasoning`* | 1M | $1.25/$2.50 | no | Utility-role default; six plans after safe tool continuation, three pending |
| `grok-build-0.1` | 256K | $1.00/$2.00 | yes | Coding model; five plans after safe tool continuation, four pending |

> Note: prices are for prompts under 200K tokens; at or above that threshold xAI applies the higher long-context rate to every token in the request.

**LiteLLM Integration**: Set `XAI_PROVIDER=xai` to enable model name prefixing when using default PentAGI configurations with LiteLLM proxy. Leave empty for direct API usage.


## Advanced Setup

### Langfuse Integration

Langfuse provides advanced capabilities for monitoring and analyzing AI agent operations.

1. Configure Langfuse environment variables in existing `.env` file.

<details>
    <summary>Langfuse valuable environment variables</summary>

### Database Credentials
- `LANGFUSE_POSTGRES_USER` and `LANGFUSE_POSTGRES_PASSWORD` - Langfuse PostgreSQL credentials
- `LANGFUSE_CLICKHOUSE_USER` and `LANGFUSE_CLICKHOUSE_PASSWORD` - ClickHouse credentials
- `LANGFUSE_REDIS_AUTH` - Redis password

### Encryption and Security Keys
- `LANGFUSE_SALT` - Salt for hashing in Langfuse Web UI
- `LANGFUSE_ENCRYPTION_KEY` - Encryption key (32 bytes in hex)
- `LANGFUSE_NEXTAUTH_SECRET` - Secret key for NextAuth

### Admin Credentials
- `LANGFUSE_INIT_USER_EMAIL` - Admin email
- `LANGFUSE_INIT_USER_PASSWORD` - Admin password
- `LANGFUSE_INIT_USER_NAME` - Admin username

### API Keys and Tokens
- `LANGFUSE_INIT_PROJECT_PUBLIC_KEY` - Project public key (used from PentAGI side too)
- `LANGFUSE_INIT_PROJECT_SECRET_KEY` - Project secret key (used from PentAGI side too)

### S3 Storage
- `LANGFUSE_S3_ACCESS_KEY_ID` - S3 access key ID
- `LANGFUSE_S3_SECRET_ACCESS_KEY` - S3 secret access key

</details>

2. Enable integration with Langfuse for PentAGI service in `.env` file.

```bash
LANGFUSE_BASE_URL=http://langfuse-web:3000
LANGFUSE_PROJECT_ID= # default: value from ${LANGFUSE_INIT_PROJECT_ID}
LANGFUSE_PUBLIC_KEY= # default: value from ${LANGFUSE_INIT_PROJECT_PUBLIC_KEY}
LANGFUSE_SECRET_KEY= # default: value from ${LANGFUSE_INIT_PROJECT_SECRET_KEY}
```

3. Run the Langfuse stack:

```bash
curl -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose-langfuse.yml
docker compose -f docker-compose.yml -f docker-compose-langfuse.yml up -d
```

Visit [localhost:4000](http://localhost:4000) to access Langfuse Web UI with credentials from `.env` file:

- `LANGFUSE_INIT_USER_EMAIL` - Admin email
- `LANGFUSE_INIT_USER_PASSWORD` - Admin password

### Monitoring and Observability

For detailed system operation tracking, integration with monitoring tools is available.

1. Enable integration with OpenTelemetry and all observability services for PentAGI in `.env` file.

```bash
OTEL_HOST=otelcol:8148
```

2. Run the observability stack:

```bash
curl -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose-observability.yml
docker compose -f docker-compose.yml -f docker-compose-observability.yml up -d
```

Visit [localhost:3000](http://localhost:3000) to access Grafana Web UI.

> [!NOTE]
> If you want to use Observability stack with Langfuse, you need to enable integration in `.env` file to set `LANGFUSE_OTEL_EXPORTER_OTLP_ENDPOINT` to `http://otelcol:4318`.
>
> To run all available stacks together (Langfuse, Graphiti, and Observability):
>
> ```bash
> docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml up -d
> ```
>
> You can also register aliases for these commands in your shell to run it faster:
>
> ```bash
> alias pentagi="docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml"
> alias pentagi-up="docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml up -d"
> alias pentagi-down="docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml down"
> ```

### Knowledge Graph Integration (Graphiti)

> [!IMPORTANT]
> Graphiti is an optional **beta** integration and is disabled by default. Review [Limitations and Security](#limitations-and-security) before enabling it in production.

PentAGI integrates with [Graphiti](https://github.com/vxcontrol/pentagi-graphiti), a temporal knowledge graph system powered by Neo4j, to provide advanced semantic understanding and relationship tracking for AI agent operations. The vxcontrol fork provides custom entity and edge types that are specific to pentesting purposes.

#### What is Graphiti?

Graphiti asynchronously extracts structured knowledge from agent interactions and builds a graph of entities, relationships, evidence, and temporal context. PentAGI sends agent responses and tool executions to Graphiti and exposes the `graphiti_search` tool to enabled agents. Graphiti complements the primary pgvector memory; it does not replace it.

- **Semantic Memory**: Store and recall relationships between tools, targets, vulnerabilities, and techniques
- **Contextual Understanding**: Track how different pentesting actions relate to each other over time
- **Flow-Scoped Recall**: Reuse knowledge within the active flow without exposing data from other engagements by default
- **Advanced Querying**: Search temporal context, relationships, successful tools, recent episodes, and entities by type

When enabled, PentAGI captures agent responses, tool execution details, and flow/task/subtask context. Ingestion is asynchronous, so newly submitted events can take time to become searchable.

#### Deployment Modes and Enabling

Graphiti can run as the bundled Neo4j + Graphiti stack, as an external service, or remain disabled.

For the bundled stack, configure `.env`:

```bash
GRAPHITI_ENABLED=true
GRAPHITI_TIMEOUT=30
GRAPHITI_URL=http://graphiti:8000
GRAPHITI_LLM_CLIENT_TYPE=openai

# Reused by the Graphiti OpenAI preset
OPEN_AI_KEY=your_openai_api_key
OPEN_AI_SERVER_URL=https://api.openai.com/v1

# Bundled Neo4j
NEO4J_USER=neo4j
NEO4J_DATABASE=neo4j
NEO4J_PASSWORD=replace_with_a_strong_password
NEO4J_URI=bolt://neo4j:7687
```

Download the optional compose file and the Neo4j settings with the APOC plugin Graphiti needs when installing manually, then start both stacks:

```bash
curl -O https://raw.githubusercontent.com/vxcontrol/pentagi/master/docker-compose-graphiti.yml
mkdir -p neo4j/conf neo4j/plugins
for f in conf/neo4j.conf conf/apoc.conf plugins/apoc-5.26.19-core.jar; do
  curl -fsSL -o "neo4j/$f" "https://raw.githubusercontent.com/vxcontrol/pentagi/master/examples/neo4j/$f"
done
docker compose -f docker-compose.yml -f docker-compose-graphiti.yml up -d
```

The base stack must create the external `pentagi-network` before the Graphiti stack can start. The installer handles stack ordering automatically.

For an external Graphiti deployment, set `GRAPHITI_ENABLED=true` and point `GRAPHITI_URL` to its API. Do not start `docker-compose-graphiti.yml`; configure providers, embeddings, the graph database, and ingest tuning on the external service itself.

PentAGI enables its client only when both `GRAPHITI_ENABLED=true` and `GRAPHITI_URL` is non-empty. At startup it performs three health-check attempts with a two-second backoff. If they all fail, PentAGI logs a warning and continues with Graphiti disabled.

#### LLM Provider and Model Presets

`GRAPHITI_LLM_CLIENT_TYPE` selects one deployment-wide preset. Model names and call parameters are not environment variables; they live in `graphiti/<provider>.yaml`.

| Preset | Credentials and endpoint | Shipped main model |
| --- | --- | --- |
| `openai` | `OPEN_AI_KEY`, `OPEN_AI_SERVER_URL` | `openai/gpt-5-mini` |
| `gemini` | `GEMINI_API_KEY`, `GEMINI_SERVER_URL` | `gemini/gemini-3.5-flash-lite` |
| `custom` | `LLM_SERVER_KEY`, `LLM_SERVER_URL` | `Qwen/Qwen3.6-27B-FP8` |
| `litellm` | `GRAPHITI_LITELLM_API_KEY`, `GRAPHITI_LITELLM_BASE_URL` | `openrouter/openai/gpt-oss-20b` |

The Gemini preset uses Graphiti's LiteLLM/OpenAI-compatible client path. Point `GEMINI_SERVER_URL` at a compatible gateway if the native Gemini endpoint does not provide the required OpenAI-compatible API.

Each preset file must contain a matching `provider` plus `MODEL_NAME` and `SMALL_MODEL_NAME` mappings. The small model is used for reranking and lighter calls. Supported call settings include temperature, token limits, sampling and penalty parameters, JSON mode, reasoning effort, verbosity, pricing metadata, and provider-specific `extra_body`.

The installer copies [`examples/graphiti`](examples/graphiti) beside the installation as `./graphiti`. The compose mount is controlled by:

```bash
GRAPHITI_CONFIG_PATH=./graphiti
GRAPHITI_CONFIG_DIR=llm_configs
```

`GRAPHITI_CONFIG_PATH` may point directly to `./examples/graphiti` for development. `GRAPHITI_CONFIG_DIR=llm_configs` activates the mounted presets. If an older `.env` omits that variable, a newer compose file mounts an empty host directory at the unused `configs` path instead of hiding the presets built into the image.

> [!NOTE]
> `GRAPHITI_MODEL_NAME` is obsolete and ignored. Edit the active YAML preset instead, then restart the Graphiti container.

#### Graphiti Embedding Configuration

By default, Graphiti uses the active LLM preset's credentials and its default OpenAI embedding model. To use PentAGI's shared embedding endpoint explicitly:

```bash
GRAPHITI_SEPARATE_EMBEDDING=true
EMBEDDING_URL=https://embedding.example.com/v1
EMBEDDING_KEY=your_embedding_api_key
EMBEDDING_MODEL=openai/text-embedding-3-large
```

Graphiti's embedder is OpenAI-compatible. `EMBEDDING_PROVIDER` is used by PentAGI but is not passed to Graphiti, so a non-OpenAI-compatible embedding provider cannot be shared directly.

#### Ingestion and Extraction Tuning

The supplied defaults prioritize flow isolation and limit expensive extraction to useful events.

Ingest policy actions:

- `REJECT`: do not store the episode.
- `SKIP_LLM`: store the episode for retrieval but do not extract nodes or edges.
- `PROCESS`: store the episode and run full LLM extraction.

| Variable | Default | When to change it |
| --- | --- | --- |
| `GRAPHITI_INGEST_POLICY_RULES` | `{"graphiti_search":"REJECT","tool_execution_terminal":"PROCESS","tool_execution_file":"PROCESS"}` | Add narrow, case-insensitive name/source patterns when specific events need different handling |
| `GRAPHITI_INGEST_POLICY_FIELD` | `both` | Restrict matching to `name` or `source_description` only when event naming is controlled |
| `GRAPHITI_INGEST_POLICY_DEFAULT_ACTION` | `SKIP_LLM` | Use `PROCESS` only when every unmatched event justifies extraction cost |
| `GRAPHITI_INGEST_USE_GROUP_ACTORS` | `true` | Keep enabled to preserve FIFO ordering independently for each flow |
| `GRAPHITI_INGEST_WORKER_COUNT` | `16` | Raise for more concurrent flows when the LLM and database have capacity; lower to control load |
| `GRAPHITI_INGEST_LOCK_BY_GROUP_ID` | `true` | Used only in shared-pool mode; ignored when group actors are enabled |
| `GRAPHITI_INGEST_TASK_MAX_RETRIES` | `1` (`0`-`5`) | Increase for transient LLM/network failures |
| `GRAPHITI_INGEST_TASK_RETRY_DELAY_SEC` | `2.0` (`0.5`-`60`) | Increase when an upstream service needs more recovery time |
| `GRAPHITI_INGEST_TASK_TIMEOUT_SEC` | `0` (`0`-`3600`) | Set a finite value to prevent one stalled request from blocking a flow; `0` disables the timeout |
| `GRAPHITI_INGEST_QUEUE_MAX_SIZE` | `0` | Set a bound to return HTTP 429 instead of allowing an unlimited backlog |
| `GRAPHITI_INGEST_DEAD_LETTER_ENABLED` | `false` | Enable when failed episodes must be retained for operational review |

Extraction uses the following fallback order: full combined extraction (nodes, attributes, summaries, and edges in one call), regular combined extraction (nodes and edges), then separate node/edge extraction. Empty or failed combined results automatically fall back; these log messages are expected during normal operation.

| Variable | Default | Effect |
| --- | --- | --- |
| `GRAPHITI_TAXONOMY_LAYER_PROFILE` | `STRUCTURAL,EVIDENCE,PROGRESS,ATTEMPT` | Controls which edge classes appear in prompts and pass validation; `full`/`all` enables every class and `minimal` selects the core attack graph |
| `GRAPHITI_USE_COMBINED_FULL_EXTRACTION` | `true` | Enables the most compact single-call extraction path |
| `GRAPHITI_USE_COMBINED_EXTRACTION` | `true` | Enables the regular combined fallback |
| `GRAPHITI_COMBINED_FULL_GATING_ENABLED` | `true` | Skips expensive full extraction for low-signal administrative/search events |
| `GRAPHITI_COMBINED_DIAGNOSTIC_SAMPLES` | `false` | Includes content samples in diagnostics; keep disabled because pentest output can contain credentials |
| `GRAPHITI_ANCHOR_NODE_MODE` | `smart` | `smart` loads all key entities plus limited high-volume types; `limit` applies one total cap |
| `GRAPHITI_ANCHOR_NODE_LIMIT` | `25` (`1`-`500`) | Total anchor cap in `limit` mode |
| `GRAPHITI_ANCHOR_MASS_TYPE_LIMIT` | `10` (`1`-`100`) | Per-type cap in `smart` mode; `0` is invalid |
| `GRAPHITI_ANCHOR_QUERY_TIMEOUT` | `10` (`1`-`60`) | Bounds anchor lookup; timeout degrades gracefully to no anchors |

Anchors connect entities across episodes and are used by the separate extraction path. Combined extraction has already produced its edges and does not perform this anchor lookup.

These flags are passed as process environment variables by the bundled compose file. This is important for combined extraction because Graphiti reads those flags when Python modules are imported.

#### Runtime, Logging, and Neo4j

The values below are PentAGI's recommended `.env.example`/compose defaults, not the raw Graphiti image fallbacks. Running a freshly pulled image behind an old compose file can instead enable telemetry and global search, use one shared-pool worker with `PROCESS` as the unmatched ingest action, enable the full taxonomy, and disable combined extraction. Keep the image, compose file, `.env`, and presets in sync.

| Variable | Default | Guidance |
| --- | --- | --- |
| `GRAPHITI_CPUS`, `GRAPHITI_MEMORY` | `2.0`, `2G` | Container limits; raise together with concurrency only after observing CPU and memory pressure |
| `GRAPHITI_SEMAPHORE_LIMIT` | `20` | Limits parallel Graphiti coroutines; it is separate from ingest worker concurrency |
| `GRAPHITI_TELEMETRY_ENABLED` | `false` | Enables anonymous Graphiti telemetry when set to `true` |
| `GRAPHITI_LOG_LEVEL` | `INFO` | Use `DEBUG` temporarily; it can produce sensitive and high-volume output |
| `GRAPHITI_LOG_STDOUT` | `events` | `off`, `events`, or `full`; `events` is recommended for containers |
| `GRAPHITI_FLOW_LOGGER_WARN_COUNT` | `256` | Warns about growth of cached per-flow loggers; `0` disables the warning |
| `GRAPHITI_DEBUG_RUNTIME_RESOURCES` | `false` | Enables `/debug/runtime-resources`; expose only to trusted operators |
| `GRAPHITI_SEARCH_SCOPE` | `flowid` | Keep for flow/tenant isolation; `all` enables global searches and can expose other engagements |
| `GRAPHITI_LOG_FORMAT` | `json` | Reserved by the current deployment contract; the Graphiti logger does not yet apply it |
| `NEO4J_CPUS`, `NEO4J_MEMORY` | `4.0`, `4G` | Neo4j container limits; use `neo4j-admin server memory-recommendation --docker` for production sizing |
| `NEO4J_SHM_SIZE` | `4g` | `/dev/shm` limit; actual use counts toward the container memory limit |
| `NEO4J_NOFILE` | `65536` | Open-file soft/hard limit, suitable for many indexes and concurrent connections |
| `NEO4J_HEAP_INITIAL_SIZE`, `NEO4J_HEAP_MAX_SIZE`, `NEO4J_PAGECACHE_SIZE` | `2G`, `2G`, `1G` | JVM heap/page cache sizing; `NEO4J_CPUS`/`NEO4J_MEMORY` only cap the container, the JVM does not reliably size itself to fit inside that cap on its own. Defaults favor heap over page cache — suited to a low write-throughput deployment with occasional wide reads, since query execution/result materialization lives in heap while a small dataset is already comfortably held by 1G of page cache; re-run `neo4j-admin server memory-recommendation --docker` once real data volume is known |
| `NEO4J_TRANSACTION_MAX` | `1G` | Caps a single transaction's memory (`db.memory.transaction.max`) so one runaway/unbounded query (e.g. a Cartesian product or an unbounded variable-length path before a `LIMIT`) fails cleanly with an out-of-memory Cypher error instead of exhausting the whole heap and taking down every other query on the server |
| `NEO4J_BOLT_ADVERTISED_ADDRESS` | empty | Set only when Neo4j Browser and the Bolt connector are reverse-proxied on different public domains (e.g. behind Guarder with CORS/cookie-group support for cross-origin bolt access); format `host:port`. Left empty, Neo4j's discovery endpoint advertises whatever `Host` header the request arrived with, which is correct only when both share one domain |
| `NEO4J_HTTP_ADVERTISED_ADDRESS` | empty | Set only when a reverse proxy in front of Neo4j Browser strips or rewrites the `Host` header, making the dynamic Host-header-based advertised address incorrect; format `host:port`. Leave empty in the common case (proxy forwards `Host` unchanged) |

`NEO4J_USER`, `NEO4J_PASSWORD`, `NEO4J_URI`, and `NEO4J_DATABASE` configure the bundled connection. Neo4j Community Edition supports only its default database; do not configure a separate database name that requires Enterprise multi-database support.

The installer copies [`examples/neo4j`](examples/neo4j) beside the installation as `./neo4j`. It contains static, non-`.env`-tunable settings that don't have a `NEO4J_*` variable: `conf/neo4j.conf` and `conf/apoc.conf`, plus a version-pinned `plugins/apoc-*-core.jar`. The compose mount is controlled by:

```bash
NEO4J_DIR=./neo4j
```

`NEO4J_DIR` may point directly to `./examples/neo4j` for development. Both `conf/` and `plugins/` are mounted read-only; the stack still starts on Neo4j's built-in defaults (without APOC) if the directory is absent, since Docker creates an empty one automatically — but Graphiti writes relationships through APOC, so its graph then stays empty while PentAGI logs no error. Do not duplicate any `NEO4J_*` variable from the table above inside `conf/neo4j.conf` — the Neo4j Docker entrypoint always strips a matching line from the mounted file and re-appends the environment variable's value, so a duplicated setting in the file would be silently ignored.

The bundled stack currently wires Neo4j only. The Graphiti image contains FalkorDB support, but using it requires a separately configured deployment because the stock compose file does not expose `GRAPHITI_GRAPH_BACKEND` or `FALKORDB_*`.

#### Verification and Troubleshooting

Check service health, queue state, and logs:

```bash
docker compose -f docker-compose.yml -f docker-compose-graphiti.yml ps graphiti neo4j
docker compose -f docker-compose.yml -f docker-compose-graphiti.yml logs -f graphiti
curl -fsS http://localhost:8000/healthcheck
curl -fsS http://localhost:8000/queue-size
```

Neo4j Browser is available at `http://localhost:7474`; the Graphiti OpenAPI UI is at `http://localhost:8000/docs`. Both are bound to localhost by the stock compose file.

Common failures:

- A missing API key or base URL for the selected preset, a missing YAML file, or a YAML `provider` mismatch causes the Graphiti container to fail startup validation.
- `LLM_CLIENT_TYPE=openai` rejects local/custom model prefixes; use the `custom` preset for an OpenAI-compatible local server.
- In `flowid` search mode, requests without a group ID are rejected. PentAGI supplies the flow-derived group ID automatically.
- A bounded full queue returns HTTP 429. `/queue-size` reports waiting, processing, active-group, and dropped counters.
- Invalid retry, timeout, or anchor ranges fail startup rather than being silently normalized.

Update `.env`, `docker-compose-graphiti.yml`, the Graphiti image, and the `graphiti` preset directory together. Pulling only a new image can retain older compose defaults and silently change extraction behavior.

#### Limitations and Security

- Graphiti is beta and has no in-app graph explorer.
- One provider preset is active for the entire Graphiti deployment; it is not selected per PentAGI agent or flow.
- Graphiti extraction, reranking, and embeddings incur billing independently of the model used by the main PentAGI flow.
- Search is flow-scoped by default. Cross-flow reuse requires an explicit global-search design and must not be enabled on shared or multi-tenant deployments without additional isolation.
- The Graphiti HTTP API has no authentication layer in the bundled service. The stock compose binds it and Neo4j to `127.0.0.1`; secure external deployments with network controls and authentication at a trusted reverse proxy.
- Agent and tool output may contain credentials and exploitation evidence. Protect Neo4j data, logs, dead letters, diagnostics, and backups accordingly.
- If Graphiti is unavailable, PentAGI continues with its primary memory and vector store after logging the failed startup health check. Set `GRAPHITI_ENABLED=false` to disable the integration explicitly.

### GitHub and Google OAuth Integration

OAuth integration with GitHub and Google allows users to authenticate using their existing accounts on these platforms. This provides several benefits:

- Simplified login process without need to create separate credentials
- Enhanced security through trusted identity providers
- Access to user profile information from GitHub/Google accounts
- Seamless integration with existing development workflows

PentAGI uses `PUBLIC_URL` as the public origin/base URL for OAuth redirects. In the default deployment, both GitHub and Google callbacks are handled by:

```text
${PUBLIC_URL}/api/v1/auth/login-callback
```

For GitHub OAuth:

1. Create a new OAuth App in your GitHub account.
2. Set **Homepage URL** to your `PUBLIC_URL`.
3. Set **Authorization callback URL** to `${PUBLIC_URL}/api/v1/auth/login-callback`.
4. Add the client credentials to your `.env` file:

```bash
PUBLIC_URL=https://pentagi.example.com
OAUTH_GITHUB_CLIENT_ID=your_github_client_id
OAUTH_GITHUB_CLIENT_SECRET=your_github_client_secret
```

For Google OAuth:

1. Create OAuth credentials in your Google Cloud project.
2. Use the same callback endpoint: `${PUBLIC_URL}/api/v1/auth/login-callback`.
3. Add the client credentials to your `.env` file:

```bash
PUBLIC_URL=https://pentagi.example.com
OAUTH_GOOGLE_CLIENT_ID=your_google_client_id
OAUTH_GOOGLE_CLIENT_SECRET=your_google_client_secret
```

Make sure `PUBLIC_URL` matches the externally accessible HTTPS address of your PentAGI instance and does not include the callback path itself. If the URL configured in the OAuth provider does not exactly match the callback generated by PentAGI, the provider will reject the login attempt with a redirect URI mismatch error.

### Docker Image Configuration

PentAGI allows you to configure Docker image selection for executing various tasks. The system automatically chooses the most appropriate image based on the task type, but you can constrain this selection by specifying your preferred images:

| Variable                           | Default                | Description                                                 |
| ---------------------------------- | ---------------------- | ----------------------------------------------------------- |
| `PENTAGI_IMAGE`                    | `vxcontrol/pentagi:latest` | Docker image used for the main PentAGI application service |
| `DOCKER_DEFAULT_IMAGE`             | `debian:latest`        | Default Docker image for general tasks and ambiguous cases  |
| `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` | `vxcontrol/kali-linux` | Default Docker image for security/penetration testing tasks |
| `DOCKER_ALLOWED_IMAGES`            | empty                  | Comma-separated allow-list the selected image must belong to |
| `DOCKER_IMAGE_SELECTION_MODE`      | `llm`                  | `llm` lets the model pick the image, `fixed` always uses the pentest image |

`PENTAGI_IMAGE` changes the image used by the main `pentagi` service in `docker-compose.yml`. The `DOCKER_DEFAULT_IMAGE` and `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` variables only affect automatic worker image selection for task execution inside PentAGI. They do not rewrite the rest of the Compose stack, so services such as `pgvector`, `scraper`, and the optional `graphiti` stack still use the image references defined in the compose files.

`DOCKER_DEFAULT_IMAGE` and `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` are suggestions the model receives in the image selection prompt, not limits: on their own they do not stop a model from answering with some other image. Two settings turn them into limits:

- **`DOCKER_ALLOWED_IMAGES`** restricts the choice. An answer outside the list is discarded and the pentest image is used instead. An entry without a tag admits any tag of that repository, so `vxcontrol/kali-linux` also admits `vxcontrol/kali-linux:2026.1`. While the list is empty the model may pick any image, which is the default behaviour.
- **`DOCKER_IMAGE_SELECTION_MODE=fixed`** removes the choice altogether: the flow always runs in `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` and no model call is made. Use it when weaker or local models answer with the wrong environment.

Regardless of these settings, an answer that is not a usable image reference never reaches Docker: the flow falls back to the pentest image and the rejected answer is logged.

Restricting the choice is useful for:

- **Security Enforcement**: Restricting usage to only verified and trusted images
- **Environment Standardization**: Using corporate or customized images across all operations
- **Performance Optimization**: Utilizing pre-built images with necessary tools already installed

Configuration examples:

```bash
# Using a custom PentAGI application image
PENTAGI_IMAGE=registry.example.com/security/pentagi:latest

# Using a custom image for general tasks
DOCKER_DEFAULT_IMAGE=mycompany/custom-debian:latest

# Using a specialized image for penetration testing
DOCKER_DEFAULT_IMAGE_FOR_PENTEST=mycompany/pentest-tools:v2.0
```

> [!NOTE]
> If a user explicitly specifies a particular Docker image in their task, the system will try to use that exact image, ignoring these settings. These variables only affect the system's automatic image selection process.

For an advanced OpenVAS/GVM experiment that uses a custom pentest image, see [OpenVAS via a Custom Pentest Image](examples/guides/openvas-custom-image.md).

#### Restricted Networks, Docker Mirrors, and Proxies

If your environment cannot reach Docker Hub (`docker.io`) directly, changing PentAGI environment variables is usually not enough to fix image download failures. PentAGI still relies on Docker's own registry access for Compose-managed services, and the installer network checks also validate Docker Hub reachability.

For restricted networks:

1. Confirm that the host can resolve and reach `docker.io`.
2. If your environment requires an outbound proxy for PentAGI or installer HTTP traffic, set the `PROXY_URL` environment variable. To route Docker image pulls through a proxy, configure the Docker daemon or Docker Desktop proxy separately — Docker does not use PentAGI's `PROXY_URL` for registry access.
3. If Docker Hub is blocked or heavily rate-limited, configure an organization-approved registry mirror or registry proxy before running the installer or `docker compose up`.
4. Restart Docker after changing the daemon configuration, then rerun the installer checks or Compose startup.

Example Docker daemon mirror configuration:

```json
{
  "registry-mirrors": ["https://mirror.example.com"]
}
```

On Linux, this is typically configured in `/etc/docker/daemon.json`. On Docker Desktop, use the equivalent Docker Engine or proxy settings. A Docker Hub mirror covers Docker Hub-hosted images such as `vxcontrol/*`, but the main Compose stack already includes `quay.io/prometheuscommunity/postgres-exporter`, the optional observability stack includes `gcr.io/cadvisor/cadvisor`, and the optional Langfuse stack includes `cgr.dev/chainguard/minio`. Those registries still need direct access or individually approved proxy/mirror paths.

See the official Docker documentation for [registry mirrors](https://docs.docker.com/docker-hub/image-library/mirror/) and [daemon proxy configuration](https://docs.docker.com/engine/daemon/proxy/).

#### Troubleshooting: "failed to select primary docker image via llm call"

A flow that fails immediately with `failed to select primary docker image via llm call` usually indicates a problem with the configured LLM backend, not with Docker or the image registry. Older PentAGI versions reported the same failure as `failed to get primary docker image`, which led users to debug Docker even though the registry was healthy.

When a flow starts, PentAGI makes its first LLM call to choose the primary Docker image for the task. This image-selection call runs through the `simple` agent type, so a failure here points at the model assigned to that agent type rather than at Docker. A message such as `API returned unexpected status code: 502` or `404` in this context is returned by the LLM backend, not by Docker Hub.

This is distinct from the registry reachability problems described above: if Docker pulls succeed and the Compose stack starts, but flow creation still fails at image selection, investigate the LLM backend rather than Docker.

To diagnose:

1. Check PentAGI logs first: `docker logs pentagi`.
2. Check the logs of your configured LLM backend (the server behind your provider or `LLM_SERVER_URL`).
3. Verify that the base URL, API key, and model name in [Custom LLM Provider Configuration](#custom-llm-provider-configuration) are correct and reachable from the container. If you assign different models per agent type, check the model used by the `simple` agent type, since image selection runs through it.
4. For custom, OpenAI-compatible, vLLM, or SGLang backends, confirm that the model supports tool calling (function calling) and that the matching tool-call parser is enabled. A missing or mismatched tool-call parser is a known cause of this failure.

## Development

### Development Requirements

- golang
- nodejs
- docker
- postgres

### Environment Setup

#### Backend Setup

Run once `cd backend && go mod download` to install needed packages.

For generating swagger files have to run

```bash
swag init -g ../../pkg/server/router.go -o pkg/server/docs/ --parseDependency --parseInternal --parseDepth 2 -d cmd/pentagi
```

before installing `swag` package via

```bash
go install github.com/swaggo/swag/cmd/swag@v1.8.7
```

For generating graphql resolver files have to run

```bash
go run github.com/99designs/gqlgen --config ./gqlgen/gqlgen.yml
```

after that you can see the generated files in `pkg/graph` folder.

For generating ORM methods (database package) from sqlc configuration

```bash
docker run --rm -v $(pwd):/src -w /src --network pentagi-network -e DATABASE_URL="{URL}" sqlc/sqlc:1.27.0 generate -f sqlc/sqlc.yml
```

For generating Langfuse SDK from OpenAPI specification

```bash
fern generate --local
```

and to install fern-cli

```bash
pnpm add fern-api
```

#### Testing

For running tests `cd backend && go test -v ./...`

#### Frontend Setup

Run once `cd frontend && pnpm install` to install needed packages.

For generating graphql files have to run `pnpm run graphql:generate` which using `graphql-codegen.ts` file.

Be sure that you have `graphql-codegen` installed globally:

```bash
pnpm add graphql-codegen
```

After that you can run:
* `pnpm run prettier` to check if your code is formatted correctly
* `pnpm run prettier:fix` to fix it
* `pnpm run lint` to check if your code is linted correctly
* `pnpm run lint:fix` to fix it

For generating SSL certificates you need to run `pnpm run ssl:generate` which using `generate-ssl.ts` file or it will be generated automatically when you run `pnpm run dev`.

#### Backend Configuration

Edit the configuration for `backend` in `.vscode/launch.json` file:
- `DATABASE_URL` - PostgreSQL database URL (eg. `postgres://postgres:postgres@localhost:5432/pentagidb?sslmode=disable`)
- `DOCKER_HOST` - Docker SDK API (eg. for macOS `DOCKER_HOST=unix:///Users/<my-user>/Library/Containers/com.docker.docker/Data/docker.raw.sock`) [more info](https://stackoverflow.com/a/62757128/5922857)

Optional:
- `SERVER_PORT` - Port to run the server (default: `8443`)
- `SERVER_USE_SSL` - Enable SSL for the server (default: `false`)

##### PostgreSQL / pgvector connection pool sizing

PentAGI opens two independent connection pools to the same Postgres instance:

| Pool | Env var | Default | Used by |
|---|---|---|---|
| Shared `sql.DB` | `DATABASE_MAX_OPEN_CONNS` | `25` | All sqlc queries and GORM handlers share a single `*sql.DB` |
| Shared `pgxpool` | `DATABASE_VECTOR_MAX_CONNS` | `10` | All pgvector stores (agent memory + knowledge API) share a single pool |

Additional tuning knob:
- `DATABASE_MAX_IDLE_CONNS` — maximum idle connections kept open in the `sql.DB` pool between requests (default: `5`).

**Budget for the stock `vxcontrol/pgvector` image** (`max_connections = 100`, `superuser_reserved_connections = 3`):

```
Available for client connections  = 97
  pentagi sql.DB  (DATABASE_MAX_OPEN_CONNS)   = 25
  pentagi pgxpool (DATABASE_VECTOR_MAX_CONNS) = 10
  pgexporter                                  =  3
  autovacuum workers                          =  3
  ─────────────────────────────────────────
  Total consumed                              = 41
  Free buffer                                 = 56  (≈ 58 %)
```

The defaults are sized for **10 parallel flows** with concurrent API requests. If you run more flows or deploy multiple PentAGI instances against the same Postgres, raise `max_connections` via the `command` override in `docker-compose.yml` and increase the pool sizes proportionally:

```yaml
pgvector:
  image: vxcontrol/pgvector:latest
  command: postgres -c max_connections=200
```

To inspect the live connection budget on a running deployment:

```bash
# Postgres limits
docker exec pgvector sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT name, setting FROM pg_settings
   WHERE name IN ('"'"'max_connections'"'"', '"'"'superuser_reserved_connections'"'"');"'

# Current usage vs. available
docker exec pgvector sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT max_conn, used, max_conn - used AS available
   FROM (SELECT current_setting('"'"'max_connections'"'"')::int AS max_conn,
                count(*) AS used FROM pg_stat_activity) t;"'

# Breakdown by client
docker exec pgvector sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c \
  "SELECT application_name, client_addr, state, count(*)
   FROM pg_stat_activity
   WHERE pid <> pg_backend_pid()
   GROUP BY 1, 2, 3 ORDER BY count DESC;"'
```

##### External PostgreSQL and schema handling

`DATABASE_URL` may point at any PostgreSQL instance, not only the bundled `pgvector` container. Two extra knobs apply when — and only when — `TENANT_ID` is set, because that is when PentAGI creates its own schema and rewrites the connection's `search_path`:

| Env var | Default | Purpose |
|---|---|---|
| `DATABASE_EXTENSIONS_SCHEMA` | `public` | Schema holding the shared `vector` and `pg_trgm` extensions that every tenant's `search_path` must reach |
| `DATABASE_SEARCH_PATH_VIA_OPTIONS` | `false` | Send the tenant `search_path` inside the `options` startup parameter instead of as a bare connection parameter |

**Supabase (cloud or self-hosted)** needs both of them considered, and is the reason they exist:

- Supabase installs its bundled extensions into an `extensions` schema instead of `public`, so set `DATABASE_EXTENSIONS_SCHEMA=extensions`. Without it, startup aborts with an error naming the schema where `vector` was actually found — no need to move a provider-managed extension with `ALTER EXTENSION`.
- Supabase's pooler (Supavisor) does not reliably forward a bare `search_path` connection parameter. Prefer a **direct** PostgreSQL connection: self-hosted, expose the `db` service port and bypass the `supavisor` service; cloud, use the "Direct connection" string (or the IPv4 add-on on IPv4-only networks). If the pooler cannot be bypassed, use its session mode and try `DATABASE_SEARCH_PATH_VIA_OPTIONS=true` — PentAGI verifies the effective schema on boot and refuses to start if it did not take effect, so a silent cross-tenant data mix-up is not possible.

Both settings are managed by the installer under *Server Settings*, next to `TENANT_ID` — see [Running Several Instances](#running-several-instances-tenant_id) for that scenario, and [Multi-Instance Deployment](backend/docs/config.md#multi-instance-deployment-tenant_id) for the full matrix, including the PgBouncer recipe (`pool_mode = session`, `ignore_startup_parameters`, per-tenant `connect_query`).

#### Frontend Configuration

Edit the configuration for `frontend` in `.vscode/launch.json` file:
- `VITE_API_URL` - Backend API URL. *Omit* the URL scheme (e.g., `localhost:8080` *NOT* `http://localhost:8080`)
- `VITE_USE_HTTPS` - Enable SSL for the server (default: `false`)
- `VITE_PORT` - Port to run the server (default: `8000`)
- `VITE_HOST` - Host to run the server (default: `0.0.0.0`)

### Running the Application

#### Backend

Run the command(s) in `backend` folder:
- Use `.env` file to set environment variables like a `source .env`
- Run `go run cmd/pentagi/main.go` to start the server

> [!NOTE]
> The first run can take a while as dependencies and docker images need to be downloaded to setup the backend environment.

#### Frontend

Run the command(s) in `frontend` folder:
- Run `pnpm install` to install the dependencies
- Run `pnpm run dev` to run the web app
- Run `pnpm run build` to build the web app

Open your browser and visit the web app URL.

## Testing LLM Agents

PentAGI includes a powerful utility called `ctester` for testing and validating LLM agent capabilities. This tool helps ensure your LLM provider configurations work correctly with different agent types, allowing you to optimize model selection for each specific agent role.

The utility features parallel testing of multiple agents, detailed reporting, and flexible configuration options.

### Key Features

- **Parallel Testing**: Tests multiple agents simultaneously for faster results
- **Comprehensive Test Suite**: Evaluates basic completion, JSON responses, function calling, and penetration testing knowledge
- **Detailed Reporting**: Generates markdown reports with success rates and performance metrics
- **Flexible Configuration**: Test specific agents or test groups as needed
- **Specialized Test Groups**: Includes domain-specific tests for cybersecurity and penetration testing scenarios

### Usage Scenarios

#### For Developers (with local Go environment)

If you've cloned the repository and have Go installed:

```bash
# Default configuration with .env file
cd backend
go run cmd/ctester/*.go -verbose

# Custom provider configuration
go run cmd/ctester/*.go -config ../examples/configs/openrouter.provider.yml -verbose

# Generate a report file
go run cmd/ctester/*.go -config ../examples/configs/deepinfra.provider.yml -report ../test-report.md

# Test specific agent types only
go run cmd/ctester/*.go -agents simple,simple_json,primary_agent -verbose

# Test specific test groups only
go run cmd/ctester/*.go -groups basic,advanced -verbose
```

#### For Users (using Docker image)

If you prefer to use the pre-built Docker image without setting up a development environment:

```bash
# Using Docker to test with default environment
docker run --rm -v $(pwd)/.env:/opt/pentagi/.env vxcontrol/pentagi /opt/pentagi/bin/ctester -verbose

# Test with your custom provider configuration
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  -v $(pwd)/my-config.yml:/opt/pentagi/config.yml \
  vxcontrol/pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/config.yml -agents simple,primary_agent,coder -verbose

# Generate a detailed report
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  -v $(pwd):/opt/pentagi/output \
  vxcontrol/pentagi /opt/pentagi/bin/ctester -report /opt/pentagi/output/report.md
```

#### Using Pre-configured Providers

The Docker image comes with built-in support for major providers (OpenAI, Anthropic, Gemini, Ollama) and pre-configured provider files for additional services (OpenRouter, OpenCode, Atlas, OrcaRouter, DeepInfra, DeepSeek, Moonshot, Novita, xAI):

```
/* Detailed source-code truncated for AI context efficiency. */
```

To use these configurations, your `.env` file only needs to contain:

```
/* Detailed source-code truncated for AI context efficiency. */
```

#### Using OpenAI with Unverified Organizations

For OpenAI accounts with unverified organizations that don't have access to the latest reasoning models (o1, o3, o4-mini), you need to use a custom configuration.

To use OpenAI with unverified organization accounts, configure your `.env` file as follows:

```bash
LLM_SERVER_URL=https://api.openai.com/v1
LLM_SERVER_KEY=your_openai_api_key
LLM_SERVER_MODEL=                                # Leave empty, models are specified in config
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom-openai.provider.yml
```

This configuration uses the pre-built `custom-openai.provider.yml` file that maps all agent types to models available for unverified organizations, using `o3-mini` instead of models like `o1`, `o3`, and `o4-mini`.

You can test this configuration using:

```bash
# Test with custom OpenAI configuration for unverified accounts
docker exec -it pentagi /opt/pentagi/bin/ctester -config /opt/pentagi/conf/custom-openai.provider.yml
```

#### Using LiteLLM Proxy

When using LiteLLM proxy to access various LLM providers, model names are prefixed with the provider name (e.g., `moonshot/kimi-2.5` instead of `kimi-2.5`). To use the same provider configuration files with both direct API access and LiteLLM proxy, set the `LLM_SERVER_PROVIDER` variable:

```bash
# Direct access to Moonshot API
LLM_SERVER_URL=https://api.moonshot.ai/v1
LLM_SERVER_KEY=your_moonshot_api_key
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/moonshot.provider.yml
LLM_SERVER_PROVIDER=                             # Empty for direct access

# Access via LiteLLM proxy
LLM_SERVER_URL=http://litellm-proxy:4000
LLM_SERVER_KEY=your_litellm_api_key
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/moonshot.provider.yml
LLM_SERVER_PROVIDER=moonshot                     # Provider prefix for LiteLLM
```

With `LLM_SERVER_PROVIDER=moonshot`, the system automatically prefixes all model names from the configuration file with `moonshot/`, making them compatible with LiteLLM's model naming convention.

**LiteLLM Provider Name Mapping:**

When using LiteLLM proxy, set the corresponding `*_PROVIDER` variable to enable model prefixing:

- `deepseek` - for DeepSeek models (`DEEPSEEK_PROVIDER=deepseek` → `deepseek/deepseek-v4-flash`)
- `zai` - for GLM models (`GLM_PROVIDER=zai` → `zai/glm-4`)
- `moonshot` - for Kimi models (`KIMI_PROVIDER=moonshot` → `moonshot/kimi-k3`)
- `dashscope` - for Qwen models (`QWEN_PROVIDER=dashscope` → `dashscope/qwen-plus`)
- `openai`, `anthropic`, `gemini` - for major cloud providers
- `opencode` - for OpenCode Go plan
- `openrouter` - for OpenRouter aggregator
- `orcarouter` - for OrcaRouter aggregator
- `deepinfra` - for DeepInfra hosting
- `novita` - for Novita AI
- `xai` - for xAI (`grok-*` models)
- Any other provider name configured in your LiteLLM instance

**Example with LiteLLM:**
```bash
# Use DeepSeek models via LiteLLM proxy with model prefixing
DEEPSEEK_API_KEY=your_litellm_proxy_key
DEEPSEEK_SERVER_URL=http://litellm-proxy:4000
DEEPSEEK_PROVIDER=deepseek  # Models become deepseek/deepseek-v4-flash, deepseek/deepseek-v4-pro for LiteLLM

# Direct DeepSeek API usage (no prefix needed)
DEEPSEEK_API_KEY=your_deepseek_api_key
DEEPSEEK_SERVER_URL=https://api.deepseek.com
# Leave DEEPSEEK_PROVIDER empty
```

This approach allows you to:
- Use the same configuration files for both direct and proxied access
- Switch between providers without modifying configuration files
- Easily test different routing strategies with LiteLLM

#### Running Tests in a Production Environment

If you already have a running PentAGI container and want to test the current configuration:

```bash
# Run ctester in an existing container using current environment variables
docker exec -it pentagi /opt/pentagi/bin/ctester -verbose

# Test specific agent types with deterministic ordering
docker exec -it pentagi /opt/pentagi/bin/ctester -agents simple,primary_agent,pentester -groups basic,knowledge -verbose

# Generate a report file inside the container
docker exec -it pentagi /opt/pentagi/bin/ctester -report /opt/pentagi/data/agent-test-report.md

# Access the report from the host
docker cp pentagi:/opt/pentagi/data/agent-test-report.md ./
```

### Command-line Options

The utility accepts several options:

- `-env <path>` - Path to environment file (default: `.env`)
- `-type <provider>` - Provider type: `custom`, `openai`, `anthropic`, `ollama`, `bedrock`, `gemini` (default: `custom`)
- `-config <path>` - Path to custom provider config (default: from `LLM_SERVER_CONFIG_PATH` env variable)
- `-tests <path>` - Path to custom tests YAML file (optional)
- `-report <path>` - Path to write the report file (optional)
- `-agents <list>` - Comma-separated list of agent types to test (default: `all`)
- `-groups <list>` - Comma-separated list of test groups to run (default: `all`)
- `-verbose` - Enable verbose output with detailed test results for each agent

### Available Agent Types

Agents are tested in the following deterministic order:

1. **simple** - Basic completion tasks
2. **simple_json** - JSON-structured responses
3. **primary_agent** - Main reasoning agent
4. **assistant** - Interactive assistant mode
5. **generator** - Content generation
6. **refiner** - Content refinement and improvement
7. **adviser** - Expert advice and consultation
8. **reflector** - Self-reflection and analysis
9. **searcher** - Information gathering and search
10. **enricher** - Data enrichment and expansion
11. **coder** - Code generation and analysis
12. **installer** - Installation and setup tasks
13. **pentester** - Penetration testing and security assessment

### Available Test Groups

- **basic** - Fundamental completion and prompt response tests
- **advanced** - Complex reasoning and function calling tests
- **json** - JSON format validation and structure tests (specifically designed for `simple_json` agent)
- **knowledge** - Domain-specific cybersecurity and penetration testing knowledge tests

> **Note**: The `json` test group is specifically designed for the `simple_json` agent type, while all other agents are tested with `basic`, `advanced`, and `knowledge` groups. This specialization ensures optimal testing coverage for each agent's intended purpose.

### Example Provider Configuration

Provider configuration defines which models to use for different agent types:

```yaml
simple:
  model: "provider/model-name"
  temperature: 0.7
  top_p: 0.95
  n: 1
  max_tokens: 4000

simple_json:
  model: "provider/model-name"
  temperature: 0.7
  top_p: 1.0
  n: 1
  max_tokens: 4000
  json: true

# ... other agent types ...
```

### Optimization Workflow

1. **Create a baseline**: Run tests with default configuration to establish benchmark performance
2. **Analyze agent-specific performance**: Review the deterministic agent ordering to identify underperforming agents
3. **Test specialized configurations**: Experiment with different models for each agent type using provider-specific configs
4. **Focus on domain knowledge**: Pay special attention to knowledge group tests for cybersecurity expertise
5. **Validate function calling**: Ensure tool-based tests pass consistently for critical agent types
6. **Compare results**: Look for the best success rate and performance across all test groups
7. **Deploy optimal configuration**: Use in production with your optimized setup

This tool helps ensure your AI agents are using the most effective models for their specific tasks, improving reliability while optimizing costs.

## Embedding Configuration and Testing

PentAGI uses vector embeddings for semantic search, knowledge storage, and memory management. The system supports multiple embedding providers that can be configured according to your needs and preferences.

### Supported Embedding Providers

PentAGI supports the following embedding providers:

- **OpenAI** (default): Uses OpenAI's text embedding models
- **Ollama**: Local embedding model through Ollama
- **Mistral**: Mistral AI's embedding models
- **Jina**: Jina AI's embedding service
- **HuggingFace**: Models from HuggingFace
- **GoogleAI**: Google's embedding models
- **VoyageAI**: VoyageAI's embedding models

> **OpenAI-compatible third parties**: any provider exposing OpenAI's `/embeddings` API can be plugged in via `EMBEDDING_PROVIDER=openai` with a custom `EMBEDDING_URL`. For example, **Qwen DashScope** offers `text-embedding-v4` through the `/compatible-mode/v1` endpoint (International and Chinese Mainland regions only — the US region does not expose embeddings). See the [Qwen Alternative Integrations](#alternative-integrations) subsection for the full configuration snippet.

<details>
<summary><b>Embedding Provider Configuration</b> (click to expand)</summary>

### Environment Variables

To configure the embedding provider, set the following environment variables in your `.env` file:

```bash
# Primary embedding configuration
EMBEDDING_PROVIDER=openai       # Provider type (openai, ollama, mistral, jina, huggingface, googleai, voyageai)
EMBEDDING_MODEL=text-embedding-3-small  # Model name to use
EMBEDDING_URL=                  # Optional custom API endpoint
EMBEDDING_KEY=                  # API key for the provider (if required)
EMBEDDING_BATCH_SIZE=100        # Number of documents to process in a batch
EMBEDDING_STRIP_NEW_LINES=true  # Whether to remove new lines from text before embedding
EMBEDDING_MAX_TEXT_BYTES=8192   # Max bytes of text sent to embedding model per document (byte proxy for token limit)

# Advanced settings
PROXY_URL=                      # Optional proxy for all backend LLM, embedding, search and update-check calls
HTTP_CLIENT_TIMEOUT=600         # Timeout in seconds for external API calls (default: 600, 0 = no timeout)
TERMINAL_TOOL_TIMEOUT=1200      # Default timeout in seconds for terminal tool commands when timeout=0 or negative (range: 1–10800; values <= 0 or above 10800 are clamped to 10800 = 3 hours)

# SSL/TLS Certificate Configuration (for external communication with LLM backends and tool servers)
EXTERNAL_SSL_CA_PATH=           # Path to custom CA certificate file (PEM format) inside the container
                                # Must point to /opt/pentagi/ssl/ directory (e.g., /opt/pentagi/ssl/ca-bundle.pem)
EXTERNAL_SSL_INSECURE=false     # Skip certificate verification (use only for testing)
```

<details>
<summary><b>How to Add Custom CA Certificates</b> (click to expand)</summary>

If you see this error: `tls: failed to verify certificate: x509: certificate signed by unknown authority`

**Step 1:** Get your CA certificate bundle in PEM format (can contain multiple certificates)

**Step 2:** Place the file in the SSL directory on your host machine:
```bash
# Default location (if PENTAGI_SSL_DIR is not set)
cp ca-bundle.pem ./pentagi-ssl/

# Or custom location (if using PENTAGI_SSL_DIR in docker-compose.yml)
cp ca-bundle.pem /path/to/your/ssl/dir/
```

**Step 3:** Set the path in `.env` file (path must be inside the container):
```bash
# The volume pentagi-ssl is mounted to /opt/pentagi/ssl inside the container
EXTERNAL_SSL_CA_PATH=/opt/pentagi/ssl/ca-bundle.pem
EXTERNAL_SSL_INSECURE=false
```

**Step 4:** Restart PentAGI:
```bash
docker compose restart pentagi
```

**Notes:**
- The `pentagi-ssl` volume is mounted to `/opt/pentagi/ssl` inside the container
- You can change host directory using `PENTAGI_SSL_DIR` variable in docker-compose.yml
- File supports multiple certificates and intermediate CAs in one PEM file
- Use `EXTERNAL_SSL_INSECURE=true` only for testing (not recommended for production)

</details>

### Provider-Specific Limitations

Each provider has specific limitations and supported features:

- **OpenAI**: Supports all configuration options
- **Ollama**: Does not support `EMBEDDING_KEY` as it uses local models
- **Mistral**: Supports all configuration options
- **Jina**: Supports all configuration options
- **HuggingFace**: Requires `EMBEDDING_KEY` and supports all other options
- **GoogleAI**: Does not support `EMBEDDING_URL`, requires `EMBEDDING_KEY`
- **VoyageAI**: Supports all configuration options

If `EMBEDDING_URL` and `EMBEDDING_KEY` are both left empty, the system uses the corresponding LLM provider's server and key together (e.g., `OPEN_AI_SERVER_URL` and `OPEN_AI_KEY` when `EMBEDDING_PROVIDER=openai`). A key set in `EMBEDDING_KEY` without `EMBEDDING_URL` goes to the provider's public endpoint, and an `EMBEDDING_URL` set without a key receives the LLM provider's key only when it is that provider's own server. So an OpenAI-compatible endpoint of another vendor needs its key in `EMBEDDING_KEY`, and a keyless local server (LocalAI, vLLM, TEI) takes any non-empty value there. Whenever the Mistral embedder uses `MISTRAL_SERVER_URL`, it also names the model the way that server expects: `MISTRAL_PROVIDER=mistral` turns `mistral-embed` into `mistral/mistral-embed`, and a model that already starts with `mistral/` is sent as is.

### Why Consistent Embedding Providers Matter

It's crucial to use the same embedding provider consistently because:

1. **Vector Compatibility**: Different providers produce vectors with different dimensions and mathematical properties
2. **Semantic Consistency**: Changing providers can break semantic similarity between previously embedded documents
3. **Memory Corruption**: Mixed embeddings can lead to poor search results and broken knowledge base functionality

If you change your embedding provider, you should flush and reindex your entire knowledge base (see `etester` utility below).

</details>

### Embedding Tester Utility (etester)

PentAGI includes a specialized `etester` utility for testing, managing, and debugging embedding functionality. This tool is essential for diagnosing and resolving issues related to vector embeddings and knowledge storage.

<details>
<summary><b>Etester Commands</b> (click to expand)</summary>

```bash
# Test embedding provider and database connection
cd backend
go run cmd/etester/main.go test -verbose

# Show statistics about the embedding database
go run cmd/etester/main.go info

# Delete all documents from the embedding database (use with caution!)
go run cmd/etester/main.go flush

# Recalculate embeddings for all documents (after changing provider)
go run cmd/etester/main.go reindex

# Search for documents in the embedding database
go run cmd/etester/main.go search -query "How to install PostgreSQL" -limit 5
```

### Using Docker

If you're running PentAGI in Docker, you can use etester from within the container:

```bash
# Test embedding provider
docker exec -it pentagi /opt/pentagi/bin/etester test

# Show detailed database information
docker exec -it pentagi /opt/pentagi/bin/etester info -verbose
```

### Advanced Search Options

The `search` command supports various filters to narrow down results:

```bash
# Filter by document type
docker exec -it pentagi /opt/pentagi/bin/etester search -query "Security vulnerability" -doc_type guide -threshold 0.8

# Filter by flow ID
docker exec -it pentagi /opt/pentagi/bin/etester search -query "Code examples" -doc_type code -flow_id 42

# All available search options
docker exec -it pentagi /opt/pentagi/bin/etester search -help
```

Available search parameters:
- `-query STRING`: Search query text (required)
- `-doc_type STRING`: Filter by document type (answer, memory, guide, code)
- `-flow_id NUMBER`: Filter by flow ID (positive number)
- `-answer_type STRING`: Filter by answer type (guide, vulnerability, code, tool, other)
- `-guide_type STRING`: Filter by guide type (install, configure, use, pentest, development, other)
- `-limit NUMBER`: Maximum number of results (default: 3)
- `-threshold NUMBER`: Similarity threshold (0.0-1.0, default: 0.7)

### Memory Lifecycle Across Flows

PentAGI stores several kinds of vector documents, and they serve different purposes:

- `memory` captures flow-specific execution history such as tool results and agent observations
- `guide`, `answer`, and `code` are intended for reusable knowledge that can help future runs

If you want to inspect what happened in one engagement, search the vector store with the related `flow_id`. If you want knowledge to survive beyond a single run, store the durable result explicitly as a `guide`, `answer`, or `code` document instead of relying on execution memory alone.

For example, if a target has recurring setup notes, authentication quirks, or target-specific testing methodology, instruct the agent to save that information as a `guide` and search for it at the beginning of the next engagement. This is the safest current workflow when you want a new flow to start with reusable context.

Flow deletion removes the flow from normal queries through PentAGI's soft-delete mechanism, so reusable knowledge should be treated as a separate concern from per-flow execution history. If you enable the optional Graphiti knowledge graph described earlier in this README, treat its current search context as scoped to the active flow or engagement unless you explicitly build a separate cross-flow reuse workflow.

### Common Troubleshooting Scenarios

1. **After changing embedding provider**: Always run `flush` or `reindex` to ensure consistency
2. **Poor search results**: Try adjusting the similarity threshold or check if embeddings are correctly generated
3. **Database connection issues**: Verify PostgreSQL is running with pgvector extension installed
4. **Missing API keys**: Check environment variables for your chosen embedding provider

</details>

### Troubleshooting: Flow Stalls or Hangs Without Progress

If a flow starts but then appears to wait indefinitely with no subtasks progressing, a common cause is an embedding provider that is misconfigured or unreachable. PentAGI uses the embedding provider to store and search vector memory while a flow runs, so embedding calls that fail or hang can leave a flow waiting instead of advancing.

**1. Check the container logs first.** Embedding errors surface in the PentAGI logs:

```bash
docker logs pentagi
```

Look for embedding-related failures such as authentication errors (401/403), wrong-model or not-found errors (404), connection timeouts, or TLS certificate errors. These point at the embedding provider configuration rather than at the flow itself.

**2. Validate the provider with etester.** The [Embedding Tester Utility (etester)](#embedding-tester-utility-etester) checks both the embedding provider and the database connection without starting a flow:

```bash
docker exec -it pentagi /opt/pentagi/bin/etester test -verbose
```

A failing `test` confirms the problem is in the embedding configuration rather than in the flow.

**3. Verify the configuration.** Check the following in your `.env` file against the [Supported Embedding Providers](#supported-embedding-providers) list and each provider's documented limitations:

- `EMBEDDING_PROVIDER` is one of the supported providers (default `openai`).
- `EMBEDDING_MODEL` is a valid model name for that provider.
- `EMBEDDING_URL` and `EMBEDDING_KEY` are correct for the provider. If both are left empty, PentAGI falls back to the matching LLM provider settings (for example `OPEN_AI_KEY` and `OPEN_AI_SERVER_URL` when `EMBEDDING_PROVIDER=openai`), so a missing or wrong key there can break embeddings too.
- The endpoint is reachable from inside the container. If outbound calls go through a proxy, confirm `PROXY_URL` is set; if calls hang rather than fail quickly, `HTTP_CLIENT_TIMEOUT` controls how long PentAGI waits on the provider before giving up.

> **Changing provider?** If you switch embedding providers after data has already been indexed, run `flush` or `reindex` with etester so old and new vectors are not mixed. See [Why Consistent Embedding Providers Matter](#why-consistent-embedding-providers-matter) above.

## Function Testing with ftester

PentAGI includes a versatile utility called `ftester` for debugging, testing, and developing specific functions and AI agent behaviors. While `ctester` focuses on testing LLM model capabilities, `ftester` allows you to directly invoke individual system functions and AI agent components with precise control over execution context.

### Key Features

- **Direct Function Access**: Test individual functions without running the entire system
- **Mock Mode**: Test functions without a live PentAGI deployment using built-in mocks
- **Interactive Input**: Fill function arguments interactively for exploratory testing
- **Detailed Output**: Color-coded terminal output with formatted responses and errors
- **Context-Aware Testing**: Debug AI agents within the context of specific flows, tasks, and subtasks
- **Observability Integration**: All function calls are logged to Langfuse and Observability stack

### Usage Modes

#### Command Line Arguments

Run ftester with specific function and arguments directly from the command line:

```bash
# Basic usage with mock mode
cd backend
go run cmd/ftester/main.go [function_name] -[arg1] [value1] -[arg2] [value2]

# Example: Test terminal command in mock mode
go run cmd/ftester/main.go terminal -command "ls -la" -message "List files"

# Using a real flow context
go run cmd/ftester/main.go -flow 123 terminal -command "whoami" -message "Check user"

# Testing AI agent in specific task/subtask context
go run cmd/ftester/main.go -flow 123 -task 456 -subtask 789 pentester -message "Find vulnerabilities"
```

#### Interactive Mode

Run ftester without arguments for a guided interactive experience:

```bash
# Start interactive mode
go run cmd/ftester/main.go [function_name]

# For example, to interactively fill browser tool arguments
go run cmd/ftester/main.go browser
```

<details>
<summary><b>Available Functions</b> (click to expand)</summary>

### Environment Functions
- **terminal**: Execute commands in a container and return the output
- **file**: Perform file operations (read, write, list) in a container

### Search Functions
- **browser**: Access websites and capture screenshots
- **web_search**: Unified search orchestrator that agents actually call — pass a `query` and a `mode` (`links`, `answer`, `research`, `exploit`) and it auto-selects, retries, and falls back across the engines below, so you never name an engine explicitly
- **google**: Search the web using Google Custom Search
- **duckduckgo**: Search the web using DuckDuckGo
- **tavily**: Search using Tavily AI search engine
- **firecrawl**: Search using Firecrawl with main-content markdown scraping
- **traversaal**: Search using Traversaal AI search engine
- **perplexity**: Search using Perplexity AI
- **sploitus**: Search for security exploits, vulnerabilities (CVEs), and pentesting tools
- **searxng**: Search using SearXNG meta search engine (aggregates results from multiple engines)
- **internal** *(ftester-only debug function, not an agent tool)*: Opt-in browser-analytics fallback engine that discovers links, scrapes each page, and summarizes the result; requires `WEB_SEARCH_INTERNAL_ENABLED=true`, a configured scraper, and at least one available link engine

### Vector Database Functions
- **search_in_memory**: Search for information in vector database
- **search_guide**: Find guidance documents in vector database
- **search_answer**: Find answers to questions in vector database
- **search_code**: Find code examples in vector database

### AI Agent Functions
- **advice**: Get expert advice from an AI agent
- **coder**: Request code generation or modification
- **maintenance**: Run system maintenance tasks
- **memorist**: Store and organize information in vector database
- **pentester**: Perform security tests and vulnerability analysis
- **search**: Complex search across multiple sources

### Utility Functions
- **describe**: Show information about flows, tasks, and subtasks

</details>

<details>
<summary><b>Debugging Flow Context</b> (click to expand)</summary>

The `describe` function provides detailed information about tasks and subtasks within a flow. This is particularly useful for diagnosing issues when PentAGI encounters problems or gets stuck.

```bash
# List all flows in the system
go run cmd/ftester/main.go describe

# Show all tasks and subtasks for a specific flow
go run cmd/ftester/main.go -flow 123 describe

# Show detailed information for a specific task
go run cmd/ftester/main.go -flow 123 -task 456 describe

# Show detailed information for a specific subtask
go run cmd/ftester/main.go -flow 123 -task 456 -subtask 789 describe

# Show verbose output with full descriptions and results
go run cmd/ftester/main.go -flow 123 describe -verbose
```

This function allows you to identify the exact point where a flow might be stuck and resume processing by directly invoking the appropriate agent function.

</details>

<details>
<summary><b>Function Help and Discovery</b> (click to expand)</summary>

Each function has a help mode that shows available parameters:

```bash
# Get help for a specific function
go run cmd/ftester/main.go [function_name] -help

# Examples:
go run cmd/ftester/main.go terminal -help
go run cmd/ftester/main.go browser -help
go run cmd/ftester/main.go describe -help
```

You can also run ftester without arguments to see a list of all available functions:

```bash
go run cmd/ftester/main.go
```

</details>

<details>
<summary><b>Output Format</b> (click to expand)</summary>

The `ftester` utility uses color-coded output to make interpretation easier:

- **Blue headers**: Section titles and key names
- **Cyan [INFO]**: General information messages
- **Green [SUCCESS]**: Successful operations
- **Red [ERROR]**: Error messages
- **Yellow [WARNING]**: Warning messages
- **Yellow [MOCK]**: Indicates mock mode operation
- **Magenta values**: Function arguments and results

JSON and Markdown responses are automatically formatted for readability.

</details>

<details>
<summary><b>Advanced Usage Scenarios</b> (click to expand)</summary>

### Debugging Stuck AI Flows

When PentAGI gets stuck in a flow:

1. Pause the flow through the UI
2. Use `describe` to identify the current task and subtask
3. Directly invoke the agent function with the same task/subtask IDs
4. Examine the detailed output to identify the issue
5. Resume the flow or manually intervene as needed

### Testing Environment Variables

Verify that API keys and external services are configured correctly:

```bash
# Test Google search API configuration
go run cmd/ftester/main.go google -query "pentesting tools"

# Test browser access to external websites
go run cmd/ftester/main.go browser -url "https://example.com"
```

### Developing New AI Agent Behaviors

When developing new prompt templates or agent behaviors:

1. Create a test flow in the UI
2. Use ftester to directly invoke the agent with different prompts
3. Observe responses and adjust prompts accordingly
4. Check Langfuse for detailed traces of all function calls

### Verifying Docker Container Setup

Ensure containers are properly configured:

```bash
go run cmd/ftester/main.go -flow 123 terminal -command "env | grep -i proxy" -message "Check proxy settings"
```

</details>

<details>
<summary><b>Docker Container Usage</b> (click to expand)</summary>

If you have PentAGI running in Docker, you can use ftester from within the container:

```bash
# Run ftester inside the running PentAGI container
docker exec -it pentagi /opt/pentagi/bin/ftester [arguments]

# Examples:
docker exec -it pentagi /opt/pentagi/bin/ftester -flow 123 describe
docker exec -it pentagi /opt/pentagi/bin/ftester -flow 123 terminal -command "ps aux" -message "List processes"
```

This is particularly useful for production deployments where you don't have a local development environment.

</details>

<details>
<summary><b>Integration with Observability Tools</b> (click to expand)</summary>

All function calls made through ftester are logged to:

1. **Langfuse**: Captures the entire AI agent interaction chain, including prompts, responses, and function calls
2. **OpenTelemetry**: Records metrics, traces, and logs for system performance analysis
3. **Terminal Output**: Provides immediate feedback on function execution

To access detailed logs:

- Check Langfuse UI for AI agent traces (typically at `http://localhost:4000`)
- Use Grafana dashboards for system metrics (typically at `http://localhost:3000`)
- Examine terminal output for immediate function results and errors

</details>

### Command-line Options

The main utility accepts several options:

- `-env <path>` - Path to environment file (optional, default: `.env`)
- `-provider <type>` - Provider type to use (default: `custom`, options: `openai`, `anthropic`, `gemini`, `bedrock`, `ollama`, `deepseek`, `glm`, `kimi`, `qwen`, `minimax`, `custom`)
- `-flow <id>` - Flow ID for testing functions that require it (0 means using mocks, default: `0`)
- `-user <id>` - User ID for testing functions that require it (default: `0`; `1` is the default admin user)
- `-task <id>` - Task ID for agent context (optional)
- `-subtask <id>` - Subtask ID for agent context (optional)

Function-specific arguments are passed after the function name using `-name value` format.

### Pentesting Prompt Methodology

When refining prompts for offensive security work, give the agent a clear methodology instead of a flat list of payloads:

1. Start with explicit scope, authorization, and success criteria
2. Map the application first: roles, routes, parameters, uploads, integrations, and trust boundaries
3. Prioritize attack surfaces systematically instead of testing everything at once
4. Validate findings with reproducible evidence before escalating to deeper exploitation
5. Finish with report-ready notes that capture impact, prerequisites, and next steps

For PentAGI-specific prompt guidance, see [`backend/docs/prompt_engineering_pentagi.md`](backend/docs/prompt_engineering_pentagi.md). For a practical starting point, reuse and adapt [`examples/prompts/base_web_pentest.md`](examples/prompts/base_web_pentest.md) to match the target application, technology stack, and engagement scope.

## Building

### Building Docker Image

The Docker build process automatically embeds version information from git tags. To properly version your build, use the provided scripts:

#### Linux/macOS

```bash
# Load version variables
source ./scripts/version.sh

# Standard build
docker build \
  --build-arg PACKAGE_VER=$PACKAGE_VER \
  --build-arg PACKAGE_REV=$PACKAGE_REV \
  -t pentagi:$PACKAGE_VERSION_FULL .

# Multi-platform build
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --build-arg PACKAGE_VER=$PACKAGE_VER \
  --build-arg PACKAGE_REV=$PACKAGE_REV \
  -t pentagi:$PACKAGE_VERSION_FULL .

# Build and push
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --build-arg PACKAGE_VER=$PACKAGE_VER \
  --build-arg PACKAGE_REV=$PACKAGE_REV \
  -t myregistry/pentagi:$PACKAGE_VERSION_FULL \
  --push .
```

#### Windows (PowerShell)

```powershell
# Load version variables
. .\scripts\version.ps1

# Standard build
docker build `
  --build-arg PACKAGE_VER=$env:PACKAGE_VER `
  --build-arg PACKAGE_REV=$env:PACKAGE_REV `
  -t pentagi:$env:PACKAGE_VERSION_FULL .

# Multi-platform build
docker buildx build `
  --platform linux/amd64,linux/arm64 `
  --build-arg PACKAGE_VER=$env:PACKAGE_VER `
  --build-arg PACKAGE_REV=$env:PACKAGE_REV `
  -t pentagi:$env:PACKAGE_VERSION_FULL .
```

#### Quick build without version

For development builds without version tracking:

```bash
docker build -t pentagi:dev .
```

> [!NOTE]
> - The build scripts automatically determine version from git tags
> - Release builds (on tag commit) have no revision suffix
> - Development builds (after tag) include commit hash as revision (e.g., `1.1.0-ce.hbc6e800`); the edition is `ce` for Community and `ee` for Enterprise, and a release build is `1.1.0-ce`
> - To use the built image locally, update the image name in `docker-compose.yml` or use the `build` option

## Credits

This project is made possible thanks to the following research and developments:
- [Emerging Architectures for LLM Applications](https://lilianweng.github.io/posts/2023-06-23-agent)
- [A Survey of Autonomous LLM Agents](https://arxiv.org/abs/2403.08299)
- [Codel](https://github.com/semanser/codel) by Andriy Semenets - initial architectural inspiration for agent-based automation

## License

**PentAGI** is licensed under the [MIT License](LICENSE).

Copyright (c) 2025 PentAGI Development Team

### Third-Party Dependencies

All third-party dependencies use MIT-compatible licenses. See [licenses/](licenses/) directory for detailed license reports.

### VXControl Cloud Services

⚠️ **Note:** While the VXControl Cloud SDK code is MIT licensed, accessing **VXControl Cloud Services** (threat intelligence, AI support, premium features) requires a separate License Key and compliance with [Terms of Service](https://github.com/vxcontrol/cloud#license-and-terms).

The SDK code itself is free to use - service access requires registration.

For questions contact: **info@pentagi.com** or **info@vxcontrol.com**

---

### Backend Docs Installer Installer Overview (`backend/docs/installer/installer-overview.md`)

# PentAGI Installer Overview

> Comprehensive guide to the PentAGI installer - a robust Terminal User Interface (TUI) for configuring and deploying PentAGI services.

## 🎯 **Project Overview**

The PentAGI installer provides a modern, interactive Terminal User Interface for configuring and deploying the PentAGI autonomous penetration testing platform. Built using the [Charm](https://charm.sh/) tech stack, it implements responsive design patterns optimized for terminal environments.

### **Core Purpose**
- **Configuration Management**: Interactive setup of LLM providers, monitoring, and security settings
- **Environment Setup**: Automated configuration of Docker services and environment variables
- **User Experience**: Professional TUI with intuitive navigation and real-time validation
- **Production Ready**: Robust error handling, state persistence, and graceful degradation

### **Build Command**
```bash
# From backend/ directory
go build -o ../build/installer ./cmd/installer/main.go

# Monitor debug output
tail -f log.json | jq '.'
```

## 🏗️ **Technology Stack**

### **Core Technologies**
- **TUI Framework**: BubbleTea (Model-View-Update pattern)
- **Styling**: Lipgloss (CSS-like styling for terminals)
- **Components**: Bubbles (viewport, textinput, etc.)
- **Markdown**: Glamour (markdown rendering)
- **Language**: Go 1.21+

### **Architecture Components**
- **Navigation**: Type-safe screen routing with parameter passing
- **State Management**: Persistent configuration with environment variable integration
- **Layout System**: Responsive design with breakpoint-based layouts
- **Form System**: Dynamic forms with validation and auto-completion
- **Controller Layer**: Business logic abstraction from UI components

## 🎯 **Key Features**

### **Responsive Design**
- **Adaptive Layout**: Automatically adjusts to terminal size
- **Breakpoint System**: Horizontal/vertical layouts based on terminal width
- **Content Hiding**: Graceful degradation when space is insufficient
- **Dynamic Sizing**: Form fields and panels resize automatically

### **Interactive Configuration**
- **LLM Providers**: Support for OpenAI, Anthropic, Gemini, Bedrock, DeepSeek, GLM, Kimi, Qwen, Ollama, Custom endpoints
- **Monitoring Setup**: Langfuse integration for LLM observability
- **Observability**: Complete monitoring stack with Grafana, VictoriaMetrics, Jaeger
- **Summarization**: Advanced context management for LLM interactions

### **Professional UX**
- **Auto-Scrolling Forms**: Fields automatically scroll into view when focused
- **Tab Completion**: Boolean fields offer `true`/`false` suggestions
- **Real-time Validation**: Immediate feedback with human-readable error messages
- **Resource Estimation**: Live calculation of token usage and memory requirements
- **State Persistence**: Navigation and form state preserved across sessions

## 🏗️ **Architecture Overview**

### **Directory Structure**
```
backend/cmd/installer/
├── main.go                 # Application entry point
├── wizard/
│   ├── app.go             # Main application controller
│   ├── controller/        # Business logic layer
│   │   └── controller.go
│   ├── locale/            # Localization constants
│   │   └── locale.go
│   ├── logger/            # TUI-safe logging
│   │   └── logger.go
│   ├── models/            # Screen implementations
│   │   ├── welcome.go     # Welcome screen
│   │   ├── eula.go        # EULA acceptance
│   │   ├── main_menu.go   # Main navigation
│   │   ├── llm_providers.go
│   │   ├── llm_provider_form.go
│   │   ├── summarizer.go
│   │   ├── summarizer_form.go
│   │   ├── update_overview.go   # What an update changes (informational)
│   │   ├── installer_update.go  # Installer build download
│   │   └── types.go       # Shared types
│   ├── styles/            # Styling and layout
│   │   └── styles.go
│   └── window/            # Terminal size management
│       └── window.go
```

### **Component Responsibilities**

#### **App Layer** (`app.go`)
- Global navigation management
- Screen lifecycle (creation, initialization, cleanup)
- Unified header and footer rendering
- Window size distribution to models
- Global event handling (ESC, Ctrl+C, resize)

#### **Models Layer** (`models/`)
- Screen-specific logic and state
- User interaction handling
- Content rendering (content area only)
- Local state management

#### **Controller Layer** (`controller/`)
- Business logic abstraction
- Environment variable management
- Configuration persistence
- State validation

#### **Styles Layer** (`styles/`)
- Centralized styling and theming
- Dimension management (singleton pattern)
- Memoised glamour renderers: one per wrap width, plus the default 80-column one
- Responsive style calculations

#### **Window Layer** (`window/`)
- Terminal size management
- Content area size calculations
- Dimension change coordination

## 🎯 **Navigation System**

### **Composite ScreenID Architecture**
The installer implements a sophisticated navigation system using composite screen IDs:

```go
// Format: "screen§arg1§arg2§..."
type ScreenID string

// Examples:
"welcome"                    // Simple screen
"main_menu§llm_providers"    // Menu with selection
"llm_provider_form§openai"   // Form with provider type
"summarizer_form§general"    // Form with configuration type
```

### **Navigation Features**
- **Parameter Preservation**: Arguments maintained across navigation
- **Stack Management**: Proper back navigation without loops
- **State Persistence**: Complete navigation state restoration
- **Universal ESC**: Always returns to welcome screen
- **Type Safety**: Compile-time validation of screen IDs

### **Navigation Flow Example**
```
1. Start: ["welcome"]
2. Continue: ["welcome", "main_menu"]
3. LLM Providers: ["welcome", "main_menu§llm_providers", "llm_providers"]
4. OpenAI Form: [..., "llm_provider_form§openai"]
5. GoBack: [..., "llm_providers§openai"]
6. ESC: ["welcome"]
```

## 🎯 **Form System Architecture**

### **Advanced Form Patterns**
- **Boolean Fields**: Tab completion with `true`/`false` suggestions
- **Integer Fields**: Range validation with human-readable formatting
- **Environment Integration**: Direct EnvVar integration with presence detection
- **Smart Cleanup**: Automatic removal of cleared environment variables
- **Resource Estimation**: Real-time calculation of token/memory usage

### **Dynamic Field Generation**
Forms adapt based on configuration type:
```go
// Type-specific field generation
switch m.configType {
case "general":
    m.addBooleanField("use_qa", "Use QA Pairs", envVar)
    m.addIntegerField("max_sections", "Max Sections", envVar, 1, 50)
case "assistant":
    m.addIntegerField("keep_sections", "Keep Sections", envVar, 1, 10)
}
```

### **Viewport-Based Scrolling**
Forms automatically scroll to keep focused fields visible:
- **Auto-scroll**: Focused field automatically stays visible
- **Smart positioning**: Calculates field heights for precise scroll positioning
- **No extra hotkeys**: Uses existing navigation keys

## 🎯 **Configuration Management**

### **Supported Configurations**

#### **LLM Providers**
- **OpenAI**: GPT-4, GPT-3.5-turbo with API key configuration
- **Anthropic**: Claude-3, Claude-2 with API key configuration
- **Google Gemini**: Gemini Pro, Ultra with API key configuration
- **AWS Bedrock**: Multi-model support with AWS credentials
- **DeepSeek/GLM/Kimi/Qwen**: Base URL + API Key + Provider Name (optional, for LiteLLM)
- **Ollama**: Local model server integration
- **Custom**: OpenAI-compatible endpoint configuration

#### **Monitoring & Observability**
- **Langfuse**: LLM observability (embedded or external)
- **Observability Stack**: Grafana, VictoriaMetrics, Jaeger, Loki
- **Performance Monitoring**: System metrics and health checks

#### **Summarization Settings**
- **General**: Global conversation context management
- **Assistant**: Specialized settings for AI assistant contexts
- **Token Estimation**: Real-time calculation of context size

## 🎯 **Update Screens**

Two screens under **Maintenance** stand outside the generic operation form: neither edits configuration, and neither is an operation the form framework describes for itself. They sit at different distances from the form system — the overview is a plain viewport over a rendered document, while the installer update is still a `BaseScreen`, built with an empty field list, which is what gives it the help panel and the vertical/horizontal layout.

The two entries appear under different conditions. **Update Installer** shows only when a newer build is on offer *and* the update server answered. **Update PentAGI** shows whenever an installed stack is not known to be up to date — which includes a check that failed, because a failed check resets every per-stack verdict instead of leaving a stale one in place. So the overview is reachable with nothing yet known to need doing, and that is exactly why it has to be able to say the check did not complete.

### **Update Overview** (`models/update_overview.go`)

Sits **between** the "Update PentAGI" menu entry and the apply form, which the entry used to point at directly — so the first thing the user saw was "are you sure?" about something the interface had never described. The overview describes it, then continues to the same form, confirmation and all.

- **Informational only**: the document is built from the update check already in memory, so the screen makes no network call and cannot fail
- **Per stack**: the version move (`current → target`), a note when components come from different releases and the version shown is therefore the oldest of them, and the per-component plan
- **Three component verdicts, spelled out in words**: `will be updated`, `unchanged`, and `cannot verify — nothing to compare against`. The third is not a synonym for the second — a component with nothing to compare against is not a component that is current, and collapsing the two is how an installation gets told it is up to date on no evidence at all
- **The release notes of every release the update crosses**, oldest first, each with its date and a `(preview)` marker when it is not stable. An answer that carries no release list falls back to the target release's changelog and release notes
- **Stacks with nothing to do** are gathered in one list at the bottom rather than omitted: silence about a stack reads as "not installed"
- **A check that did not complete** says so instead of saying "up to date" — the screen is reachable from a menu drawn by an earlier check, and "up to date" there would be the opposite claim
- **Keys**: `enter` continues to the apply form; `↑`/`↓`, `pgup`/`pgdown`, `home`/`end` scroll. Deliberately *not* the EULA's `enter`-pages-down: this screen asks for no consent, so the one unambiguous key belongs to the one action it offers
- **Width-aware markdown**: rendered through `styles.GetRendererForWidth()` (clamped to a readable band, memoised) instead of the fixed 80-column renderer, and re-rendered only when the wrap width actually changes — a drag-resize delivers a `WindowSizeMsg` per column and glamour is not cheap

### **Installer Update** (`models/installer_update.go`)

Its own screen rather than an operation form, because the operation is a single verified download and the generic form described it as replacing the running binary and exiting.

- **Says what it will do before it does it**: running version, offered version, platform (`os/arch`), download size and the full path the file will be written to. The package description is fetched on entry so the size is on screen *before* the user agrees to the download; the answer is reused by the download itself rather than asked for twice
- **Download and verification are the whole operation.** Length, `sha256` and signature are checked in a single pass as the bytes arrive; anything that fails is deleted rather than left behind under a release-looking name
- **The installer never replaces itself**, and that is deliberate rather than unfinished: a process cannot reliably overwrite the binary it is executing — Linux refuses the write, Windows holds the file locked — and failing halfway leaves the user with no installer at all. Instead the screen prints the exact command: `mv "<downloaded file>" "<running installer>"`, or `move /Y` on Windows plus a note to close the installer first. Both paths are quoted, so an installation under a path with a space does not become two arguments; when the running executable's path cannot be determined, no command is printed at all, because a command with a guessed destination is worse than none
- **One confirmation, for one thing**: a file already sitting under that exact name — a previous download, or one interrupted partway. Writing over empty space is not a question, so it is not asked. Declining still prints the move command, since the file the user chose to keep is the build they came for
- **Where the file lands**: beside the environment file, named `installer_<version>` (`.exe` on Windows), written directly under its final name — staging an executable through the system temp directory and moving it is a pattern antivirus and EDR products score heavily. A download that would land on the running binary is refused outright
- **Progress** is reported against the size from the package description rather than a `Content-Length`, which the encrypted response body does not carry, and printed once per 10% rather than once per read — a line every few kilobytes would bury everything around it in a panel that scrolls, and the path and the verification result are printed after the last of them

## 🎯 **Localization Architecture**

### **Centralized Constants**
All user-visible text stored in `locale/locale.go`:
```go
// Screen-specific constants
const (
    WelcomeTitle = "PentAGI Installer"
    WelcomeGreeting = "Welcome to PentAGI!"

    // Form help text with practical guidance
    LLMFormOpenAIHelp = `OpenAI provides access to GPT models...

Get your API key from:
https://platform.openai.com/api-keys`
)
```

### **Multi-line Help Text**
Detailed guidance integrated into forms:
- Provider-specific setup instructions
- Configuration recommendations
- Troubleshooting tips
- Best practices

## 🎯 **Error Handling & Recovery**

### **Graceful Degradation**
- **Dimension Fallbacks**: Handles invalid terminal sizes
- **Content Fallbacks**: Shows loading states and error messages
- **Network Resilience**: Offline operation support
- **State Recovery**: Automatic restoration from corrupted state

### **User-Friendly Error Messages**
- **Validation Errors**: Real-time feedback with clear guidance
- **System Errors**: Plain English explanations with suggested fixes
- **Network Errors**: Offline alternatives and retry mechanisms

## 🎯 **Performance Considerations**

### **Optimizations**
- **Lazy Loading**: Content loaded on-demand when screens accessed
- **Memoised Renderers**: one cached glamour instance per wrap width, built once instead of per render
- **Efficient Scrolling**: Viewport-based rendering for large content
- **Memory Management**: Proper cleanup and resource sharing

### **Responsive Performance**
- **Breakpoint-Based**: Layout decisions based on terminal capabilities
- **Content Adaptation**: Hide non-essential content on small screens
- **Progressive Enhancement**: Full features on capable terminals

## 🎯 **Development Workflow**

### **File Organization**
- **One Model Per File**: Clear separation of screen logic
- **Shared Constants**: Type definitions in `types.go`
- **Centralized Locale**: All text in `locale.go`
- **Clean Dependencies**: Business logic isolated in controllers

### **Code Style**
- **Compact Syntax**: Where appropriate for readability
- **Expanded Logic**: For complex business rules
- **Comments**: Explain "why" and "how", not "what"
- **Error Handling**: Graceful degradation with user guidance

### **Testing Strategy**
- **Build Testing**: Successful compilation verification
- **Manual Testing**: Interactive validation on various terminal sizes
- **Dimension Testing**: Minimum (80x24) to large terminal support
- **Navigation Testing**: Complete flow validation

This overview provides the foundation for understanding the PentAGI installer's architecture, features, and development approach. The system prioritizes user experience, maintainability, and production reliability.

---

### Backend Docs Analytics Api (`backend/docs/analytics_api.md`)

# Analytics API

GraphQL API for system analytics and statistics monitoring. This document covers three main use cases for developers building dashboards and analytics tools.

## Important Data Notes

### Duration Storage and Calculation

**Database columns (as of migration 20260129_120000):**
- `toolcalls.duration_seconds`: Pre-calculated duration for completed toolcalls
  - Automatically set during migration: `EXTRACT(EPOCH FROM (updated_at - created_at))`
  - Set to 0.0 for incomplete toolcalls (`received`, `running`)
  - Column is `NOT NULL DEFAULT 0.0`
  - **Incremental updates**: Backend calculates delta time and passes it to SQL
  - Formula: `duration_seconds = duration_seconds + duration_delta` (parameter from Go code)
  - Updated by SQL queries: `UpdateToolcallStatus`, `UpdateToolcallFinishedResult`, `UpdateToolcallFailedResult`
  
- `msgchains.duration_seconds`: Pre-calculated duration for message chains
  - Automatically set during migration: `EXTRACT(EPOCH FROM (updated_at - created_at))`
  - Set to 0.0 for incomplete msgchains
  - Column is `NOT NULL DEFAULT 0.0`
  - **Incremental updates**: Backend calculates delta time and passes it to SQL
  - Formula: `duration_seconds = duration_seconds + duration_delta` (parameter from Go code)
  - Updated by SQL queries: `UpdateMsgChain`, `UpdateMsgChainUsage`

**Benefits of pre-calculated durations:**
- Faster query performance (no real-time `EXTRACT(EPOCH FROM ...)` calculations)
- Consistent duration values (accumulated over multiple updates)
- Simpler SQL queries for analytics (just `tc.duration_seconds` instead of complex expressions)
- Reduced database load for dashboard queries
- Incremental tracking: accurately captures time even when records are updated multiple times
- All analytics SQL queries updated to use `duration_seconds` column directly

**Incremental update logic:**
- Backend code tracks execution time and calculates delta in seconds
- Delta is passed as parameter to SQL update queries
- SQL adds delta to existing `duration_seconds` value
- Prevents overwrites and ensures accurate cumulative duration
- Works correctly for long-running operations with multiple status updates
- Backend has full control over time measurement (can use high-precision timers)

### Execution Time Metrics (`totalDurationSeconds`)

**What the data shows:**
- **Subtask duration**: Linear wall-clock time from subtask start to finish (created_at → updated_at)
  - Excludes subtasks in `created` or `waiting` status (not yet started)
  - Includes only `running`, `finished`, `failed` subtasks
  - **NOT** the sum of toolcalls (avoids double-counting nested agent calls)
  
- **Task duration**: Total execution time including:
  - Generator Agent execution (runs before subtasks)
  - All subtasks execution (with overlap compensation for batch-created subtasks)
  - Refiner Agent execution(s) (runs between subtasks)
  
- **Flow duration**: Complete flow execution time including:
  - All tasks duration (generator + subtasks + refiner)
  - Flow-level toolcalls (e.g., Assistant toolcalls without task/subtask binding)
  - **EXCLUDES** Assistant msgchain lifetime (only active toolcall time counted)

**Why not sum of toolcalls:**
- Primary Agent calls Coder Agent → Coder calls Terminal
- Summing toolcalls would count Terminal execution twice (in Coder time AND separately)
- Linear time gives accurate wall-clock duration

**Overlap compensation:**
- Generator creates multiple subtasks simultaneously (same created_at)
- Subtasks execute sequentially, not in parallel
- System compensates for this overlap to show real execution time

### Toolcalls Count Metrics (`totalCount`, `totalToolcallsCount`)

**What the data shows:**
- Only **completed** toolcalls (status = `finished` or `failed`)
- Excludes `received` (not started) and `running` (in progress) toolcalls
- Represents actual completed operations, not attempted or pending

**Use this for:**
- Counting successful/failed operations
- Calculating success rate: `finished_count / (finished_count + failed_count)`
- Understanding actual work performed

### Agent Type Breakdown

**Important distinction:**
- `primary_agent`: Main orchestrator for subtasks
- `generator`: Creates subtask plans (runs once per task, before subtasks)
- `refiner`: Updates subtask plans (runs between subtasks, can run multiple times)
- `reporter`: Generates final task reports
- `assistant`: Interactive mode (can have long idle periods)
- Specialist agents: `coder`, `pentester`, `installer`, `searcher`, `memorist`, `adviser`

**Usage interpretation:**
- High `generator` usage = complex task decomposition
- High `refiner` usage = adaptive planning (many plan adjustments)
- High specialist usage = delegated work (Primary Agent using team members)

### Agent vs Non-Agent Tools (`isAgent` field)

The `isAgent` field in `FunctionToolcallsStats` categorizes tools into two types:

**Agent Tools (`isAgent: true`):**
- Delegation to AI agents (e.g., `coder`, `pentester`, `searcher`)
- Store results from agent execution (e.g., `coder_result`, `hack_result`)
- Execution time represents delegation overhead + agent decision time
- Does NOT include nested tool calls made by the agent
- Examples: `coder`, `pentester`, `installer`, `searcher`, `memorist`, `adviser`

**Non-Agent Tools (`isAgent: false`):**
- Direct tool execution (e.g., `terminal`, `browser`, `file`)
- Search engines (e.g., `google`, `duckduckgo`, `tavily`, `firecrawl`, `sploitus`, `searxng`)
- Vector database operations (e.g., `search_in_memory`, `store_guide`)
- Environment operations (e.g., `terminal`, `file`)
- Execution time represents actual operation duration
- Examples: `terminal`, `browser`, `file`, `google`, `search_in_memory`

**Use this distinction for:**
- Analyzing delegation overhead vs direct execution time
- Identifying which agents are most frequently used
- Optimizing agent selection strategies
- Understanding the balance between AI-driven and direct operations
- Dashboard visualizations: color-code or separate agent vs non-agent tools

## Common Fragments

Define these fragments once and reuse across queries:

```graphql
fragment UsageStats on UsageStats {
  totalUsageIn
  totalUsageOut
  totalUsageCacheIn
  totalUsageCacheOut
  totalUsageCostIn
  totalUsageCostOut
}

fragment ToolcallsStats on ToolcallsStats {
  totalCount
  totalDurationSeconds
}

fragment FlowsStats on FlowsStats {
  totalFlowsCount
  totalTasksCount
  totalSubtasksCount
  totalAssistantsCount
}

fragment FlowStats on FlowStats {
  totalTasksCount
  totalSubtasksCount
  totalAssistantsCount
}

fragment FunctionToolcallsStats on FunctionToolcallsStats {
  functionName
  isAgent
  totalCount
  totalDurationSeconds
  avgDurationSeconds
}

fragment SubtaskExecutionStats on SubtaskExecutionStats {
  subtaskId
  subtaskTitle
  totalDurationSeconds
  totalToolcallsCount
}

fragment TaskExecutionStats on TaskExecutionStats {
  taskId
  taskTitle
  totalDurationSeconds
  totalToolcallsCount
  subtasks {
    ...SubtaskExecutionStats
  }
}

fragment FlowExecutionStats on FlowExecutionStats {
  flowId
  flowTitle
  totalDurationSeconds
  totalToolcallsCount
  totalAssistantsCount
  tasks {
    ...TaskExecutionStats
  }
}
```

---

## Use Case 1: Time-Filtered Dashboard

**Purpose:** Display system activity over time periods (week/month/quarter) with day-by-day breakdowns and execution metrics.

### Query

```graphql
query TimeDashboard($period: UsageStatsPeriod!) {
  # LLM token usage over time
  usageStatsByPeriod(period: $period) {
    date
    stats { ...UsageStats }
  }
  
  # Toolcalls activity over time
  toolcallsStatsByPeriod(period: $period) {
    date
    stats { ...ToolcallsStats }
  }
  
  # Flows/tasks/subtasks created over time
  flowsStatsByPeriod(period: $period) {
    date
    stats { ...FlowsStats }
  }
  
  # Flow execution times with full hierarchy
  flowsExecutionStatsByPeriod(period: $period) {
    ...FlowExecutionStats
  }
}
```

**Variables:**
```json
{ "period": "week" }  // or "month", "quarter"
```

### Data Interpretation

**Daily Usage Trends:**
- `usageStatsByPeriod`: Track token consumption patterns (input/output, cache hits, costs)
- Chart: Line graph showing daily token usage and costs
- Insights: Identify peak usage days, cost optimization opportunities

**Toolcalls Performance:**
- `toolcallsStatsByPeriod`: Monitor tool execution frequency and duration
- **Count:** Only completed operations (finished/failed), excludes pending/running
- **Duration:** Sum of individual toolcall execution times (created_at → updated_at)
- Chart: Dual-axis chart (count bars + duration line)
- Insights: Detect performance degradation, identify bottlenecks
- **Note:** Duration here IS sum of toolcalls (unlike execution stats which use wall-clock)

**Flow Activity:**
- `flowsStatsByPeriod`: Track system load (flows/tasks/subtasks/assistants created per day)
- Chart: Stacked bar chart showing hierarchy depth
- Insights: Understand workload distribution, capacity planning, assistant usage patterns

**Execution Breakdown:**
- `flowsExecutionStatsByPeriod`: Hierarchical view of execution times
- Chart: Treemap or sunburst showing time distribution across flows → tasks → subtasks
- **Important:** Duration is **wall-clock time**, not sum of toolcalls
  - Subtask: Linear time from start to finish (excludes created/waiting subtasks)
  - Task: Subtasks + Generator + Refiner agents
  - Flow: Tasks + Flow-level toolcalls (Assistant active time, NOT lifetime)
- **Count:** Only completed toolcalls (finished/failed status)
- Insights: Identify slow flows/tasks, optimize critical paths
- **Note:** Batch-created subtasks have overlap compensation applied

**Cross-Correlations:**
- Compare token usage vs execution time (efficiency metric)
- Correlate toolcall count with flow complexity
- Cost per flow: `totalUsageCost / flowsCount`
- Average toolcalls per task: `totalToolcallsCount / totalTasksCount`

---

## Use Case 2: Overall System Statistics

**Purpose:** Get comprehensive system-wide metrics without time filters.

### Query

```graphql
query SystemOverview {
  # Total LLM usage
  usageStatsTotal { ...UsageStats }
  
  # Usage breakdown by provider
  usageStatsByProvider {
    provider
    stats { ...UsageStats }
  }
  
  # Usage breakdown by model
  usageStatsByModel {
    model
    provider
    stats { ...UsageStats }
  }
  
  # Usage breakdown by agent type
  usageStatsByAgentType {
    agentType
    stats { ...UsageStats }
  }
  
  # Total toolcalls stats
  toolcallsStatsTotal { ...ToolcallsStats }
  
  # Toolcalls breakdown by function
  toolcallsStatsByFunction {
    ...FunctionToolcallsStats
  }
  
  # Total flows/tasks/subtasks
  flowsStatsTotal { ...FlowsStats }
}
```

### Data Interpretation

**Token Economics:**
- `usageStatsTotal`: Overall system cost and token consumption
- Metrics: Total spend, cache efficiency (`cacheIn / (cacheIn + usageIn)`)
- Dashboard KPI: Display as headline metrics

**Provider Distribution:**
- `usageStatsByProvider`: Cost and usage per LLM provider
- Chart: Pie chart showing provider share by cost
- Insights: Identify most/least cost-effective providers

**Model Efficiency:**
- `usageStatsByModel`: Granular per-model breakdown
- Chart: Table sorted by cost with usage metrics
- Metrics:
  - Cost per token: `(costIn + costOut) / (usageIn + usageOut)`
  - Cache hit rate per model
- Insights: Choose optimal models for different tasks

**Agent Performance:**
- `usageStatsByAgentType`: Resource consumption by agent role
- Chart: Horizontal bar chart showing usage by agent
- Insights: Understand which agents consume most resources

**Tool Usage Patterns:**
- `toolcallsStatsByFunction`: Top tool functions by usage and duration
- **Agent Classification:** The `isAgent` field indicates if the function is an agent delegation tool
  - Agent tools (e.g., `coder`, `pentester`) show their own execution time
  - Does NOT include time of nested calls they make
  - Example: `coder` toolcall = time for Coder Agent to decide + delegate, not terminal commands
  - Non-agent tools (e.g., `terminal`, `browser`) show direct execution time
- Chart: Table with sortable columns (count, duration, average, agent type)
- Metrics:
  - Slowest tools: Sort by `avgDurationSeconds`
  - Most used tools: Sort by `totalCount` (only completed)
  - Time sinks: Sort by `totalDurationSeconds`
  - Agent vs non-agent breakdown: Filter by `isAgent`
- Insights: Optimize frequently-used slow tools, identify unused tools, distinguish delegation overhead from direct execution

**System Scale:**
- `flowsStatsTotal`: Total entities in system
- Metrics:
  - Tasks per flow: `totalTasksCount / totalFlowsCount`
  - Subtasks per task: `totalSubtasksCount / totalTasksCount`
  - Assistants per flow: `totalAssistantsCount / totalFlowsCount`
- Insights: Understand average flow complexity and assistant usage

---

## Use Case 3: Flow-Specific Dashboard

**Purpose:** Deep dive into a specific flow's metrics.

### Query

```graphql
query FlowAnalytics($flowId: ID!) {
  # Basic flow info
  flow(flowId: $flowId) {
    id
    title
    status
    createdAt
    updatedAt
  }
  
  # LLM usage for this flow
  usageStatsByFlow(flowId: $flowId) { ...UsageStats }
  
  # Agent usage breakdown
  usageStatsByAgentTypeForFlow(flowId: $flowId) {
    agentType
    stats { ...UsageStats }
  }
  
  # Toolcalls stats
  toolcallsStatsByFlow(flowId: $flowId) { ...ToolcallsStats }
  
  # Tool function breakdown
  toolcallsStatsByFunctionForFlow(flowId: $flowId) {
    ...FunctionToolcallsStats
  }
  
  # Example: Separate agent and non-agent tools
  # Filter client-side: stats.filter(s => s.isAgent) for agent tools
  # Filter client-side: stats.filter(s => !s.isAgent) for direct execution tools
  
  # Flow structure stats
  flowStatsByFlow(flowId: $flowId) { ...FlowStats }
}
```

**Variables:**
```json
{ "flowId": "123" }
```

### Data Interpretation

**Flow Performance Summary:**
- `usageStatsByFlow`: Total LLM costs for this flow (all msgchains)
- `toolcallsStatsByFlow`: Execution metrics (duration, toolcall count)
  - **Count**: Only completed toolcalls (finished/failed)
  - **Duration**: Sum of individual toolcall times
- KPIs:
  - Cost per toolcall: `totalUsageCost / totalToolcallsCount`
  - Average toolcall duration: `totalDurationSeconds / totalCount`
  - Cost efficiency: tokens per second
- **Note:** For actual flow wall-clock time, use `flowsExecutionStatsByPeriod`

**Agent Activity:**
- `usageStatsByAgentTypeForFlow`: Which agents were most active
- Chart: Donut chart showing token distribution by agent
- Insights: Understand which agents drive flow execution

**Tool Usage Analysis:**
- `toolcallsStatsByFunctionForFlow`: Detailed breakdown per tool
- Chart: Bubble chart (x=count, y=avgDuration, size=totalDuration, color=isAgent)
- Metrics:
  - Identify bottleneck tools (high avgDuration)
  - Find frequently-used tools (high totalCount)
  - Calculate tool efficiency scores
  - Separate agent delegation overhead from direct execution time using `isAgent` field

**Flow Complexity:**
- `flowStatsByFlow`: Structural metrics
- Metrics:
  - Subtasks per task: `totalSubtasksCount / totalTasksCount`
  - Toolcalls per task: `toolcallsCount / tasksCount`
  - Assistants count: `totalAssistantsCount`
- Insights: Compare against average complexity to identify outliers, track assistant usage per flow

**Cross-Flow Comparisons:**
Fetch multiple flows and compare:
- Cost efficiency (cost per task)
- Execution speed (duration per task)
- Tool utilization patterns
- Agent composition differences

---

## Understanding Metric Differences

### Execution Stats vs Toolcalls Stats

These two metric types measure different aspects of system performance:

**Execution Stats (`flowsExecutionStatsByPeriod`):**
- **Purpose:** Measure real wall-clock time for flows/tasks/subtasks
- **Duration calculation:** Linear time (start → end timestamp)
- **What it shows:** How long a flow/task/subtask actually ran
- **Use for:** Performance analysis, SLA monitoring, user-facing progress
- **Example:** Subtask ran for 100 seconds (even if it made 10 toolcalls)

**Toolcalls Stats (`toolcallsStatsByPeriod`, `toolcallsStatsByFunction`):**
- **Purpose:** Measure individual tool execution metrics
- **Duration calculation:** Sum of toolcall durations (each toolcall's created_at → updated_at)
- **What it shows:** Aggregate time spent in specific tools
- **Use for:** Tool optimization, identifying slow functions, resource attribution
- **Example:** 50 terminal toolcalls totaling 300 seconds

**Key difference:**
```
Flow execution time = 100 seconds (wall-clock)
Toolcalls total time = 150 seconds (sum of all toolcalls)

Why different? 
- Flow time is LINEAR (real time elapsed)
- Toolcalls time INCLUDES OVERLAPS (nested agent calls counted in parent time)
```

**When to use which:**
- User wants to know "how long did my pentest take?" → Use **Execution Stats**
- Developer wants to optimize slow tools → Use **Toolcalls Stats**
- Manager wants to see system utilization → Use **Toolcalls Stats**
- SLA monitoring → Use **Execution Stats**

### Subtask Status and Inclusion

**Included in metrics (counted):**
- `running`: Currently executing (duration = created_at → now)
- `finished`: Completed successfully (duration = created_at → updated_at)
- `failed`: Terminated with error (duration = created_at → updated_at)

**Excluded from metrics (NOT counted):**
- `created`: Generated but not started yet (duration = 0)
- `waiting`: Paused for user input (duration = 0)

**Why this matters:**
- Generator creates 10 subtasks at once, only 1 starts executing
- You'll see 1 subtask in stats, not 10
- As more subtasks execute, they appear in metrics
- Final stats include only executed subtasks

### Assistant Time Accounting

**Assistant msgchains:**
- Can exist for days/weeks (created once, used intermittently)
- Their **lifetime is NOT counted** in flow duration
- Only their **active toolcalls** are counted

**Example:**
```
Assistant created: Monday 9 AM
User asks question: Monday 10 AM (toolcall 1: 5 seconds)
User asks question: Tuesday 3 PM (toolcall 2: 3 seconds)

Flow duration contribution: 8 seconds (5 + 3)
NOT: 30+ hours (Monday to Tuesday)
```

### Generator and Refiner Inclusion

**Task execution includes:**
1. **Generator Agent** (runs once at task start):
   - Creates initial subtask plan
   - Has msgchain with task_id, NO subtask_id
   - Time is added to task duration
   
2. **Subtasks** (execute sequentially):
   - Each has Primary Agent msgchain
   - Individual durations with overlap compensation
   
3. **Refiner Agent** (runs between subtasks):
   - Updates subtask plan based on results
   - Can run multiple times per task
   - Each run has msgchain with task_id, NO subtask_id
   - Total refiner time added to task duration

**Example task timeline:**
```
Generator (5s) → Subtask 1 (10s) → Refiner (3s) → Subtask 2 (15s) → Refiner (2s) → Subtask 3 (8s)

Task duration = 5 + 10 + 3 + 15 + 2 + 8 = 43 seconds
```

---

## Advanced Analytics Patterns

### 1. Cost Optimization Dashboard

Combine queries to identify cost reduction opportunities:

```graphql
query CostOptimization {
  usageStatsByModel {
    model
    provider
    stats { ...UsageStats }
  }
  toolcallsStatsByFunction {
    functionName
    isAgent
    totalCount
    avgDurationSeconds
  }
}
```

**Analysis:**
- Expensive models with high usage → candidates for cheaper alternatives
- Slow tools called frequently → optimization targets
- Calculate ROI per model: `performance_gain / cost_increase`

### 2. Performance Monitoring

Track system responsiveness:

```graphql
query PerformanceMetrics($period: UsageStatsPeriod!) {
  toolcallsStatsByPeriod(period: $period) {
    date
    stats { ...ToolcallsStats }
  }
  flowsExecutionStatsByPeriod(period: $period) {
    flowTitle
    totalDurationSeconds
    totalToolcallsCount
  }
}
```

**Metrics:**
- Average execution time per flow: `totalDurationSeconds / flowsCount`
- Toolcalls per day trend (detect performance degradation)
- P95/P99 flow durations (for SLA monitoring)

**Important for performance analysis:**
- Use `flowsExecutionStatsByPeriod` for wall-clock time (what users experience)
- Compare with `toolcallsStatsByPeriod` to detect overhead (high ratio = optimization needed)
- Ratio > 2.0 suggests significant nested agent call overhead

### 3. Resource Attribution

Understand resource consumption patterns:

```graphql
query ResourceAttribution {
  usageStatsByAgentType {
    agentType
    stats { ...UsageStats }
  }
  toolcallsStatsByFunction {
    functionName
    isAgent
    totalDurationSeconds
  }
}
```

**Analysis:**
- Which agents consume most resources
- Tool time distribution (execution time budget)
- Cost attribution by capability (pentesting vs coding vs searching)

---

## Implementation Tips

**Caching Strategy:**
- Cache `*StatsTotal` queries (update every 5-10 minutes)
- Cache `*StatsByPeriod` per period (update hourly)
- Real-time for flow-specific queries

**Visualization Libraries:**
- Time series: Recharts, Chart.js (daily trends)
- Hierarchical: D3 treemap/sunburst (execution breakdown)
- Tables: TanStack Table with sorting/filtering (function stats)

**Performance Optimization:**
- Use query batching for multiple flows
- Implement pagination for large datasets
- Add loading states for slow queries (execution stats can be heavy)

**Data Refresh:**
- Overall stats: Manual refresh or 10min polling
- Time-filtered: Auto-refresh on period change
- Flow-specific: Subscribe to flow updates for real-time metrics

---

## Practical Examples

### Example 1: Understanding a Slow Flow

**Scenario:** Flow took 300 seconds but only has 50 toolcalls

**Investigation:**
```graphql
query InvestigateSlowFlow($flowId: ID!) {
  flowsExecutionStatsByPeriod(period: week) {
    flowId
    flowTitle
    totalDurationSeconds
    totalToolcallsCount
    tasks {
      taskTitle
      totalDurationSeconds
      totalToolcallsCount
      subtasks {
        subtaskTitle
        totalDurationSeconds
        totalToolcallsCount
      }
    }
  }
}
```

**Analysis:**
1. Check task breakdown: Which task took longest?
2. Check subtask breakdown: Which subtasks in slow task took longest?
3. Check toolcall count: High duration + low count = slow individual operations
4. Check toolcalls by function: Which tools are slow?

**Common causes:**
- Long-running terminal commands (compilation, scanning)
- Slow search engine responses (tavily, perplexity)
- Large file operations
- Network latency for browser tool

### Example 2: Cost Attribution

**Scenario:** Need to understand cost per capability

**Query:**
```graphql
query CostAttribution {
  usageStatsByAgentType {
    agentType
    stats {
      totalUsageCostIn
      totalUsageCostOut
    }
  }
  
  toolcallsStatsByFunction {
    functionName
    totalCount
    avgDurationSeconds
  }
}
```

**Interpretation:**
- `primary_agent`: Orchestration overhead
- `generator`: Planning cost (usually low, runs once per task)
- `refiner`: Replanning cost (high value = many adjustments)
- `pentester`: Security testing operations
- `coder`: Development work
- `searcher`: Research and information gathering

**Cost optimization:**
- High generator cost → simplify task descriptions
- High refiner cost → improve initial planning (fewer adjustments needed)
- High searcher cost → use memory tools more (cheaper than web search)

### Example 3: Detecting Inefficient Flows

**Red flags:**
```
Flow A: 100 toolcalls, 500 seconds → 5s per toolcall (GOOD)
Flow B: 100 toolcalls, 5000 seconds → 50s per toolcall (INVESTIGATE)
```

**Check:**
```graphql
query DetectInefficiency {
  toolcallsStatsByFunction {
    functionName
    isAgent
    totalCount
    avgDurationSeconds
  }
}
```

**Common issues:**
- High `terminal` avg → long commands, consider timeout tuning
- High `browser` avg → slow websites, consider caching
- High `tavily`/`perplexity` avg → deep research, optimize queries
- High agent tools (`coder`, `pentester`) avg → complex delegated work

### Example 4: Understanding Task Complexity

**Scenario:** Why do some tasks take so long?

**Metrics to check:**
```
Task complexity indicators:
- High subtask count → decomposition into many steps
- High refiner calls → adaptive planning (many plan changes)
- High generator time → complex initial planning
- Low subtask count + high duration → individual subtasks are slow
```

**Query:**
```graphql
query TaskComplexity($flowId: ID!) {
  flowsExecutionStatsByPeriod(period: week) {
    flowId
    tasks {
      taskTitle
      totalDurationSeconds
      totalToolcallsCount
      subtasks {
        subtaskTitle
        totalDurationSeconds
      }
    }
  }
  
  usageStatsByAgentTypeForFlow(flowId: $flowId) {
    agentType
    stats { totalUsageIn totalUsageOut }
  }
}
```

**Analysis patterns:**
- Many subtasks + low generator usage → simple decomposition
- Many subtasks + high generator usage → complex planning
- High refiner usage → dynamic adaptation (plan changed during execution)
- Few subtasks + high duration → intensive work per subtask

---

## Data Quality Guarantees

### Accuracy

**Time measurements:**
- ✅ No double-counting of nested agent calls
- ✅ Overlap compensation for batch-created subtasks
- ✅ Excludes non-started subtasks (created/waiting)
- ✅ Includes all execution phases (generator, subtasks, refiner)

**Count measurements:**
- ✅ Only completed operations (finished/failed)
- ✅ Excludes pending (received) and in-progress (running)
- ✅ Consistent across all queries

**Cost measurements:**
- ✅ Aggregated from msgchains (source of truth for LLM calls)
- ✅ Includes cache hits and misses
- ✅ Separate input/output costs

### Known Limitations

**Current limitations:**
1. **Historical data:** Subtasks created before this update may have:
   - Missing primary_agent subtask_id (known bug, now fixed)
   - Use linear time fallback (still accurate)

2. **Running entities:** Duration calculated as `created_at → now`
   - Updates as entity continues execution
   - Final duration set when status changes to finished/failed

3. **Assistant lifetime:** Long-lived assistants
   - Only active toolcall time counted (correct behavior)
   - Msgchain lifetime NOT included in flow duration

**Edge cases handled:**
- ✅ Batch-created subtasks (overlap compensation)
- ✅ Missing primary_agent msgchain (fallback to linear time)
- ✅ Subtasks in waiting status (excluded from duration)
- ✅ Flow-level toolcalls without task binding (counted separately)
- ✅ Generator/Refiner without subtask binding (counted in task duration)

---

## Troubleshooting

### "My flow shows 0 duration but has toolcalls"

**Possible causes:**
1. All subtasks are in `created` or `waiting` status
2. Flow just started (no completed subtasks yet)

**Check:**
```graphql
query CheckFlowStatus($flowId: ID!) {
  flow(flowId: $flowId) {
    status
  }
  tasks(flowId: $flowId) {
    status
    subtasks {
      status
    }
  }
}
```

### "Task duration seems low compared to subtasks"

**This is normal if:**
- Subtasks were created in batch (overlap compensation applied)
- Example: 3 subtasks created at 10:00:00, finished at 10:00:10, 10:00:20, 10:00:30
- Naive sum: 10 + 20 + 30 = 60 seconds
- Actual time: 30 seconds (overlap compensated)

### "Toolcalls duration > execution duration"

**This is expected:**
- **Toolcalls duration:** Sum of all toolcall times (includes nested calls)
- **Execution duration:** Wall-clock time (linear)
- Nested agent calls cause toolcalls > execution
- Example: Primary Agent (100s) calls Coder (30s), toolcalls = 130s, execution = 100s

### "Count doesn't match my expectations"

**Remember:**
- Only **completed** toolcalls counted (finished/failed)
- Received (pending) and running (in-progress) excluded
- Check toolcall status distribution if counts seem low

---

## Best Practices

### Dashboard Design

**Real-time monitoring:**
- Use execution stats for user-facing progress
- Show flow/task/subtask hierarchy with durations
- Update as status changes (subscribe to updates)

**Historical analysis:**
- Use toolcalls stats for tool performance
- Use usage stats for cost tracking
- Group by period for trend analysis

**Cost optimization:**
- Compare cost per agent type
- Identify expensive models with low value
- Track cache hit rates for efficiency

### Query Optimization

**For large datasets:**
```graphql
# Don't fetch full hierarchy if not needed
query LightweightStats {
  toolcallsStatsTotal { totalCount totalDurationSeconds }
  usageStatsTotal { totalUsageCostIn totalUsageCostOut }
}

# Instead of:
query HeavyStats($period: UsageStatsPeriod!) {
  flowsExecutionStatsByPeriod(period: $period) {
    # Full hierarchy - expensive for many flows
    tasks { subtasks { ... } }
  }
}
```

**Batch requests:**
```graphql
query BatchedAnalytics {
  # Fetch multiple metrics in one request
  usageStatsTotal { ... }
  toolcallsStatsTotal { ... }
  flowsStatsTotal { ... }
}
```

---

## Data Refresh Strategy

### Real-time (WebSocket subscriptions)
- Flow status changes
- Task/subtask creation
- Toolcall completion
- **Use for:** Live flow monitoring

### Polling (every 1-5 minutes)
- Execution stats for running flows
- Toolcalls stats for active periods
- **Use for:** Dashboard auto-refresh

### Cached (refresh every 10-30 minutes)
- Historical period stats
- Total stats (system-wide)
- Provider/model breakdowns
- **Use for:** Reports and analytics

### On-demand (user action)
- Flow-specific deep dives
- Custom period queries
- Export operations
- **Use for:** Detailed investigation

---

## Migration Notes

### Duration Calculation Changes (20260129_120000)

**Previous behavior:**
- Durations calculated on-the-fly in SQL queries: `EXTRACT(EPOCH FROM (updated_at - created_at))`
- Slower query performance due to real-time calculations
- Required complex SQL expressions in every analytics query

**New behavior (improved performance):**
- Pre-calculated `duration_seconds` columns in `toolcalls` and `msgchains` tables
- Duration calculated once during migration for existing records
- Analytics queries use simple column references: `tc.duration_seconds`
- Significant performance improvement for dashboard queries (simpler execution plans)

**Migration steps:**
1. Add `duration_seconds DOUBLE PRECISION NULL` column
2. Calculate duration for existing records: `EXTRACT(EPOCH FROM (updated_at - created_at))`
3. Set remaining NULL values to 0.0
4. Alter column to `NOT NULL`
5. Set default value to 0.0 for future records

**For developers:**
- All SQL queries in `backend/sqlc/models/toolcalls.sql` updated to use `duration_seconds`
- Update queries accept `duration_delta` parameter from Go code
- Updated queries with new signatures:
  - `UpdateToolcallStatus(status, duration_delta, id)`: adds delta to duration when status changes
  - `UpdateToolcallFinishedResult(result, duration_delta, id)`: adds final delta when toolcall finishes
  - `UpdateToolcallFailedResult(result, duration_delta, id)`: adds final delta when toolcall fails
  - `UpdateMsgChain(chain, duration_delta, id)`: adds delta when chain is updated
  - `UpdateMsgChainUsage(usage_in, usage_out, usage_cache_in, usage_cache_out, usage_cost_in, usage_cost_out, duration_delta, id)`: adds delta when usage updated
- Backend code must calculate `duration_delta` in seconds before calling update
- Use `time.Since(startTime).Seconds()` or similar for accurate measurements
- SQL simply adds the delta: `duration_seconds = duration_seconds + $duration_delta`

---

### Backend Docs Chain Ast (`backend/docs/chain_ast.md`)

# ChainAST Documentation

## Table of Contents

- [ChainAST Documentation](#chainast-documentation)
  - [Table of Contents](#table-of-contents)
  - [Introduction](#introduction)
  - [Structure Overview](#structure-overview)
  - [Constants and Default Values](#constants-and-default-values)
  - [Size Tracking Features](#size-tracking-features)
  - [Creating and Using ChainAST](#creating-and-using-chainast)
    - [Basic Creation](#basic-creation)
    - [Using Constructors](#using-constructors)
    - [Getting Messages](#getting-messages)
    - [Body Pair Validation](#body-pair-validation)
    - [Common Validation Rules](#common-validation-rules)
  - [Modifying Message Chains](#modifying-message-chains)
    - [Adding Elements](#adding-elements)
    - [Adding Human Messages](#adding-human-messages)
    - [Working with Tool Calls](#working-with-tool-calls)
  - [Testing Utilities](#testing-utilities)
    - [Message Builders](#message-builders)
    - [Predefined Test Chains](#predefined-test-chains)
    - [Generating Custom Test Chains](#generating-custom-test-chains)
  - [Message Chain Structure in LLM Providers](#message-chain-structure-in-llm-providers)
    - [Message Roles](#message-roles)
    - [Message Content](#message-content)
  - [Provider-Specific Requirements](#provider-specific-requirements)
    - [Reasoning Signatures](#reasoning-signatures)
      - [Gemini (Google AI)](#gemini-google-ai)
      - [Anthropic (Claude)](#anthropic-claude)
      - [Kimi/Moonshot (OpenAI-compatible)](#kimimoonshot-openai-compatible)
    - [Helper Functions](#helper-functions)
  - [Best Practices](#best-practices)
  - [Common Use Cases](#common-use-cases)
    - [1. Chain Validation and Repair](#1-chain-validation-and-repair)
    - [2. Chain Summarization](#2-chain-summarization)
    - [3. Adding Tool Responses](#3-adding-tool-responses)
    - [4. Building a Conversation](#4-building-a-conversation)
    - [5. Using Summarization in Conversation](#5-using-summarization-in-conversation)
  - [Example Usage](#example-usage)

## Introduction

ChainAST is a structured representation of message chains used in Large Language Model (LLM) conversations. It organizes conversations into a logical hierarchy, making it easier to analyze, modify, and validate message sequences, especially when they involve tool calls and their responses.

The structure helps address common issues in LLM conversations such as:
- Validating proper conversation flow
- Managing tool calls and their responses
- Handling conversation sections and state changes
- Ensuring consistent conversation structure
- Efficient size tracking for summarization and context management

## Structure Overview

ChainAST represents a message chain as an abstract syntax tree with the following components:

```
ChainAST
├── Sections[] (ChainSection)
    ├── Header
    │   ├── SystemMessage (optional)
    │   ├── HumanMessage (optional)
    │   └── sizeBytes (total header size in bytes)
    ├── sizeBytes (total section size in bytes)
    └── Body[] (BodyPair)
        ├── Type (RequestResponse, Completion, or Summarization)
        ├── AIMessage
        ├── ToolMessages[] (for RequestResponse and Summarization types)
        └── sizeBytes (total body pair size in bytes)
```

Components:
- **ChainAST**: The root structure containing an array of sections
- **ChainSection**: A logical unit of conversation, starting with a header and containing multiple body pairs
  - Includes `sizeBytes` tracking total section size in bytes
- **Header**: Contains system and/or human messages that initiate a section
  - Includes `sizeBytes` tracking total header size in bytes
- **BodyPair**: Represents an AI response, which may include tool calls and their responses
  - Includes `sizeBytes` tracking total body pair size in bytes
- **RequestResponse**: A type of body pair where the AI message contains tool calls requiring responses
- **Completion**: A simple AI message without tool calls
- **Summarization**: A special type of body pair containing a tool call to the summarization tool

## Constants and Default Values

ChainAST provides several important constants:
- `fallbackRequestArgs`: Default arguments (`{}`) for tool calls without specified arguments
- `FallbackResponseContent`: Default response content ("the call was not handled, please try again") for missing tool responses when using force=true
- `SummarizationToolName`: Name of the special summarization tool ("execute_task_and_return_summary")
- `SummarizationToolArgs`: Default arguments for the summarization tool (`{"question": "delegate and execute the task, then return the summary of the result"}`)

## Size Tracking Features

ChainAST includes built-in size tracking to support efficient summarization algorithms and context management:

```go
// Get the size of a section in bytes
sizeInBytes := section.Size()

// Get the size of a body pair in bytes
sizeInBytes := bodyPair.Size()

// Get the size of a header in bytes
sizeInBytes := header.Size()

// Get the total size of the entire ChainAST
totalSize := ast.Size()
```

Size calculation considers all content types including:
- Text content (string length)
- Image URLs (URL string length)
- Binary data (byte count)
- Tool calls (ID, type, name, and arguments length)
- Tool call responses (ID, name, and content length)

The `sizeBytes` values are automatically maintained when:
- Creating a new ChainAST from a message chain
- Appending human messages
- Adding tool responses
- Creating elements with constructors

## Creating and Using ChainAST

### Basic Creation

```go
// Create from an existing message chain
ast, err := NewChainAST(messageChain, false)
if err != nil {
    // Handle validation error
}

// Get messages (flattened chain)
flatChain := ast.Messages()
```

The `force` parameter in `NewChainAST` determines how the function handles inconsistencies:
- `force=false`: Strict validation, returns errors for any inconsistency
- `force=true`: Attempts to repair problems by:
  - Merging consecutive human messages into a single message with multiple content parts
  - Adding missing tool responses with placeholder content ("the call was not handled, please try again")
  - Skipping invalid messages like unexpected tool messages without preceding AI messages

During creation, the size of all components is calculated automatically.

### Using Constructors

ChainAST provides constructors to create elements with automatic size calculation:

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Getting Messages

Each component provides a method to get its messages in the correct order:

```go
// Get all messages from a header (system first, then human)
headerMsgs := header.Messages()

// Get all messages from a body pair (AI first, then tools)
bodyPairMsgs := bodyPair.Messages()

// Get all messages from a section
sectionMsgs := section.Messages()

// Get all messages from the ChainAST
allMsgs := ast.Messages()
```

### Body Pair Validation

The `IsValid()` method checks if a BodyPair follows the structure rules:

```go
// Check if a body pair is valid
isValid := bodyPair.IsValid()
```

Validation rules depend on the body pair type:
- For **Completion**: No tool messages allowed
- For **RequestResponse**: Must have at least one tool message
- For **Summarization**: Must have exactly one tool message
- For all types: All tool calls must have matching responses and vice versa

The `GetToolCallsInfo()` method returns detailed information about tool calls:

```go
// Get information about tool calls and responses
toolCallsInfo := bodyPair.GetToolCallsInfo()

// Check for pending or unmatched tool calls
if len(toolCallsInfo.PendingToolCallIDs) > 0 {
    // There are tool calls without responses
}

if len(toolCallsInfo.UnmatchedToolCallIDs) > 0 {
    // There are tool responses without matching tool calls
}

// Access completed tool calls
for id, pair := range toolCallsInfo.CompletedToolCalls {
    // Use tool call and response information
    toolCall := pair.ToolCall
    response := pair.Response
}
```

### Common Validation Rules

When `force=false`, NewChainAST enforces these rules:
1. First message must be System or Human
2. No consecutive Human messages
3. Tool calls must have matching responses
4. Tool responses must reference valid tool calls
5. System messages can't appear in the middle of a chain
6. AI messages with tool calls must have responses before another AI message
7. Summarization body pairs must have exactly one tool message

## Modifying Message Chains

### Adding Elements

```go
// Add a section to the ChainAST
ast.AddSection(section)

// Add a body pair to a section
section.AddBodyPair(bodyPair)
```

### Adding Human Messages

```go
// Append a new human message
ast.AppendHumanMessage("Tell me more about this topic")
```

The function follows these rules:
1. If chain is empty: Creates a new section with this message as HumanMessage
2. If the last section has body pairs (AI responses): Creates a new section with this message
3. If the last section has no body pairs and no HumanMessage: Adds this message to that section
4. If the last section has no body pairs but has HumanMessage: Appends content to the existing message

Section and header sizes are automatically updated when human messages are added or modified.

### Working with Tool Calls

```go
// Add a response to a tool call
err := ast.AddToolResponse("tool-call-id", "tool-name", "Response content")
if err != nil {
    // Handle error (tool call not found)
}

// Find all responses for a specific tool call
responses := ast.FindToolCallResponses("tool-call-id")
```

The `AddToolResponse` function:
- Searches for the specified tool call ID in AI messages
- If the tool call is found and already has a response, updates the existing response content
- If the tool call is found but doesn't have a response, adds a new response
- If the tool call is not found, returns an error

Body pair and section sizes are automatically updated when tool responses are added or modified.

## Testing Utilities

The package tests build their message chains from `pkg/cast/fixtures_test.go`. Every helper there is a function that returns a fresh chain on each call: parsing shares the `Parts` of the input messages with the AST, and `AddToolResponse`, `NormalizeToolCallIDs`, `ClearReasoning` and `SanitizeToolCallArguments` write into those parts, so a chain kept in a shared variable would change under the next test. The file is a `_test.go` file of package `cast`, so only the tests of this package can use it.

### Message Builders

```go
chainASTSystem(text)             // a system message with one text part
chainASTHuman(text)              // a human message with one text part
chainASTAnswer(text)             // an AI message with one text part: a completion
chainASTText(role, text)         // a message of any role with one text part
chainASTAI(parts...)             // an AI message with the given parts
chainASTCall(id, name)           // a "function" tool call with the arguments {}
chainASTCallWith(id, name, args) // the same call with the given arguments
chainASTTool(id, name, content)  // a tool message holding one response
```

### Predefined Test Chains

```go
// Basic chains
emptyChain()             // no messages
systemOnlyChain()        // only a system message
humanOnlyChain()         // only a human message
systemHumanChain()       // system + human messages
basicConversationChain() // system + human + AI completion

// Tool-related chains
chainWithTool()               // a call to get_weather (tool-1) with no response
chainWithSingleToolResponse() // the same call and its response
chainWithMultipleTools()      // calls to get_weather (tool-1) and get_time (tool-2), neither answered

// Complex chains
chainWithMultipleSections()    // a completion, then a second section with a call and its response
chainWithMissingToolResponse() // the two calls of chainWithMultipleTools with only tool-1 answered
chainWithUnexpectedTool()      // a completion followed by a tool message that answers no call

// Summarization chains
chainWithSummarizationAndOtherPairs() // a summarization pair, a completion, then a call and its response

// Provider switching
providerSwitchChain(withReasoning) // a scan in three sections, described below
```

`providerSwitchChain` is the chain a provider switch is tested against: `NormalizeToolCallIDs` and `ClearReasoning` run on it each on its own and, clearing first, one after the other; `NormalizeToolCallIDs`, `SanitizeToolCallArguments` and `ClearReasoning` run in the order `pkg/providers` runs them on a variant of it that needs a forced repair. Its first section holds a text preamble beside two parallel calls, one with an id already in the Anthropic format (`toolu_…`) and one with an OpenAI-style id, their two responses and a completion; the second section opens with a completion before a call whose tool message carries a text note beside the response; the third holds a summarization pair. With `withReasoning` set, the preamble, the first completion, the note and every call except the one already in the Anthropic format carry reasoning with a signature. Without it the chain is otherwise identical, which is what `ClearReasoning` must turn the first form into; both forms have the same sizes, because reasoning is not counted.

### Generating Custom Test Chains

For chains with more sections, pairs and calls than the predefined ones, use the generators:

```go
// A system prompt, one question and its completion
config := DefaultChainConfig()

// Custom configuration
config := ChainConfig{
    IncludeSystem:           true,
    Sections:                3,                                      // 3 conversation turns
    BodyPairsPerSection:     []int{1, 2, 2},                         // AI responses per section
    ToolsForBodyPairs:       []bool{false, true, false, true, true}, // which responses, counted across the chain, make calls
    ToolCallsPerBodyPair:    []int{0, 2, 0, 3, 1},                   // how many calls each of those responses makes
    IncludeAllToolResponses: true,                                   // whether every call gets its response
}

// Generate chain based on config
chain := GenerateChain(config)

// Five sections, a response with three calls in the first, third and fifth and a completion in the
// others, with the first seven tool responses of the chain dropped
complexChain := GenerateComplexChain(
    5, // number of sections
    3, // number of calls per tool-using response
    7, // number of tool responses to drop, counted from the start of the chain
)
```

The `ChainConfig` struct controls the generated chain:
- `IncludeSystem`: whether the chain opens with the system message "You are a helpful assistant."
- `Sections`: the number of conversation turns; turn n opens with the human message "Question n"
- `BodyPairsPerSection`: the number of AI responses in each section; a section past the end of the slice has one
- `ToolsForBodyPairs`: which AI responses make tool calls, indexed by the position of the response in the whole chain; a response past the end of the slice is the completion "Response to question n"
- `ToolCallsPerBodyPair`: how many calls each tool-using response makes, indexed the same way; a response past the end of the slice makes one
- `IncludeAllToolResponses`: whether every call is followed by its response; without it no call is answered

Calls are numbered `tool-1`, `tool-2`, … across the whole chain. The k-th call of a response is `get_data_k` with the arguments `{"query": "Test query k"}`, and its response, one tool message per call in the order of the calls, reads "Response for tool-N".

`GenerateComplexChain(sections, calls, missing)` gives every section one AI response, a response with `calls` calls in the first, third, fifth… section and a completion in the others, answers every call, and then drops the first `missing` tool messages of the chain.

## Message Chain Structure in LLM Providers

ChainAST is designed to work with message chains that follow common conventions in LLM providers:

### Message Roles

- **System**: Provides context or instructions to the model
- **Human/User**: User input messages
- **AI/Assistant**: Model responses
- **Tool**: Results of tool calls executed by the system

### Message Content

Messages can contain different types of content:
- **TextContent**: Simple text messages
- **ToolCall**: Function call requests from the model
- **ToolCallResponse**: Results returned from executing tools

## Provider-Specific Requirements

### Reasoning Signatures

Different LLM providers have specific requirements for reasoning content in function calls:

#### Gemini (Google AI)

Gemini requires **thought signatures** (`thought_signature`) for function calls, especially in multi-turn conversations with tool use. These signatures:

- Are cryptographic representations of the model's internal reasoning process
- Are strictly validated only for the **current turn** (defined as all messages after the last user message with text content)
- Must be preserved when summarizing content that contains them
- Can use fake signatures when creating summarized content: `"skip_thought_signature_validator"`

**Example:**
```go
// Check if original content had reasoning
hasReasoning := ContainsReasoning(originalMessages)

// Create summarized content with fake signature if needed
summaryPair := NewBodyPairFromSummarization(summaryText, tcIDTemplate, hasReasoning)
```

#### Anthropic (Claude)

Anthropic uses **extended thinking** with cryptographic signatures that:

- Are automatically removed from previous turns (not counted in context window)
- Are only required for the current tool use loop

#### Kimi/Moonshot (OpenAI-compatible)

Kimi reasoning models require **reasoning_content in TextContent** before ToolCall:

- Reasoning must be present in a TextContent part before any ToolCall when thinking is enabled
- Error: "thinking is enabled but reasoning_content is missing in assistant tool call message"
- Use `ExtractReasoningMessage()` to preserve reasoning TextContent when summarizing
- Combine with fake ToolCall signatures for full multi-provider compatibility

**Example structure:**
```go
AIMessage.Parts = [
    TextContent{Text: "...", Reasoning: {Content: "..."}},  // Required by Kimi
    ToolCall{..., Reasoning: {Signature: []byte("...")}},  // Required by Gemini
]
```

**Critical Rule:** Never summarize the last body pair in a section, as this preserves reasoning signatures required by Gemini, Anthropic, and Kimi.

### Helper Functions

```go
// Check if messages contain reasoning signatures in ToolCall parts
// Returns true if any message contains Reasoning in ToolCall (NOT TextContent)
// This is specific to function calling scenarios which require thought_signature
hasToolCallReasoning := ContainsToolCallReasoning(messages)

// Extract reasoning message from AI messages
// Returns the first AI message with TextContent containing reasoning (or nil)
// Useful for preserving reasoning content for providers like Kimi (Moonshot)
reasoningMsg := ExtractReasoningMessage(messages)

// Create summarization with conditional fake signature and reasoning message
addFakeSignature := ContainsToolCallReasoning(originalMessages)
reasoningMsg := ExtractReasoningMessage(originalMessages)
summaryPair := NewBodyPairFromSummarization(summaryText, tcIDTemplate, addFakeSignature, reasoningMsg)
```

## Best Practices

1. **Validation First**: Use `NewChainAST` with `force=false` to validate chains before processing
2. **Defensive Programming**: Always check for errors from ChainAST functions
3. **Complete Tool Calls**: Ensure all tool calls have corresponding responses before sending to an LLM
4. **Section Management**: Use sections to organize conversation turns logically
5. **Testing**: Use the predefined chains and the generators of `fixtures_test.go` to test the code of this package that manipulates message chains
6. **Size Management**: Leverage size tracking to maintain efficient context windows
7. **Reasoning Preservation**: 
   - Use `ContainsToolCallReasoning()` to check if fake signatures are needed (checks only ToolCall.Reasoning)
   - Use `ExtractReasoningMessage()` to preserve reasoning TextContent for Kimi/Moonshot
8. **Last Pair Protection**: Never summarize the last (most recent) body pair in a section to preserve reasoning signatures
9. **Multi-Provider Support**: When summarizing for current turn, preserve both ToolCall and TextContent reasoning for maximum compatibility

## Common Use Cases

### 1. Chain Validation and Repair

```go
// Try to parse with strict validation
ast, err := NewChainAST(chain, false)
if err != nil {
    // If validation fails, try with repair enabled
    ast, err = NewChainAST(chain, true)
    if err != nil {
        // Handle severe structural errors
    }
    // Log that the chain was repaired
}
```

### 2. Chain Summarization

```go
// Create AST from chain
ast, _ := NewChainAST(chain, true)

// Analyze total size and section sizes
totalSize := ast.Size()
if totalSize > maxContextSize {
    // Select sections to summarize
    oldestSections := ast.Sections[:len(ast.Sections)-1] // Keep last section

    // Summarize sections
    summaryText := generateSummary(oldestSections)

    // Create a new AST with the summary
    newAST := &ChainAST{Sections: []*ChainSection{}}

    // Copy system message if exists
    var systemMsg *llms.MessageContent
    if len(ast.Sections) > 0 && ast.Sections[0].Header.SystemMessage != nil {
        systemMsgCopy := *ast.Sections[0].Header.SystemMessage
        systemMsg = &systemMsgCopy
    }

    // Create header and section
    header := NewHeader(systemMsg, nil)
    section := NewChainSection(header, []*BodyPair{})
    newAST.AddSection(section)

    // Add summarization body pair
    summaryPair := NewBodyPairFromSummarization(summaryText)
    section.AddBodyPair(summaryPair)

    // Copy the most recent section
    lastSection := ast.Sections[len(ast.Sections)-1]
    // Add appropriate logic to copy the last section

    // Get the summarized chain
    summarizedChain := newAST.Messages()
}
```

### 3. Adding Tool Responses

```go
// Parse a chain with tool calls
ast, _ := NewChainAST(chain, false)

// Find unresponded tool calls and add responses
for _, section := range ast.Sections {
    for _, bodyPair := range section.Body {
        if bodyPair.Type == RequestResponse {
            for _, part := range bodyPair.AIMessage.Parts {
                if toolCall, ok := part.(llms.ToolCall); ok {
                    // Execute the tool
                    result := executeToolCall(toolCall)

                    // Add the response
                    ast.AddToolResponse(toolCall.ID, toolCall.FunctionCall.Name, result)
                }
            }
        }
    }
}

// Get the updated chain
updatedChain := ast.Messages()
```

### 4. Building a Conversation

```go
// Create an empty AST
ast := &ChainAST{Sections: []*ChainSection{}}

// Add system message
sysMsg := &llms.MessageContent{
    Role: llms.ChatMessageTypeSystem,
    Parts: []llms.ContentPart{llms.TextContent{Text: "You are a helpful assistant"}},
}
header := NewHeader(sysMsg, nil)
section := NewChainSection(header, []*BodyPair{})
ast.AddSection(section)

// Add a human message
ast.AppendHumanMessage("Hello, how can you help me?")

// Add an AI response
aiMsg := &llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{llms.TextContent{Text: "I can answer questions, help with tasks, and more."}},
}
bodyPair := NewBodyPair(aiMsg, nil)
section.AddBodyPair(bodyPair)

// Continue the conversation
ast.AppendHumanMessage("Can you help me find information?")

// Get the message chain
chain := ast.Messages()
```

### 5. Using Summarization in Conversation

```go
// Create an empty AST
ast := &ChainAST{Sections: []*ChainSection{}}

// Create a new header with a system message
sysMsg := &llms.MessageContent{
    Role:  llms.ChatMessageTypeSystem,
    Parts: []llms.ContentPart{llms.TextContent{Text: "You are a helpful assistant."}},
}
header := NewHeader(sysMsg, nil)

// Create a new section with the header
section := NewChainSection(header, []*BodyPair{})
ast.AddSection(section)

// Add a human message requesting a summary
ast.AppendHumanMessage("Can you summarize our discussion?")

// Create a summarization body pair
summaryPair := NewBodyPairFromSummarization("This is a summary of our previous conversation about weather and travel plans.")
section.AddBodyPair(summaryPair)

// Get the message chain
chain := ast.Messages()
```

## Example Usage

```go
// Parse a conversation chain with summarization
ast, err := NewChainAST(conversationChain, true)
if err != nil {
    log.Fatalf("Failed to parse chain: %v", err)
}

// Check if any body pairs are summarization pairs
for _, section := range ast.Sections {
    for _, bodyPair := range section.Body {
        if bodyPair.Type == Summarization {
            fmt.Println("Found a summarization body pair")

            // Extract the summary text from the tool response
            for _, toolMsg := range bodyPair.ToolMessages {
                for _, part := range toolMsg.Parts {
                    if resp, ok := part.(llms.ToolCallResponse); ok &&
                       resp.Name == SummarizationToolName {
                        fmt.Printf("Summary content: %s\n", resp.Content)
                    }
                }
            }
        }
    }
}
```

---

### Backend Docs Chain Summary (`backend/docs/chain_summary.md`)

# Enhanced Chain Summarization Algorithm

## Table of Contents

- [Enhanced Chain Summarization Algorithm](#enhanced-chain-summarization-algorithm)
  - [Table of Contents](#table-of-contents)
  - [Overview](#overview)
  - [Architectural Overview](#architectural-overview)
  - [Fundamental Concepts](#fundamental-concepts)
    - [ChainAST Structure with Size Tracking](#chainast-structure-with-size-tracking)
    - [ChainAST Construction Process](#chainast-construction-process)
    - [Tool Call ID Normalization](#tool-call-id-normalization)
    - [Reasoning Content Cleanup](#reasoning-content-cleanup)
    - [Summarization Types](#summarization-types)
  - [Configuration Parameters](#configuration-parameters)
  - [Algorithm Operation](#algorithm-operation)
  - [Key Algorithm Components](#key-algorithm-components)
    - [1. Section Summarization](#1-section-summarization)
      - [Reasoning Signature Handling](#reasoning-signature-handling)
    - [2. Individual Body Pair Size Management](#2-individual-body-pair-size-management)
    - [3. Last Section Rotation](#3-last-section-rotation)
    - [4. QA Pair Management](#4-qa-pair-management)
  - [Summary Generation](#summary-generation)
  - [Helper Functions](#helper-functions)
    - [Content Detection Functions](#content-detection-functions)
  - [Code Architecture](#code-architecture)
  - [Full Process Overview](#full-process-overview)
  - [Usage Example](#usage-example)
  - [Edge Cases and Handling](#edge-cases-and-handling)
  - [Performance Considerations](#performance-considerations)
  - [Limitations](#limitations)

## Overview

The Enhanced Chain Summarization Algorithm manages context growth in conversation chains by selectively summarizing older message content while preserving recent interactions. The algorithm maintains conversation coherence by creating summarized body pairs rather than modifying existing messages. It uses configurable parameters to optimize context retention based on use cases and introduces byte-size tracking for precise content management.

Key features of the enhanced algorithm:

- **Size-aware processing** - Tracks byte size of all content to make optimal retention decisions
- **Section summarization** - Ensures all sections except the last `KeepQASections` ones consist of a header and a single body pair
- **Last section rotation** - Intelligently manages active conversation sections with size limits
- **QA pair summarization** - Focuses on question-answer sections when enabled, **preserving last `KeepQASections` sections unconditionally**
- **Body pair type preservation** - Maintains appropriate type for summarized content based on original types
- **Keep QA Sections** - Preserves a configurable number of recent QA sections without summarization, **even if they exceed `MaxQABytes`** (critical for agent state preservation)
- **Concurrent processing** - Uses goroutines for efficient parallel summarization of sections and body pairs
- **Idempotent operation** - Multiple consecutive calls do not modify already summarized content
- **Last BodyPair protection** - The last BodyPair in a section is never summarized to preserve reasoning signatures

## Architectural Overview

```mermaid
flowchart TD
    A[Input Message Chain] --> B[Convert to ChainAST]
    B --> C{Empty chain or\nsingle section?}
    C -->|Yes| O[Return Original Chain]
    C -->|No| D[Apply Section Summarization]
    D --> E{PreserveLast\nenabled?}
    E -->|Yes| F[Apply Last Section Rotation]
    E -->|No| G{UseQA\nenabled?}
    F --> G
    G -->|Yes| H[Apply QA Summarization]
    G -->|No| I[Convert AST to Messages]
    H --> I
    I --> O[Return Summarized Chain]
```

## Fundamental Concepts

### ChainAST Structure with Size Tracking

The algorithm operates on ChainAST structure from the `pentagi/pkg/cast` package that includes size tracking:

```
ChainAST
├── Sections[] (ChainSection)
    ├── Header
    │   ├── SystemMessage (optional)
    │   ├── HumanMessage (optional)
    │   └── Size() method
    ├── Body[] (BodyPair)
    │   ├── Type (Completion | RequestResponse | Summarization)
    │   ├── AIMessage
    │   ├── ToolMessages[] (for RequestResponse type)
    │   └── Size() method
    └── Size() method
```

```mermaid
classDiagram
    class ChainAST {
        +Sections[] ChainSection
        +Size() int
        +Messages() []MessageContent
        +AddSection(section)
    }

    class ChainSection {
        +Header Header
        +Body[] BodyPair
        +Size() int
        +Messages() []MessageContent
    }

    class Header {
        +SystemMessage *MessageContent
        +HumanMessage *MessageContent
        +Size() int
        +Messages() []MessageContent
    }

    class BodyPair {
        +Type BodyPairType
        +AIMessage *MessageContent
        +ToolMessages[] MessageContent
        +Size() int
        +Messages() []MessageContent
    }

    class BodyPairType {
        <<enumeration>>
        Completion
        RequestResponse
        Summarization
    }

    ChainAST "1" *-- "*" ChainSection : contains
    ChainSection "1" *-- "1" Header : has
    ChainSection "1" *-- "*" BodyPair : contains
    BodyPair --> BodyPairType : has type
```

Each component (ChainAST, ChainSection, Header, BodyPair) provides a Size() method that enables precise content management decisions. Size calculation is handled internally by the cast package and considers all content types including text, binary data, and images.

The body pair types are critical for understanding the structure:
- **Completion**: Contains a single AI message with text content
- **RequestResponse**: Contains an AI message with tool calls and corresponding tool response messages
- **Summarization**: Contains a summary of previous messages

The algorithm leverages the cast package's constructor methods to ensure proper size calculation:

```go
// Creating components with automatic size calculation
header := cast.NewHeader(systemMsg, humanMsg)       // New header with size tracking
section := cast.NewChainSection(header, bodyPairs)  // New section with size tracking
summaryPair := cast.NewBodyPairFromCompletion(text)      // New Completion pair with text content
summaryPair := cast.NewBodyPairFromSummarization(text)   // New Summarization pair with text content
```

### ChainAST Construction Process

```mermaid
flowchart TD
    A[Input MessageContent Array] --> B[Create Empty ChainAST]
    B --> C[Process Messages Sequentially]
    C --> D{Is System Message?}
    D -->|Yes| E[Add to Current/New Section Header]
    D -->|No| F{Is Human Message?}
    F -->|Yes| G[Create New Section with Human Message in Header]
    F -->|No| H{Is AI or Tool Message?}
    H -->|Yes| I[Add to Current Section's Body]
    H -->|No| J[Skip Message]
    E --> C
    G --> C
    I --> C
    J --> C
    C --> K[Calculate Sizes for All Components]
    K --> L[Return Populated ChainAST]
```

The ChainAST construction process analyzes the roles and types of messages in the chain, grouping them into logical sections with headers and body pairs.

### Tool Call ID Normalization

When switching between different LLM providers (e.g., from Gemini to Anthropic), tool call IDs may have different formats that are incompatible with the new provider's API. The `NormalizeToolCallIDs` method addresses this by validating and replacing incompatible IDs:

```mermaid
flowchart TD
    A[ChainAST with Tool Calls] --> B[Iterate Through Sections]
    B --> C{Has RequestResponse or\nSummarization?}
    C -->|No| D[Skip Section]
    C -->|Yes| E[Extract Tool Call IDs]
    E --> F{Validate ID Against\nNew Template}
    F -->|Valid| G[Keep Existing ID]
    F -->|Invalid| H[Generate New ID]
    H --> I[Create ID Mapping]
    I --> J[Update Tool Call ID]
    J --> K[Update Corresponding\nTool Response IDs]
    G --> L[Continue to Next]
    K --> L
    D --> L
    L --> M{More Sections?}
    M -->|Yes| B
    M -->|No| N[Return Normalized AST]
```

The normalization process:
1. **Validates** each tool call ID against the new provider's template using `ValidatePattern`
2. **Generates** new IDs only for those that don't match the template
3. **Preserves** IDs that already match to avoid unnecessary changes
4. **Updates** both tool calls and their corresponding responses to maintain consistency
5. **Supports** all body pair types: RequestResponse and Summarization

**Example Usage:**

```go
// After restoring a chain that may contain tool calls from a different provider
ast, err := cast.NewChainAST(chain, true)
if err != nil {
    return err
}

// Normalize to new provider's format (e.g., from "call_*" to "toolu_*")
err = ast.NormalizeToolCallIDs("toolu_{r:24:b}")
if err != nil {
    return err
}

// Chain now has compatible tool call IDs
normalizedChain := ast.Messages()
```

**Template Format Examples:**

| Provider | Template Format | Example ID |
|----------|----------------|------------|
| OpenAI/Gemini | `call_{r:24:x}` | `call_abc123def456ghi789jkl` |
| Anthropic | `toolu_{r:24:b}` | `toolu_A1b2C3d4E5f6G7h8I9j0K1l2` |
| Custom | `{prefix}_{r:N:charset}` | Defined per provider |

This feature is critical for assistant providers that may switch between different LLM providers while maintaining conversation history.

### Reasoning Content Cleanup

When switching between providers, reasoning content must also be cleared because it contains provider-specific data:

```mermaid
flowchart TD
    A[ChainAST with Reasoning] --> B[Iterate Through Sections]
    B --> C[Process Header Messages]
    C --> D[Clear SystemMessage Reasoning]
    D --> E[Clear HumanMessage Reasoning]
    E --> F[Process Body Pairs]
    F --> G[Clear AI Message Reasoning]
    G --> H[Clear Tool Messages Reasoning]
    H --> I{More Sections?}
    I -->|Yes| B
    I -->|No| J[Return Cleaned AST]
```

The cleanup process:
1. **Iterates** through all sections, headers, and body pairs
2. **Clears** `Reasoning` field from `TextContent` parts
3. **Clears** `Reasoning` field from `ToolCall` parts
4. **Preserves** all other content (text, arguments, function names, etc.)

**Why this is needed:**
- Reasoning content includes cryptographic signatures (especially Anthropic's extended thinking)
- These signatures are validated by the provider and will fail if sent to a different provider
- Reasoning blocks may contain provider-specific metadata

**Example Usage:**

```go
// After restoring and normalizing a chain
ast, err := cast.NewChainAST(chain, true)
if err != nil {
    return err
}

// First normalize tool call IDs
err = ast.NormalizeToolCallIDs(newTemplate)
if err != nil {
    return err
}

// Then clear provider-specific reasoning
err = ast.ClearReasoning()
if err != nil {
    return err
}

// Chain is now safe to use with the new provider
cleanedChain := ast.Messages()
```

**What gets cleared:**
- `TextContent.Reasoning` - Extended thinking signatures and content
- `ToolCall.Reasoning` - Per-tool reasoning (used by some providers)

**What stays preserved:**
- All text content
- Tool call IDs (after normalization)
- Function names and arguments
- Tool responses

This operation is automatically performed in `restoreChain()` when switching providers, ensuring compatibility across different LLM providers.

### Summarization Types

The algorithm supports three types of summarization:

1. **Section Summarization** - Ensures all sections except the last N ones consist of a header and a single body pair
2. **Last Section Rotation** - Manages size of the last (active) section by summarizing oldest pairs when size limits are exceeded
3. **QA Pair Summarization** - Creates a summary section containing essential question-answer exchanges when enabled

## Configuration Parameters

Summarization behavior is controlled through the `SummarizerConfig` structure:

```go
type SummarizerConfig struct {
    PreserveLast   bool  // Whether to manage the last section size
    UseQA          bool  // Whether to use QA pair summarization
    SummHumanInQA  bool  // Whether to summarize human messages in QA pairs
    LastSecBytes   int   // Maximum byte size for last section
    MaxBPBytes     int   // Maximum byte size for a single body pair
    MaxQASections  int   // Maximum QA pair sections to preserve
    MaxQABytes     int   // Maximum byte size for QA pair sections
    KeepQASections int   // Number of recent QA sections to keep without summarization
}
```

These parameters have default values defined as constants:

| Parameter | Field in SummarizerConfig | Default Constant | Default Value | Description |
|-----------|---------------------------|------------------|---------------|-------------|
| Max last section size | `LastSecBytes` | `maxLastSectionByteSize` | 50 KB | Maximum size for the last section |
| Max single body pair size | `MaxBPBytes` | `maxSingleBodyPairByteSize` | 16 KB | Maximum size for a single body pair |
| Max QA sections | `MaxQASections` | `maxQAPairSections` | 10 | Maximum QA sections to keep |
| Max QA byte size | `MaxQABytes` | `maxQAPairByteSize` | 64 KB | Maximum size for QA sections |
| Last section reserve percentage | N/A | `lastSectionReservePercentage` | 25% | Percentage of section size to reserve for future messages |
| Keep QA sections | `KeepQASections` | `keepMinLastQASections` | 1 | Number of most recent QA sections to preserve without summarization, even if they exceed MaxQABytes |

## Algorithm Operation

The enhanced algorithm operates in these sequential phases:

1. Convert input chain to ChainAST with size tracking
2. Apply section summarization to all sections except the last `KeepQASections` sections (with concurrent processing)
3. Apply last section rotation to multiple recent sections if enabled and size limits are exceeded
4. Apply QA pair summarization if enabled and limits are exceeded, **preserving the last `KeepQASections` sections**
5. Return the modified chain if it saves space

**Critical Guarantees:**
- The last `KeepQASections` sections are **NEVER** summarized by section or QA summarization, even if they exceed `MaxQABytes`
- The last BodyPair in a section is **NEVER** summarized by `summarizeOversizedBodyPairs` or `summarizeLastSection` to preserve reasoning signatures
- **Idempotent**: calling `SummarizeChain` multiple times on already summarized content does not change it further

The primary algorithm is implemented through the `Summarizer` interface in the `pentagi/pkg/csum` package:

```go
// Summarizer interface for chain summarization
type Summarizer interface {
    SummarizeChain(
        ctx context.Context,
        handler tools.SummarizeHandler,
        chain []llms.MessageContent,
    ) ([]llms.MessageContent, error)
}

// Implementation is created using the NewSummarizer constructor
func NewSummarizer(config SummarizerConfig) Summarizer {
    // Sets defaults if not specified
    if config.PreserveLast {
        if config.LastSecBytes <= 0 {
            config.LastSecBytes = maxLastSectionByteSize
        }
    }

    if config.UseQA {
        if config.MaxQASections <= 0 {
            config.MaxQASections = maxQAPairSections
        }
        if config.MaxQABytes <= 0 {
            config.MaxQABytes = maxQAPairByteSize
        }
    }

    if config.MaxBPBytes <= 0 {
        config.MaxBPBytes = maxSingleBodyPairByteSize
    }

    if config.KeepQASections <= 0 {
        config.KeepQASections = keepMinLastQASections
    }

    return &summarizer{config: config}
}
```

The main algorithm flow:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Key Algorithm Components

### 1. Section Summarization

For all sections except the last `KeepQASections` sections, ensure they consist of a header and a single body pair:

```mermaid
flowchart TD
    A[For each section except last KeepQASections] --> B{Has single body pair that\nis already summarized?}
    B -->|Yes| A
    B -->|No| C[Collect all messages from body pairs]
    C --> D[Add human message if it exists]
    D --> E[Start concurrent goroutine for summary generation]
    E --> F[Determine appropriate body pair type]
    F --> G[Create new body pair with summary]
    G --> H[Replace all body pairs with summary pair]
    H --> I[Wait for all goroutines to complete]
    I --> J[Check for any errors from parallel processing]
    J --> A
    A --> K[Return updated AST]
```

```
/* Detailed source-code truncated for AI context efficiency. */
```

The `determineTypeToSummarizedSection` function decides which type to use for the summarized content based on the original section's body pair types:

```go
// determineTypeToSummarizedSection determines the type of each body pair to summarize
// based on the type of the body pairs in the section
// if all body pairs are Completion, return Completion, otherwise return Summarization
func determineTypeToSummarizedSection(section *cast.ChainSection) cast.BodyPairType {
    summarizedType := cast.Completion
    for _, pair := range section.Body {
        if pair.Type == cast.Summarization || pair.Type == cast.RequestResponse {
            summarizedType = cast.Summarization
            break
        }
    }

    return summarizedType
}
```

#### Reasoning Signature Handling

When summarizing content that originally contained reasoning, the algorithm:

1. **Detects ToolCall Reasoning**: Uses `cast.ContainsToolCallReasoning()` to check if messages contain reasoning in ToolCall parts (NOT TextContent)
2. **Extracts TextContent Reasoning**: Uses `cast.ExtractReasoningMessage()` to preserve reasoning TextContent for providers like Kimi/Moonshot
3. **Adds Fake Signatures**: Adds a fake signature to the summarized ToolCall if ToolCall reasoning was present
4. **Preserves Reasoning Message**: Prepends reasoning TextContent before ToolCall in the summarized content
5. **Provider Compatibility**: Ensures the summarized chain remains compatible with all provider APIs

```go
// Check if the original pair contained reasoning signatures in ToolCall parts
addFakeSignature := cast.ContainsToolCallReasoning(pairMessages)

// Extract reasoning message for Kimi/Moonshot compatibility
reasoningMsg := cast.ExtractReasoningMessage(pairMessages)

// Create summarization with conditional fake signature AND preserved reasoning
bodyPairsSummarized[i] = cast.NewBodyPairFromSummarization(summaryText, tcIDTemplate, addFakeSignature, reasoningMsg)
```

**Why this is needed:**

- **Gemini**: Validates `thought_signature` presence for function calls in the current turn. Removing signatures would cause 400 errors: "Function call is missing a thought_signature". Fake signatures satisfy the API validation.
- **Kimi/Moonshot**: Requires `reasoning_content` in TextContent before ToolCall when thinking is enabled. Without it: "thinking is enabled but reasoning_content is missing". Preserving reasoning message satisfies this requirement.
- **Anthropic**: Extended thinking with cryptographic signatures, automatically removed from previous turns.

**Important:** This reasoning preservation is **only applied to current turn** (last section). Previous turns are summarized without fake signatures to save tokens, as they are not validated by provider APIs.

### 2. Individual Body Pair Size Management

Before handling the overall last section size, manage individual oversized body pairs:

**CRITICAL PRESERVATION RULES**:

1. **Never Summarize Last Pair**: The last (most recent) body pair in a section is **NEVER** summarized to preserve reasoning signatures required by providers like Gemini (thought_signature) and Anthropic (cryptographic signatures). Summarizing the last pair would remove these signatures and cause API errors.

2. **Preserve Reasoning Requirements**: When summarizing body pairs that contain reasoning signatures:
   - The algorithm checks if the original content contained reasoning using `cast.ContainsReasoning()`
   - If reasoning was present, a fake signature is added to the summarized content
   - For Gemini: uses `"skip_thought_signature_validator"`
   - This ensures API compatibility when the chain continues with the same provider

```mermaid
flowchart TD
    A[Start with Section's Body Pairs] --> B[Initialize concurrent processing]
    B --> C[For each body pair]
    C --> CA{Is this the\nlast body pair?}
    CA -->|Yes| CB[SKIP - Never summarize last pair]
    CA -->|No| D{Is pair oversized AND\nnot already summarized?}
    CB --> C
    D -->|Yes| E[Start goroutine for pair processing]
    D -->|No| C
    E --> F[Get messages from pair]
    F --> G[Add human message if exists]
    G --> H[Generate summary]
    H --> I{Was summary generation successful?}
    I -->|No| J[Skip this pair - handled by next step]
    I -->|Yes| K{What is the original pair type?}
    K -->|RequestResponse| L[Create Summarization pair]
    K -->|Other| M[Create Completion pair]
    L --> N[Add to modified pairs map with mutex]
    M --> N
    N --> O[Wait for all goroutines to complete]
    O --> P{Any pairs summarized?}
    P -->|Yes| Q[Update section with new pairs]
    P -->|No| R[Return unchanged]
    Q --> S[Return updated section]
```

```
/* Detailed source-code truncated for AI context efficiency. */
```

### 3. Last Section Rotation

For the specified section (which can be any of the last N sections in the active conversation), when it exceeds size limits:

**CRITICAL PRESERVATION RULE**: The last (most recent) body pair in a section is **ALWAYS** kept without summarization. This ensures:
1. **Reasoning signatures** (Gemini's thought_signature, Anthropic's cryptographic signatures) are preserved
2. **Latest tool calls** maintain their complete context including thinking content
3. **API compatibility** when the chain continues with the same provider

```mermaid
flowchart TD
    A[Get Specified Section by Index] --> B[Summarize Oversized Individual Pairs]
    B --> C{Is section size still\nexceeding limit?}
    C -->|No| Z[Return Unchanged]
    C -->|Yes| D[Determine which pairs to keep vs. summarize]
    D --> E{Any pairs to summarize?}
    E -->|No| Z
    E -->|Yes| F[Collect messages from pairs to summarize]
    F --> G[Add human message if it exists]
    G --> H[Generate summary text]
    H --> I{Summary generation successful?}
    I -->|No| J[Keep only recent pairs]
    I -->|Yes| K[Determine type for summary pair]
    K --> L[Create appropriate body pair]
    L --> M[Create new body with summary pair first]
    M --> N[Add kept pairs after summary]
    N --> O[Create new section with updated body]
    O --> P[Update specified section in AST]
    J --> P
    P --> Z[Return Updated AST]
```

```
/* Detailed source-code truncated for AI context efficiency. */
```

The `determineLastSectionPairs` function is a critical piece that decides which pairs to keep and which to summarize:

```mermaid
flowchart TD
    A[Start with Section] --> B[Calculate header size]
    B --> C{Are there any body pairs?}
    C -->|No| D[Return empty arrays]
    C -->|Yes| E[Always keep last pair]
    E --> F[Calculate threshold with reserve]
    F --> G[Process remaining pairs in reverse order]
    G --> H{Would pair fit within threshold?}
    H -->|Yes| I[Add to pairs to keep]
    H -->|No| J[Add to pairs to summarize]
    I --> G
    J --> G
    G --> K[Return pairs to keep and summarize]
```

```go
// determineLastSectionPairs splits the last section's pairs into those to keep and those to summarize
func determineLastSectionPairs(
    section *cast.ChainSection,
    maxBytes int,
    reservePercent int,
) ([]*cast.BodyPair, []*cast.BodyPair) {
    // Implementation details...
    // Returns two slices: pairsToKeep and pairsToSummarize
}
```

### 4. QA Pair Management

When QA pair summarization is enabled, the algorithm **preserves the last `KeepQASections` sections** without summarization, even if they exceed `MaxQABytes`. This ensures that:
- Recent reasoning blocks are preserved for AI agent continuation
- Tool calls in the most recent sections maintain their full context
- Agent state remains intact for multi-turn conversations

```mermaid
flowchart TD
    A[Check limits] --> B{Do QA sections exceed limits?}
    B -->|No| Z[Return unchanged]
    B -->|Yes| C[Prepare sections for summarization]
    C --> D{Any human/AI messages to summarize?}
    D -->|No| Z
    D -->|Yes| E{Human messages exist?}
    E -->|Yes| F{Summarize human messages?}
    F -->|Yes| G[Generate human summary]
    F -->|No| H[Concatenate human messages]
    E -->|No| I[No human message needed]
    G --> J[Generate AI summary]
    H --> J
    I --> J
    J --> K[Determine summary pair type]
    K --> L[Create new AST with summary section]
    L --> M[Add system message if it exists]
    M --> N[Create summary section header]
    N --> O[Create summary section with summary pair]
    O --> P[Determine how many recent sections to keep]
    P --> Q[Add those sections to new AST]
    Q --> R[Replace original sections with new ones]
    R --> Z
```

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Summary Generation

The algorithm uses a `GenerateSummary` function to create summaries:

```mermaid
flowchart TD
    A[GenerateSummary Called] --> B{Handler is nil?}
    B -->|Yes| C[Return error]
    B -->|No| D{No messages to summarize?}
    D -->|Yes| C
    D -->|No| E[Convert messages to prompt]
    E --> F[Call handler to generate summary]
    F --> G{Handler error?}
    G -->|Yes| C
    G -->|No| H[Return summary text]
```

```go
// GenerateSummary generates a summary of the provided messages
func GenerateSummary(
    ctx context.Context,
    handler tools.SummarizeHandler,
    humanMessages []llms.MessageContent,
    aiMessages []llms.MessageContent,
) (string, error) {
    if handler == nil {
        return "", fmt.Errorf("summarizer handler cannot be nil")
    }

    if len(humanMessages) == 0 && len(aiMessages) == 0 {
        return "", fmt.Errorf("cannot summarize empty message list")
    }

    // Convert messages to text format optimized for summarization
    text := messagesToPrompt(humanMessages, aiMessages)

    // Generate the summary using provided summarizer handler
    summary, err := handler(ctx, text)
    if err != nil {
        return "", fmt.Errorf("summarization failed: %w", err)
    }

    return summary, nil
}
```

The `messagesToPrompt` function handles different summarization scenarios:

```mermaid
flowchart TD
    A[messagesToPrompt] --> B[Convert human messages to text]
    A --> C[Convert AI messages to text]
    B --> D{Both human and AI messages exist?}
    C --> D
    D -->|Yes| E[Use Case 1: Human as context for AI]
    D -->|No| F{Only AI messages?}
    F -->|Yes| G[Use Case 2: AI without context]
    F -->|No| H{Only human messages?}
    H -->|Yes| I[Use Case 3: Human as instructions]
    H -->|No| J[Return empty string]
    E --> K[Format with appropriate instructions]
    G --> K
    I --> K
    K --> L[Return formatted prompt]
```

```go
// messagesToPrompt converts a slice of messages to a text representation
func messagesToPrompt(humanMessages []llms.MessageContent, aiMessages []llms.MessageContent) string {
    var buffer strings.Builder

    humanMessagesText := humanMessagesToText(humanMessages)
    aiMessagesText := aiMessagesToText(aiMessages)

    // Different cases based on available messages
    // case 1: use human messages as a context for ai messages
    if len(humanMessages) > 0 && len(aiMessages) > 0 {
        instructions := getSummarizationInstructions(1)
        buffer.WriteString(fmt.Sprintf("<instructions>%s</instructions>\n\n", instructions))
        buffer.WriteString(humanMessagesText)
        buffer.WriteString(aiMessagesText)
    }

    // case 2: use ai messages as a content to summarize without context
    if len(aiMessages) > 0 && len(humanMessages) == 0 {
        instructions := getSummarizationInstructions(2)
        buffer.WriteString(fmt.Sprintf("<instructions>%s</instructions>\n\n", instructions))
        buffer.WriteString(aiMessagesText)
    }

    // case 3: use human messages as a instructions to summarize them
    if len(humanMessages) > 0 && len(aiMessages) == 0 {
        instructions := getSummarizationInstructions(3)
        buffer.WriteString(fmt.Sprintf("<instructions>%s</instructions>\n\n", instructions))
        buffer.WriteString(humanMessagesText)
    }

    return buffer.String()
}
```

The algorithm includes detailed instructions for each summarization scenario through the `getSummarizationInstructions` function, which ensures appropriate summaries for different contexts.

## Helper Functions

The algorithm includes several important helper functions that support the summarization process:

### Content Detection Functions

```go
// containsSummarizedContent checks if a body pair contains summarized content
// Local helper function to avoid naming conflicts with test utilities
func containsSummarizedContent(pair *cast.BodyPair) bool {
    if pair == nil {
        return false
    }

    switch pair.Type {
    case cast.Summarization:
        return true
    case cast.RequestResponse:
        return false
    case cast.Completion:
        if pair.AIMessage == nil || len(pair.AIMessage.Parts) == 0 {
            return false
        }

        textContent, ok := pair.AIMessage.Parts[0].(llms.TextContent)
        if !ok {
            return false
        }

        if strings.HasPrefix(textContent.Text, SummarizedContentPrefix) {
            return true
        }

        return false
    default:
        return false
    }
}
```

This function is crucial for:
- **Avoiding double summarization**: Prevents already summarized content from being summarized again
- **Type-aware detection**: Handles different body pair types appropriately
- **Content prefix detection**: Recognizes summarized content by checking for the `SummarizedContentPrefix` marker
- **Robust checking**: Safely handles nil values and missing content

The function replaces the previous logic that only checked for `cast.Summarization` type, providing more comprehensive detection of summarized content across all body pair types.

## Code Architecture

```mermaid
classDiagram
    class SummarizerConfig {
        +bool PreserveLast
        +bool UseQA
        +bool SummHumanInQA
        +int LastSecBytes
        +int MaxBPBytes
        +int MaxQASections
        +int MaxQABytes
        +int KeepQASections
    }

    class Summarizer {
        <<interface>>
        +SummarizeChain(ctx, handler, chain) []llms.MessageContent, error
    }

    class summarizer {
        -config SummarizerConfig
        +SummarizeChain(ctx, handler, chain) []llms.MessageContent, error
    }

    class SummarizeHandler {
        <<function>>
        +invoke(ctx, text) string, error
    }

    class tools.SummarizeHandler {
        <<function>>
        +invoke(ctx, text) string, error
    }

    Summarizer <|.. summarizer : implements
    summarizer -- SummarizerConfig : uses
    summarizer -- SummarizeHandler : calls
    SummarizeHandler -- tools.SummarizeHandler : alias
```

The algorithm is implemented through the `Summarizer` interface in the `pentagi/pkg/csum` package, which provides the `SummarizeChain` method. The implementation leverages the `ChainAST` structure from the `pentagi/pkg/cast` package for managing the chain structure.

## Full Process Overview

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Usage Example

```go
// Create a summarizer with custom configuration
config := csum.SummarizerConfig{
    PreserveLast:   true,
    LastSecBytes:   40 * 1024,
    MaxBPBytes:     16 * 1024,
    UseQA:          true,
    MaxQASections:  5,
    MaxQABytes:     30 * 1024,
    SummHumanInQA:  false,
    KeepQASections: 2,
}
summarizer := csum.NewSummarizer(config)

// Define a summary handler function
summaryHandler := func(ctx context.Context, text string) (string, error) {
    // Use your preferred LLM or summarization method here
    return llmClient.Summarize(ctx, text)
}

// Apply summarization to a message chain
newChain, err := summarizer.SummarizeChain(ctx, summaryHandler, originalChain)
if err != nil {
    log.Fatalf("Failed to summarize chain: %v", err)
}

// Use the summarized chain
for _, msg := range newChain {
    fmt.Printf("[%s] %s\n", msg.Role, getMessageText(msg))
}
```

## Edge Cases and Handling

| Edge Case | Handling Strategy |
|-----------|-------------------|
| Empty chain | Return unchanged immediately without processing |
| Very short chains | Return unchanged after section count check |
| Single section chains | Return unchanged after section count check |
| Empty sections to process | Skip summarization |
| Last section over size limit | Create a new section with summary pair followed by recent pairs |
| QA pairs over limit | Create summary section and keep most recent sections |
| KeepQASections larger than number of sections | No summarization performed, preserves all sections |
| Last KeepQASections sections exceed MaxQABytes | Sections are kept anyway to preserve reasoning and agent state |
| Summary generation fails | Keep the most recent content and log the error |
| Chain with already summarized content | Detected during processing and handled appropriately (idempotent) |
| Multiple consecutive summarization calls | Idempotent - no changes after first summarization |

## Performance Considerations

1. **Token Efficiency**
   - Summarization creates body pairs that reduce overall token count
   - Size-aware decisions prevent context growth while maintaining conversation coherence
   - Multiple last section rotation prevents unbounded growth in active conversations
   - Individual oversized pair handling prevents single large pairs from affecting summarization decisions
   - KeepQASections parameter preserves recent context while summarizing older content

2. **Memory Efficiency**
   - Leverages cast package's size tracking for precise memory management
   - Creates new components only when needed (using constructors)
   - Uses Messages() methods to extract content without duplication

3. **Processing Optimization**
   - **Concurrent Processing**: Uses goroutines for parallel summarization of sections and body pairs, significantly improving performance for large chains
   - **Error Handling**: Robust error collection and handling from parallel operations using channels and error joining
   - Short-circuit logic avoids unnecessary processing for simple chains
   - Handles empty or single-section chains efficiently
   - Uses built-in size tracking methods rather than recalculating sizes
   - Selective summarization with KeepQASections avoids redundant processing
   - **Multiple Last Sections**: Processes multiple recent sections in sequence for better active conversation management

## Limitations

1. **Semantic Coherence**
   - Quality of summaries depends entirely on the provided summarizer handler
   - Summarized content may lose detailed reasoning or discussion context

2. **Content Processing**
   - Binary and image content has size tracked but content isn't semantically analyzed
   - Tool calls and responses are included in text representation for summarization

3. **Implementation Considerations**
   - Depends on ChainAST's accuracy for section and message management
   - API changes in the cast package may require updates to summarization code
   - KeepQASections parameter may need balancing between context preservation and token efficiency

---

### Backend Docs Charm (`backend/docs/charm.md`)

# Charm.sh Ecosystem - Personal Cheat Sheet

> Personal reference for building TUI applications with the Charm stack.

## 📦 **Core Libraries Overview**

### Core Packages
- **`bubbletea`**: Event-driven TUI framework (MVU pattern)
- **`lipgloss`**: Styling and layout engine
- **`bubbles`**: Pre-built components (viewport, textinput, etc.)
- **`huh`**: Advanced form builder
- **`glamour`**: Markdown renderer

## 🫧 **BubbleTea (MVU Pattern)**

### Model-View-Update Lifecycle
```go
// Model holds all state
type Model struct {
    content string
    ready   bool
}

// Update handles events and returns new state
func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.WindowSizeMsg:
        // Handle resize
        return m, nil
    case tea.KeyMsg:
        // Handle keyboard input
        return m, nil
    }
    return m, nil
}

// View renders current state
func (m Model) View() string {
    return "content"
}

// Init returns initial command
func (m Model) Init() tea.Cmd {
    return nil
}
```

### Commands and Messages
```go
// Commands return future messages
func loadDataCmd() tea.Msg {
    return DataLoadedMsg{data: "loaded"}
}

// Async operations
return m, tea.Cmd(func() tea.Msg {
    time.Sleep(time.Second)
    return TimerMsg{}
})
```

### Critical Patterns
```go
// Model interface implementation
type Model struct {
    styles *styles.Styles  // ALWAYS use shared styles
}

func (m Model) Init() tea.Cmd {
    // ALWAYS reset state completely
    m.content = ""
    m.ready = false
    return m.loadContent
}

func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.WindowSizeMsg:
        // NEVER store dimensions in model - use styles.SetSize()
        // Model gets dimensions via m.styles.GetSize()
    case tea.KeyMsg:
        switch msg.String() {
        case "enter": return m, navigateCmd
        }
    }
}
```

## 🎨 **Lipgloss (Styling & Layout)**

**Purpose**: CSS-like styling for terminal interfaces
**Key Insight**: Height() vs MaxHeight() behavior difference!

### Critical Height Control
```go
// ❌ WRONG: Height() sets MINIMUM height (can expand!)
style := lipgloss.NewStyle().Height(1).Border(lipgloss.NormalBorder())

// ✅ CORRECT: MaxHeight() + Inline() for EXACT height
style := lipgloss.NewStyle().MaxHeight(1).Inline(true)

// ✅ PRODUCTION: Background approach for consistent 1-line footers
footer := lipgloss.NewStyle().
    Width(width).
    Background(borderColor).
    Foreground(textColor).
    Padding(0, 1, 0, 1).  // Only horizontal padding
    Render(text)

// FOOTER APPROACH - PRODUCTION READY (✅ PROVEN SOLUTION)
// ❌ WRONG: Border approach (inconsistent height)
style.BorderTop(true).Height(1)

// ✅ CORRECT: Background approach (always 1 line)
style.Background(color).Foreground(textColor).Padding(0,1,0,1)
```

### Layout Patterns
```go
// LAYOUT COMPOSITION
lipgloss.JoinVertical(lipgloss.Left, header, content, footer)
lipgloss.JoinHorizontal(lipgloss.Top, left, right)
lipgloss.Place(width, height, lipgloss.Center, lipgloss.Top, content)

// Horizontal layout
left := lipgloss.NewStyle().Width(leftWidth).Render(leftContent)
right := lipgloss.NewStyle().Width(rightWidth).Render(rightContent)
combined := lipgloss.JoinHorizontal(lipgloss.Top, left, right)

// Vertical layout with consistent spacing
sections := []string{header, content, footer}
combined := lipgloss.JoinVertical(lipgloss.Left, sections...)

// Centering content
centered := lipgloss.Place(width, height, lipgloss.Center, lipgloss.Center, content)

// Responsive design
verticalStyle := lipgloss.NewStyle().Width(width).Padding(0, 2, 0, 2)
if width < 80 {
    // Vertical layout for narrow screens
}
```

### Responsive Patterns
```go
// Breakpoint-based layout
width, height := m.styles.GetSize()  // ALWAYS from styles
if width < 80 {
    return lipgloss.JoinVertical(lipgloss.Left, panels...)
} else {
    return lipgloss.JoinHorizontal(lipgloss.Top, panels...)
}

// Dynamic width allocation
leftWidth := width / 3
rightWidth := width - leftWidth - 4
```

## 📺 **Bubbles (Interactive Components)**

**Purpose**: Pre-built interactive components
**Key Components**: viewport, textinput, list, table

### Viewport - Critical for Scrolling
```go
import "github.com/charmbracelet/bubbles/viewport"

// Setup
viewport := viewport.New(width, height)
viewport.Style = lipgloss.NewStyle() // Clean style prevents conflicts

// Modern scroll methods (use these!)
viewport.ScrollUp(1)     // Replaces LineUp()
viewport.ScrollDown(1)   // Replaces LineDown()
viewport.ScrollLeft(2)   // Horizontal, 2 steps for forms
viewport.ScrollRight(2)

// Deprecated (avoid)
vp.LineUp(lines)       // ❌ Deprecated
vp.LineDown(lines)     // ❌ Deprecated

// Status tracking
viewport.ScrollPercent() // 0.0 to 1.0
viewport.AtBottom()      // bool
viewport.AtTop()         // bool

// State checking
isScrollable := !(vp.AtTop() && vp.AtBottom())
progress := vp.ScrollPercent()

// Content management
viewport.SetContent(content)
viewport.View() // Renders visible portion

// Update in message handling
var cmd tea.Cmd
m.viewport, cmd = m.viewport.Update(msg)
```

### TextInput
```go
import "github.com/charmbracelet/bubbles/textinput"

ti := textinput.New()
ti.Placeholder = "Enter text..."
ti.Focus()
ti.EchoMode = textinput.EchoPassword  // For masked input
ti.CharLimit = 100
```

## 📝 **Huh (Forms)**

**Purpose**: Advanced form builder for complex user input
```go
import "github.com/charmbracelet/huh"

form := huh.NewForm(
    huh.NewGroup(
        huh.NewInput().
            Key("api_key").
            Title("API Key").
            Password().  // Masked input
            Validate(func(s string) error {
                if len(s) < 10 {
                    return errors.New("API key too short")
                }
                return nil
            }),

        huh.NewSelect[string]().
            Key("provider").
            Title("Provider").
            Options(
                huh.NewOption("OpenAI", "openai"),
                huh.NewOption("Anthropic", "anthropic"),
            ),
    ),
)

// Integration with bubbletea
func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    var cmd tea.Cmd
    m.form, cmd = m.form.Update(msg)

    if m.form.State == huh.StateCompleted {
        // Form submitted - access values
        apiKey := m.form.GetString("api_key")
        provider := m.form.GetString("provider")
    }

    return m, cmd
}
```

## ✨ **Glamour (Markdown Rendering)**

**Purpose**: Beautiful markdown rendering in terminal
**CRITICAL**: Create renderer ONCE in styles.New(), reuse everywhere

```go
// ✅ CORRECT: Single renderer instance (prevents freezing)
// styles.go
type Styles struct {
    renderer *glamour.TermRenderer
}

func New() *Styles {
    renderer, _ := glamour.NewTermRenderer(
        glamour.WithAutoStyle(),
        glamour.WithWordWrap(80),
    )
    return &Styles{renderer: renderer}
}

func (s *Styles) GetRenderer() *glamour.TermRenderer {
    return s.renderer
}

// Usage in models
rendered, err := m.styles.GetRenderer().Render(markdown)

// ❌ WRONG: Creating new renderer each time (can freeze!)
renderer, _ := glamour.NewTermRenderer(...)
```

### Safe Rendering with Fallback
```go
// Safe rendering with fallback
rendered, err := renderer.Render(content)
if err != nil {
    // Fallback to plain text
    rendered = fmt.Sprintf("# Content\n\n%s\n\n*Render error: %v*", content, err)
}
```

## 🏗️ **Production Architecture Patterns**

### 1. Centralized Styles & Dimensions

**CRITICAL**: Never store width/height in models - use styles singleton

```go
// ✅ CORRECT: Centralized in styles
type Styles struct {
    width    int
    height   int
    renderer *glamour.TermRenderer
    // ... all styles
}

func (s *Styles) SetSize(width, height int) {
    s.width = width
    s.height = height
    s.updateStyles()  // Recalculate responsive styles
}

func (s *Styles) GetSize() (int, int) {
    return s.width, s.height
}

// Models use styles for dimensions
func (m *Model) updateViewport() {
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        return  // Graceful handling
    }
    // ... viewport setup
}
```

### 2. TUI-Safe Logging System

**Problem**: fmt.Printf breaks TUI rendering
**Solution**: File-based logger

```go
// logger.Log() writes to log.json
logger.Log("[Component] ACTION: details %v", value)
logger.Errorf("[Component] ERROR: %v", err)

// Development monitoring (separate terminal)
tail -f log.json

// ❌ WRONG: Console output in TUI
fmt.Printf("Debug: %v\n", value)  // Breaks rendering
```

### 3. Unified Header/Footer Management
```go
// app.go - Central layout control
func (a *App) View() string {
    header := a.renderHeader()
    footer := a.renderFooter()
    content := a.currentModel.View()

    contentHeight := max(height - headerHeight - footerHeight, 0)
    contentArea := a.styles.Content.Height(contentHeight).Render(content)

    return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}

func (a *App) renderHeader() string {
    switch a.navigator.Current() {
    case WelcomeScreen:
        return a.styles.RenderASCIILogo()
    default:
        return a.styles.Header.Render(title)
    }
}

func (a *App) renderFooter() string {
    actions := []string{}
    // Dynamic actions based on screen state
    if canContinue {
        actions = append(actions, "Enter: Continue")
    }
    if hasScrollableContent {
        actions = append(actions, "↑↓: Scroll")
    }

    return lipgloss.NewStyle().
        Width(width).
        Background(borderColor).
        Foreground(textColor).
        Padding(0, 1, 0, 1).
        Render(strings.Join(actions, " • "))
}

// locale.go - Helper functions
func BuildCommonActions() []string {
    return []string{NavBack, NavExit}
}

func BuildEULAActions(atEnd bool) []string {
    if !atEnd {
        return []string{EULANavScrollInstructions}
    }
    return []string{EULANavAcceptReject}
}

// Usage
actions := locale.BuildCommonActions()
actions = append(actions, specificActions...)
```

### 4. Type-Safe Navigation with Composite ScreenIDs

**Critical Pattern**: Use typed screen IDs with argument support
```go
type ScreenID string
const (
    WelcomeScreen ScreenID = "welcome"
    EULAScreen   ScreenID = "eula"
    MainMenuScreen ScreenID = "main_menu"
    LLMProviderFormScreen ScreenID = "llm_provider_form"
)

// ScreenID methods for composite support
func (s ScreenID) GetScreen() string {
    parts := strings.Split(string(s), "§")
    return parts[0]
}

func (s ScreenID) GetArgs() []string {
    parts := strings.Split(string(s), "§")
    if len(parts) <= 1 {
        return []string{}
    }
    return parts[1:]
}

func CreateScreenID(screen string, args ...string) ScreenID {
    if len(args) == 0 {
        return ScreenID(screen)
    }
    parts := append([]string{screen}, args...)
    return ScreenID(strings.Join(parts, "§"))
}

type NavigationMsg struct {
    Target ScreenID  // Can be simple or composite!
    GoBack bool
}

// Usage - Simple screen
return m, func() tea.Msg {
    return NavigationMsg{Target: EULAScreen}
}

// Usage - Composite screen with arguments
return m, func() tea.Msg {
    return NavigationMsg{Target: CreateScreenID("llm_provider_form", "openai")}
}
```

### 5. Model State Management

**Pattern**: Complete reset on Init() for predictable behavior
```go
func (m *Model) Init() tea.Cmd {
    logger.Log("[Model] INIT")
    // ALWAYS reset ALL state
    m.content = ""
    m.ready = false
    m.scrolled = false
    m.scrolledToEnd = false
    m.error = nil
    return m.loadContent
}

// Force re-render after async operations
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case ContentLoadedMsg:
        m.content = msg.Content
        m.ready = true
        // Force view update
        return m, func() tea.Msg { return nil }
    }
    return m, nil
}
```

## 🐛 **Key Debugging Techniques**

### 1. TUI-Safe Debug Output
```go
// ❌ NEVER: Breaks TUI rendering
fmt.Println("debug")
log.Println("debug")

// ✅ ALWAYS: File-based logging
logger.Log("[Component] Event: %v", msg)
logger.Log("[Model] UPDATE: key=%s", msg.String())
logger.Log("[Model] VIEWPORT: %dx%d ready=%v", width, height, m.ready)

// Monitor in separate terminal
tail -f log.json
```

### 2. Dimension Handling
```go
func (m Model) View() string {
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        return "Loading..." // Graceful fallback
    }
    // ... normal rendering
}

// Log dimension changes
logger.Log("[Model] RESIZE: %dx%d", width, height)
```

### 3. Content Loading Debug
```go
func (m *Model) loadContent() tea.Msg {
    logger.Log("[Model] LOAD: start")
    content, err := source.GetContent()
    if err != nil {
        logger.Errorf("[Model] LOAD: error: %v", err)
        return ErrorMsg{err}
    }
    logger.Log("[Model] LOAD: success (%d chars)", len(content))
    return ContentLoadedMsg{content}
}
```

## 🎯 **Advanced Navigation with Composite ScreenIDs**

### Composite ScreenID Pattern
**Problem**: Need to pass parameters to screens (e.g., which provider to configure)
**Solution**: Composite ScreenIDs with `§` separator

```go
// Format: "screen§arg1§arg2§..."
type ScreenID string

// Methods for parsing composite IDs
func (s ScreenID) GetScreen() string {
    parts := strings.Split(string(s), "§")
    return parts[0]
}

func (s ScreenID) GetArgs() []string {
    parts := strings.Split(string(s), "§")
    if len(parts) <= 1 {
        return []string{}
    }
    return parts[1:]
}

// Helper for creating composite IDs
func CreateScreenID(screen string, args ...string) ScreenID {
    if len(args) == 0 {
        return ScreenID(screen)
    }
    parts := append([]string{screen}, args...)
    return ScreenID(strings.Join(parts, "§"))
}
```

### Usage Examples
```go
// Simple screen (no arguments)
welcome := WelcomeScreen  // "welcome"

// Composite screen (with arguments)
providerForm := CreateScreenID("llm_provider_form", "openai")  // "llm_provider_form§openai"

// Navigation with arguments
return m, func() tea.Msg {
    return NavigationMsg{
        Target: CreateScreenID("llm_provider_form", "anthropic"),
        Data:   FormData{ProviderID: "anthropic"},
    }
}

// In createModelForScreen - extract arguments
func (a *App) createModelForScreen(screenID ScreenID, data any) tea.Model {
    baseScreen := screenID.GetScreen()
    args := screenID.GetArgs()

    switch ScreenID(baseScreen) {
    case LLMProviderFormScreen:
        providerID := "openai" // default
        if len(args) > 0 {
            providerID = args[0]
        }
        return NewLLMProviderFormModel(providerID, ...)
    }
}
```

### State Persistence
```go
// Stack automatically preserves composite IDs
navigator.Push(CreateScreenID("llm_provider_form", "gemini"))

// State contains: ["welcome", "main_menu", "llm_providers", "llm_provider_form§gemini"]
// On restore: user returns to Gemini provider form, not default OpenAI
```

## 🎯 **Advanced Form Scrolling with Viewport**

### Auto-Scrolling Forms Pattern
**Problem**: Forms with many fields don't fit on smaller terminals, focused fields go off-screen
**Solution**: Viewport component with automatic scroll-to-focus behavior

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Key Benefits of Viewport Forms
- **Automatic Clipping**: Viewport handles content that exceeds available space
- **Smooth Scrolling**: Fields slide into view without jarring jumps
- **Focus Preservation**: Focused field always remains visible
- **No Extra Hotkeys**: Uses standard navigation (Tab, arrows)
- **Terminal Friendly**: Works on any terminal size

### Critical Implementation Details
1. **Field Height Tracking**: Must calculate actual rendered height of each field
2. **Scroll Timing**: Call `ensureFocusVisible()` after every focus change
3. **Content Updates**: Re-render form content when input values change
4. **Viewport Sizing**: Account for padding, headers, footers in size calculation

This pattern is essential for professional TUI applications with complex forms.

## ⚠️ **Common Pitfalls & Solutions**

### 1. Glamour Renderer Freezing
**Problem**: Creating new renderer instances can freeze
**Solution**: Single shared renderer in styles.New()

```go
// ❌ WRONG: New renderer each time
func (m *Model) renderMarkdown(content string) string {
    renderer, _ := glamour.NewTermRenderer(...)  // Can freeze!
    return renderer.Render(content)
}

// ✅ CORRECT: Shared renderer instance
func (m *Model) renderMarkdown(content string) string {
    return m.styles.GetRenderer().Render(content)
}
```

### 2. Footer Height Inconsistency
**Problem**: Border-based footers vary in height
**Solution**: Background approach with padding

```go
// ❌ WRONG: Border approach (height varies)
footer := lipgloss.NewStyle().
    Height(1).
    Border(lipgloss.Border{Top: true}).
    Render(text)

// ✅ CORRECT: Background approach (exactly 1 line)
footer := lipgloss.NewStyle().
    Background(borderColor).
    Foreground(textColor).
    Padding(0, 1, 0, 1).
    Render(text)
```

### 3. Dimension Synchronization
**Problem**: Models store their own width/height, get out of sync
**Solution**: Centralize dimensions in styles singleton

```go
// ❌ WRONG: Models managing their own dimensions
type Model struct {
    width, height int
}

// ✅ CORRECT: Centralized dimension management
type Model struct {
    styles *styles.Styles  // Access via styles.GetSize()
}
```

### 4. TUI Rendering Corruption
**Problem**: Console output breaks rendering
**Solution**: File-based logger, never fmt.Printf

```go
// ❌ NEVER: Use tea.ClearScreen during navigation
return a, tea.Batch(cmd, tea.ClearScreen)

// ✅ CORRECT: Let model Init() handle clean state
return a, a.currentModel.Init()
```

### 5. Navigation State Issues
**Problem**: Models retain state between visits
**Solution**: Complete state reset in Init()

```go
// ❌ WRONG: String-based navigation (typo-prone)
return NavigationMsg{Target: "main_menu"}

// ❌ WRONG: Manual string concatenation for arguments
return NavigationMsg{Target: ScreenID("llm_provider_form/openai")}

// ✅ CORRECT: Type-safe constants
return NavigationMsg{Target: MainMenuScreen}

// ✅ CORRECT: Composite ScreenID with helper
return NavigationMsg{Target: CreateScreenID("llm_provider_form", "openai")}
```

## 🚀 **Performance & Best Practices**

### Proven Patterns
```go
// ✅ DO: Shared renderer
rendered, _ := m.styles.GetRenderer().Render(content)

// ✅ DO: Centralized dimensions
width, height := m.styles.GetSize()

// ✅ DO: File logging
logger.Log("[Component] ACTION: %v", data)

// ✅ DO: Complete state reset
func (m *Model) Init() tea.Cmd {
    m.resetAllState()
    return m.loadContent
}

// ✅ DO: Graceful dimension handling
if width <= 0 || height <= 0 {
    return "Loading..."
}
```

### Anti-Patterns to Avoid
```go
// ❌ DON'T: New renderer instances
renderer, _ := glamour.NewTermRenderer(...)

// ❌ DON'T: Model dimensions
type Model struct {
    width  int  // Store in styles instead
    height int  // Store in styles instead
}

// ❌ DON'T: Console output
fmt.Printf("Debug: %v\n", value)

// ❌ DON'T: Partial state reset
func (m *Model) Init() tea.Cmd {
    // Only resetting some fields - incomplete!
    m.content = ""
    // Missing: m.ready, m.scrolled, etc.
}
```

### Key Best Practices Summary
- **Single glamour renderer**: Prevents freezing, faster rendering
- **Centralized dimensions**: Eliminates sync issues, simplifies models
- **Background footer**: Consistent height, modern appearance
- **Type-safe navigation**: Compile-time error prevention
- **File-based logging**: Debug without breaking TUI
- **Complete state reset**: Predictable model behavior
- **Graceful fallbacks**: Handle edge cases elegantly
- **Resource estimation**: Real-time calculation of token/memory usage
- **Environment integration**: Proper EnvVar handling with cleanup
- **Value formatting**: Consistent human-readable displays (formatBytes, formatNumber)

---

*This cheat sheet contains battle-tested solutions for TUI development in the Charm ecosystem, proven in production use.*

## 🎯 **Advanced Form Field Patterns**

### **Boolean Fields with Tab Completion**
**Innovation**: Auto-completion for boolean values with suggestions

```go
import "github.com/charmbracelet/bubbles/textinput"

func createBooleanField() textinput.Model {
    input := textinput.New()
    input.Prompt = ""
    input.ShowSuggestions = true
    input.SetSuggestions([]string{"true", "false"})  // Enable tab completion

    // Show default value in placeholder
    input.Placeholder = "true (default)"  // Or "false (default)"

    return input
}

// Tab completion handler in Update()
func (m *FormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.KeyMsg:
        switch msg.String() {
        case "tab":
            // Complete boolean suggestion
            if m.focusedField.Input.ShowSuggestions {
                suggestion := m.focusedField.Input.CurrentSuggestion()
                if suggestion != "" {
                    m.focusedField.Input.SetValue(suggestion)
                    m.focusedField.Input.CursorEnd()
                    return m, nil
                }
            }
        }
    }
    return m, nil
}
```

### **Integer Fields with Range Validation**
**Innovation**: Real-time validation with human-readable formatting

```
/* Detailed source-code truncated for AI context efficiency. */
```

### **Value Formatting Utilities**
**Critical**: Consistent formatting across all forms

```go
// Universal byte formatting for configuration values
func formatBytes(bytes int) string {
    if bytes >= 1048576 {
        return fmt.Sprintf("%.1fMB", float64(bytes)/1048576)
    } else if bytes >= 1024 {
        return fmt.Sprintf("%.1fKB", float64(bytes)/1024)
    }
    return fmt.Sprintf("%d bytes", bytes)
}

// Universal number formatting for display
func formatNumber(num int) string {
    if num >= 1000000 {
        return fmt.Sprintf("%.1fM", float64(num)/1000000)
    } else if num >= 1000 {
        return fmt.Sprintf("%.1fK", float64(num)/1000)
    }
    return strconv.Itoa(num)
}

// Usage in forms and info panels
sections = append(sections, fmt.Sprintf("• Memory Limit: %s", formatBytes(memoryLimit)))
sections = append(sections, fmt.Sprintf("• Estimated tokens: ~%s", formatNumber(tokenCount)))
```

### **Environment Variable Integration Pattern**
**Innovation**: Direct EnvVar integration with presence detection

```go
// EnvVar wrapper (from loader package)
type EnvVar struct {
    Key     string
    Value   string  // Current value in environment
    Default string  // Default value from config
}

func (e EnvVar) IsPresent() bool {
    return e.Value != "" // Check if actually set in environment
}

// Form field creation from EnvVar
func (m *FormModel) addFieldFromEnvVar(envVarName, fieldKey, title, description string) {
    envVar, _ := m.controller.GetVar(envVarName)

    // Track initially set fields for cleanup logic
    m.initiallySetFields[fieldKey] = envVar.IsPresent()

    input := textinput.New()
    input.Prompt = ""

    // Show default in placeholder if not set
    if !envVar.IsPresent() {
        input.Placeholder = fmt.Sprintf("%s (default)", envVar.Default)
    } else {
        input.SetValue(envVar.Value) // Set current value
    }

    field := FormField{
        Key:         fieldKey,
        Title:       title,
        Description: description,
        Input:       input,
        EnvVarName:  envVarName,
    }

    m.fields = append(m.fields, field)
}
```

### **Smart Field Cleanup Pattern**
**Innovation**: Environment variable cleanup for empty values

```go
func (m *FormModel) saveConfiguration() error {
    // First pass: Remove cleared fields from environment
    for _, field := range m.fields {
        value := strings.TrimSpace(field.Input.Value())

        // If field was initially set but now empty, remove it
        if value == "" && m.initiallySetFields[field.Key] {
            if err := m.controller.SetVar(field.EnvVarName, ""); err != nil {
                return fmt.Errorf("failed to clear %s: %w", field.EnvVarName, err)
            }
            logger.Log("[FormModel] SAVE: cleared %s", field.EnvVarName)
        }
    }

    // Second pass: Save only non-empty values
    for _, field := range m.fields {
        value := strings.TrimSpace(field.Input.Value())
        if value == "" {
            continue // Skip empty - use defaults
        }

        // Validate before saving
        if err := m.validateFieldValue(field, value); err != nil {
            return fmt.Errorf("validation failed for %s: %w", field.Key, err)
        }

        // Save validated value
        if err := m.controller.SetVar(field.EnvVarName, value); err != nil {
            return fmt.Errorf("failed to set %s: %w", field.EnvVarName, err)
        }
        logger.Log("[FormModel] SAVE: set %s=%s", field.EnvVarName, value)
    }

    return nil
}
```

### **Resource Estimation Pattern**
**Innovation**: Real-time calculation of resource usage

```
/* Detailed source-code truncated for AI context efficiency. */
```

### **Current Configuration Preview Pattern**
**Innovation**: Live display of current settings in info panel

```
/* Detailed source-code truncated for AI context efficiency. */
```

### **Type-Based Dynamic Forms**
**Innovation**: Conditional field generation based on selection

```go
func (m *FormModel) buildDynamicForm() {
    m.fields = []FormField{} // Reset

    // Common fields for all types
    m.addFieldFromEnvVar("ENABLED", "enabled", "Enable Service", "Enable or disable this service")
    m.addFieldFromEnvVar("MAX_MEMORY", "max_memory", "Memory Limit", "Maximum memory usage in bytes")

    // Type-specific fields
    switch m.configType {
    case "database":
        m.addFieldFromEnvVar("MAX_CONNECTIONS", "max_connections", "Max Connections", "Maximum database connections")
        m.addFieldFromEnvVar("CACHE_SIZE", "cache_size", "Cache Size", "Database cache size in bytes")

    case "worker":
        m.addFieldFromEnvVar("WORKER_COUNT", "worker_count", "Worker Count", "Number of worker processes")
        m.addFieldFromEnvVar("QUEUE_SIZE", "queue_size", "Queue Size", "Maximum queue size")

    case "api":
        m.addFieldFromEnvVar("RATE_LIMIT", "rate_limit", "Rate Limit", "API requests per minute")
        m.addFieldFromEnvVar("TIMEOUT", "timeout", "Request Timeout", "Request timeout in seconds")
    }

    // Set focus on first field
    if len(m.fields) > 0 {
        m.fields[0].Input.Focus()
    }
}

// Environment variable naming helper
func (m *FormModel) getEnvVarName(configType, suffix string) string {
    prefix := strings.ToUpper(configType) + "_"
    return prefix + suffix
}
```

These advanced patterns enable:
- **Smart Validation**: Real-time feedback with user-friendly error messages
- **Resource Awareness**: Live estimation of memory, CPU, or token usage
- **Environment Integration**: Proper handling of defaults, presence detection, and cleanup
- **Type Safety**: Compile-time validation and runtime error handling
- **User Experience**: Auto-completion, formatting, and intuitive navigation

## 🎯 **Production Form Architecture Patterns**

### Form Model Structure (Latest Pattern)
**Based on successful llm_provider_form.go and summarizer_form.go implementations**

```go
type FormModel struct {
    controller *controllers.StateController
    styles     *styles.Styles
    window     *window.Window

    // Core form state
    fields       []FormField
    focusedIndex int
    showValues   bool
    hasChanges   bool
    args         []string // Arguments from composite ScreenID

    // Enhanced state tracking (from summarizer implementation)
    initialized        bool
    configType        string
    typeName          string
    initiallySetFields map[string]bool // Track fields for cleanup

    // Viewport as permanent property for forms
    viewport     viewport.Model
    formContent  string
    fieldHeights []int
}

// Constructor pattern - args from composite ScreenID
func NewFormModel(
    controller *controllers.StateController, styles *styles.Styles,
    window *window.Window, args []string,
) *FormModel {
    // Extract primary argument (e.g., provider ID)
    primaryArg := "default"
    if len(args) > 0 && args[0] != "" {
        primaryArg = args[0]
    }

    return &FormModel{
        controller: controller,
        styles:     styles,
        window:     window,
        args:       args,
        viewport:   viewport.New(window.GetContentSize()), // Permanent viewport
    }
}
```

### Key Form Implementation Patterns

#### 1. Proper Navigation Hotkeys
```go
// Modern form navigation (Production Pattern - from summarizer_form.go)
switch msg.String() {
case "down":           // ↓: Next field
    m.focusNext()
    m.ensureFocusVisible()

case "up":             // ↑: Previous field
    m.focusPrev()
    m.ensureFocusVisible()

case "tab":            // Tab: Complete suggestion (boolean auto-complete)
    m.completeSuggestion()

case "ctrl+h":         // Ctrl+H: Toggle show/hide masked values
    m.toggleShowValues()

case "ctrl+s":         // Ctrl+S: Save configuration only
    return m.saveConfiguration()

case "ctrl+r":         // Ctrl+R: Reset form to defaults
    m.resetForm()
    return m, nil

case "enter":          // Enter: Save and return (GoBack navigation)
    return m.saveAndReturn()
}

// Enhanced field navigation with auto-scroll
func (m *FormModel) focusNext() {
    if len(m.fields) == 0 {
        return
    }
    m.fields[m.focusedIndex].Input.Blur()
    m.focusedIndex = (m.focusedIndex + 1) % len(m.fields)
    m.fields[m.focusedIndex].Input.Focus()
    m.updateFormContent()
}
```

#### 2. Suggestions and Auto-completion
```go
// Boolean field with suggestions (from summarizer_form.go)
func (m *FormModel) addBooleanField(key, title, description string, envVar loader.EnvVar) {
    input := textinput.New()
    input.Prompt = ""
    input.PlaceholderStyle = m.styles.FormPlaceholder
    input.ShowSuggestions = true
    input.SetSuggestions([]string{"true", "false"})

    // Show default in placeholder
    if envVar.Default == "true" {
        input.Placeholder = "true (default)"
    } else {
        input.Placeholder = "false (default)"
    }

    // Set value only if actually present in environment
    if envVar.Value != "" && envVar.IsPresent() {
        input.SetValue(envVar.Value)
    }
}

// Tab completion handler
func (m *FormModel) completeSuggestion() {
    if m.focusedIndex < len(m.fields) {
        suggestion := m.fields[m.focusedIndex].Input.CurrentSuggestion()
        if suggestion != "" {
            m.fields[m.focusedIndex].Input.SetValue(suggestion)
            m.fields[m.focusedIndex].Input.CursorEnd()
            m.fields[m.focusedIndex].Value = suggestion
            m.hasChanges = true
            m.updateFormContent()
        }
    }
}
```

#### 3. Dynamic Input Width Calculation
```go
// Adaptive input sizing
func (m *FormModel) getInputWidth() int {
    viewportWidth, _ := m.getViewportSize()
    inputWidth := viewportWidth - 6  // Account for padding
    if m.isVerticalLayout() {
        inputWidth = viewportWidth - 4  // Less padding in vertical
    }
    return inputWidth
}

func (m *FormModel) getViewportSize() (int, int) {
    contentWidth, contentHeight := m.window.GetContentSize()
    if contentWidth <= 0 || contentHeight <= 0 {
        return 0, 0
    }

    if m.isVerticalLayout() {
        return contentWidth - PaddingWidth/2, contentHeight - PaddingHeight
    } else {
        leftWidth := MinMenuWidth
        extraWidth := contentWidth - leftWidth - MinInfoWidth - PaddingWidth
        if extraWidth > 0 {
            leftWidth = min(leftWidth+extraWidth/2, MaxMenuWidth)
        }
        return leftWidth, contentHeight - PaddingHeight
    }
}
```

#### 4. Viewport as Permanent Form Property
```go
// ✅ CORRECT: Viewport as permanent property for forms
type FormModel struct {
    viewport viewport.Model  // Permanent - preserves scroll position
}

// Update viewport dimensions on resize
func (m *FormModel) updateViewport() {
    formContentHeight := lipgloss.Height(m.formContent) + 2
    viewportWidth, viewportHeight := m.getViewportSize()
    m.viewport.Width = viewportWidth
    m.viewport.Height = min(viewportHeight, formContentHeight)
    m.viewport.SetContent(m.formContent)
}

// ❌ WRONG: Creating viewport in View() - loses scroll state
func (m *FormModel) View() string {
    vp := viewport.New(width, height) // State lost on re-render!
    return vp.View()
}
```

### Layout Architecture (Two-Column Pattern)

#### 1. Layout Constants (Production Values)
```go
const (
    MinMenuWidth  = 38  // Minimum left panel width
    MaxMenuWidth  = 66  // Maximum left panel width (prevents too wide)
    MinInfoWidth  = 34  // Minimum right panel width
    PaddingWidth  = 8   // Total horizontal padding
    PaddingHeight = 2   // Vertical padding
)
```

#### 2. Adaptive Layout Logic
```go
func (m *Model) isVerticalLayout() bool {
    contentWidth := m.window.GetContentWidth()
    return contentWidth < (MinMenuWidth + MinInfoWidth + PaddingWidth)
}

// Horizontal layout with dynamic width allocation
func (m *Model) renderHorizontalLayout(leftPanel, rightPanel string, width, height int) string {
    leftWidth, rightWidth := MinMenuWidth, MinInfoWidth
    extraWidth := width - leftWidth - rightWidth - PaddingWidth

    // Distribute extra space, but cap left panel at MaxMenuWidth
    if extraWidth > 0 {
        leftWidth = min(leftWidth+extraWidth/2, MaxMenuWidth)
        rightWidth = width - leftWidth - PaddingWidth/2
    }

    leftStyled := lipgloss.NewStyle().Width(leftWidth).Padding(0, 2, 0, 2).Render(leftPanel)
    rightStyled := lipgloss.NewStyle().Width(rightWidth).PaddingLeft(2).Render(rightPanel)

    // Use viewport for final layout rendering
    viewport := viewport.New(width, height-PaddingHeight)
    viewport.SetContent(lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled))
    return viewport.View()
}
```

#### 3. Content Hiding When Space Insufficient
```go
// Vertical layout with conditional content hiding
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
    verticalStyle := lipgloss.NewStyle().Width(width).Padding(0, 4, 0, 2)

    leftStyled := verticalStyle.Render(leftPanel)
    rightStyled := verticalStyle.Render(rightPanel)

    // Hide right panel if both don't fit
    if lipgloss.Height(leftStyled)+lipgloss.Height(rightStyled)+2 < height {
        return lipgloss.JoinVertical(lipgloss.Left,
            leftStyled,
            verticalStyle.Height(1).Render(""),
            rightStyled,
        )
    }

    // Show only essential left panel
    return leftStyled
}
```

### Composite ScreenID Navigation (Production Pattern)

#### 1. Proper ScreenID Creation for Navigation
```go
// Navigation from menu with argument preservation
func (m *MenuModel) handleSelection() (tea.Model, tea.Cmd) {
    selectedItem := m.getSelectedItem()

    // Create composite ScreenID with current selection for stack preservation
    return m, func() tea.Msg {
        return NavigationMsg{
            Target: CreateScreenID(string(targetScreen), selectedItem.ID),
        }
    }
}

// Form navigation back - use GoBack to avoid stack loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    model, cmd := m.saveConfiguration()
    if cmd != nil {
        return model, cmd
    }

    // ✅ CORRECT: Use GoBack to return to previous screen
    return m, func() tea.Msg {
        return NavigationMsg{GoBack: true}
    }
}

// ❌ WRONG: Direct navigation creates stack loops
return m, func() tea.Msg {
    return NavigationMsg{Target: LLMProvidersScreen} // Creates loop!
}
```

#### 2. Constructor with Args Pattern
```go
// Model constructor receives args from composite ScreenID
func NewModel(
    controller *controllers.StateController, styles *styles.Styles,
    window *window.Window, args []string,
) *Model {
    // Initialize with selection from args
    selectedIndex := 0
    if len(args) > 1 && args[1] != "" {
        // Find matching item and set selectedIndex
        for i, item := range items {
            if item.ID == args[1] {
                selectedIndex = i
                break
            }
        }
    }

    return &Model{
        controller:    controller,
        selectedIndex: selectedIndex,
        args:          args,
    }
}

// No separate SetSelected* methods needed
func (m *Model) Init() tea.Cmd {
    logger.Log("[Model] INIT: args=%s", strings.Join(m.args, " § "))

    // Selection already set in constructor from args
    m.loadData()
    return nil
}
```

### Viewport Usage Patterns

#### 1. Forms: Permanent Viewport Property
```go
// ✅ For forms with user interaction and scroll state
type FormModel struct {
    viewport viewport.Model  // Permanent - preserves scroll position
}

func (m *FormModel) ensureFocusVisible() {
    // Auto-scroll to focused field
    focusY := m.calculateFieldPosition(m.focusedIndex)
    if focusY < m.viewport.YOffset {
        m.viewport.YOffset = focusY
    }
    // ... scroll logic
}
```

#### 2. Layout: Temporary Viewport Creation
```go
// ✅ For final layout rendering only
func (m *Model) renderHorizontalLayout(left, right string, width, height int) string {
    content := lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)

    // Create viewport just for layout rendering
    vp := viewport.New(width, height-PaddingHeight)
    vp.SetContent(content)
    return vp.View()
}
```

### Screen Architecture (App.go Integration)

#### 1. Content Area Only Pattern
```go
// Screen models ONLY handle content area
func (m *Model) View() string {
    // ✅ CORRECT: Only content, no header/footer
    leftPanel := m.renderForm()
    rightPanel := m.renderHelp()

    if m.isVerticalLayout() {
        return m.renderVerticalLayout(leftPanel, rightPanel, width, height)
    }
    return m.renderHorizontalLayout(leftPanel, rightPanel, width, height)
}

// ❌ WRONG: Handling header/footer in screen
func (m *Model) View() string {
    header := m.renderHeader()    // App.go handles this!
    footer := m.renderFooter()    // App.go handles this!
    // ...
}
```

#### 2. App.go Layout Management
```go
// App.go manages complete layout structure
func (a *App) View() string {
    header := a.renderHeader()    // Screen-specific header
    footer := a.renderFooter()    // Dynamic footer with actions
    content := a.currentModel.View()  // Content from model

    // Calculate content area size
    contentWidth, contentHeight := a.window.GetContentSize()
    contentArea := a.styles.Content.
        Width(contentWidth).
        Height(contentHeight).
        Render(content)

    return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```

### Field Configuration Best Practices

#### 1. Clean Input Setup
```go
// Modern input field setup
func (m *FormModel) addInputField(config *Config, fieldType string) {
    input := textinput.New()
    input.Prompt = ""  // Clean appearance
    input.PlaceholderStyle = m.styles.FormPlaceholder

    // Dynamic width - set during rendering
    // input.Width NOT set here - calculated in updateFormContent()

    if fieldType == "password" {
        input.EchoMode = textinput.EchoPassword
    }

    if fieldType == "boolean" {
        input.ShowSuggestions = true
        input.SetSuggestions([]string{"true", "false"})
    }

    // Set value from config
    if config != nil {
        input.SetValue(config.GetValue(fieldType))
    }
}
```

#### 2. Dynamic Width Application
```go
// Apply width during content update
func (m *FormModel) updateFormContent() {
    inputWidth := m.getInputWidth()

    for i, field := range m.fields {
        // Apply dynamic width to input
        field.Input.Width = inputWidth - 3  // Account for borders
        field.Input.SetValue(field.Input.Value())  // Trigger width update

        // Render with consistent styling
        inputStyle := m.styles.FormInput.Width(inputWidth)
        if i == m.focusedIndex {
            inputStyle = inputStyle.BorderForeground(styles.Primary)
        }

        renderedInput := inputStyle.Render(field.Input.View())
        sections = append(sections, renderedInput)
    }
}
```

### State Management Best Practices

#### 1. Configuration vs Status Separation
```go
// ✅ SIMPLIFIED: Single status field
type ProviderInfo struct {
    ID          string
    Name        string
    Description string
    Configured  bool  // Single status - provider has required fields
}

// Load status logic
func (m *Model) loadProviders() {
    configs := m.controller.GetLLMProviders()

    provider := ProviderInfo{
        ID:         "openai",
        Name:       locale.LLMProviderOpenAI,
        Configured: configs["openai"].Configured,  // From controller
    }
}

// ❌ COMPLEX: Multiple status fields (removed)
type ProviderInfo struct {
    Configured bool
    Enabled    bool  // Removed - controller handles this
}
```

#### 2. GoBack Navigation Pattern
```go
// ✅ CORRECT: GoBack prevents navigation loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    if err := m.saveConfiguration(); err != nil {
        return m, nil  // Stay on form if save fails
    }

    // Return to previous screen (from navigation stack)
    return m, func() tea.Msg {
        return NavigationMsg{GoBack: true}
    }
}

// Navigation stack automatically maintained:
// ["main_menu§llm_providers", "llm_providers§openai", "llm_provider_form§openai"]
// GoBack removes current and returns to: "llm_providers§openai"
```

This production architecture ensures:
- **Clean separation**: Forms handle content, app.go handles layout
- **Persistent state**: Viewport scroll positions maintained
- **Adaptive design**: Content hides gracefully when space insufficient
- **Type-safe navigation**: Arguments preserved in composite ScreenIDs
- **No navigation loops**: GoBack pattern prevents stack corruption

---

### Backend Docs Config (`backend/docs/config.md`)

# PentAGI Configuration Guide

This document serves as a comprehensive guide to the configuration system in PentAGI, primarily aimed at developers. It details all available configuration options, their purposes, default values, and how they're used throughout the application.

## Table of Contents

- [PentAGI Configuration Guide](#pentagi-configuration-guide)
  - [Table of Contents](#table-of-contents)
  - [Configuration Basics](#configuration-basics)
    - [Current Web Settings Coverage](#current-web-settings-coverage)
    - [Still Server-Managed](#still-server-managed)
  - [General Settings](#general-settings)
    - [Multi-Instance Deployment (`TENANT_ID`)](#multi-instance-deployment-tenant_id)
      - [What the application namespaces automatically](#what-the-application-namespaces-automatically)
      - [What stays the operator's responsibility](#what-stays-the-operators-responsibility)
      - [Deployment topologies](#deployment-topologies)
      - [Extensions installed outside `public` (`DATABASE_EXTENSIONS_SCHEMA`)](#extensions-installed-outside-public-database_extensions_schema)
      - [Multi-tenant PostgreSQL access through PgBouncer](#multi-tenant-postgresql-access-through-pgbouncer)
      - [Multi-tenant PostgreSQL through Supabase's Supavisor pooler (`DATABASE_SEARCH_PATH_VIA_OPTIONS`)](#multi-tenant-postgresql-through-supabases-supavisor-pooler-database_search_path_via_options)
    - [PentAGI Cloud API Settings](#pentagi-cloud-api-settings)
      - [`UPDATE_STRATEGY`](#update_strategy)
      - [`UPDATE_CHECK_INTERVAL`](#update_check_interval)
      - [`UPDATE_SERVER_HOST`](#update_server_host)
      - [`SUPPORT_SERVER_HOST`](#support_server_host)
    - [Usage Details](#usage-details)
  - [Docker Settings](#docker-settings)
    - [Worker Docker Access (`DOCKER_INSIDE_*`)](#worker-docker-access-docker_inside_)
    - [Usage Details](#usage-details-1)
  - [Server Settings](#server-settings)
    - [Usage Details](#usage-details-2)
  - [Frontend Settings](#frontend-settings)
    - [Usage Details](#usage-details-3)
  - [Authentication Settings](#authentication-settings)
    - [Usage Details](#usage-details-4)
  - [Web Scraper Settings](#web-scraper-settings)
    - [Usage Details](#usage-details-5)
  - [LLM Provider Settings](#llm-provider-settings)
    - [OpenAI](#openai)
    - [Anthropic](#anthropic)
      - [Workload Identity Federation](#workload-identity-federation)
    - [Ollama LLM Provider](#ollama-llm-provider)
    - [Google AI (Gemini) LLM Provider](#google-ai-gemini-llm-provider)
    - [AWS Bedrock LLM Provider](#aws-bedrock-llm-provider)
    - [DeepSeek LLM Provider](#deepseek-llm-provider)
    - [GLM LLM Provider](#glm-llm-provider)
    - [Kimi LLM Provider](#kimi-llm-provider)
    - [Qwen LLM Provider](#qwen-llm-provider)
    - [MiniMax LLM Provider](#minimax-llm-provider)
    - [Mistral LLM Provider](#mistral-llm-provider)
    - [xAI (Grok) LLM Provider](#xai-grok-llm-provider)
    - [Custom LLM Provider](#custom-llm-provider)
    - [Usage Details](#usage-details-6)
  - [Embedding Settings](#embedding-settings)
    - [Usage Details](#usage-details-7)
  - [Summarizer Settings](#summarizer-settings)
    - [Usage Details and Impact on System Behavior](#usage-details-and-impact-on-system-behavior)
      - [Core Summarization Strategies and Their Parameters](#core-summarization-strategies-and-their-parameters)
      - [Deep Dive: Parameter Impact and Recommendations](#deep-dive-parameter-impact-and-recommendations)
    - [Summarization Effects on Agent Behavior](#summarization-effects-on-agent-behavior)
    - [Implementation Details](#implementation-details)
    - [Recommended Settings for Different Use Cases](#recommended-settings-for-different-use-cases)
  - [Assistant Settings](#assistant-settings)
    - [Usage Details](#usage-details-8)
    - [Recommended Assistant Settings for Different Use Cases](#recommended-assistant-settings-for-different-use-cases)
  - [Functions Configuration](#functions-configuration)
    - [DisableFunction Structure](#disablefunction-structure)
    - [ExternalFunction Structure](#externalfunction-structure)
    - [Usage Details](#usage-details-9)
    - [Example Configuration](#example-configuration)
    - [Security Considerations](#security-considerations)
    - [Built-in Functions Reference](#built-in-functions-reference)
  - [Search Engine Settings](#search-engine-settings)
    - [DuckDuckGo Search](#duckduckgo-search)
    - [Sploitus Search](#sploitus-search)
    - [Google Search](#google-search)
    - [Traversaal Search](#traversaal-search)
    - [Tavily Search](#tavily-search)
    - [Firecrawl Search](#firecrawl-search)
    - [Perplexity Search](#perplexity-search)
    - [Searxng Search](#searxng-search)
    - [Internal Analytics Engine](#internal-analytics-engine)
    - [Usage Details](#usage-details-10)
  - [Network and Proxy Settings](#network-and-proxy-settings)
    - [Usage Details](#usage-details-11)
  - [Graphiti Knowledge Graph Settings](#graphiti-knowledge-graph-settings)
    - [PentAGI Configuration Boundary](#pentagi-configuration-boundary)
    - [Client Lifecycle and Failure Behavior](#client-lifecycle-and-failure-behavior)
    - [Data Flow, Search, and Tenancy](#data-flow-search-and-tenancy)
    - [Deployment Ownership](#deployment-ownership)
  - [Agent Supervision Settings](#agent-supervision-settings)
    - [Usage Details](#usage-details-12)
    - [Supervision System Integration](#supervision-system-integration)
    - [Recommended Settings](#recommended-settings)
  - [Observability Settings](#observability-settings)
    - [Telemetry](#telemetry)
    - [Langfuse](#langfuse)
    - [Usage Details](#usage-details-13)

## Configuration Basics

PentAGI uses environment variables for configuration, with support for `.env` files through the `godotenv` package. The configuration is defined in the `Config` struct in `pkg/config/config.go` and is loaded using the `NewConfig()` function.

```go
func NewConfig() (*Config, error) {
    godotenv.Load()

    var config Config
    if err := env.ParseWithOptions(&config, env.Options{
        RequiredIfNoDef: false,
        FuncMap: map[reflect.Type]env.ParserFunc{
            reflect.TypeOf(&url.URL{}): func(s string) (interface{}, error) {
                if s == "" {
                    return nil, nil
                }
                return url.Parse(s)
            },
        },
    }); err != nil {
        return nil, err
    }

    return &config, nil
}
```

This function automatically loads environment variables from a `.env` file if present, then parses them into the `Config` struct using the `env` package from `github.com/caarlos0/env/v10`.

### Current Web Settings Coverage

The running PentAGI instance already exposes several settings areas in the web UI:

- **Settings -> Providers**: Manage user-defined provider profiles, per-agent model and runtime options, and provider test actions for provider types supported by the running server.
- **Settings -> Prompts**: Manage system, human, and tool prompt templates.
- **Settings -> PentAGI API**: Create, revoke, and delete PentAGI API tokens.
- **Other UI-managed preferences**: Favorite flows are stored as user preferences, and theme selection is handled client-side from the main sidebar/profile controls.

These web-console features do not replace the environment variables in this guide for provider credentials, endpoints, or external integrations.

### Still Server-Managed

The environment variables documented below remain the source of truth for configuration that is not currently editable from the web console:

- **LLM credentials and connection settings**: API keys, base URLs, auth modes, and provider-specific connection settings for OpenAI, Anthropic, Bedrock, Ollama, custom providers, and similar backends; config-path settings apply only where supported, such as `OLLAMA_SERVER_CONFIG_PATH`, `LLM_SERVER_CONFIG_PATH`, and `BEDROCK_CONFIG_PATH`.
- **Search provider credentials and options**: DuckDuckGo, Google, Tavily, Traversaal, Perplexity, SearXNG, Sploitus, and related search configuration.
- **Third-party integrations**: Langfuse, Graphiti, and other external observability or knowledge services.

## General Settings

These settings control basic application behavior and are foundational for the system's operation.

| Option           | Environment Variable        | Default Value                                                                | Description                                                              |
| ---------------- | --------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| DatabaseURL      | `DATABASE_URL`              | `postgres://pentagiuser:pentagipass@pgvector:5432/pentagidb?sslmode=disable` | Connection string for the PostgreSQL database with pgvector extension    |
| DatabaseExtensionsSchema | `DATABASE_EXTENSIONS_SCHEMA` | `public`                                                              | Schema every tenant's search_path must include for shared extensions (`vector`, `pg_trgm`) to resolve. Only used when `TenantID` is set. Override for databases that install extensions elsewhere by convention, e.g. Supabase uses `extensions`. See [Extensions installed outside `public`](#extensions-installed-outside-public-database_extensions_schema). |
| DatabaseSearchPathViaOptions | `DATABASE_SEARCH_PATH_VIA_OPTIONS` | `false`                                                          | Sends the tenant search_path as `options=--search_path=<value>` instead of a bare `search_path` parameter. Only used when `TenantID` is set. For poolers that forward `options` but drop an unrecognized bare `search_path` (e.g. some Supabase Supavisor versions). See [Multi-tenant PostgreSQL through Supabase's Supavisor pooler](#multi-tenant-postgresql-through-supabases-supavisor-pooler-database_search_path_via_options). |
| DBMaxOpenConns   | `DATABASE_MAX_OPEN_CONNS`   | `25`                                                                         | Maximum open connections in the shared `sql.DB` pool (sqlc + GORM combined). See [database.md §Connection Pooling](database.md#connection-pooling). |
| DBMaxIdleConns   | `DATABASE_MAX_IDLE_CONNS`   | `5`                                                                          | Maximum idle connections kept open between requests                      |
| DBVectorMaxConns | `DATABASE_VECTOR_MAX_CONNS` | `10`                                                                         | Maximum connections in the shared `pgxpool` for all pgvector stores      |
| Debug            | `DEBUG`                     | `false`                                                                      | Enables debug mode with additional logging                               |
| DataDir          | `DATA_DIR`                  | `./data`                                                                     | Directory for storing persistent data                                    |
| AskUser          | `ASK_USER`                  | `false`                                                                      | When enabled, requires explicit user confirmation for certain operations |
| TenantID         | `TENANT_ID`                 | *(empty)*                                                                    | Namespaces every externally-visible artifact this instance creates so several PentAGI instances can share one host and one set of backing services. Empty = single-instance behavior, unchanged. See [Multi-Instance Deployment](#multi-instance-deployment-tenant_id). |
| DockerPortsBase  | `DOCKER_PORTS_BASE`         | `0` (means `28000`)                                                          | First host port for per-flow sandbox port publishing; each instance owns `[base, base+2000)` |
| InstallationID   | `INSTALLATION_ID`           | *(none)*                                                                     | Unique installation identifier for PentAGI Cloud API communication       |
| LicenseKey       | `LICENSE_KEY`               | *(none)*                                                                     | License key for PentAGI Cloud API authentication and feature activation  |
| UpdateStrategy   | `UPDATE_STRATEGY`         | `preview`                                                                    | Which builds this installation is offered: `nightly`, `preview` or `stable`. An unrecognised value falls back to `preview`. See [PentAGI Cloud API Settings](#pentagi-cloud-api-settings). |
| UpdateCheckInterval | `UPDATE_CHECK_INTERVAL` | `3h`                                                                         | How often the running server reports its own state to the Cloud API. `0` disables the report. See [PentAGI Cloud API Settings](#pentagi-cloud-api-settings). |
| UpdateServerHost | `UPDATE_SERVER_HOST`      | `update.pentagi.com`                                                         | PentAGI Cloud API endpoint as `host[:port]`, **without a scheme** — the connection is always HTTPS. See [PentAGI Cloud API Settings](#pentagi-cloud-api-settings). |
| SupportServerHost | `SUPPORT_SERVER_HOST`    | `support.pentagi.com`                                                        | PentAGI Support API endpoint as `host[:port]`, same shape as `UPDATE_SERVER_HOST`. See [PentAGI Cloud API Settings](#pentagi-cloud-api-settings). |

### Multi-Instance Deployment (`TENANT_ID`)

`TENANT_ID` namespaces the artifacts a PentAGI instance creates in **shared external services**, so that several independent installations can use one PostgreSQL server, one worker node (Docker daemon), one Neo4j/Graphiti and one Langfuse without colliding.

**Primary use case: several management instances, shared resources.** The typical deployment puts each PentAGI management backend on its own server, while the heavy shared resources — the worker node where sandbox containers run, and the database — are common. `TENANT_ID` is what keeps those instances from writing over each other.

Running several management instances on **one** server is also possible (for example behind an nginx reverse proxy), but that is not what the installer or the stock `docker-compose.yml` are built for — see *Deployment topologies* below.

**When empty (the default) nothing changes.** Every helper degrades to an identity function: container names stay `pentagi-terminal-<id>`, the database schema stays `public`, Graphiti group ids stay `flow-<id>`, and the session cookie stays `auth`. This is enforced in one place (`backend/pkg/config/tenant.go`) and covered by tests.

**Validation.** `TENANT_ID` must match `^[a-z][a-z0-9_]{0,31}$` — the intersection of the constraints imposed by every consumer:

| Consumer | Constraint |
| --- | --- |
| Docker object name | `[a-zA-Z0-9][a-zA-Z0-9_.-]*` |
| PostgreSQL identifier | ≤ 63 bytes; unquoted-safe as `[a-z_][a-z0-9_]*` |
| Graphiti / Neo4j group id | parsed on a hyphen boundary, so no hyphens |

Hyphens are excluded deliberately: they separate the tenant from the rest of a group id or object name. An invalid value **aborts startup** rather than being normalised, because collapsing two distinct tenants onto one namespace is exactly the collision tenancy exists to prevent.

#### What the application namespaces automatically

| Area | Effect |
| --- | --- |
| PostgreSQL | A schema named after the tenant is created on boot and `search_path` is set to `<tenant>,<DATABASE_EXTENSIONS_SCHEMA>`; the DSN is rewritten once so sqlc, GORM, goose and the pgvector pool all follow. Extensions (`vector`, `pg_trgm`) stay shared in `DATABASE_EXTENSIONS_SCHEMA` (default `public`). |
| Worker containers | Sandbox container names become `<tenant>-pentagi-terminal-<flow>`; the per-flow volume and the container hostname derive from that name automatically. Both are labelled `pentagi.tenant`, so daemon-wide sweeps can filter by owner. |
| Host ports | Per-flow sandbox ports are allocated from `DOCKER_PORTS_BASE` (default `28000`), giving each instance the window `[base, base+2000)`. |
| Knowledge graph | Graphiti/Neo4j group ids become `<tenant>-flow-<id>`. The GraphQL API contract is unchanged — clients still send `flow-<id>` and the server rebuilds the namespaced key internally. |
| Auth | Cookie and JWT keys are derived from `COOKIE_SIGNING_SALT` **plus** the tenant, and the session cookie is renamed, so a session or API token minted by one instance is rejected by another. |
| Telemetry | OTel resources carry `service.instance.id` and, with a tenant set, `tenant_id`; Langfuse traces carry the native `environment` field plus a `tenant:<id>` tag and tenant-prefixed trace/session names. |

#### What stays the operator's responsibility

`TENANT_ID` does **not** rewrite infrastructure-level settings. The following must be given distinct values per instance by whoever provisions it:

| Setting | Why it is not derived from `TENANT_ID` |
| --- | --- |
| `DATA_DIR` | The data directory is chosen by the operator. Flow file caches, the container `/work` bind mount, screenshots and `installation_id` all live under it, and two instances pointed at the same directory **will overwrite each other's flow data**. Give each instance its own path or its own volume. |
| `DOCKER_NETWORK` | Instances may legitimately share one Docker network; the application never renames it. Set it explicitly if you want separate networks. |
| Published ports | `PENTAGI_LISTEN_PORT`, `PGVECTOR_LISTEN_PORT`, `SCRAPER_LISTEN_PORT`, `PPROF_ADDR` and `DOCKER_PORTS_BASE` are host-level resources, not string namespaces. |
| `INSTALLATION_ID` | Unique per installation, or left empty to be generated once and cached in `DATA_DIR`. |
| Database privileges | The configured user needs `CREATE SCHEMA`, and on first boot `CREATE EXTENSION` — unless an administrator pre-installed `vector` and `pg_trgm` into `DATABASE_EXTENSIONS_SCHEMA` (default `public`). |

The values actually in effect are written to the startup log under `Instance identity` (`tenant_id`, `data_dir`, `schema`, `installation_id`), which is the quickest way to confirm two instances are not sharing something they should not.

`DOCKER_INSIDE` is orthogonal to tenancy and may be enabled alongside it.

#### Deployment topologies

| Topology | Supported by |
| --- | --- |
| One instance per server, shared PostgreSQL and/or worker node | The installer and the stock `docker-compose.yml`, with `TENANT_ID` set per server. This is the intended setup. |
| Several instances on one server | Manual configuration only. The stock `docker-compose.yml` uses fixed `container_name` and network names, so copying it verbatim will not start a second stack. Adapt it to your network and infrastructure — for example, distinct container names behind a shared nginx — and give each instance its own `DATA_DIR` and ports. |

The installer provisions **one** instance per server; it does not manage several side by side. Note that it does scope the sandbox resources it cleans up — worker containers and volumes are matched by the tenant prefix — so a purge run for one management instance will not remove another's sandboxes from a shared worker node.

**Upgrading an existing deployment.** Leave `TENANT_ID` empty. The instance keeps using `public` and its current data directory; no migration is required. Setting `TENANT_ID` on an existing installation points it at a **new, empty schema** — the data in `public` is not migrated and will appear to be gone. Do not point an existing `public` deployment at a `search_path` that lists another tenant's schema.

#### Extensions installed outside `public` (`DATABASE_EXTENSIONS_SCHEMA`)

`ensureTenantSchema` requires `vector` and `pg_trgm` to already live in (or be creatable in) one schema common to every tenant, because it must be part of every tenant's `search_path`. That schema defaults to `public`, which is where a stock PostgreSQL/`docker-compose.yml` install keeps them.

Managed providers do not always follow that convention. **Supabase** (cloud and self-hosted) installs its bundled extensions into a dedicated `extensions` schema instead, and its default roles get `extensions` added to their `search_path` for exactly that reason. Pointing `DATABASE_URL` at such a database with `TENANT_ID` set fails fast on boot:

```
Tenant schema initialization failed: extension "vector" is installed in schema "extensions",
but multi-tenant mode requires it in "public" so every tenant can reach it; either run
ALTER EXTENSION vector SET SCHEMA public, or set DATABASE_EXTENSIONS_SCHEMA=extensions
to match where it already lives
```

Set `DATABASE_EXTENSIONS_SCHEMA=extensions` (or whatever schema the error reports) instead of moving the extension with `ALTER EXTENSION ... SET SCHEMA` — the schema only needs to be part of the `search_path` PentAGI computes (`<tenant>,<DATABASE_EXTENSIONS_SCHEMA>`), moving a provider-managed extension out of its documented location is unnecessary and risks breaking whatever else that provider expects to find it there.

#### Multi-tenant PostgreSQL access through PgBouncer

`DATABASE_URL` can point at a PgBouncer instance instead of PostgreSQL directly, but three things have to be true, independent of each other:

1. **`pool_mode = session` on the PgBouncer side.** PentAGI holds a `pg_advisory_lock`/`pg_advisory_unlock` pair on one dedicated connection across the whole tenant-bootstrap + migration sequence (`backend/pkg/database/tenant.go`), and `pgx` (used by the pgvector pool) caches server-side prepared statements by default. Both break silently under `transaction`/`statement` pooling, because PgBouncer is then free to hand the client a different backend connection between statements. This requirement is unrelated to tenancy — it applies even with `TENANT_ID` empty.
2. **`ignore_startup_parameters = search_path` in `pgbouncer.ini`.** With a tenant configured, PentAGI's own DSN already carries `?search_path=<tenant>,<DATABASE_EXTENSIONS_SCHEMA>` (see the table above). PgBouncer validates startup parameters from the client against a small built-in allowlist and rejects anything else with `unsupported startup parameter: search_path` unless it is explicitly ignored. Ignoring it does **not** apply the value — it only stops PgBouncer from rejecting the connection — so this step alone is not sufficient; see the next point.
3. **A `connect_query` per tenant in PgBouncer's `[databases]` section**, since PentAGI's own `search_path` startup parameter is ignored per point 2 above. `connect_query` runs on PgBouncer's own connection to PostgreSQL before any client statement, so it is not subject to the client-facing startup-parameter allowlist and works regardless of pool mode:

   ```ini
   [databases]
   pentagi_testing = host=pgvector port=5432 dbname=pentagidb pool_mode=session connect_query='SET search_path TO testing,public'
   pentagi_acme    = host=pgvector port=5432 dbname=pentagidb pool_mode=session connect_query='SET search_path TO acme,public'
   ```

   Point each instance's `DATABASE_URL` at its own virtual database name (`pentagi_testing`, `pentagi_acme`, ...) rather than the shared `pentagidb` — PgBouncer pools per `(user, dbname)` pair, so distinct virtual names are what keeps the tenants' pools, and therefore their `connect_query`, apart.

   (`track_extra_parameters = search_path` is PgBouncer's other mechanism for this, but it only works when PostgreSQL reports `search_path` changes back to the client, which requires PostgreSQL 18+ or Citus 12+ — not an option against the PostgreSQL 16/17 that ships in `docker-compose.yml`.)

#### Multi-tenant PostgreSQL through Supabase's Supavisor pooler (`DATABASE_SEARCH_PATH_VIA_OPTIONS`)

Supabase's shared/self-hosted pooler (Supavisor) is not PgBouncer and none of its `[databases]`/`connect_query` configuration exists for it, so the PgBouncer recipe above does not apply. Reports on whether Supavisor forwards a tenant's `search_path` at all are inconsistent — see [supabase/supavisor#206](https://github.com/supabase/supavisor/issues/206) — and depend on the Supavisor version (a parsing fix landed in [PR #768](https://github.com/supabase/supavisor/pull/768)).

If bypassing the pooler entirely (connecting straight to the underlying PostgreSQL, or to a Supabase project's "Direct connection"/IPv4-add-on string) is not an option, set:

```
DATABASE_SEARCH_PATH_VIA_OPTIONS=true
```

This sends the tenant's search_path as `options=--search_path=<tenant>,<DATABASE_EXTENSIONS_SCHEMA>` instead of a bare `search_path=` parameter. Some poolers forward the `options` startup parameter through to the real backend while silently dropping an unrecognized bare `search_path` — this is exactly the workaround reported to work against some Supavisor versions. **It is not guaranteed** — verify it actually took effect by checking that the app starts (`verifySearchPath` fails fast with a clear error if it did not) rather than assuming success from the flag alone.

This flag changes nothing for a direct PostgreSQL connection or a PgBouncer setup already following the recipe above; both accept `search_path` and `options` equally, so there is no reason to enable it outside a Supavisor-fronted deployment.

### PentAGI Cloud API Settings

These four are **shared**: one value in one `.env`, read by the installer and by the running PentAGI server alike. The server parses them into the `Config` struct as `UpdateServerHost`, `UpdateStrategy`, `UpdateCheckInterval` and `SupportServerHost`, and `docker-compose.yml` passes all four into the container; the installer's own calls read them straight from the file, and its *Server Settings* form edits every one of them except `UPDATE_CHECK_INTERVAL`, which concerns only the server and is set by hand. There is deliberately no per-component variant of any of them — an installation that pointed the installer at one endpoint and the server at another would be talking about itself in two places at once.

`UPDATE_SERVER_HOST`, together with `INSTALLATION_ID` and `LICENSE_KEY`, is behind every call either side makes to the PentAGI Cloud API: the update check the installer runs while gathering the state of the machine, the package info and download calls behind the installer build it offers, the same pair of calls used to fetch the Jaeger ClickHouse storage plugin whenever the observability stack is updated — that plugin is a file mounted into the container rather than an image, so a broken `UPDATE_SERVER_HOST` stops it from being installed as surely as it stops a check — and the running server's own periodic report, described under [`UPDATE_CHECK_INTERVAL`](#update_check_interval). `UPDATE_STRATEGY` travels only in the update check; the package calls name a component, a version and a platform, and nothing else.

Both sides honour `PROXY_URL`. The installer's own proxy setting takes precedence over `HTTP_PROXY`/`HTTPS_PROXY` in the environment, because it is the one the operator configured deliberately and often the only one carrying credentials. See [Network and Proxy Settings](#network-and-proxy-settings).

#### `UPDATE_STRATEGY`

Which builds this installation is offered. The value is trimmed and lower-cased before it is matched, so `NIGHTLY` and `  preview  ` are accepted. Empty means `preview`; an unrecognised value logs a warning and also falls back to `preview`, deliberately, because a typo must not leave an installation quietly unable to hear about security updates.

| Value | What the installation is offered |
| --- | --- |
| `preview` (default) | Pre-release builds from the channels this installation already follows, offered as soon as they are published, without waiting for a release to be cut. The version, changelog and release notes that come with the answer still describe the stack's latest *release*, so they name the last release crossed and not the preview build itself. |
| `stable` | Only artefacts belonging to the latest curated stable release. The whole installation moves onto one released set, which is what lets it be described by a single version. A component that release does not cover is answered `unknown` with a reason rather than a newer build from elsewhere — offering part of an installation from outside the curated set is exactly the promise `stable` makes. |
| `nightly` | The newest build published, with no release attached to it. The answer carries no version metadata, so the update overview can report that a newer build exists and little else about it. |

The default is `preview` rather than `stable`, and the difference is not caution versus eagerness. Under `stable` the answer is measured against curated releases, so a stack with no published release is offered nothing at all — which reads as "no updates exist" when it means "we are not publishing releases for this yet". `preview` follows the channels the client is already on, which is what an installation wants until releases are something it can rely on.

The strategy is sent with each update check and applies to every stack in the answer, including the installer's own build — so it also decides which installer version is offered for download. Offered, not applied: the installer downloads that build next to the environment file under a name carrying its version, verifies it, and prints the command that moves it into place. It never writes over the binary it is running — a process cannot reliably replace its own executable, so "download the installer" and "update the installer" are the same operation, and putting the new build in place stays a deliberate step the operator takes.

#### `UPDATE_CHECK_INTERVAL`

How often the **running server** reports its own state to the Cloud API, as a Go duration (`3h` by default, `90m` and `1h30m` are the same value). `0`, or any negative value, disables the report entirely.

This is the other half of the picture the update check cannot see. An installer's check describes the artefacts on the machine and says nothing about the server itself, so an installation deployed once and never started looks exactly like one in daily use. The report closes that gap: on each tick it sends the build that is running, the platform it was compiled for, the sha256 of that binary, and a summary of how the server is configured — quantities and closed vocabularies produced by `GetInstanceSummary`, nothing that identifies a person and nothing anybody wrote. Those first three are read off this process and travel **inside** the summary as well as beside it, because the same summary is what `-info` prints for a caller that will forward it: whoever carries a description cannot produce the platform of the process that wrote it, and the executable's digest is not something an image digest can stand in for.

The answer is kept in memory and shown in the web UI as a version badge — in the main sidebar header and at the bottom of the settings sidebar — read through the `versionInfo` GraphQL query (permission `settings.view`). The badge shows a yellow **↓** when the service names a newer version, and a yellow **?** whenever no verdict is available: the first check has not completed yet, the service could not be reached, it answered without naming a version to move to, it could not match the running build to a published release, or checks are disabled. A verdict, once reached, outlives the failures that follow it — an update that was available an hour ago is still available — so a badge keeps its arrow through an outage and the popover dates both the last answer and the last failure. Nothing is applied from the UI: the tooltip and the popover say to run the installer.

Two properties are worth knowing when reading logs. The first send is delayed by a random wait of up to two minutes, so a fleet restarted at once does not arrive as one burst. And nothing here can interrupt the product: a missing or malformed `INSTALLATION_ID` means the service is never built, every later failure is a debug line, and the next tick simply tries again — an endpoint that is down or a licence that has expired are ordinary states of the world for something the server does not depend on.

#### `UPDATE_SERVER_HOST`

The address is `host[:port]` with **no scheme** — the connection is always TLS, so writing one is redundant. Empty, which is the normal case, means `update.pentagi.com`.

A full URL is still accepted because this setting historically held one: an `https://` prefix is stripped, so is anything after the host (path, query string, fragment), surrounding whitespace is trimmed, and the remainder is lower-cased. What has to be left over is a host, and five shapes are refused outright rather than repaired:

| Value | Result |
| --- | --- |
| `http://update.example.com` | **Rejected.** The transport is HTTPS-only, and silently upgrading a value someone deliberately wrote as `http` would hide the misconfiguration instead of reporting it. |
| Any other scheme (`ftp://`, `tcp://`, ...) | **Rejected** as unsupported. |
| `https://` or `https:///some/path` | **Rejected** — dropping the scheme and the path leaves nothing, so there is no host to connect to. |
| `user:pass@update.example.com` | **Rejected** — credentials in the host are not supported. Authentication to the Cloud API is `LICENSE_KEY`, and a proxy's credentials belong in `PROXY_URL`. |
| `update .example.com` (a space or a tab inside the host) | **Rejected.** Whitespace around the value is trimmed, but whitespace inside it is a typo rather than an address. |

Writing a scheme is the common mistake here, and `https://…` being tolerated makes `http://…` look like it should work too. A rejected value does not abort the installer: the client is never built, the update check is recorded as failed with the reason `unreachable`, and the reason is written to the installer log. The visible symptom is therefore an unreachable update server, which is worth checking the log about before concluding the server is actually down.

The repairs and the refusals above are the installer's. The running server hands the value to the Cloud SDK as written — nothing normalises it and nothing reports it — so a value the SDK cannot build a client from costs one debug line at startup and then no product report at all. Write a bare host and both sides agree.

#### `SUPPORT_SERVER_HOST`

The PentAGI Support API endpoint, in exactly the shape `UPDATE_SERVER_HOST` takes: `host[:port]`, no scheme, empty meaning `support.pentagi.com`. It is a separate service from the update endpoint and is configured separately, so that an installation reaching the Cloud API through a mirror or an internal proxy can point the two at different places.

It is written by the installer's *Server Settings* form and parsed by the server, and no call targets it yet — the support integration that will use it is not in place. Setting it today changes nothing; leaving it empty costs nothing either.

### Usage Details

- **DatabaseURL**: This is a critical setting used throughout the application for all database connections. It is used to:
  - Initialize the single shared `sql.DB` connection pool in `main.go` (used by both sqlc `Queries` and GORM)
  - Seed the shared `pgxpool.Pool` for all pgvector stores (agent memory + knowledge API)

  For connection pool sizing and operational monitoring commands see [database.md §Connection Pooling](database.md#connection-pooling).

```go
// In main.go — one pool shared by sqlc Queries and GORM
db, err := sql.Open("postgres", cfg.DatabaseURL)
db.SetMaxOpenConns(cfg.DBMaxOpenConns)
db.SetMaxIdleConns(cfg.DBMaxIdleConns)
db.SetConnMaxLifetime(time.Hour)

queries := database.New(db)
orm, err := database.NewGorm(db)   // GORM wraps the same *sql.DB

// Shared pgxpool for all pgvector stores
pgPoolConfig, _ := pgxpool.ParseConfig(cfg.DatabaseURL)
pgPoolConfig.MaxConns = int32(cfg.DBVectorMaxConns)
pgPool, _ := pgxpool.NewWithConfig(ctx, pgPoolConfig)
cfg.PgxPool = pgPool               // passed to tools and router
```

- **Debug**: Controls debug mode throughout the application, enabling additional logging and development features:
  - Activates detailed logging in the router setup
  - Can enable development endpoints and tools

```go
// In router.go for enabling debug mode
if cfg.Debug {
    // Enable debug features
}
```

- **DataDir**: Specifies where PentAGI stores persistent data. This is used across multiple components:
  - In `docker/client.go` for container volume mapping
  - For screenshots storage in `services.NewScreenshotService`
  - In tools for file operations and data persistence
  - In Docker container management for mapping volumes

```go
// In docker/client.go
dataDir, err := filepath.Abs(cfg.DataDir)

// In router.go for screenshot service
screenshotService := services.NewScreenshotService(orm, cfg.DataDir)

// In tools.go for various tools
dataDir: fte.cfg.DataDir
```

- **AskUser**: A safety feature that, when enabled, requires explicit user confirmation before executing potentially destructive operations:
  - Used in tools to prompt for confirmation before executing commands
  - Serves as a safeguard for sensitive operations

```go
// In tools.go
if fte.cfg.AskUser {
    // Prompt user for confirmation before executing
}
```

- **InstallationID**: A unique identifier for the PentAGI installation used for cloud API communication:
  - Generated automatically during installation or can be manually set
  - Required for certain cloud-based features and integrations

```go
// Used in cloud SDK initialization
if cfg.InstallationID != "" {
    // Initialize cloud API client with installation ID
}
```

- **LicenseKey**: Authentication key for PentAGI Cloud API and premium feature activation:
  - Validates license and enables licensed features
  - Required for enterprise features and support
  - Used for authentication with PentAGI Cloud services

```go
// Used in cloud SDK initialization
if cfg.LicenseKey != "" {
    // Validate license and activate premium features
}
```

## Docker Settings

These settings control how PentAGI interacts with Docker, which is used for terminal isolation and executing commands in a controlled environment. They're crucial for the security and functionality of tool execution.

| Option                       | Environment Variable               | Default Value          | Description |
| ---------------------------- | ---------------------------------- | ---------------------- | ----------- |
| DockerInside                 | `DOCKER_INSIDE`                    | `false`                | Set to `true` if PentAGI runs inside Docker and needs to access the host Docker daemon. |
| DockerNetAdmin               | `DOCKER_NET_ADMIN`                 | `false`                | Set to `true` to grant the primary container NET_ADMIN capability for advanced networking. |
| DockerSocket                 | `DOCKER_SOCKET`                    | *(none)*               | Path to Docker socket for container management |
| DockerInsideHost             | `DOCKER_INSIDE_HOST`               | *(none)*               | Docker daemon endpoint given to worker containers; also disables host-socket autodetection. See [Worker Docker Access](#worker-docker-access-docker_inside_) |
| DockerInsideTLSVerify        | `DOCKER_INSIDE_TLS_VERIFY`         | *(none)*               | TLS verification for the worker container's Docker connection |
| DockerInsideCertPath         | `DOCKER_INSIDE_CERT_PATH`          | *(none)*               | TLS certificate directory **on the worker node**, mounted read-only into worker containers |
| DockerInsidePolicyTests      | `DOCKER_INSIDE_POLICY_TESTS`       | `false`                | Test the sandbox once at startup: the daemon must be separate from PentAGI's and must refuse host-escape requests. Its only effect is whether agents get Docker — a failure turns the sandbox off and PentAGI still starts. Off by default |
| DockerDefaultImageForTest    | `DOCKER_DEFAULT_IMAGE_FOR_TEST`    | `vxcontrol/kali-linux:test` | Worker the startup test runs in, and the image it pulls into the sandbox |
| DockerNetwork                | `DOCKER_NETWORK`                   | *(none)*               | Docker network name for bridge mode, or `host` for host network mode. See network modes below. |
| DockerPublicIP               | `DOCKER_PUBLIC_IP`                 | `0.0.0.0`              | Public IP address for Docker containers' port bindings (bridge mode only) |
| DockerWorkDir                | `DOCKER_WORK_DIR`                  | *(none)*               | Custom working directory inside Docker containers |
| DockerDefaultImage           | `DOCKER_DEFAULT_IMAGE`             | `debian:latest`        | Default Docker image for containers when specific images fail |
| DockerDefaultImageForPentest | `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` | `vxcontrol/kali-linux` | Default Docker image for penetration testing tasks |
| TerminalToolTimeout          | `TERMINAL_TOOL_TIMEOUT`            | `1200`                 | Default execution timeout in seconds applied when an agent requests `timeout=0` or a negative value. Accepted range: `1`–`10800` (3 hours). Values `<= 0` or above `10800` are clamped to the 3-hour maximum. Negative values are treated identically to `0`. |

### Worker Docker Access (`DOCKER_INSIDE_*`)

`DOCKER_HOST`, `DOCKER_TLS_VERIFY` and `DOCKER_CERT_PATH` tell **PentAGI itself** which daemon to create worker containers on. The `DOCKER_INSIDE_*` trio is the mirror image: it tells a **worker container** which daemon *it* may talk to. Both sets are independent — sandboxes can be pointed at a different daemon than the one that spawned them.

They are read only when `DOCKER_INSIDE=true`; with it disabled the sandbox gets no Docker access and no Docker configuration whatsoever.

**How the socket is chosen.** With `DOCKER_INSIDE=true`:

| `DOCKER_SOCKET` | `DOCKER_INSIDE_HOST` | Result |
| --- | --- | --- |
| set | empty | That socket is bind-mounted at `/var/run/docker.sock`; this legacy mode does not run the worker policy preflight. |
| set | set | Configuration error: the socket could bypass the designated worker daemon. |
| empty | set | **Nothing is mounted.** The sandbox reaches `DOCKER_INSIDE_HOST` and must pass the Docker API policy preflight before agent use. |
| empty | empty | A local Docker host keeps historical socket autodetection; with a remote `DOCKER_HOST`, initialization fails until a worker endpoint or explicit socket is configured. |

The separate-endpoint row is the point of the feature: mounting the host socket into a sandbox can give an autonomous agent control of the daemon running PentAGI itself, including every other flow's containers. A designated worker daemon must be distinct from the orchestration daemon and enforce the [Docker API policy checked at worker startup](docker.md#docker-api-policy-preflight). An explicit socket remains an unverified legacy exception.

**What is injected.** Every non-empty `DOCKER_INSIDE_*` value is passed into the container as an environment variable with the `_INSIDE_` segment removed, so the Docker CLI inside picks it up with no extra configuration:

| Configured | Seen inside the worker container |
| --- | --- |
| `DOCKER_INSIDE_HOST=tcp://dind:2376` | `DOCKER_HOST=tcp://dind:2376` |
| `DOCKER_INSIDE_TLS_VERIFY=1` | `DOCKER_TLS_VERIFY=1` |
| `DOCKER_INSIDE_CERT_PATH=/certs/client` | `DOCKER_CERT_PATH=/certs/client` |

Empty values are omitted rather than injected blank.

When `DOCKER_INSIDE_CERT_PATH` is set, that directory is additionally bind-mounted **read-only at the same path** inside the container, so the injected `DOCKER_CERT_PATH` resolves unchanged. The path is resolved on the **worker node** — the machine whose daemon creates sandboxes — which may not be the machine running PentAGI or the installer. For that reason the installer does not verify it exists.

**Example — sandboxes use a DinD sidecar over mutual TLS:**

```bash
DOCKER_INSIDE=true
DOCKER_SOCKET=                      # leave empty so the host socket is not mounted
DOCKER_INSIDE_HOST=tcp://dind:2376
DOCKER_INSIDE_TLS_VERIFY=1
DOCKER_INSIDE_CERT_PATH=/certs/client
```

### Usage Details

The Docker settings are primarily used in `pkg/docker/client.go` which implements the Docker client interface used throughout the application. This client is responsible for creating, managing, and executing commands in Docker containers:

- **DockerInside**: Signals whether PentAGI is running inside a Docker container itself, which affects how volumes and sockets are mounted:
  ```go
  inside := cfg.DockerInside
  ```

- **DockerSocket**: Specifies the path to the Docker socket, which is crucial for container management:
  ```go
  if cfg.DockerSocket != "" {
      socket = cfg.DockerSocket
  }
  ```

- **TerminalToolTimeout**: Sets the default execution timeout for terminal tool commands when the tool call uses `timeout=0` or a negative value:
  ```go
  term := NewTerminalTool(
      flowID,
      taskID,
      subtaskID,
      containerID,
      containerLID,
      dockerClient,
      termLogProvider,
      time.Duration(cfg.TerminalToolTimeout)*time.Second,
  )
  ```

  The value is clamped inside the terminal tool: values `<= 0` or above `10800` s (3 hours) are silently raised/capped to the 3-hour maximum — agents always receive a finite timeout. Negative values are accepted at the environment level and treated identically to `0` (both resolve to the 3-hour ceiling). Explicit `timeout` values provided by the tool call override this default when they are within the `1`–`10800` s range.

- **DockerNetwork**: Controls the network isolation mode for containers. Supports two modes:
  
  **Bridge Mode** (custom network name, e.g., `pentagi-network`):
  - Containers run in an isolated bridge network
  - Port forwarding maps container ports to host ports
  - Enhanced security through network isolation
  - Recommended for most deployments
  
  **Host Mode** (special value: `host`):
  - Containers share the host's network stack directly
  - No port forwarding - services bind directly to host interfaces
  - Required for advanced network testing (raw packets, custom protocols)
  - Reduced isolation - use with caution
  
  ```go
  network := cfg.DockerNetwork

  // Host network mode
  if dc.network == "host" {
      hostConfig.NetworkMode = container.NetworkMode("host")
      // No port bindings needed
  } else if dc.network != "" {
      // Bridge mode with custom network
      networkingConfig = &network.NetworkingConfig{
          EndpointsConfig: map[string]*network.EndpointSettings{
              dc.network: {},
          },
      }
      // Port bindings are configured
  }
  ```

- **DockerPublicIP**: Defines the IP address to bind container ports to, making services accessible:
  ```go
  publicIP := cfg.DockerPublicIP

  // Used when setting up port bindings
  hostConfig.PortBindings[natPort] = []nat.PortBinding{
      {
          HostIP:   dc.publicIP,
          HostPort: fmt.Sprintf("%d", port),
      },
  }
  ```

- **DockerWorkDir**: Provides a custom working directory path to use inside containers:
  ```go
  hostDir := getHostDataDir(ctx, cli, dataDir, cfg.DockerWorkDir)
  ```

- **DockerDefaultImage**: Specifies the fallback image to use when requested images aren't available:
  ```go
  defImage := strings.ToLower(cfg.DockerDefaultImage)
  if defImage == "" {
      defImage = defaultImage
  }
  ```

This client is used by the tools executor to run commands in isolated containers, providing a secure environment for AI agents to execute terminal commands.

## Server Settings

These settings control the HTTP and GraphQL server that forms the backend API of PentAGI.

| Option       | Environment Variable | Default Value | Description                      |
| ------------ | -------------------- | ------------- | -------------------------------- |
| ServerPort   | `SERVER_PORT`        | `8080`        | Port for the HTTP server         |
| ServerHost   | `SERVER_HOST`        | `0.0.0.0`     | Host address for the HTTP server |
| ServerUseSSL | `SERVER_USE_SSL`     | `false`       | Enable SSL for the HTTP server   |
| ServerSSLKey | `SERVER_SSL_KEY`     | *(none)*      | Path to SSL key file             |
| ServerSSLCrt | `SERVER_SSL_CRT`     | *(none)*      | Path to SSL certificate file     |

| Option         | Environment Variable | Default Value | Description                                                        |
| -------------- | -------------------- | ------------- | ------------------------------------------------------------------ |
| TrustedProxies | `TRUSTED_PROXIES`    | *(none)*      | Comma-separated proxies whose `X-Forwarded-For` the server believes |

Left empty behind a reverse proxy, every request is attributed to the proxy's own
address, so the per-address login rate limit becomes a single budget shared by all
users. Set it to the addresses or CIDRs of the proxies in front of PentAGI.

### Usage Details

These settings are used in `main.go` to configure and start the HTTP server:

```go
// Build the listen address from host and port
listen := net.JoinHostPort(cfg.ServerHost, strconv.Itoa(cfg.ServerPort))

// Conditionally use TLS based on SSL configuration
if cfg.ServerUseSSL && cfg.ServerSSLCrt != "" && cfg.ServerSSLKey != "" {
    err = r.RunTLS(listen, cfg.ServerSSLCrt, cfg.ServerSSLKey)
} else {
    err = r.Run(listen)
}
```

The settings determine:
- The IP address and port the server listens on
- Whether to use HTTPS (SSL/TLS) for secure connections
- The location of the SSL certificate and key files (when SSL is enabled)

These configurations are crucial for production deployments where proper server binding and secure communication are required.

## Frontend Settings

These settings control how the server serves frontend assets and handles Cross-Origin Resource Sharing (CORS) for API requests from browsers.

| Option      | Environment Variable | Default Value | Description                                                              |
| ----------- | -------------------- | ------------- | ------------------------------------------------------------------------ |
| StaticURL   | `STATIC_URL`         | *(none)*      | URL to serve static frontend assets from (enables reverse proxy mode)    |
| StaticDir   | `STATIC_DIR`         | `./fe`        | Directory containing frontend static files (used when not in proxy mode) |
| CorsOrigins | `CORS_ORIGINS`       | `*`           | Allowed origins for CORS requests (comma-separated)                      |

### Usage Details

The frontend settings are extensively used in `pkg/server/router.go` for configuring how the application serves the frontend:

- **StaticURL**: When set, enables reverse proxy mode where static assets are served from an external URL:
  ```go
  if cfg.StaticURL != nil && cfg.StaticURL.Scheme != "" && cfg.StaticURL.Host != "" {
      // Set up reverse proxy for static assets
      router.NoRoute(func(c *gin.Context) {
          req := c.Request.Clone(c.Request.Context())
          req.URL.Scheme = cfg.StaticURL.Scheme
          req.URL.Host = cfg.StaticURL.Host
          // ...
      })
  }
  ```

- **StaticDir**: When StaticURL is not set, specifies the local directory containing static frontend assets:
  ```go
  // Serve static files from local directory
  router.Use(static.Serve("/", static.LocalFile(cfg.StaticDir, true)))

  // Also used for finding index.html for SPA routes
  indexPath := filepath.Join(cfg.StaticDir, "index.html")
  ```

- **CorsOrigins**: Configures CORS policy for the API, controlling which origins can make requests:
  ```go
  // In GraphQL service initialization
  graphqlService := services.NewGraphqlService(db, baseURL, cfg.CorsOrigins, providers, controller, subscriptions)

  // In CORS middleware configuration
  if !slices.Contains(cfg.CorsOrigins, "*") {
      config.AllowCredentials = true
  }
  config.AllowOrigins = cfg.CorsOrigins
  ```

These settings are essential for:
- Supporting different deployment architectures (single server vs. separate frontend/backend)
- Enabling proper SPA routing for frontend applications
- Configuring security policies for cross-origin requests

## Authentication Settings

These settings control authentication mechanisms, including cookie-based sessions and OAuth providers for user login.

| Option                  | Environment Variable         | Default Value | Description                                            |
| ----------------------- | ---------------------------- | ------------- | ------------------------------------------------------ |
| CookieSigningSalt       | `COOKIE_SIGNING_SALT`        | *(none)*      | Salt for signing and securing cookies used in sessions |
| PublicURL               | `PUBLIC_URL`                 | *(none)*      | Public origin/base URL used to build OAuth callback URLs such as `/api/v1/auth/login-callback` |
| OAuthGoogleClientID     | `OAUTH_GOOGLE_CLIENT_ID`     | *(none)*      | Google OAuth client ID for authentication              |
| OAuthGoogleClientSecret | `OAUTH_GOOGLE_CLIENT_SECRET` | *(none)*      | Google OAuth client secret                             |
| OAuthGithubClientID     | `OAUTH_GITHUB_CLIENT_ID`     | *(none)*      | GitHub OAuth client ID for authentication              |
| OAuthGithubClientSecret | `OAUTH_GITHUB_CLIENT_SECRET` | *(none)*      | GitHub OAuth client secret                             |

### Usage Details

The authentication settings are used in `pkg/server/router.go` to set up authentication middleware and OAuth providers:

- **CookieSigningSalt**: Used to secure cookies for session management:
  ```go
  // Used in auth middleware for authentication checks
  authMiddleware := auth.NewAuthMiddleware(baseURL, cfg.CookieSigningSalt, tokenCache, userCache)

  // Used for cookie store creation
  cookieStore := cookie.NewStore(auth.MakeCookieStoreKey(cfg.CookieSigningSalt)...)
  router.Use(sessions.Sessions("auth", cookieStore))
  ```

- **PublicURL**: The public origin/base URL for OAuth callback endpoints, crucial for redirects after authentication:
  ```go
  publicURL, err := url.Parse(cfg.PublicURL)
  ```

  The router builds the login callback path under the API base URL:
  ```go
  oauthLoginCallbackURL := "/auth/login-callback"

  publicURL, err := url.Parse(cfg.PublicURL)
  if err == nil {
      publicURL.Path = path.Join(baseURL, oauthLoginCallbackURL)
  }
  ```

  In the default deployment, configure your OAuth providers with:
  - **Homepage URL**: `PUBLIC_URL`
  - **Authorization callback URL / Redirect URI**: `${PUBLIC_URL}/api/v1/auth/login-callback`

  Example:
  ```bash
  PUBLIC_URL=https://pentagi.example.com
  OAUTH_GITHUB_CLIENT_ID=your_github_client_id
  OAUTH_GITHUB_CLIENT_SECRET=your_github_client_secret
  OAUTH_GOOGLE_CLIENT_ID=your_google_client_id
  OAUTH_GOOGLE_CLIENT_SECRET=your_google_client_secret
  ```

- **OAuth Provider Settings**: Used to configure authentication with Google and GitHub:
  ```go
  // Google OAuth setup
  if publicURL != nil && cfg.OAuthGoogleClientID != "" && cfg.OAuthGoogleClientSecret != "" {
      googleClient := oauth.NewGoogleOAuthClient(
          cfg.OAuthGoogleClientID,
          cfg.OAuthGoogleClientSecret,
          publicURL.String(),
      )
      // ...
  }

  // GitHub OAuth setup
  if publicURL != nil && cfg.OAuthGithubClientID != "" && cfg.OAuthGithubClientSecret != "" {
      githubClient := oauth.NewGithubOAuthClient(
          cfg.OAuthGithubClientID,
          cfg.OAuthGithubClientSecret,
          publicURL.String(),
      )
      // ...
  }
  ```

  Google and GitHub both use the same PentAGI login callback endpoint. `PUBLIC_URL` should be the externally reachable base URL only, without an extra path suffix. If the URL configured in the provider console does not exactly match the generated callback URL, authentication will fail with a redirect URI mismatch error.

These settings are essential for:
- Secure user authentication and session management
- Supporting social login through OAuth providers
- Enabling proper redirects in the authentication flow

## Web Scraper Settings

These settings control the web scraper service used for browsing websites and taking screenshots, which allows AI agents to interact with web content.

| Option            | Environment Variable  | Default Value | Description                                               |
| ----------------- | --------------------- | ------------- | --------------------------------------------------------- |
| ScraperPublicURL  | `SCRAPER_PUBLIC_URL`  | *(none)*      | Public URL for accessing the scraper service from clients |
| ScraperPrivateURL | `SCRAPER_PRIVATE_URL` | *(none)*      | Private URL for internal scraper service access           |

### Usage Details

The scraper settings are extensively used in the tools executor to provide web browsing capabilities to AI agents:

```go
// In various tool functions in pkg/tools/tools.go
browseTool = &functions.BrowseFunc{
    scPrvURL: fte.cfg.ScraperPrivateURL,
    scPubURL: fte.cfg.ScraperPublicURL,
    // ...
}

screenshotTool = &functions.ScreenshotFunc{
    scPrvURL: fte.cfg.ScraperPrivateURL,
    scPubURL: fte.cfg.ScraperPublicURL,
    // ...
}
```

These URLs serve different purposes:
- **ScraperPublicURL**: Used when generating URLs that will be accessed by the client (browser)
- **ScraperPrivateURL**: Used for internal communication between the backend and the scraper service

The scraper settings enable critical functionality:
- Web browsing capabilities for AI agents
- Screenshot capturing for web content analysis
- Web information gathering for research tasks

## LLM Provider Settings

These settings control the integration with various Large Language Model (LLM) providers, including OpenAI, Anthropic, and custom providers.

### OpenAI

| Option          | Environment Variable | Default Value               | Description                        |
| --------------- | -------------------- | --------------------------- | ---------------------------------- |
| OpenAIKey       | `OPEN_AI_KEY`        | *(none)*                    | API key for OpenAI services        |
| OpenAIServerURL | `OPEN_AI_SERVER_URL` | `https://api.openai.com/v1` | Server URL for OpenAI API requests |

### Anthropic

| Option             | Environment Variable   | Default Value                  | Description                           |
| ------------------ | ---------------------- | ------------------------------ | ------------------------------------- |
| AnthropicAPIKey    | `ANTHROPIC_API_KEY`    | *(none)*                       | API key for Anthropic Claude services |
| AnthropicServerURL | `ANTHROPIC_SERVER_URL` | `https://api.anthropic.com/v1` | Server URL for Anthropic API requests |

#### Workload Identity Federation

Instead of a long-lived key, the Anthropic provider can exchange a short-lived identity token issued by your identity provider (AWS, Google Cloud, Azure, GitHub Actions, Kubernetes, any OIDC issuer) for a Claude API access token, refreshed before it expires. The variables are the ones the vendor SDKs read; see [Workload Identity Federation](https://platform.claude.com/docs/en/manage-claude/workload-identity-federation) for creating the service account, issuer and rule.

| Option                     | Environment Variable            | Default Value | Description                                                                                  |
| -------------------------- | ------------------------------- | ------------- | -------------------------------------------------------------------------------------------- |
| AnthropicFederationRuleID  | `ANTHROPIC_FEDERATION_RULE_ID`  | *(none)*      | Federation rule to exchange under (`fdrl_...`)                                               |
| AnthropicOrganizationID    | `ANTHROPIC_ORGANIZATION_ID`     | *(none)*      | Anthropic organization UUID                                                                  |
| AnthropicServiceAccountID  | `ANTHROPIC_SERVICE_ACCOUNT_ID`  | *(none)*      | Service account the minted token acts as (`svac_...`)                                        |
| AnthropicWorkspaceID       | `ANTHROPIC_WORKSPACE_ID`        | *(none)*      | Workspace to scope the token to; required when the rule covers more than one workspace       |
| AnthropicIdentityTokenFile | `ANTHROPIC_IDENTITY_TOKEN_FILE` | *(none)*      | Path to the identity token, re-read on every exchange; takes precedence over the token below |
| AnthropicIdentityToken     | `ANTHROPIC_IDENTITY_TOKEN`      | *(none)*      | The identity token itself; read once, so only for runs shorter than its lifetime             |

Federation is used only while `ANTHROPIC_API_KEY` is empty: a non-empty key shadows it, and the server log says so at startup. A partial federation config leaves the provider disabled and the log names the missing variables. The token file path is read inside the pentagi container, so mount the file there. The exchange goes to `ANTHROPIC_SERVER_URL` + `/oauth/token`, so federation works against the Anthropic API itself and not through a gateway.

Every provider, flow and settings test that uses the same federation rule, organization, service account, workspace and server URL shares one access token, exchanged once and refreshed the way the vendor SDKs do: from two minutes before it expires a failed exchange keeps serving the cached token and logs a warning that names the time calls start to fail, and from 30 seconds before it the call fails. Both points move closer for a token that lives only a few minutes, so it is not exchanged again on every call.

An identity token that carries a `jti` claim, as Kubernetes service-account and GitHub Actions tokens do, can be exchanged only once; presenting it again is refused with `jti_reused`. The token file must therefore hold a new token before each refresh, so the platform has to rotate it well within the lifetime of the minted token: the rule's `token_lifetime_seconds`, capped at twice the remaining lifetime of the identity token. A Kubernetes projected token is rotated at 80% of its `expirationSeconds`, about 48 minutes by default, which is too slow for a rule that mints ten-minute tokens.

`ANTHROPIC_IDENTITY_TOKEN` is read once at startup and cannot be rotated: once the identity token expires, and after its first exchange if it carries `jti`, the provider cannot mint a new access token. It suits only runs shorter than the identity token's lifetime, the server log warns about it at startup and names the expiry, and a long-running server needs the token file.

**Note on Google Vertex AI**: PentAGI does not currently expose a dedicated Vertex AI configuration path for Anthropic Claude in `.env`. The variables above target the direct Anthropic API. To run Claude through a non-Anthropic-hosted backend, use one of:

- **AWS Bedrock**: see the [AWS Bedrock LLM Provider](#aws-bedrock-llm-provider) section below and configure the `BEDROCK_*` variables.
- **OpenAI-compatible gateway in front of Vertex AI**: expose Vertex AI through a proxy or gateway that translates requests into the Chat Completions format while preserving the chat and tool-call behavior PentAGI requires, then configure it as a [custom LLM provider](#custom-llm-provider) (`LLM_SERVER_URL`, `LLM_SERVER_KEY`, `LLM_SERVER_MODEL`). Reliability of this path depends on the gateway you choose.

There is no `VERTEX_API_KEY` or `GOOGLE_APPLICATION_CREDENTIALS` variable wired into PentAGI's provider initialization today.

### Ollama LLM Provider

| Option                        | Environment Variable                | Default Value        | Description                                                       |
| ----------------------------- | ----------------------------------- | -------------------- | ----------------------------------------------------------------- |
| OllamaServerURL               | `OLLAMA_SERVER_URL`                 | *(none)*             | Ollama server URL (local or cloud https://ollama.com)             |
| OllamaServerAPIKey            | `OLLAMA_SERVER_API_KEY`             | *(none)*             | Ollama Cloud API key (optional, required for https://ollama.com)  |
| OllamaServerModel             | `OLLAMA_SERVER_MODEL`               | *(none)*             | Default model to use for inference                                |
| OllamaServerConfig            | `OLLAMA_SERVER_CONFIG_PATH`         | *(none)*             | Path to config file for Ollama provider options                   |
| OllamaServerPullModelsTimeout | `OLLAMA_SERVER_PULL_MODELS_TIMEOUT` | `600`                | Timeout in seconds for model downloads                            |
| OllamaServerPullModelsEnabled | `OLLAMA_SERVER_PULL_MODELS_ENABLED` | `false`              | Automatically download required models on startup                 |
| OllamaServerLoadModelsEnabled | `OLLAMA_SERVER_LOAD_MODELS_ENABLED` | `false`              | Load available models list from server API                        |

**Deployment Scenarios**: 
- **Local Server**: Set `OLLAMA_SERVER_URL` to local endpoint (e.g., `http://ollama-server:11434`), leave `OLLAMA_SERVER_API_KEY` empty
- **Ollama Cloud**: Set `OLLAMA_SERVER_URL=https://ollama.com` and provide `OLLAMA_SERVER_API_KEY` from https://ollama.com/settings/keys

**Note:** When `OllamaServerLoadModelsEnabled=false`, only the default model is available. Enable this to see all installed models in the UI.

### Google AI (Gemini) LLM Provider

| Option          | Environment Variable | Default Value                               | Description                           |
| --------------- | -------------------- | ------------------------------------------- | ------------------------------------- |
| GeminiAPIKey    | `GEMINI_API_KEY`     | *(none)*                                    | API key for Google AI Gemini services |
| GeminiServerURL | `GEMINI_SERVER_URL`  | `https://generativelanguage.googleapis.com` | Server URL for Gemini API requests    |

### AWS Bedrock LLM Provider

| Option              | Environment Variable        | Default Value | Description                                                                                                              |
| ------------------- | --------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------ |
| BedrockRegion       | `BEDROCK_REGION`            | `us-east-1`   | AWS region for Bedrock service                                                                                           |
| BedrockDefaultAuth  | `BEDROCK_DEFAULT_AUTH`      | `false`       | Use default AWS SDK credential chain (environment variables, EC2 role, ~/.aws/credentials) - highest priority            |
| BedrockBearerToken  | `BEDROCK_BEARER_TOKEN`      | *(none)*      | Bearer token for authentication - takes priority over static credentials                                                 |
| BedrockAccessKey    | `BEDROCK_ACCESS_KEY_ID`     | *(none)*      | AWS access key ID for static credentials authentication                                                                  |
| BedrockSecretKey    | `BEDROCK_SECRET_ACCESS_KEY` | *(none)*      | AWS secret access key for static credentials authentication                                                              |
| BedrockSessionToken | `BEDROCK_SESSION_TOKEN`     | *(none)*      | AWS session token for temporary credentials (optional, used with static credentials for STS/assumed roles)               |
| BedrockServerURL    | `BEDROCK_SERVER_URL`        | *(none)*      | Optional custom endpoint URL for Bedrock service (VPC endpoints, local testing)                                          |
| BedrockConfig       | `BEDROCK_CONFIG_PATH`       | *(none)*      | Path to a custom YAML config that replaces the built-in Bedrock per-agent config (model assignments, prices)             |

**Authentication Priority**: `BedrockDefaultAuth` (highest) → `BedrockBearerToken` → `BedrockAccessKey`+`BedrockSecretKey` (lowest)

### DeepSeek LLM Provider

| Option            | Environment Variable  | Default Value              | Description                                              |
| ----------------- | --------------------- | -------------------------- | -------------------------------------------------------- |
| DeepSeekAPIKey    | `DEEPSEEK_API_KEY`    | *(none)*                   | DeepSeek API key for authentication                      |
| DeepSeekServerURL | `DEEPSEEK_SERVER_URL` | `https://api.deepseek.com` | DeepSeek API endpoint URL                                |
| DeepSeekProvider  | `DEEPSEEK_PROVIDER`   | *(none)*                   | Provider name prefix for LiteLLM integration (optional)  |

**LiteLLM Integration**: Set `DEEPSEEK_PROVIDER=deepseek` to enable model prefixing (e.g., `deepseek/deepseek-flash`) when using LiteLLM proxy with default PentAGI configs.

### GLM LLM Provider

| Option         | Environment Variable | Default Value                  | Description                                              |
| -------------- | -------------------- | ------------------------------ | -------------------------------------------------------- |
| GLMAPIKey      | `GLM_API_KEY`        | *(none)*                       | GLM API key for authentication                           |
| GLMServerURL   | `GLM_SERVER_URL`     | `https://api.z.ai/api/paas/v4` | GLM API endpoint URL (international)                     |
| GLMProvider    | `GLM_PROVIDER`       | *(none)*                       | Provider name prefix for LiteLLM integration (optional)  |

**Alternative Endpoints**:
- International: `https://api.z.ai/api/paas/v4` (default)
- China: `https://open.bigmodel.cn/api/paas/v4`
- Coding-specific: `https://api.z.ai/api/coding/paas/v4`

**LiteLLM Integration**: Set `GLM_PROVIDER=zai` to enable model prefixing (e.g., `zai/glm-4`) when using LiteLLM proxy with default PentAGI configs.

### Kimi LLM Provider

| Option          | Environment Variable | Default Value                 | Description                                              |
| --------------- | -------------------- | ----------------------------- | -------------------------------------------------------- |
| KimiAPIKey      | `KIMI_API_KEY`       | *(none)*                      | Kimi API key for authentication                          |
| KimiServerURL   | `KIMI_SERVER_URL`    | `https://api.moonshot.ai/v1`  | Kimi API endpoint URL (international)                    |
| KimiProvider    | `KIMI_PROVIDER`      | *(none)*                      | Provider name prefix for LiteLLM integration (optional)  |

**Alternative Endpoints**:
- International: `https://api.moonshot.ai/v1` (default)
- China: `https://api.moonshot.cn/v1`

**LiteLLM Integration**: Set `KIMI_PROVIDER=moonshot` to enable model prefixing (e.g., `moonshot/kimi-k2.6`) when using LiteLLM proxy with default PentAGI configs.

### Qwen LLM Provider

| Option          | Environment Variable | Default Value                                          | Description                                              |
| --------------- | -------------------- | ------------------------------------------------------ | -------------------------------------------------------- |
| QwenAPIKey      | `QWEN_API_KEY`       | *(none)*                                               | Qwen Cloud, DashScope, or gateway API key                |
| QwenServerURL   | `QWEN_SERVER_URL`    | `https://dashscope-us.aliyuncs.com/compatible-mode/v1` | Direct DashScope or OpenAI-compatible gateway URL        |
| QwenProvider    | `QWEN_PROVIDER`      | *(none)*                                               | Gateway model prefix: `qwen_cloud` or `dashscope`        |

**Alternative Endpoints**:
- US: `https://dashscope-us.aliyuncs.com/compatible-mode/v1` (default)
- Singapore: `https://dashscope-intl.aliyuncs.com/compatible-mode/v1`
- China: `https://dashscope.aliyuncs.com/compatible-mode/v1`

**LiteLLM Integration**: Set `QWEN_PROVIDER=qwen_cloud` for Qwen Cloud routes or `QWEN_PROVIDER=dashscope` for Alibaba Cloud routes. The bundled role configuration uses only model IDs shared by both platforms.

### MiniMax LLM Provider

| Option           | Environment Variable | Default Value               | Description                                             |
| ---------------- | -------------------- | --------------------------- | ------------------------------------------------------- |
| MiniMaxAPIKey    | `MINIMAX_API_KEY`    | *(none)*                    | MiniMax API key for authentication                      |
| MiniMaxServerURL | `MINIMAX_SERVER_URL` | `https://api.minimax.io/v1` | MiniMax API endpoint URL                                |
| MiniMaxProvider  | `MINIMAX_PROVIDER`   | *(none)*                    | Provider name prefix for LiteLLM integration (optional) |

**LiteLLM Integration**: Set `MINIMAX_PROVIDER=minimax` to enable model prefixing (e.g., `minimax/MiniMax-M3`) when using LiteLLM proxy with default PentAGI configs.

### Mistral LLM Provider

| Option           | Environment Variable | Default Value                | Description                                             |
| ---------------- | -------------------- | ---------------------------- | ------------------------------------------------------- |
| MistralAPIKey    | `MISTRAL_API_KEY`    | *(none)*                     | Mistral API key for authentication                      |
| MistralServerURL | `MISTRAL_SERVER_URL` | `https://api.mistral.ai/v1`  | Mistral API endpoint URL                                |
| MistralProvider  | `MISTRAL_PROVIDER`   | *(none)*                     | Provider name prefix for LiteLLM integration (optional) |

**LiteLLM Integration**: Set `MISTRAL_PROVIDER=mistral` to enable model prefixing (e.g., `mistral/mistral-medium-latest`) when using LiteLLM proxy with default PentAGI configs.

### xAI (Grok) LLM Provider

| Option       | Environment Variable | Default Value          | Description                                             |
| ------------ | -------------------- | ---------------------- | ------------------------------------------------------- |
| XAIAPIKey    | `XAI_API_KEY`        | *(none)*               | xAI API key for authentication                          |
| XAIServerURL | `XAI_SERVER_URL`     | `https://api.x.ai/v1`  | xAI API endpoint URL                                    |
| XAIProvider  | `XAI_PROVIDER`       | *(none)*               | Provider name prefix for LiteLLM integration (optional) |

**LiteLLM Integration**: Set `XAI_PROVIDER=xai` to enable model prefixing (e.g., `xai/grok-4.3`) when using LiteLLM proxy with default PentAGI configs.

### Custom LLM Provider

| Option                     | Environment Variable            | Default Value | Description                                                                  |
| -------------------------- | ------------------------------- | ------------- | ---------------------------------------------------------------------------- |
| LLMServerURL               | `LLM_SERVER_URL`                | *(none)*      | Server URL for custom LLM provider                                           |
| LLMServerKey               | `LLM_SERVER_KEY`                | *(none)*      | API key for custom LLM provider                                              |
| LLMServerModel             | `LLM_SERVER_MODEL`              | *(none)*      | Model name for custom LLM provider                                           |
| LLMServerConfig            | `LLM_SERVER_CONFIG_PATH`        | *(none)*      | Path to config file for custom LLM provider options                          |
| LLMServerProvider          | `LLM_SERVER_PROVIDER`           | *(none)*      | Provider name prefix for model names (useful for LiteLLM proxy)              |
| LLMServerPreserveReasoning | `LLM_SERVER_PRESERVE_REASONING` | `false`       | Preserve reasoning content in multi-turn conversations (required by some providers) |
| LLMServerAPIType           | `LLM_SERVER_API_TYPE`           | *(none)*      | `azure` or `azure_ad` to talk to an Azure OpenAI deployment; empty for a plain OpenAI-compatible endpoint |
| LLMServerAPIVersion        | `LLM_SERVER_API_VERSION`        | `2024-10-21`  | Azure OpenAI API version, used only when `LLM_SERVER_API_TYPE` names an Azure mode |

### Usage Details

The LLM provider settings are used in `pkg/providers` modules to initialize and configure the appropriate language model providers:

- **OpenAI Settings**: Used in `pkg/providers/openai/openai.go` to create the OpenAI client:
  ```go
  baseURL := cfg.OpenAIServerURL

  client, err := openai.New(
      openai.WithToken(cfg.OpenAIKey),
      openai.WithModel(OpenAIAgentModel),
      openai.WithBaseURL(baseURL),
      // ...
  )
  ```

- **Anthropic Settings**: Used in `pkg/providers/anthropic/anthropic.go` to create the Anthropic client:
  ```go
  baseURL := cfg.AnthropicServerURL

  client, err := anthropic.New(
      anthropic.WithToken(cfg.AnthropicAPIKey),
      anthropic.WithBaseURL(baseURL),
      // ...
  )
  ```

- **Ollama Settings**: Used in `pkg/providers/ollama/ollama.go` to create the Ollama client:
  ```go
  serverURL := cfg.OllamaServerURL

  client, err := ollama.New(
      ollama.WithServerURL(serverURL),
      ollama.WithHTTPClient(httpClient),
      ollama.WithModel(OllamaAgentModel),
      ollama.WithPullModel(),
  )

  // Load provider options from config file if specified
  if cfg.OllamaServerConfig != "" {
      configData, err := os.ReadFile(cfg.OllamaServerConfig)
      providerConfig, err := BuildProviderConfig(cfg, configData)
      // ...
  }
  ```

- **Gemini Settings**: Used in `pkg/providers/gemini/gemini.go` to create the Google AI client:
  ```go
  opts := []googleai.Option{
      googleai.WithRest(),
      googleai.WithAPIKey(cfg.GeminiAPIKey),
      googleai.WithEndpoint(cfg.GeminiServerURL),
      googleai.WithDefaultModel(GeminiAgentModel),
  }

  client, err := googleai.New(context.Background(), opts...)
  ```

- **Bedrock Settings**: Used in `pkg/providers/bedrock/bedrock.go` to create the AWS Bedrock client:
  ```go
  opts := []func(*bconfig.LoadOptions) error{
      bconfig.WithRegion(cfg.BedrockRegion),
      bconfig.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
          cfg.BedrockAccessKey,
          cfg.BedrockSecretKey,
          cfg.BedrockSessionToken,
      )),
  }

  if cfg.BedrockServerURL != "" {
      opts = append(opts, bconfig.WithBaseEndpoint(cfg.BedrockServerURL))
  }

  bcfg, err := bconfig.LoadDefaultConfig(context.Background(), opts...)
  bclient := bedrockruntime.NewFromConfig(bcfg)

  client, err := bedrock.New(
      bedrock.WithClient(bclient),
      bedrock.WithModel(BedrockAgentModel),
      bedrock.WithConverseAPI(),
  )
  ```

  The `BedrockSessionToken` is optional and only required when using temporary AWS credentials (e.g., from STS, assumed roles, or MFA-enabled IAM users). For permanent IAM user credentials, leave this field empty.

- **Custom LLM Settings**: Used in `pkg/providers/custom/custom.go` to create a custom LLM client:
  ```go
  baseKey := cfg.LLMServerKey
  baseURL := cfg.LLMServerURL
  baseModel := cfg.LLMServerModel

  client, err := openai.New(
      openai.WithToken(baseKey),
      openai.WithModel(baseModel),
      openai.WithBaseURL(baseURL),
      // ...
  )

  // Load provider options from config file if specified
  if cfg.LLMServerConfig != "" {
      providerConfig, err := LoadConfig(cfg.LLMServerConfig, simple)
      // ...
  }
  ```

- **LLMServerPreserveReasoning**: Controls whether reasoning content is preserved and sent back in multi-turn conversations:
  ```go
  // Used in custom provider to preserve reasoning content
  if cfg.LLMServerPreserveReasoning {
      // Preserves and returns reasoning_content in assistant messages
  }
  ```
  - `false` (default): Reasoning content is not preserved in conversation history
  - `true`: Reasoning content is preserved and sent in subsequent API calls

This setting is required by some LLM providers (e.g., Moonshot) that return errors like "thinking is enabled but reasoning_content is missing in assistant tool call message" when reasoning content is not included in multi-turn conversations. Enable this setting if your provider requires reasoning content to be preserved across conversation turns.

Provider registration is table-driven. `pkg/providers/registry.go` holds one entry per
provider type and `pkg/providers/providers.go` walks it; nothing enumerates providers by hand:

```go
// pkg/providers/registry.go
type registryEntry struct {
    Type        provider.ProviderType
    Name        provider.ProviderName
    Enabled     func(*config.Config) bool
    NewConfig   func(*config.Config) (*pconfig.ProviderConfig, error)
    New         func(*config.Config, provider.ProviderName, *pconfig.ProviderConfig) (provider.Provider, error)
    BuildConfig func(*config.Config, []byte) (*pconfig.ProviderConfig, error)
}

var providerRegistry = []registryEntry{
    {
        Type:        provider.ProviderOpenAI,
        Name:        provider.DefaultProviderNameOpenAI,
        Enabled:     func(c *config.Config) bool { return c.OpenAIKey != "" },
        NewConfig:   ignoreConfig(openai.DefaultProviderConfig),
        New:         openai.New,
        BuildConfig: fromData(openai.BuildProviderConfig),
    },
    // ... one entry per provider type
}
```

`Enabled` is where the credential gate lives — a deployment without a given key simply ends
up with fewer providers rather than a failed startup:

```go
// pkg/providers/providers.go
for _, e := range providerRegistry {
    if !e.Enabled(cfg) {
        continue
    }

    p, err := e.New(cfg, e.Name, defaultConfigs[e.Type])
    if err != nil {
        return nil, fmt.Errorf("failed to create %s provider: %w", e.Type, err)
    }

    providers[e.Name] = p
}
```

`defaultConfigs` comes from the same table: `buildDefaultConfigs` calls every entry's
`NewConfig` beforehand, and a config that fails to load aborts startup unless that provider
is disabled anyway, in which case the reason is recorded and reported later. `BuildConfig`
is the same construction from stored bytes, used through `entryForType` when a user-defined
provider or a provider test rebuilds one at request time.

An entry may also wrap its constructor. The custom provider's does, to hand the transport the
same catalogue enrichment the picker sees — its models come from the gateway at runtime, so
the reasoning metadata the bundled `models.yml` files carry has to be folded in on the way.

Adding a provider is therefore adding one entry; the full checklist is in the repository root
`CLAUDE.md` under "Adding a New LLM Provider".

These settings are critical for:
- Connecting to various LLM providers for AI capabilities
- Supporting multiple model options for different tasks
- Enabling custom or self-hosted LLM solutions
- Configuring specific model behaviors and parameters

## Embedding Settings

These settings control the vector embedding service used for semantic search and similarity matching, which is fundamental for PentAGI's intelligent search capabilities.

| Option                 | Environment Variable        | Default Value | Description                                                                |
| ---------------------- | --------------------------- | ------------- | -------------------------------------------------------------------------- |
| EmbeddingURL           | `EMBEDDING_URL`             | *(none)*      | Server URL for embedding provider (overrides provider-specific URLs)       |
| EmbeddingKey           | `EMBEDDING_KEY`             | *(none)*      | API key for embedding provider (overrides provider-specific keys)          |
| EmbeddingModel         | `EMBEDDING_MODEL`           | *(none)*      | Model name for embedding generation                                        |
| EmbeddingStripNewLines | `EMBEDDING_STRIP_NEW_LINES` | `true`        | Whether to strip newlines before embedding (improves quality)              |
| EmbeddingBatchSize     | `EMBEDDING_BATCH_SIZE`      | `512`         | Batch size for embedding operations (affects memory usage and performance) |
| EmbeddingProvider      | `EMBEDDING_PROVIDER`        | `openai`      | Provider for embeddings (openai, ollama, mistral, jina, huggingface)       |
| EmbeddingMaxTextBytes  | `EMBEDDING_MAX_TEXT_BYTES`  | `8192`        | Maximum byte size of text sent to the embedding model per document. Acts as a byte-level proxy for token limits (e.g. 8192 tokens for OpenAI models). When a stored document exceeds this limit the heavy content field (Guide/Answer/Code) is truncated to fit before computing the vector; the full original text is always preserved in the database. Reduce if your model has a smaller context window. |

### Usage Details

The embedding settings are extensively used in `pkg/providers/embeddings/embedder.go` to configure the vector embedding service:

- **EmbeddingProvider**: Determines which embedding provider to use:
  ```go
  switch cfg.EmbeddingProvider {
  case "openai":
      return newOpenAIEmbedder(ctx, cfg)
  case "ollama":
      return newOllamaEmbedder(ctx, cfg)
  case "mistral":
      return newMistralEmbedder(ctx, cfg)
  case "jina":
      return newJinaEmbedder(ctx, cfg)
  case "huggingface":
      return newHuggingFaceEmbedder(ctx, cfg)
  default:
      return &embedder{nil}, fmt.Errorf("unsupported embedding provider: %s", cfg.EmbeddingProvider)
  }
  ```

- **Provider-specific configurations**: Used to configure each embedding provider with appropriate options:
  ```go
  // Example for OpenAI embeddings
  if cfg.EmbeddingURL != "" {
      opts = append(opts, openai.WithBaseURL(cfg.EmbeddingURL))
  } else if cfg.OpenAIServerURL != "" {
      opts = append(opts, openai.WithBaseURL(cfg.OpenAIServerURL))
  }

  if cfg.EmbeddingKey != "" {
      opts = append(opts, openai.WithToken(cfg.EmbeddingKey))
  } else if cfg.OpenAIKey != "" {
      opts = append(opts, openai.WithToken(cfg.OpenAIKey))
  }

  if cfg.EmbeddingModel != "" {
      opts = append(opts, openai.WithEmbeddingModel(cfg.EmbeddingModel))
  }
  ```

- **Embedding behavior configuration**: Controls how text is processed for embeddings:
  ```go
  embeddings.WithStripNewLines(cfg.EmbeddingStripNewLines),
  embeddings.WithBatchSize(cfg.EmbeddingBatchSize),
  ```

These settings are essential for:
- Configuring semantic search capabilities
- Determining which embedding model to use
- Optimizing embedding performance and quality
- Supporting multiple embedding providers for flexibility

## Summarizer Settings

These settings control the text summarization behavior used for condensing long conversations and improving context management in AI interactions. The summarization system is a critical component that allows PentAGI to maintain coherent, long-running conversations while managing token usage effectively.

| Option                   | Environment Variable             | Default Value | Description                                                |
| ------------------------ | -------------------------------- | ------------- | ---------------------------------------------------------- |
| SummarizerPreserveLast   | `SUMMARIZER_PRESERVE_LAST`       | `true`        | Preserve the last message in summarization                 |
| SummarizerUseQA          | `SUMMARIZER_USE_QA`              | `true`        | Use question-answer format for summarization               |
| SummarizerSumHumanInQA   | `SUMMARIZER_SUM_MSG_HUMAN_IN_QA` | `false`       | Include human messages in QA summaries                     |
| SummarizerLastSecBytes   | `SUMMARIZER_LAST_SEC_BYTES`      | `51200`       | Bytes to preserve from the last section (50KB)             |
| SummarizerMaxBPBytes     | `SUMMARIZER_MAX_BP_BYTES`        | `16384`       | Maximum bytes for bullet points summarization (16KB)       |
| SummarizerMaxQASections  | `SUMMARIZER_MAX_QA_SECTIONS`     | `10`          | Maximum QA sections to include                             |
| SummarizerMaxQABytes     | `SUMMARIZER_MAX_QA_BYTES`        | `65536`       | Maximum bytes for QA summarization (64KB)                  |
| SummarizerKeepQASections | `SUMMARIZER_KEEP_QA_SECTIONS`    | `1`           | Number of recent QA sections to keep without summarization |

### Usage Details and Impact on System Behavior

The summarizer settings map directly to the `SummarizerConfig` structure that controls the chain summarization algorithm in `pkg/csum`. These settings work together to implement a sophisticated, multi-strategy approach to managing conversation context:

#### Core Summarization Strategies and Their Parameters

1. **Section Summarization** - Always active, ensures all older sections (except the last one) consist of a single summarized pair
   - No specific parameters control this as it's a fundamental part of the algorithm
   - Prevents unbounded growth by consolidating completed conversation sections

2. **Last Section Management** (`SummarizerPreserveLast` and `SummarizerLastSecBytes`)
   - Controls how the current/active conversation section is managed
   - When `SummarizerPreserveLast = true`, older messages within the last section will be summarized when the section exceeds `SummarizerLastSecBytes` bytes
   - A reserve space of 25% is automatically maintained to accommodate new messages without triggering frequent re-summarization
   - Individual oversized pairs are summarized separately if they exceed `SummarizerMaxBPBytes`

3. **QA Pair Summarization** (`SummarizerUseQA`, `SummarizerMaxQASections`, `SummarizerMaxQABytes`, `SummarizerSumHumanInQA`)
   - When `SummarizerUseQA = true`, creates larger summarization units focused on question-answer patterns
   - Preserves the most recent `SummarizerMaxQASections` sections as long as they don't exceed `SummarizerMaxQABytes` total
   - If `SummarizerSumHumanInQA = true`, human messages are also summarized; otherwise, they're preserved verbatim

#### Deep Dive: Parameter Impact and Recommendations

**`SummarizerPreserveLast`** (Default: `true`)
- **Purpose**: Controls whether the last (active) section has size management applied
- **Impact**: When enabled, prevents the active conversation from growing indefinitely
- **When to adjust**:
  - Enable (default) for production systems and long-running conversations
  - Disable only for debugging or when you need to preserve the complete conversation history regardless of size

**`SummarizerLastSecBytes`** (Default: `51200` - 50KB)
- **Purpose**: Maximum byte size for the last (active) section before summarization begins
- **Impact**: Directly controls how much conversation history is preserved verbatim in the active section
- **When to adjust**:
  - Increase for models with larger context windows to maintain more conversation detail
  - Decrease for models with smaller context to prevent token limits from being exceeded
  - Balance with `SummarizerMaxBPBytes` to ensure coherent summarization

**`SummarizerMaxBPBytes`** (Default: `16384` - 16KB)
- **Purpose**: Maximum byte size for individual body pairs (typically AI responses)
- **Impact**: Controls when individual large responses get summarized, even if the overall section is under limit
- **When to adjust**:
  - Increase if your use case involves long but important AI responses that shouldn't be summarized
  - Decrease if you want more aggressive summarization of lengthy responses

**`SummarizerUseQA`** (Default: `true`)
- **Purpose**: Enables question-answer style summarization that creates more cohesive summaries
- **Impact**: When enabled, creates a new first section with a summary of older interactions, preserving recent sections
- **When to adjust**:
  - Enable (default) for more coherent, organized summaries focused on main topics
  - Disable if you prefer simpler, section-by-section summarization without cross-section analysis

**`SummarizerMaxQASections`** (Default: `10`)
- **Purpose**: Maximum number of recent sections to preserve when using QA-style summarization
- **Impact**: Directly controls how many conversation turns remain intact after QA summarization
- **When to adjust**:
  - Increase to preserve more recent conversation context (at the cost of token usage)
  - Decrease to create more compact conversation histories, focusing on only the very recent exchanges

**`SummarizerMaxQABytes`** (Default: `65536` - 64KB)
- **Purpose**: Maximum total byte size for preserved sections in QA-style summarization
- **Impact**: Sets an upper bound on memory used by preserved sections, regardless of section count
- **When to adjust**:
  - Increase for models with larger context windows or when detailed context is essential
  - Decrease for smaller context models or when prioritizing efficiency over context preservation

**`SummarizerSumHumanInQA`** (Default: `false`)
- **Purpose**: Controls whether human messages are summarized in QA-style summarization
- **Impact**: When false, human messages are preserved verbatim; when true, they are also summarized
- **When to adjust**:
  - Keep disabled (default) to preserve the exact wording of user queries
  - Enable only when human messages are very verbose and token efficiency is critical

**`SummarizerKeepQASections`** (Default: `1`)
- **Purpose**: Controls the number of recent QA sections to keep without summarization
- **Impact**: Directly controls how many recent conversation turns are preserved verbatim
- **When to adjust**:
  - Increase to preserve more recent conversation context
  - Decrease to create more compact conversation histories, focusing on only the very recent exchanges

### Summarization Effects on Agent Behavior

The summarization settings have significant effects on agent behavior:

1. **Context Retention vs. Token Efficiency**
   - More aggressive summarization (smaller byte limits) reduces token usage but may lose context details
   - More permissive settings (larger byte limits) preserve more context but increase token consumption

2. **Conversation Coherence**
   - Appropriate summarization helps the agent maintain a coherent understanding of the conversation
   - Over-aggressive summarization may cause the agent to lose important details or previous instructions
   - Under-aggressive summarization may lead to context overflow in longer conversations

3. **Response Quality**
   - QA-style summarization (`SummarizerUseQA = true`) typically improves response quality for complex tasks
   - Preserving human messages (`SummarizerSumHumanInQA = false`) helps maintain alignment with user intent
   - Appropriate `SummarizerMaxBPBytes` prevents loss of detailed information from complex AI responses

### Implementation Details

The summarizer settings are used in `pkg/providers/providers.go` to configure the summarization behavior:

```go
summarizer := provider.SummarizerSettings{
    PreserveLast:  cfg.SummarizerPreserveLast,
    UseQA:         cfg.SummarizerUseQA,
    SummHumanInQA: cfg.SummarizerSumHumanInQA,
    LastSecBytes:  cfg.SummarizerLastSecBytes,
    MaxBPBytes:    cfg.SummarizerMaxBPBytes,
    MaxQASections: cfg.SummarizerMaxQASections,
    MaxQABytes:    cfg.SummarizerMaxQABytes,
}
```

These settings are passed to various components through the chain summarization system:

```go
// In csum/chain_summary.go
func NewSummarizer(config SummarizerConfig) Summarizer {
    if config.PreserveLast {
        if config.LastSecBytes <= 0 {
            config.LastSecBytes = maxLastSectionByteSize
        }
    }

    if config.UseQA {
        if config.MaxQASections <= 0 {
            config.MaxQASections = maxQAPairSections
        }
        if config.MaxQABytes <= 0 {
            config.MaxQABytes = maxQAPairByteSize
        }
    }

    if config.MaxBPBytes <= 0 {
        config.MaxBPBytes = maxSingleBodyPairByteSize
    }

    return &summarizer{config: config}
}
```

### Recommended Settings for Different Use Cases

1. **Long-running Assistant Conversations**
   ```
   SummarizerPreserveLast: true
   SummarizerLastSecBytes: 51200 (50KB)
   SummarizerMaxBPBytes: 16384 (16KB)
   SummarizerUseQA: true
   SummarizerMaxQASections: 10
   SummarizerMaxQABytes: 65536 (64KB)
   SummarizerSumHumanInQA: false
   SummarizerKeepQASections: 1
   ```
   The default settings are optimized for assistant-style conversations. They maintain a good balance between context retention and token efficiency.

2. **Technical Problem-Solving with Large Context Models**
   ```
   SummarizerPreserveLast: true
   SummarizerLastSecBytes: 81920 (80KB)
   SummarizerMaxBPBytes: 32768 (32KB)
   SummarizerUseQA: true
   SummarizerMaxQASections: 15
   SummarizerMaxQABytes: 102400 (100KB)
   SummarizerSumHumanInQA: false
   SummarizerKeepQASections: 1
   ```
   Increased limits to preserve more technical details when using models with large context windows (e.g., GPT-4).

3. **Limited Context Models**
   ```
   SummarizerPreserveLast: true
   SummarizerLastSecBytes: 25600 (25KB)
   SummarizerMaxBPBytes: 8192 (8KB)
   SummarizerUseQA: true
   SummarizerMaxQASections: 5
   SummarizerMaxQABytes: 32768 (32KB)
   SummarizerSumHumanInQA: true
   SummarizerKeepQASections: 1
   ```
   More aggressive summarization for models with smaller context windows (e.g., smaller or older LLMs).

4. **Debugging or Analysis (Maximum Context Preservation)**
   ```
   SummarizerPreserveLast: false
   SummarizerUseQA: false
   SummarizerKeepQASections: 0
   ```
   Disables active summarization to preserve the complete conversation history for debugging purposes. Note that this can lead to context overflow in long conversations.

## Assistant Settings

These settings control the behavior of the AI assistant functionality, including whether to use multi-agent delegation and assistant-specific summarization settings.

| Option                            | Environment Variable                    | Default Value | Description                                                             |
| --------------------------------- | --------------------------------------- | ------------- | ----------------------------------------------------------------------- |
| AssistantUseAgents                | `ASSISTANT_USE_AGENTS`                  | `false`       | Controls the default value for agent usage when creating new assistants |
| AssistantSummarizerPreserveLast   | `ASSISTANT_SUMMARIZER_PRESERVE_LAST`    | `true`        | Whether to preserve all messages in the assistant's last section        |
| AssistantSummarizerLastSecBytes   | `ASSISTANT_SUMMARIZER_LAST_SEC_BYTES`   | `76800`       | Maximum byte size for assistant's last section (75KB)                   |
| AssistantSummarizerMaxBPBytes     | `ASSISTANT_SUMMARIZER_MAX_BP_BYTES`     | `16384`       | Maximum byte size for a single body pair in assistant context (16KB)    |
| AssistantSummarizerMaxQASections  | `ASSISTANT_SUMMARIZER_MAX_QA_SECTIONS`  | `7`           | Maximum QA sections to preserve in assistant context                    |
| AssistantSummarizerMaxQABytes     | `ASSISTANT_SUMMARIZER_MAX_QA_BYTES`     | `76800`       | Maximum byte size for assistant's QA sections (75KB)                    |
| AssistantSummarizerKeepQASections | `ASSISTANT_SUMMARIZER_KEEP_QA_SECTIONS` | `3`           | Number of recent QA sections to preserve without summarization          |

### Usage Details

The assistant settings are used to configure the behavior of the AI assistant and its context management:

- **AssistantUseAgents**: Controls the default state of the "Use Agents" toggle when creating new assistants in the UI:
  ```go
  // This setting affects the initial state when creating assistants
  // Users can always override this by toggling the "Use Agents" button in the UI
  ```
  - `false` (default): New assistants are created with agent delegation disabled by default
  - `true`: New assistants are created with agent delegation enabled by default

- **Assistant Summarizer Settings**: These provide dedicated summarization configuration for assistant instances, typically allowing for more memory retention compared to the global settings:
  ```go
  // Assistant summarizer configuration provides more context retention
  // compared to global settings, preserving more recent conversation history
  // while still ensuring efficient token usage
  ```

The assistant summarizer configuration is designed to provide more memory for context retention compared to the global settings, preserving more recent conversation history while still ensuring efficient token usage.

### Recommended Assistant Settings for Different Use Cases

1. **Standard Assistant Conversations**
   ```
   AssistantUseAgents: false
   AssistantSummarizerPreserveLast: true
   AssistantSummarizerLastSecBytes: 76800 (75KB)
   AssistantSummarizerMaxBPBytes: 16384 (16KB)
   AssistantSummarizerMaxQASections: 7
   AssistantSummarizerMaxQABytes: 76800 (75KB)
   AssistantSummarizerKeepQASections: 3
   ```
   The default settings provide a balance between context retention and performance for typical assistant interactions.

2. **Multi-Agent Assistant Workflows**
   ```
   AssistantUseAgents: true
   AssistantSummarizerPreserveLast: true
   AssistantSummarizerLastSecBytes: 102400 (100KB)
   AssistantSummarizerMaxBPBytes: 32768 (32KB)
   AssistantSummarizerMaxQASections: 10
   AssistantSummarizerMaxQABytes: 102400 (100KB)
   AssistantSummarizerKeepQASections: 5
   ```
   Enhanced settings for complex workflows that benefit from agent delegation with increased context preservation.

3. **Resource-Constrained Assistant**
   ```
   AssistantUseAgents: false
   AssistantSummarizerPreserveLast: true
   AssistantSummarizerLastSecBytes: 51200 (50KB)
   AssistantSummarizerMaxBPBytes: 16384 (16KB)
   AssistantSummarizerMaxQASections: 5
   AssistantSummarizerMaxQABytes: 51200 (50KB)
   AssistantSummarizerKeepQASections: 2
   ```
   More conservative settings for environments with limited resources or smaller context models.

## Functions Configuration

These settings control which tools are available to AI agents and allow adding custom external functions. The Functions API enables fine-grained control over agent capabilities by selectively disabling built-in tools or extending functionality with custom integrations.

For provider-neutral OSINT enrichment ideas that could be exposed through external functions, see [OSINT Integration Scenarios for PentAGI Agents](../../examples/proposals/osint-integration-scenarios.md).

| Field     | Type                 | Description                                                     |
| --------- | -------------------- | --------------------------------------------------------------- |
| token     | string (optional)    | API token for authenticating external function calls            |
| disabled  | DisableFunction[]    | List of built-in functions to disable for specific agent types  |
| functions | ExternalFunction[]   | List of custom external functions to add to agent capabilities  |

### DisableFunction Structure

Allows disabling specific built-in functions for certain agent contexts, providing security and control over agent capabilities.

| Field   | Type     | Description                                                                    |
| ------- | -------- | ------------------------------------------------------------------------------ |
| name    | string   | Name of the built-in function to disable (e.g., `terminal`, `browser`, `file`) |
| context | string[] | Agent contexts where the function should be disabled (optional)                |

**Available Agent Contexts**: `agent`, `adviser`, `coder`, `searcher`, `generator`, `memorist`, `enricher`, `reporter`, `assistant`

When `context` is empty or omitted, the function is disabled for all agents.

### ExternalFunction Structure

Allows adding custom external functions that agents can call via HTTP endpoints, enabling integration with external tools and services.

| Field   | Type          | Description                                                        |
| ------- | ------------- | ------------------------------------------------------------------ |
| name    | string        | Name of the custom function (must be unique)                       |
| url     | string        | HTTP(S) URL endpoint for the function                              |
| timeout | int64         | Timeout in seconds for function execution (default: 60)            |
| context | string[]      | Agent contexts where the function is available (optional)          |
| schema  | Schema object | JSON schema defining function parameters and description (OpenAI format) |

**Available Agent Contexts**: Same as DisableFunction (`agent`, `adviser`, `coder`, `searcher`, `generator`, `memorist`, `enricher`, `reporter`, `assistant`)

When `context` is empty or omitted, the function is available to all agents.

### Usage Details

The Functions configuration is typically provided when creating a flow through the API:

```go
// Example from pkg/tools/tools.go
type Functions struct {
    Token    *string            `json:"token,omitempty"`
    Disabled []DisableFunction  `json:"disabled,omitempty"`
    Function []ExternalFunction `json:"functions,omitempty"`
}
```

These settings are used in `pkg/tools/tools.go` to configure available tools for each agent type:

- **Token**: Used for authenticating requests to external function endpoints:
  ```go
  // The token is passed in the Authorization header when calling external functions
  req.Header.Set("Authorization", "Bearer " + *functions.Token)
  ```

- **Disabled**: Filters out built-in functions for specific agent contexts:
  ```go
  // Check if function is disabled for current agent context
  for _, disabled := range functions.Disabled {
      if disabled.Name == functionName && 
         (len(disabled.Context) == 0 || contains(disabled.Context, agentType)) {
          // Skip this function
      }
  }
  ```

- **Functions**: Adds custom external functions to agent capabilities:
  ```go
  // Register external functions as available tools
  for _, externalFunc := range functions.Function {
      if len(externalFunc.Context) == 0 || contains(externalFunc.Context, agentType) {
          definitions = append(definitions, externalFunc.Schema)
          handlers[externalFunc.Name] = createExternalHandler(externalFunc)
      }
  }
  ```

### Example Configuration

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Security Considerations

- **Token Security**: Store the `token` value securely and use HTTPS endpoints for external functions
- **Function Validation**: External functions should validate all inputs and return structured error messages
- **Timeout Configuration**: Set appropriate timeouts to prevent long-running operations from blocking agents
- **Context Restriction**: Use the `context` field to limit which agents can access sensitive functions
- **URL Validation**: Ensure external function URLs are trusted and properly secured

### Built-in Functions Reference

Common built-in functions that can be disabled:

- `terminal` - Execute shell commands in containers
- `file` - Read and write files in containers
- `browser` - Browse websites and take screenshots
- `search_in_memory` - Search vector memory store
- `search_guide` - Search knowledge guides
- `search_answer` - Search for answers
- `search_code` - Search code repositories
- `store_guide` - Store knowledge guides
- `store_answer` - Store answers
- `store_code` - Store code snippets
- `google` - Google Search
- `duckduckgo` - DuckDuckGo Search
- `tavily` - Tavily Search
- `firecrawl` - Firecrawl Search
- `traversaal` - Traversaal Search
- `perplexity` - Perplexity Search
- `searxng` - SearXNG Search
- `sploitus` - Sploitus Exploit Search
- `graphiti_search` - Graphiti Knowledge Graph Search

The specific functions available depend on the agent type and system configuration.

## Search Engine Settings

These settings control the integration with various search engines used for web search capabilities, providing AI agents with up-to-date information from the internet.

### DuckDuckGo Search

| Option               | Environment Variable    | Default Value | Description                                                                |
| -------------------- | ----------------------- | ------------- | -------------------------------------------------------------------------- |
| DuckDuckGoEnabled    | `DUCKDUCKGO_ENABLED`    | `true`        | Enable or disable DuckDuckGo Search engine                                 |
| DuckDuckGoRegion     | `DUCKDUCKGO_REGION`     | *(none)*      | Region code for search results (e.g., `us-en`, `uk-en`, `cn-zh`)           |
| DuckDuckGoSafeSearch | `DUCKDUCKGO_SAFESEARCH` | *(none)*      | Safe search filter (`off`, `moderate`, `strict`)                           |
| DuckDuckGoTimeRange  | `DUCKDUCKGO_TIME_RANGE` | *(none)*      | Time range for search results (`d`: day, `w`: week, `m`: month, `y`: year) |

### Sploitus Search

| Option          | Environment Variable | Default Value | Description                                                 |
| --------------- | -------------------- | ------------- | ----------------------------------------------------------- |
| SploitusEnabled | `SPLOITUS_ENABLED`   | `true`        | Enable or disable Sploitus exploit and vulnerability search |

### Google Search

| Option       | Environment Variable | Default Value | Description                                              |
| ------------ | -------------------- | ------------- | -------------------------------------------------------- |
| GoogleAPIKey | `GOOGLE_API_KEY`     | *(none)*      | API key for Google Search                                |
| GoogleCXKey  | `GOOGLE_CX_KEY`      | *(none)*      | Custom Search Engine ID for Google Search                |
| GoogleLRKey  | `GOOGLE_LR_KEY`      | `lang_en`     | Language restriction for Google Search (e.g., `lang_en`) |

### Traversaal Search

| Option           | Environment Variable | Default Value | Description                          |
| ---------------- | -------------------- | ------------- | ------------------------------------ |
| TraversaalAPIKey | `TRAVERSAAL_API_KEY` | *(none)*      | API key for Traversaal search engine |

### Tavily Search

| Option       | Environment Variable | Default Value | Description                      |
| ------------ | -------------------- | ------------- | -------------------------------- |
| TavilyAPIKey | `TAVILY_API_KEY`     | *(none)*      | API key for Tavily search engine |

### Firecrawl Search

| Option          | Environment Variable | Default Value               | Description                                                                    |
| --------------- | -------------------- | --------------------------- | ------------------------------------------------------------------------------ |
| FirecrawlAPIKey | `FIRECRAWL_API_KEY`  | *(none)*                    | API key for Firecrawl search engine                                            |
| FirecrawlAPIURL | `FIRECRAWL_API_URL`  | `https://api.firecrawl.dev` | Base URL for the Firecrawl API (override to point at a self-hosted deployment) |

### Perplexity Search

| Option                | Environment Variable      | Default Value | Description                                                  |
| --------------------- | ------------------------- | ------------- | ------------------------------------------------------------ |
| PerplexityAPIKey      | `PERPLEXITY_API_KEY`      | *(none)*      | API key for Perplexity search engine                         |
| PerplexityModel       | `PERPLEXITY_MODEL`        | `sonar`       | Model sent to the Perplexity chat/completions API (`sonar`, `sonar-pro`, …); passed through verbatim |
| PerplexityContextSize | `PERPLEXITY_CONTEXT_SIZE` | `low`         | Context size for Perplexity search (`low`, `medium`, `high`) |

### Searxng Search

| Option            | Environment Variable | Default Value | Description                                                         |
| ----------------- | -------------------- | ------------- | ------------------------------------------------------------------- |
| SearxngURL        | `SEARXNG_URL`        | *(none)*      | Base URL for Searxng meta search engine instance                    |
| SearxngCategories | `SEARXNG_CATEGORIES` | `general`     | Search categories to use (e.g., `general`, `news`, `web`)           |
| SearxngLanguage   | `SEARXNG_LANGUAGE`   | *(none)*      | Language filter for search results (e.g., `en`, `ch`)               |
| SearxngSafeSearch | `SEARXNG_SAFESEARCH` | `0`           | Safe search filter level (`0` = none, `1` = moderate, `2` = strict) |
| SearxngTimeRange  | `SEARXNG_TIME_RANGE` | *(none)*      | Time range filter (e.g., `day`, `month`, `year`)                    |
| SearxngTimeout    | `SEARXNG_TIMEOUT`    | *(none)*      | Request timeout in seconds for Searxng API calls                    |

### Internal Analytics Engine

An optional, opt-in fallback engine for the `web_search` tool's analytic modes (`answer`/`research`). When enabled, it discovers links via the first available link engine, fetches each page's main-content markdown through the browser scraper, and asks the summarizer to synthesize a query-focused answer — without a paid analytic API. Off by default because per-page scraping and summarization can cost more than a purpose-built third-party analytic call. Requires a configured scraper and at least one available link engine (e.g. DuckDuckGo, Google).

| Option                        | Environment Variable                 | Default Value | Description                                                   |
| ----------------------------- | ------------------------------------- | -------------- | -------------------------------------------------------------- |
| WebSearchInternalEnabled      | `WEB_SEARCH_INTERNAL_ENABLED`         | `false`        | Enable or disable the internal browser-analytics engine        |
| WebSearchInternalMaxSites     | `WEB_SEARCH_INTERNAL_MAX_SITES`       | `5`            | Maximum number of pages to fetch and summarize per query       |
| WebSearchInternalMaxSiteBytes | `WEB_SEARCH_INTERNAL_MAX_SITE_BYTES`  | `10240`        | Maximum markdown bytes read from each page before truncation   |

### Usage Details

The search engine settings are used in `pkg/tools/tools.go` to configure various search providers that AI agents can use:

```
/* Detailed source-code truncated for AI context efficiency. */
```

These settings enable:
- Access to multiple search engines for diverse information sources
- Configuration of search parameters like language, context size, and time range
- Integration of search capabilities into the AI agent's toolset
- Web information gathering with different search strategies
- Security research through Sploitus, providing access to exploit databases and CVE information
- Meta-search capabilities through Searxng, aggregating results from multiple search engines

Having multiple search engine options ensures redundancy and provides different search algorithms for varied information needs. Sploitus is specifically designed for security research, providing comprehensive exploit and vulnerability information essential for penetration testing. Searxng is particularly useful as it provides aggregated results from multiple search engines while offering enhanced privacy and customization options.

## Network and Proxy Settings

These settings control HTTP proxy, SSL configuration, and network timeouts for outbound connections, which are important for network security and access control.

| Option              | Environment Variable    | Default Value | Description                                                      |
| ------------------- | ----------------------- | ------------- | ---------------------------------------------------------------- |
| ProxyURL            | `PROXY_URL`             | *(none)*      | URL for HTTP proxy (e.g., `http://user:pass@proxy:8080`)         |
| ExternalSSLCAPath   | `EXTERNAL_SSL_CA_PATH`  | *(none)*      | Path to trusted CA certificate for external LLM SSL connections  |
| ExternalSSLInsecure | `EXTERNAL_SSL_INSECURE` | `false`       | Skip SSL certificate verification for external connections       |
| HTTPClientTimeout   | `HTTP_CLIENT_TIMEOUT`   | `600`         | Timeout in seconds for external API calls (0 = no timeout)       |

### Usage Details

The proxy settings are used in various places to configure HTTP clients for external API calls:

```go
// Example from openai.go, anthropic.go, and other provider files
if cfg.ProxyURL != "" {
    httpClient = &http.Client{
        Transport: &http.Transport{
            Proxy: func(req *http.Request) (*url.URL, error) {
                return url.Parse(cfg.ProxyURL)
            },
        },
    }
}
```

The proxy URL is also passed to various tools that make external requests:

```go
// In tools.go for search tools
googleSearch: &functions.GoogleSearchFunc{
    apiKey:    fte.cfg.GoogleAPIKey,
    cxKey:     fte.cfg.GoogleCXKey,
    lrKey:     fte.cfg.GoogleLRKey,
    proxyURL:  fte.cfg.ProxyURL,
},
```

The proxy setting is essential for:
- Routing the backend's LLM, embedding, search and update-check requests through a controlled proxy (sandbox containers, the scraper and Graphiti do not use it)
- Implementing network-level security policies
- Enabling access to external services from restricted networks
- Monitoring and auditing external API usage

The SSL settings provide additional security configuration:

- **ExternalSSLCAPath**: Specifies a custom CA certificate for validating SSL connections to external services:
  ```go
  // Used in provider initialization to configure custom CA certificates
  if cfg.ExternalSSLCAPath != "" {
      caCert, err := os.ReadFile(cfg.ExternalSSLCAPath)
      // Configure TLS with custom CA
  }
  ```
  This is useful when connecting to LLM providers with self-signed certificates or internal CAs.

- **ExternalSSLInsecure**: Allows skipping SSL certificate verification:
  ```go
  // Used in HTTP client configuration
  if cfg.ExternalSSLInsecure {
      tlsConfig.InsecureSkipVerify = true
  }
  ```
  **Warning**: Only use this in development or trusted environments. Skipping certificate verification exposes connections to man-in-the-middle attacks.

- **HTTPClientTimeout**: Sets the timeout for all external HTTP requests (LLM providers, search engines, etc.):
  ```go
  // Used in pkg/system/utils.go for HTTP client configuration
  timeout := defaultHTTPClientTimeout
  if cfg.HTTPClientTimeout > 0 {
      timeout = time.Duration(cfg.HTTPClientTimeout) * time.Second
  }
  
  httpClient := &http.Client{
      Timeout: timeout,
  }
  ```
  The default value of 600 seconds (10 minutes) is suitable for most LLM API calls, including long-running operations. Setting this to 0 disables the timeout (not recommended in production), while very low values may cause legitimate requests to fail. This setting affects:
  - All LLM provider API calls (OpenAI, Anthropic, Bedrock, etc.)
  - Search engine requests (Google, Tavily, Perplexity, etc.)
  - External tool integrations
  - Embedding generation requests

  Adjust this value based on your network conditions and the complexity of operations being performed.

## Graphiti Knowledge Graph Settings

Graphiti is an optional beta integration. This section documents the PentAGI-side configuration boundary and lifecycle; the operator-facing deployment, provider, ingestion, extraction, and Neo4j settings are maintained in [README.md — Knowledge Graph Integration](../../README.md#knowledge-graph-integration-graphiti).

### PentAGI Configuration Boundary

`pkg/config.Config` intentionally owns only the settings required by the PentAGI process:

| Field | Environment Variable | Default | Responsibility |
| --- | --- | --- | --- |
| `GraphitiEnabled` | `GRAPHITI_ENABLED` | `false` | Operator intent to enable the integration |
| `GraphitiURL` | `GRAPHITI_URL` | *(empty)* | Graphiti API base URL; embedded deployments use `http://graphiti:8000` |
| `GraphitiTimeout` | `GRAPHITI_TIMEOUT` | `30` seconds | Timeout used for Graphiti client requests and storage contexts |

All other variables with `GRAPHITI_*`, `NEO4J_*`, provider, embedding, ingest, extraction, anchor, or logging names configure `docker-compose-graphiti.yml` or the Graphiti process. They are not parsed into the Go `Config` struct. Keep that boundary explicit when adding settings: a value needed by PentAGI belongs in `pkg/config/config.go`; a value consumed only by the sidecar belongs in `.env.example` plus the compose mapping.

The provider controller applies an additional URL guard:

```go
graphitiClient, err := graphiti.NewClient(
    cfg.GraphitiURL,
    time.Duration(cfg.GraphitiTimeout)*time.Second,
    cfg.GraphitiEnabled && cfg.GraphitiURL != "",
)
```

Consequently, `GRAPHITI_ENABLED=true` with an empty URL still creates a disabled wrapper.

### Client Lifecycle and Failure Behavior

`pkg/graphiti/client.go` wraps `graphiti-go-client` and performs a synchronous health check during provider initialization. It makes three attempts with a two-second backoff. A final failure is returned to `pkg/providers/providers.go`, which logs a warning and substitutes a disabled client so the rest of PentAGI can start.

The disabled wrapper has deliberate asymmetric behavior:

- `AddMessages` is a no-op, allowing normal flow execution to continue.
- Search methods return `graphiti is not enabled`; disabled clients are normally excluded from agent tool registration and prompts through `IsEnabled`.
- Storage calls use `GraphitiTimeout`, log failures with their group ID, and return the error to the caller without making Graphiti a required persistence layer.

Do not change initialization failures into fatal PentAGI startup errors without treating that as a deployment-contract change.

### Data Flow, Search, and Tenancy

Provider performers render `backend/pkg/templates/graphiti/*.tmpl` and enqueue agent responses and tool executions with observation metadata. The sidecar applies ingest policy and performs asynchronous extraction; successful submission does not imply that the graph is immediately searchable.

Enabled agents receive the `graphiti_search` tool. Its seven modes map to temporal-window, entity-relationship, diverse-result, episode-context, successful-tool, recent-context, and entity-by-label client methods. Transport and server failures are handled as degradable tool failures where possible, while malformed arguments and validation errors remain hard errors so the LLM can repair its call.

Every request is partitioned by `Config.GroupID(flowID)`:

- without `TENANT_ID`: `flow-<id>`;
- with `TENANT_ID`: `<tenant>-flow-<id>`.

`Config.ParseGroupID` rejects identifiers belonging to another tenant. Keep `GroupID` and `ParseGroupID` exact inverses, and never accept a client-supplied namespace in place of a server-derived group ID. The stock deployment also sets `GRAPHITI_SEARCH_SCOPE=flowid`; changing it to global search bypasses this retrieval boundary inside Graphiti and is unsafe for shared deployments.

### Deployment Ownership

The bundled stack is separate from `docker-compose.yml`. The installer distinguishes disabled, external, and embedded deployments using `GRAPHITI_ENABLED` and `GRAPHITI_URL`; `http://graphiti:8000` is the embedded endpoint. It starts the Graphiti stack before PentAGI so the startup health check can succeed.

Installer assets include `docker-compose-graphiti.yml` and the `graphiti` preset directory sourced from `examples/graphiti`. The target directory is checked, repaired, and removed with the Graphiti stack. Compose mounts it through `GRAPHITI_CONFIG_PATH` and `GRAPHITI_CONFIG_DIR`; the fallback target `configs` prevents a newer compose file paired with an older `.env` from masking the image's built-in `llm_configs` with an empty host directory.

Models and call parameters belong to the provider YAML presets, selected by `GRAPHITI_LLM_CLIENT_TYPE`; they are not fields in the PentAGI Go configuration. `GRAPHITI_MODEL_NAME` is obsolete. Refer operators to the README instead of duplicating the sidecar's tuning reference here.

## Agent Supervision Settings

These settings control the agent supervision system, including execution monitoring and tool call limits for different agent types.

| Option                         | Environment Variable                | Default Value | Description                                                            |
| ------------------------------ | ----------------------------------- | ------------- | ---------------------------------------------------------------------- |
| ExecutionMonitorEnabled        | `EXECUTION_MONITOR_ENABLED`         | `false`       | Enable automatic execution monitoring (mentor/adviser supervision)     |
| ExecutionMonitorSameToolLimit  | `EXECUTION_MONITOR_SAME_TOOL_LIMIT` | `5`           | Threshold for consecutive identical tool calls before mentor review    |
| ExecutionMonitorTotalToolLimit | `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT`| `10`          | Threshold for total tool calls before mentor review                    |
| MaxGeneralAgentToolCalls       | `MAX_GENERAL_AGENT_TOOL_CALLS`      | `100`         | Maximum tool calls for general agents (Assistant, Primary, Pentester, Coder, Installer) |
| MaxLimitedAgentToolCalls       | `MAX_LIMITED_AGENT_TOOL_CALLS`      | `20`          | Maximum tool calls for limited agents (Searcher, Enricher, etc.)       |
| AgentPlanningStepEnabled       | `AGENT_PLANNING_STEP_ENABLED`       | `false`       | Enable automatic task planning for specialist agents                   |

### Usage Details

The agent supervision settings are used in `pkg/providers/providers.go` and `pkg/providers/performer.go` to configure supervision mechanisms:

- **ExecutionMonitorEnabled**: Controls whether execution monitoring (mentor) is active:
  ```go
  buildMonitor: func() *executionMonitor {
      return &executionMonitor{
          enabled:        pc.cfg.ExecutionMonitorEnabled,
          sameThreshold:  pc.cfg.ExecutionMonitorSameToolLimit,
          totalThreshold: pc.cfg.ExecutionMonitorTotalToolLimit,
      }
  }
  ```

- **ExecutionMonitorSameToolLimit**: Sets the threshold for identical consecutive tool calls:
  ```go
  // In executionMonitor.shouldInvokeMentor
  if emd.sameToolCount >= emd.sameThreshold {
      // Invoke mentor (adviser agent) for execution review
  }
  ```
  When an agent calls the same tool this many times consecutively, the execution monitor automatically invokes the mentor (adviser agent) to analyze progress and provide guidance.

- **ExecutionMonitorTotalToolLimit**: Sets the threshold for total tool calls:
  ```go
  // In executionMonitor.shouldInvokeMentor
  if emd.totalCallCount >= emd.totalThreshold {
      // Invoke mentor (adviser agent) for execution review
  }
  ```
  When an agent makes this many total tool calls since the last mentor review, the execution monitor automatically invokes the mentor to prevent inefficient loops and provide strategic guidance.

- **MaxGeneralAgentToolCalls**: Maximum iterations for general-purpose agents with full capabilities:
  ```go
  // In performAgentChain
  switch optAgentType {
  case pconfig.OptionsTypeAssistant, pconfig.OptionsTypePrimaryAgent,
      pconfig.OptionsTypePentester, pconfig.OptionsTypeCoder, pconfig.OptionsTypeInstaller:
      if fp.maxGACallsLimit <= 0 {
          maxCallsLimit = maxGeneralAgentChainIterations // fallback: 100
      } else {
          maxCallsLimit = max(fp.maxGACallsLimit, maxAgentShutdownIterations*2)
      }
  }
  ```
  General agents (Assistant, Primary Agent, Pentester, Coder, Installer) are designed for complex, multi-step workflows and have a higher tool call limit to complete sophisticated tasks.

- **MaxLimitedAgentToolCalls**: Maximum iterations for specialized, limited-scope agents:
  ```go
  // In performAgentChain
  default:
      if fp.maxLACallsLimit <= 0 {
          maxCallsLimit = maxLimitedAgentChainIterations // fallback: 20
      } else {
          maxCallsLimit = max(fp.maxLACallsLimit, maxAgentShutdownIterations*2)
      }
  }
  ```
  Limited agents (Searcher, Enricher, Memorist, Generator, Reporter, Adviser, Reflector, Planner) are designed for focused, specific tasks and have a lower tool call limit to ensure efficient execution.

- **AgentPlanningStepEnabled**: Controls automatic task planning for specialist agents:
  ```go
  // In flowProvider initialization
  planning: pc.cfg.AgentPlanningStepEnabled
  
  // Used when invoking specialist agents
  if fp.planning {
      // Generate execution plan via planner before specialist execution
  }
  ```
  When enabled, the planner (adviser in planning mode) generates a structured 3-7 step execution plan before specialist agents (Pentester, Coder, Installer) begin their work, improving task completion rates and preventing scope creep.

These settings enable:
- **Automatic supervision**: Mentor reviews execution patterns to detect loops and inefficiencies
- **Graceful termination**: Reflector guides agents to proper completion when approaching limits
- **Differentiated capabilities**: General agents have more autonomy for complex workflows
- **Efficient execution**: Limited agents stay focused on their specific scope
- **Strategic planning**: Automatic task decomposition for better execution quality

### Supervision System Integration

The supervision settings work together as a comprehensive system:

1. **Execution Monitoring** (via ExecutionMonitor settings):
   - Detects repetitive patterns (same tool called N times)
   - Detects excessive exploration (total tools called N times)
   - Automatically invokes mentor for guidance and correction
   - Resets counters after mentor review

2. **Tool Call Limits** (via MaxGeneralAgentToolCalls and MaxLimitedAgentToolCalls):
   - Prevents runaway executions with hard limits
   - Invokes reflector for graceful termination near limit
   - Different limits for different agent capabilities
   - Ensures system stability and resource efficiency

3. **Task Planning** (via AgentPlanningStepEnabled):
   - Generates structured execution plans before specialist work
   - Prevents scope creep and maintains focus
   - Improves success rates for complex tasks
   - Provides clear verification points

### Recommended Settings

1. **Production Environment**:
   ```
   ExecutionMonitorEnabled: false
   ExecutionMonitorSameToolLimit: 5
   ExecutionMonitorTotalToolLimit: 10
   MaxGeneralAgentToolCalls: 100
   MaxLimitedAgentToolCalls: 20
   AgentPlanningStepEnabled: false
   ```
   Default settings provide stable execution without beta features.

2. **High-Complexity Workflows**:
   ```
   ExecutionMonitorEnabled: false
   ExecutionMonitorSameToolLimit: 7
   ExecutionMonitorTotalToolLimit: 15
   MaxGeneralAgentToolCalls: 150
   MaxLimitedAgentToolCalls: 30
   AgentPlanningStepEnabled: false
   ```
   Increased limits for tasks requiring extensive exploration and iteration.

3. **Resource-Constrained Environment**:
   ```
   ExecutionMonitorEnabled: false
   ExecutionMonitorSameToolLimit: 3
   ExecutionMonitorTotalToolLimit: 7
   MaxGeneralAgentToolCalls: 50
   MaxLimitedAgentToolCalls: 15
   AgentPlanningStepEnabled: false
   ```
   Tighter limits to reduce resource usage.

4. **Debugging Mode**:
   ```
   ExecutionMonitorEnabled: false
   MaxGeneralAgentToolCalls: 200
   MaxLimitedAgentToolCalls: 50
   AgentPlanningStepEnabled: false
   ```
   Disabled supervision for debugging to observe natural agent behavior.

## Observability Settings

These settings control the observability and monitoring capabilities, including telemetry and trace collection for system performance and debugging.

### Telemetry

| Option            | Environment Variable | Default Value | Description                                |
| ----------------- | -------------------- | ------------- | ------------------------------------------ |
| TelemetryEndpoint | `OTEL_HOST`          | *(none)*      | Endpoint for OpenTelemetry data collection |

### Langfuse

| Option            | Environment Variable  | Default Value | Description                 |
| ----------------- | --------------------- | ------------- | --------------------------- |
| LangfuseBaseURL   | `LANGFUSE_BASE_URL`   | *(none)*      | Base URL for Langfuse API   |
| LangfuseProjectID | `LANGFUSE_PROJECT_ID` | *(none)*      | Project ID for Langfuse     |
| LangfusePublicKey | `LANGFUSE_PUBLIC_KEY` | *(none)*      | Public key for Langfuse API |
| LangfuseSecretKey | `LANGFUSE_SECRET_KEY` | *(none)*      | Secret key for Langfuse API |

### Usage Details

The observability settings are used in `main.go` and the observability package to initialize monitoring systems:

- **Telemetry Configuration**: Sets up OpenTelemetry for metrics, logs, and traces:
  ```go
  // Check if telemetry is configured
  if cfg.TelemetryEndpoint == "" {
      return nil, ErrNotConfigured
  }

  // Create telemetry client with endpoint
  otelclient, err := obs.NewTelemetryClient(ctx, cfg)
  ```

- **Langfuse Configuration**: Configures Langfuse for LLM operation monitoring:
  ```go
  // Check if Langfuse is configured
  if cfg.LangfuseBaseURL == "" {
      return nil, ErrNotConfigured
  }

  // Configure Langfuse client
  langfuse.WithBaseURL(cfg.LangfuseBaseURL),
  langfuse.WithPublicKey(cfg.LangfusePublicKey),
  langfuse.WithSecretKey(cfg.LangfuseSecretKey),
  langfuse.WithProjectID(cfg.LangfuseProjectID),
  ```

- **Integration in Application**: Used in `main.go` to initialize observability:
  ```go
  lfclient, err := obs.NewLangfuseClient(ctx, cfg)
  if err != nil && !errors.Is(err, obs.ErrNotConfigured) {
      log.Fatalf("Unable to create langfuse client: %v\n", err)
  }

  otelclient, err := obs.NewTelemetryClient(ctx, cfg)
  if err != nil && !errors.Is(err, obs.ErrNotConfigured) {
      log.Fatalf("Unable to create telemetry client: %v\n", err)
  }

  obs.InitObserver(ctx, lfclient, otelclient, []logrus.Level{
      logrus.DebugLevel,
      logrus.InfoLevel,
      logrus.WarnLevel,
      logrus.ErrorLevel,
  })
  ```

These settings enable:
- Comprehensive monitoring of system performance
- LLM-specific metrics collection via Langfuse
- Tracing of requests through the system
- Centralized logging for troubleshooting
- Performance optimization based on collected metrics

---

### Backend Docs Controller (`backend/docs/controller.md`)

# Controller Package Documentation

## Table of Contents

- [Controller Package Documentation](#controller-package-documentation)
  - [Table of Contents](#table-of-contents)
  - [Overview and Role in the System](#overview-and-role-in-the-system)
    - [Key Responsibilities](#key-responsibilities)
    - [Architectural Integration](#architectural-integration)
      - [High-Level Architecture](#high-level-architecture)
    - [Role in the System](#role-in-the-system)
  - [Core Concepts and Main Interfaces](#core-concepts-and-main-interfaces)
    - [Main Interfaces and Their Hierarchy](#main-interfaces-and-their-hierarchy)
      - [Flow Management](#flow-management)
      - [Assistant Management](#assistant-management)
      - [Task and Subtask Management](#task-and-subtask-management)
      - [Log Management](#log-management)
        - [Agent Logs](#agent-logs)
        - [Assistant Logs](#assistant-logs)
        - [Message Logs](#message-logs)
        - [Search Logs](#search-logs)
        - [Terminal Logs](#terminal-logs)
        - [Vector Store Logs](#vector-store-logs)
        - [Screenshots](#screenshots)
      - [Supporting Types and Constants](#supporting-types-and-constants)
    - [Interface Hierarchy Diagram](#interface-hierarchy-diagram)
  - [Entity Lifecycle and State Management](#entity-lifecycle-and-state-management)
    - [Flow Lifecycle](#flow-lifecycle)
      - [States](#states)
      - [State Transitions](#state-transitions)
      - [State Management](#state-management)
      - [State Diagram](#state-diagram)
    - [Assistant Lifecycle](#assistant-lifecycle)
      - [States](#states-1)
      - [State Transitions](#state-transitions-1)
      - [State Management](#state-management-1)
    - [Task Lifecycle](#task-lifecycle)
      - [States](#states-2)
      - [State Transitions](#state-transitions-2)
      - [State Management](#state-management-2)
    - [Subtask Lifecycle](#subtask-lifecycle)
      - [States](#states-3)
      - [State Transitions](#state-transitions-3)
      - [State Management](#state-management-3)
    - [Error Handling and Event Publication](#error-handling-and-event-publication)
    - [Example: State Transition (Task)](#example-state-transition-task)
  - [Log Management and Event Publication](#log-management-and-event-publication)
    - [Log Types and Their Roles](#log-types-and-their-roles)
    - [Log Lifecycle and Operations](#log-lifecycle-and-operations)
      - [Special Features](#special-features)
    - [Event Publication](#event-publication)
    - [Log Worker Creation and Management](#log-worker-creation-and-management)
  - [Integration with Providers, Tools, and Subscriptions](#integration-with-providers-tools-and-subscriptions)
    - [Providers Integration](#providers-integration)
    - [Tools Integration](#tools-integration)
    - [Subscriptions and Event Publication](#subscriptions-and-event-publication)
    - [Dependency Injection and Context Propagation](#dependency-injection-and-context-propagation)
    - [Integration Flow Diagrams](#integration-flow-diagrams)
  - [Internal Structure and Concurrency Model](#internal-structure-and-concurrency-model)
    - [Mutex Usage and Thread Safety](#mutex-usage-and-thread-safety)
      - [Example: Controller Mutex Usage](#example-controller-mutex-usage)
      - [Example: Worker Mutex Usage](#example-worker-mutex-usage)
    - [State Storage and Management](#state-storage-and-management)
    - [Worker Goroutines and Channels](#worker-goroutines-and-channels)
      - [Assistant Streaming Example](#assistant-streaming-example)
  - [Extensibility, Error Handling, and Best Practices](#extensibility-error-handling-and-best-practices)
    - [Extensibility](#extensibility)
      - [Adding New Log Types](#adding-new-log-types)
    - [Error Handling](#error-handling)
      - [Error Handling Example](#error-handling-example)
    - [Best Practices](#best-practices)

## Overview and Role in the System

The `controller` package is a central part of the backend architecture, responsible for orchestrating the lifecycle and logic of flows, assistants, tasks, subtasks, and various types of logs in the system. It acts as a high-level service layer, mediating between the database, providers, tools, and the event subscription system.

### Key Responsibilities
- Managing the creation, loading, execution, and termination of flows and their associated entities (assistants, tasks, subtasks).
- Providing thread-safe controllers and workers for each logical entity (flow, assistant, task, subtask, logs, screenshots).
- Integrating with the database layer for persistent storage and retrieval of all entities.
- Interfacing with provider abstractions for LLMs, tools, and execution environments.
- Publishing events to the subscription system for real-time updates.
- Ensuring correct state transitions and error handling for all managed entities.
- Supporting both autonomous pentesting flows and interactive assistant conversations.

### Architectural Integration

The `controller` package interacts with the following key packages:
- `database`: For all persistent storage and retrieval operations.
- `providers` and `tools`: For LLM, tool execution, and agent chain management.
- `graph/subscriptions`: For publishing real-time events about entity changes.
- `observability/langfuse`: For tracing and logging of operations.
- `config`, `docker`, and `templates`: For configuration, container management, and prompt templating.

#### High-Level Architecture

The following diagram reflects the actual architecture of the `controller` package, showing all main controllers, their relationships, and integration points with other system components.

```mermaid
flowchart TD
    subgraph Controller
        FC[FlowController]
        FW[FlowWorker]
        AW[AssistantWorker]
        TC[TaskController]
        TW[TaskWorker]
        STC[SubtaskController]
        STW[SubtaskWorker]

        subgraph LogControllers[Log Controllers]
            ALC[AgentLogController]
            ASLC[AssistantLogController]
            MLC[MsgLogController]
            SLC[SearchLogController]
            TLC[TermLogController]
            VSLC[VectorStoreLogController]
            SC[ScreenshotController]
        end
    end

    DB[(Database)]
    LLMProviders[[ProviderController]]
    ToolsExecutor[[FlowToolsExecutor]]
    GraphQLSubscriptions((Subscriptions))
    Obs>Observability]
    Docker[[Docker]]

    FC -- manages --> FW
    FW -- manages --> AW
    FW -- manages --> TC
    TC -- manages --> TW
    TW -- manages --> STC
    STC -- manages --> STW

    FC -- uses --> LogControllers
    FW -- uses --> LogControllers
    AW -- uses --> LogControllers

    FC -- uses --> DB
    FC -- uses --> LLMProviders
    FC -- uses --> ToolsExecutor
    FC -- uses --> GraphQLSubscriptions
    FC -- uses --> Obs
    FC -- uses --> Docker
```

### Role in the System

The `controller` package is the main orchestrator for all user and system-initiated operations related to flows and their sub-entities. It ensures that all business logic, state transitions, and side effects (such as event publication and logging) are handled consistently and safely. It supports two main operational modes:

1. **Autonomous Pentesting Mode**: Complete flows with automated task generation and execution
2. **Interactive Assistant Mode**: Conversational AI assistants that can optionally use agents and tools

## Core Concepts and Main Interfaces

The `controller` package is built around a set of core concepts and interfaces that encapsulate the logic for managing flows, assistants, tasks, subtasks, and various types of logs. Each logical entity is represented by a controller (managing multiple entities) and a worker (managing a single entity instance).

### Main Interfaces and Their Hierarchy

#### Flow Management

```go
// flows.go
type FlowController interface {
    CreateFlow(
        ctx context.Context,
        userID int64,
        input string,
        prvtype provider.ProviderType,
        functions *tools.Functions,
    ) (FlowWorker, error)
    CreateAssistant(
        ctx context.Context,
        userID int64,
        flowID int64,
        input string,
        useAgents bool,
        prvtype provider.ProviderType,
        functions *tools.Functions,
    ) (AssistantWorker, error)
    LoadFlows(ctx context.Context) error
    ListFlows(ctx context.Context) []FlowWorker
    GetFlow(ctx context.Context, flowID int64) (FlowWorker, error)
    StopFlow(ctx context.Context, flowID int64) error
    FinishFlow(ctx context.Context, flowID int64) error
}

// flow.go
type FlowWorker interface {
    GetFlowID() int64
    GetUserID() int64
    GetTitle() string
    GetContext() *FlowContext
    GetStatus(ctx context.Context) (database.FlowStatus, error)
    SetStatus(ctx context.Context, status database.FlowStatus) error
    AddAssistant(ctx context.Context, aw AssistantWorker) error
    GetAssistant(ctx context.Context, assistantID int64) (AssistantWorker, error)
    DeleteAssistant(ctx context.Context, assistantID int64) error
    ListAssistants(ctx context.Context) []AssistantWorker
    ListTasks(ctx context.Context) []TaskWorker
    PutInput(ctx context.Context, input string) error
    Finish(ctx context.Context) error
    Stop(ctx context.Context) error
}
```

#### Assistant Management

```go
// assistant.go
type AssistantWorker interface {
    GetAssistantID() int64
    GetUserID() int64
    GetFlowID() int64
    GetTitle() string
    GetStatus(ctx context.Context) (database.AssistantStatus, error)
    SetStatus(ctx context.Context, status database.AssistantStatus) error
    PutInput(ctx context.Context, input string, useAgents bool) error
    Finish(ctx context.Context) error
    Stop(ctx context.Context) error
}
```

#### Task and Subtask Management

```
/* Detailed source-code truncated for AI context efficiency. */
```

#### Log Management

The system includes seven different types of logs, each with its own controller and worker interfaces:

##### Agent Logs

```go
// alogs.go
type AgentLogController interface {
    NewFlowAgentLog(ctx context.Context, flowID int64, pub subscriptions.FlowPublisher) (FlowAgentLogWorker, error)
    ListFlowsAgentLog(ctx context.Context) ([]FlowAgentLogWorker, error)
    GetFlowAgentLog(ctx context.Context, flowID int64) (FlowAgentLogWorker, error)
}

// alog.go
type FlowAgentLogWorker interface {
    PutLog(
        ctx context.Context,
        initiator database.MsgchainType,
        executor database.MsgchainType,
        task string,
        result string,
        taskID *int64,
        subtaskID *int64,
    ) (int64, error)
    GetLog(ctx context.Context, msgID int64) (database.Agentlog, error)
}
```

##### Assistant Logs

```go
// aslogs.go
type AssistantLogController interface {
    NewFlowAssistantLog(
        ctx context.Context, flowID int64, assistantID int64, pub subscriptions.FlowPublisher,
    ) (FlowAssistantLogWorker, error)
    ListFlowsAssistantLog(ctx context.Context, flowID int64) ([]FlowAssistantLogWorker, error)
    GetFlowAssistantLog(ctx context.Context, flowID int64, assistantID int64) (FlowAssistantLogWorker, error)
}

// aslog.go
type FlowAssistantLogWorker interface {
    PutMsg(
        ctx context.Context,
        msgType database.MsglogType,
        taskID, subtaskID *int64,
        streamID int64,
        thinking, msg string,
    ) (int64, error)
    PutFlowAssistantMsg(
        ctx context.Context,
        msgType database.MsglogType,
        thinking, msg string,
    ) (int64, error)
    PutFlowAssistantMsgResult(
        ctx context.Context,
        msgType database.MsglogType,
        thinking, msg, result string,
        resultFormat database.MsglogResultFormat,
    ) (int64, error)
    StreamFlowAssistantMsg(
        ctx context.Context,
        chunk *providers.StreamMessageChunk,
    ) error
    UpdateMsgResult(
        ctx context.Context,
        msgID, streamID int64,
        result string,
        resultFormat database.MsglogResultFormat,
    ) error
}
```

##### Message Logs

```
/* Detailed source-code truncated for AI context efficiency. */
```

##### Search Logs

```go
// slogs.go
type SearchLogController interface {
    NewFlowSearchLog(ctx context.Context, flowID int64, pub subscriptions.FlowPublisher) (FlowSearchLogWorker, error)
    ListFlowsSearchLog(ctx context.Context) ([]FlowSearchLogWorker, error)
    GetFlowSearchLog(ctx context.Context, flowID int64) (FlowSearchLogWorker, error)
}

// slog.go
type FlowSearchLogWorker interface {
    PutLog(
        ctx context.Context,
        initiator database.MsgchainType,
        executor database.MsgchainType,
        engine database.SearchengineType,
        query string,
        result string,
        taskID *int64,
        subtaskID *int64,
    ) (int64, error)
    GetLog(ctx context.Context, msgID int64) (database.Searchlog, error)
}
```

##### Terminal Logs

```go
// termlogs.go
type TermLogController interface {
    NewFlowTermLog(ctx context.Context, flowID int64, pub subscriptions.FlowPublisher) (FlowTermLogWorker, error)
    ListFlowsTermLog(ctx context.Context) ([]FlowTermLogWorker, error)
    GetFlowTermLog(ctx context.Context, flowID int64) (FlowTermLogWorker, error)
    GetFlowContainers(ctx context.Context, flowID int64) ([]database.Container, error)
}

// termlog.go
type FlowTermLogWorker interface {
    PutMsg(ctx context.Context, msgType database.TermlogType, msg string, containerID int64) (int64, error)
    GetMsg(ctx context.Context, msgID int64) (database.Termlog, error)
    GetContainers(ctx context.Context) ([]database.Container, error)
}
```

##### Vector Store Logs

```go
// vslogs.go
type VectorStoreLogController interface {
    NewFlowVectorStoreLog(ctx context.Context, flowID int64, pub subscriptions.FlowPublisher) (FlowVectorStoreLogWorker, error)
    ListFlowsVectorStoreLog(ctx context.Context) ([]FlowVectorStoreLogWorker, error)
    GetFlowVectorStoreLog(ctx context.Context, flowID int64) (FlowVectorStoreLogWorker, error)
}

// vslog.go
type FlowVectorStoreLogWorker interface {
    PutLog(
        ctx context.Context,
        initiator database.MsgchainType,
        executor database.MsgchainType,
        filter json.RawMessage,
        query string,
        action database.VecstoreActionType,
        result string,
        taskID *int64,
        subtaskID *int64,
    ) (int64, error)
    GetLog(ctx context.Context, msgID int64) (database.Vecstorelog, error)
}
```

##### Screenshots

```go
// screenshots.go
type ScreenshotController interface {
    NewFlowScreenshot(ctx context.Context, flowID int64, pub subscriptions.FlowPublisher) (FlowScreenshotWorker, error)
    ListFlowsScreenshot(ctx context.Context) ([]FlowScreenshotWorker, error)
    GetFlowScreenshot(ctx context.Context, flowID int64) (FlowScreenshotWorker, error)
}

// screenshot.go
type FlowScreenshotWorker interface {
    PutScreenshot(ctx context.Context, name, url string) (int64, error)
    GetScreenshot(ctx context.Context, screenshotID int64) (database.Screenshot, error)
}
```

#### Supporting Types and Constants

```go
// context.go
type FlowContext struct {
    DB database.Querier
    UserID    int64
    FlowID    int64
    FlowTitle string
    Executor  tools.FlowToolsExecutor
    Provider  providers.FlowProvider
    Publisher subscriptions.FlowPublisher
    TermLog    FlowTermLogWorker
    MsgLog     FlowMsgLogWorker
    Screenshot FlowScreenshotWorker
}

type TaskContext struct {
    TaskID    int64
    TaskTitle string
    TaskInput string
    FlowContext
}

type SubtaskContext struct {
    MsgChainID         int64
    SubtaskID          int64
    SubtaskTitle       string
    SubtaskDescription string
    TaskContext
}

// Updater interfaces for status propagation
type FlowUpdater interface {
    SetStatus(ctx context.Context, status database.FlowStatus) error
}

type TaskUpdater interface {
    SetStatus(ctx context.Context, status database.TaskStatus) error
}
```

### Interface Hierarchy Diagram

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Entity Lifecycle and State Management

The `controller` package implements a strict lifecycle and state management system for all major entities: flows, assistants, tasks, and subtasks. Each entity has a well-defined set of states, and transitions are managed through controller and worker methods, with all changes persisted to the database and broadcast via the subscription system.

### Flow Lifecycle

#### States
- `Created` (database.FlowStatusCreated)
- `Running` (database.FlowStatusRunning)
- `Waiting` (database.FlowStatusWaiting)
- `Finished` (database.FlowStatusFinished)
- `Failed` (database.FlowStatusFailed)

#### State Transitions
- Flows are created in the `Created` state.
- When tasks start running, flows transition to `Running`.
- If tasks are waiting for input, flows move to `Waiting`.
- On completion of all tasks, flows become `Finished`.
- On error, flows become `Failed`.

#### State Management
- Transitions are managed via `SetStatus` (FlowWorker), with updates persisted and events published.
- Flow state is influenced by task state (e.g., if tasks are waiting, the flow is waiting).
- Flows can be stopped (graceful termination) or finished (completion of all tasks).

#### State Diagram
```mermaid
stateDiagram-v2
    [*] --> Created: CreateFlow()
    Created --> Running: PutInput() / Start Task
    Running --> Waiting: Task waiting for input
    Running --> Finished: All tasks completed successfully
    Running --> Failed: Task failed / Error
    Waiting --> Running: PutInput() / Resume Task
    Waiting --> Finished: Finish()
    Waiting --> Failed: Error
    Finished --> [*]
    Failed --> [*]
```

### Assistant Lifecycle

#### States
- `Created` (database.AssistantStatusCreated)
- `Running` (database.AssistantStatusRunning)
- `Waiting` (database.AssistantStatusWaiting)
- `Finished` (database.AssistantStatusFinished)
- `Failed` (database.AssistantStatusFailed)

#### State Transitions
- Assistants are created in the `Created` state.
- When processing input, they transition to `Running`.
- If waiting for user input, they move to `Waiting`.
- On completion or user termination, they become `Finished`.
- On error, they become `Failed`.

#### State Management
- Transitions are managed via `SetStatus` (AssistantWorker).
- Assistants support streaming responses with real-time updates.
- Multiple assistants can exist within a single flow.
- Assistant execution can be stopped or finished independently.

```mermaid
stateDiagram-v2
    [*] --> Created: CreateAssistant()
    Created --> Running: PutInput()
    Running --> Waiting: Waiting for user input
    Running --> Finished: Conversation ended
    Running --> Failed: Error in processing
    Waiting --> Running: PutInput()
    Waiting --> Finished: Finish()
    Finished --> [*]
    Failed --> [*]
```

### Task Lifecycle

#### States
- `Created` (database.TaskStatusCreated)
- `Running` (database.TaskStatusRunning)
- `Waiting` (database.TaskStatusWaiting)
- `Finished` (database.TaskStatusFinished)
- `Failed` (database.TaskStatusFailed)

#### State Transitions
- Tasks are created in the `Created` state when a flow receives input.
- They transition to `Running` when execution begins.
- If subtasks require input, tasks move to `Waiting`.
- On successful completion, tasks become `Finished`.
- On error, tasks become `Failed`.

#### State Management
- Transitions are managed via `SetStatus` (TaskWorker), with updates affecting the parent flow status.
- Task state is influenced by subtask state (e.g., if subtasks are waiting, the task is waiting).
- Tasks can be finished early or allowed to complete naturally.

```mermaid
stateDiagram-v2
    [*] --> Created: CreateTask()
    Created --> Running: Run()
    Running --> Waiting: Subtask waiting for input
    Running --> Finished: All subtasks completed
    Running --> Failed: Subtask failed / Error
    Waiting --> Running: PutInput() / Resume subtask
    Waiting --> Finished: Finish()
    Finished --> [*]
    Failed --> [*]
```

### Subtask Lifecycle

#### States
- `Created` (database.SubtaskStatusCreated)
- `Running` (database.SubtaskStatusRunning)
- `Waiting` (database.SubtaskStatusWaiting)
- `Finished` (database.SubtaskStatusFinished)
- `Failed` (database.SubtaskStatusFailed)

#### State Transitions
- Subtasks are created in the `Created` state when generated by task planning.
- They transition to `Running` when execution begins.
- If they require additional input, subtasks move to `Waiting`.
- On successful completion, subtasks become `Finished`.
- On error, subtasks become `Failed`.

#### State Management
- Transitions are managed via `SetStatus` (SubtaskWorker), with updates affecting the parent task status.
- Subtasks are executed sequentially, with refinement between executions.
- Each subtask operates with its own message chain for AI provider communication.

```mermaid
stateDiagram-v2
    [*] --> Created: GenerateSubtasks()
    Created --> Running: PopSubtask() / Run()
    Running --> Waiting: Provider waiting for input
    Running --> Finished: Provider completed successfully
    Running --> Failed: Provider failed / Error
    Waiting --> Running: PutInput()
    Waiting --> Finished: Finish()
    Finished --> [*]
    Failed --> [*]
```

### Error Handling and Event Publication

- All state transitions are atomic and include error handling with proper rollback mechanisms.
- Failed states are terminal and require manual intervention or restart.
- All state changes are published via the subscription system for real-time updates.
- Errors are logged with full context and propagated up the hierarchy.

### Example: State Transition (Task)

```go
func (tw *taskWorker) SetStatus(ctx context.Context, status database.TaskStatus) error {
    task, err := tw.taskCtx.DB.UpdateTaskStatus(ctx, database.UpdateTaskStatusParams{
        Status: status,
        ID:     tw.taskCtx.TaskID,
    })
    if err != nil {
        return fmt.Errorf("failed to set task %d status: %w", tw.taskCtx.TaskID, err)
    }

    subtasks, err := tw.taskCtx.DB.GetTaskSubtasks(ctx, tw.taskCtx.TaskID)
    if err != nil {
        return fmt.Errorf("failed to get task %d subtasks: %w", tw.taskCtx.TaskID, err)
    }

    tw.taskCtx.Publisher.TaskUpdated(ctx, task, subtasks)

    tw.mx.Lock()
    defer tw.mx.Unlock()

    switch status {
    case database.TaskStatusRunning:
        tw.completed = false
        tw.waiting = false
        err = tw.updater.SetStatus(ctx, database.FlowStatusRunning)
    case database.TaskStatusWaiting:
        tw.completed = false
        tw.waiting = true
        err = tw.updater.SetStatus(ctx, database.FlowStatusWaiting)
    case database.TaskStatusFinished, database.TaskStatusFailed:
        tw.completed = true
        tw.waiting = false
        err = tw.updater.SetStatus(ctx, database.FlowStatusWaiting)
    }

    return err
}
```

## Log Management and Event Publication

The `controller` package manages seven distinct types of logs, each serving specific purposes in the penetration testing workflow. All logs are handled through a consistent controller/worker pattern and support real-time event publication.

### Log Types and Their Roles

1. **Message Logs** (`MsgLogController`/`FlowMsgLogWorker`)
   - Records all AI agent communications and reasoning
   - Supports thinking/reasoning capture for transparency
   - Handles different message types (input, output, report, etc.)
   - Supports result formatting (plain text, markdown, JSON)

2. **Assistant Logs** (`AssistantLogController`/`FlowAssistantLogWorker`)
   - Records conversations with interactive AI assistants
   - Supports real-time streaming for live chat experiences
   - Manages multiple assistants per flow
   - Includes streaming message chunks for progressive updates

3. **Agent Logs** (`AgentLogController`/`FlowAgentLogWorker`)
   - Records interactions between different AI agents
   - Tracks task delegation and agent communication
   - Identifies initiator and executor agents
   - Links to specific tasks and subtasks

4. **Search Logs** (`SearchLogController`/`FlowSearchLogWorker`)
   - Records web searches and OSINT operations
   - Tracks different search engines (Google, DuckDuckGo, etc.)
   - Stores search queries and results
   - Essential for reconnaissance phases

5. **Terminal Logs** (`TermLogController`/`FlowTermLogWorker`)
   - Records all command-line interactions
   - Tracks input/output from pentesting tools
   - Associates with specific Docker containers
   - Critical for audit trail and debugging

6. **Vector Store Logs** (`VectorStoreLogController`/`FlowVectorStoreLogWorker`)
   - Records vector database operations
   - Tracks similarity searches and embeddings
   - Supports query filtering and metadata
   - Used for knowledge management and context retrieval

7. **Screenshots** (`ScreenshotController`/`FlowScreenshotWorker`)
   - Captures visual evidence during testing
   - Stores screenshot URLs and metadata
   - Links to specific flow contexts
   - Essential for reporting and documentation

### Log Lifecycle and Operations

All log types follow a consistent pattern:

1. **Creation**: Log workers are created per flow by their respective controllers
2. **Logging**: Messages/events are recorded via `PutLog()` or similar methods
3. **Retrieval**: Historical logs can be retrieved via `GetLog()` methods
4. **Event Publication**: Every log operation triggers real-time events
5. **Cleanup**: Log workers are cleaned up when flows are finished

#### Special Features

- **Message Truncation**: Long messages are truncated to prevent database bloat
- **Thread Safety**: All log operations are protected by mutexes
- **Streaming Support**: Assistant logs support real-time streaming updates
- **Context Linking**: Most logs can be linked to specific tasks and subtasks

### Event Publication

Every log operation publishes corresponding events:

```go
// Example events published by log workers
pub.MessageLogAdded(ctx, msgLog)
pub.AgentLogAdded(ctx, agentLog)
pub.AssistantLogAdded(ctx, assistantLog)
pub.AssistantLogUpdated(ctx, assistantLog, isStreaming)
pub.SearchLogAdded(ctx, searchLog)
pub.TerminalLogAdded(ctx, termLog)
pub.VectorStoreLogAdded(ctx, vectorLog)
pub.ScreenshotAdded(ctx, screenshot)
```

### Log Worker Creation and Management

Log controllers maintain thread-safe maps of workers:

```go
// Example from MsgLogController
func (mlc *msgLogController) NewFlowMsgLog(
    ctx context.Context,
    flowID int64,
    pub subscriptions.FlowPublisher,
) (FlowMsgLogWorker, error) {
    mlc.mx.Lock()
    defer mlc.mx.Unlock()

    flw := NewFlowMsgLogWorker(mlc.db, flowID, pub)
    mlc.flows[flowID] = flw
    return flw, nil
}
```

## Integration with Providers, Tools, and Subscriptions

The `controller` package is deeply integrated with external providers (LLM, tools), the tools execution layer, and the event subscription system. This integration is essential for orchestrating complex flows, executing tasks and subtasks, and providing real-time updates to clients.

### Providers Integration

- The package uses the `providers.ProviderController` interface to create and manage provider instances for each flow and assistant.
- Providers are responsible for LLM operations, agent chain management, and tool execution.
- Each flow is associated with a `FlowProvider`, and assistants have their own `AssistantProvider`.
- Providers are injected into contexts and are accessible to all workers.
- Support for multiple provider types (OpenAI, Anthropic, etc.) through abstract interfaces.

### Tools Integration

- The `tools.FlowToolsExecutor` is created for each flow and is responsible for executing tool calls within the flow context.
- The executor is configured with the provider's image, embedder, and all log providers.
- Tools are invoked as part of agent chain execution and are tightly coupled with the flow's lifecycle.
- Tools can access all logging capabilities for audit trails and debugging.

### Subscriptions and Event Publication

- The `subscriptions.FlowPublisher` is created for each flow and publishes all significant events.
- The publisher is injected into all log workers and used to notify subscribers in real time.
- Events include entity creation, updates, log additions, and state changes.
- The event system is decoupled from core logic, ensuring atomic operations.

### Dependency Injection and Context Propagation

All dependencies are injected through constructor parameters and context objects:

```go
type flowWorkerCtx struct {
    db     database.Querier
    cfg    *config.Config
    docker docker.DockerClient
    provs  providers.ProviderController
    subs   subscriptions.SubscriptionsController

    flowProviderControllers
}

type flowProviderControllers struct {
    mlc  MsgLogController
    aslc AssistantLogController
    alc  AgentLogController
    slc  SearchLogController
    tlc  TermLogController
    vslc VectorStoreLogController
    sc   ScreenshotController
}
```

### Integration Flow Diagrams

```mermaid
flowchart LR
    subgraph Controllers
        FC[FlowController]
        FW[FlowWorker]
        AW[AssistantWorker]
        LogCtrl[Log Controllers]
    end

    subgraph External
        DB[(Database)]
        Providers[AI Providers]
        Tools[Tools Executor]
        Subs[Subscriptions]
        Docker[Docker]
    end

    FC --> FW
    FW --> AW
    FC --> LogCtrl
    FW --> LogCtrl
    AW --> LogCtrl

    Controllers --> DB
    Controllers --> Providers
    Controllers --> Tools
    Controllers --> Subs
    Controllers --> Docker
```

## Internal Structure and Concurrency Model

The `controller` package is designed for safe concurrent operation in a multi-user, multi-flow environment. All controllers and workers use mutexes to ensure thread safety for all mutable state.

### Mutex Usage and Thread Safety

- Each controller contains a `*sync.Mutex` or `*sync.RWMutex` to guard access to internal maps and state.
- All public methods that mutate or read shared state acquire the mutex for the duration of the operation.
- Workers also use mutexes to protect their internal state, especially for status flags and log operations.
- This design prevents race conditions and ensures that all operations are atomic and consistent.

#### Example: Controller Mutex Usage
```go
type flowController struct {
    db     database.Querier
    mx     *sync.Mutex
    flows  map[int64]FlowWorker
    // ... other dependencies ...
}

func (fc *flowController) CreateFlow(...) (FlowWorker, error) {
    fc.mx.Lock()
    defer fc.mx.Unlock()
    // ... mutate fc.flows ...
}
```

#### Example: Worker Mutex Usage
```go
type flowMsgLogWorker struct {
    db     database.Querier
    mx     *sync.Mutex
    flowID int64
    pub    subscriptions.FlowPublisher
}

func (mlw *flowMsgLogWorker) PutMsg(...) (int64, error) {
    mlw.mx.Lock()
    defer mlw.mx.Unlock()
    // ... perform log operation ...
}
```

### State Storage and Management

- Controllers maintain maps of active workers for fast lookup and management.
- All access to these maps is guarded by mutexes.
- Workers encapsulate all state and dependencies for a single entity.
- Context objects (`FlowContext`, `TaskContext`, `SubtaskContext`) pass dependencies down the hierarchy.

### Worker Goroutines and Channels

- `FlowWorker` and `AssistantWorker` use goroutines and channels to process input asynchronously.
- Dedicated goroutines run worker loops, processing input and managing execution.
- Synchronization is achieved using mutexes, channels, and wait groups for clean shutdown.
- Background processing enables responsive user interactions while maintaining system stability.

#### Assistant Streaming Example
```go
// Assistant log worker supports streaming for real-time chat
func (aslw *flowAssistantLogWorker) workerMsgUpdater(
    msgID, streamID int64,
    ch chan *providers.StreamMessageChunk,
) {
    // Processes streaming chunks in background goroutine
    for chunk := range ch {
        // Update database and publish events in real-time
        processChunk(chunk)
    }
}
```

## Extensibility, Error Handling, and Best Practices

The `controller` package is designed for extensibility, robust error handling, and safe integration into larger systems.

### Extensibility

- All major entities are abstracted via interfaces, making it easy to add new types or extend existing ones.
- New log types can be added by following the established controller/worker pattern.
- The pattern is consistent across all log types: Controller manages multiple workers, Worker handles single entity.
- Dependency injection via context and constructor parameters allows for easy testing and mocking.
- The use of context objects enables flexible propagation of dependencies and state.

#### Adding New Log Types
To add a new log type, follow this pattern:
1. Create `LogController` interface and `LogWorker` interface
2. Implement controller with thread-safe worker map
3. Implement worker with mutex protection and event publication
4. Add to `flowProviderControllers` struct
5. Integrate into flow worker creation process

### Error Handling

- All public methods return errors, wrapped with context using `fmt.Errorf`.
- Errors are logged using `logrus` and propagated up the call stack.
- State transitions on error are explicit: entities are set to `Failed` states.
- Defensive checks are used throughout (nil checks, state verification, etc.).
- The `wrapErrorEndSpan` utility provides consistent error handling with observability.

#### Error Handling Example
```go
func wrapErrorEndSpan(ctx context.Context, span langfuse.Span, msg string, err error) error {
    logrus.WithContext(ctx).WithError(err).Error(msg)
    err = fmt.Errorf("%s: %w", msg, err)
    span.End(
        langfuse.WithEndSpanStatus(err.Error()),
        langfuse.WithSpanLevel(langfuse.ObservationLevelError),
    )
    return err
}
```

### Best Practices

1. **Always use contexts**: All operations accept and propagate context for cancellation and tracing.
2. **Mutex discipline**: Always use defer for mutex unlocking to prevent deadlocks.
3. **Error wrapping**: Provide meaningful error messages with full context.
4. **Event publication**: Publish events after successful database operations.
5. **State consistency**: Ensure database and in-memory state remain synchronized.
6. **Resource cleanup**: Implement proper cleanup in `Finish()` methods.
7. **Observability**: Use tracing and logging for all significant operations.

The controller package provides a robust foundation for managing complex AI-driven penetration testing workflows while maintaining reliability, observability, and extensibility.

---

### Backend Docs Database (`backend/docs/database.md`)

# Database Layer

## Overview

PentAGI stores application state in PostgreSQL and uses the `vector` extension for agent memory and knowledge search. The database layer combines:

- **sqlc v1.27.0** for the main type-safe query API (`backend/pkg/database`);
- **GORM v1** (`github.com/jinzhu/gorm`) for HTTP server models and handlers (`backend/pkg/server/models`);
- **goose v3** for embedded, ordered schema migrations (`backend/migrations`);
- **lib/pq** for the shared `database/sql` pool used by sqlc and GORM;
- **pgxpool** for every `pgvector.Store` instance.

SQL queries live in `backend/sqlc/models`. Schema history lives in `backend/migrations/sql`.

This document describes how the product uses the database. It is not a substitute for the schema itself. Sources of truth, in descending order:

1. `backend/migrations/sql/*.sql` — schema and data migrations;
2. `backend/sqlc/models/*.sql` — application queries;
3. `backend/pkg/database/*.sql.go`, `models.go`, `querier.go` — generated Go API;
4. `backend/cmd/pentagi/main.go` and `backend/pkg/database/tenant.go` — connection, migration and tenant bootstrap behavior;
5. `backend/pkg/database/{database.go,converter,knowledge}` — helpers and higher-level database services.

Environment variables, PgBouncer and Supavisor setup are documented in [config.md](config.md); this file only covers the product behavior those settings enable.

## Runtime Architecture

### Startup sequence

`backend/cmd/pentagi/main.go` initializes PostgreSQL in this order:

1. Load and validate configuration (including `TENANT_ID`).
2. If `TENANT_ID` is set, create the tenant schema, validate shared extension placement and rewrite `DATABASE_URL` with the tenant `search_path` (`database.EnsureTenantSchema`).
3. Open one `*sql.DB` through `lib/pq`.
4. Configure the shared `database/sql` connection pool (`DATABASE_MAX_OPEN_CONNS`, `DATABASE_MAX_IDLE_CONNS`, one-hour max lifetime).
5. Verify that `current_schema()` resolves to the configured tenant schema (`database.VerifySearchPath`).
6. Build sqlc `Queries` and GORM on the same `*sql.DB`.
7. Create one shared `pgxpool.Pool` for every pgvector store and attach it to `cfg.PgxPool`.
8. Configure goose with the schema-qualified version table and run embedded migrations under a PostgreSQL advisory lock (`database.RunMigrations`).
9. Start controllers and the API server.

The process refuses to serve traffic after a tenant schema mismatch or a migration failure. The same tenant helpers are reused by utility binaries (`ftester`, `etester`) and by the installer's password-reset path when they talk to PostgreSQL.

### Database clients and pools

PentAGI opens two independent pools to the same PostgreSQL database:

| Pool | Configuration | Default | Consumers |
|---|---|---:|---|
| `database/sql` (`lib/pq`) | `DATABASE_MAX_OPEN_CONNS` / `DATABASE_MAX_IDLE_CONNS` | `25` / `5` | sqlc and GORM |
| `pgxpool.Pool` | `DATABASE_VECTOR_MAX_CONNS` | `10` | agent memory and knowledge pgvector stores |

GORM does not open another pool:

```go
queries := database.New(db)
orm, err := database.NewGorm(db, cfg.Debug)
```

During normal startup the shared pgx pool is stored in `cfg.PgxPool` and passed to vector stores with `pgvector.WithConn(cfg.PgxPool)`. Tool executors and the knowledge API therefore reuse the same pool. Their utility/test fallback uses `pgvector.WithConnectionURL(cfg.DatabaseURL)` only when `PgxPool` is nil.

### Connection budget

With the defaults, one PentAGI process can use up to 35 PostgreSQL connections: 25 through `database/sql` and 10 through `pgxpool`. This is a hard application budget, not a prediction of steady-state usage.

When several PentAGI instances share one database server, add both pool limits for every instance and leave capacity for PostgreSQL reserved connections, autovacuum, monitoring clients, platform services and administrative/migration sessions.

```sql
SELECT name, setting
FROM pg_settings
WHERE name IN ('max_connections', 'superuser_reserved_connections');

SELECT application_name, client_addr, state, count(*)
FROM pg_stat_activity
WHERE pid <> pg_backend_pid()
GROUP BY 1, 2, 3
ORDER BY count(*) DESC;
```

The stock Compose stack uses `vxcontrol/pgvector:latest`. Do not assume a specific PostgreSQL major version from the image tag; inspect `SHOW server_version` on the deployed database.

## Package Layout

| Path | Responsibility |
|---|---|
| `backend/migrations/migrations.go` | Embeds all goose migration SQL into the binary |
| `backend/migrations/sql/` | Authoritative ordered schema and data migration history |
| `backend/sqlc/sqlc.yml` | sqlc input, type overrides and output configuration |
| `backend/sqlc/models/` | Hand-written parameterized SQL queries |
| `backend/pkg/database/db.go` | Generated `DBTX`, `Queries`, `New` and `WithTx` |
| `backend/pkg/database/models.go` | Generated table models and PostgreSQL enum wrappers |
| `backend/pkg/database/querier.go` | Generated `Querier` interface covering all sqlc operations |
| `backend/pkg/database/*.sql.go` | Generated query implementations and result/parameter structs |
| `backend/pkg/database/database.go` | Null helpers, UTF-8 sanitization and shared GORM initialization |
| `backend/pkg/database/tenant.go` | Tenant schema bootstrap, DSN rewrite, search-path verification and advisory locks (shared by pentagi, ftester, etester, installer) |
| `backend/pkg/database/converter/` | Conversion from database rows to GraphQL models plus execution analytics calculations |
| `backend/pkg/database/knowledge/` | Knowledge-store business logic over sqlc, pgvector embeddings and GraphQL subscriptions |
| `backend/pkg/server/models/` | GORM v1 models used by REST/server services |
| `backend/cmd/pentagi/main.go` | Production pool creation, migration execution and dependency wiring |

sqlc and GORM are both active. Controllers and GraphQL paths primarily use generated sqlc queries; REST/server services continue to use GORM models for authentication, users, logs, settings, analytics, resources, flow files and related endpoints. Both clients share the same `*sql.DB`. GORM never owns schema creation.

## Schema and Data Model

### Workflow hierarchy

```text
users
  └── flows
      ├── tasks
      │   └── subtasks
      ├── containers
      ├── assistants
      ├── msgchains
      ├── toolcalls
      ├── screenshots
      └── operational logs
```

There are no relational `actions`, `artifacts` or `memories` tables. Individual operations are represented by tool calls and specialized log tables. Agent memory and knowledge documents are stored in the LangChain pgvector tables.

### Table groups

#### Identity and authorization

| Table | Key fields / notes |
|---|---|
| `users` | Local/OAuth identity (`type`, `mail`, `hash`, `password`, `provider`), `status`, `role_id`, `password_change_required` |
| `roles` | Built-in application roles (seeded in the initial migration) |
| `privileges` | Per-role permission names used by REST and GraphQL authorization; grants evolve through later privilege migrations (not RLS) |
| `api_tokens` | `token_id`, `user_id`, `role_id`, `ttl`, `status`, soft deletion via `deleted_at` |
| `user_preferences` | One JSONB preferences document per user, including favorite-flow state |

#### Workflow and interaction

| Table | Key fields / notes |
|---|---|
| `flows` | Status, title, model, provider name/type, language, functions JSON, `tool_call_id_template`, optional `trace_id`, soft deletion |
| `tasks` | Status, title, input, result; owned by `flow_id` |
| `subtasks` | Status, title, description, result, persisted `context`; owned by `task_id` |
| `containers` | Type (`primary`/`secondary`), name, image, status, optional Docker `local_id`/`local_dir` |
| `assistants` | Flow-scoped interactive assistants with model/provider/functions, `use_agents`, optional `msgchain_id`, soft deletion |
| `msgchains` | LLM chain JSON plus usage (`usage_in`/`out`, cache, cost) and `duration_seconds` |
| `toolcalls` | `call_id`, name, args JSON, result, status, `duration_seconds` |
| `flow_templates` | User-owned reusable flow descriptions (`title`, `text`) |

`flows` and `assistants` support soft deletion. Normal API deletion marks a flow's `deleted_at` and leaves child rows in place for audit/history; foreign-key cascades only run if the flow row is physically deleted.

#### Configuration and user content

| Table | Key fields / notes |
|---|---|
| `providers` | User-owned LLM provider configs (`type`, `name`, `config` JSON), soft deletion |
| `prompts` | User-owned prompt templates keyed by `PROMPT_TYPE` |
| `user_resources` | Uploaded file/directory metadata (`hash`, `name`, `path`, `size`, `is_dir`) |

Flow and assistant rows contain model/provider selection and runtime function configuration. Prompt templates themselves are stored in `prompts`, not in current flow or assistant rows.

#### Logs and artifacts

| Table | Key fields / notes |
|---|---|
| `agentlogs` | Agent-to-agent delegation (`initiator`, `executor`, task/result text) |
| `assistantlogs` | Assistant messages with optional `thinking` and `result_format` |
| `msglogs` | General flow messages with optional `thinking` and `result_format` |
| `searchlogs` | Search engine calls (`engine`, query/result) |
| `termlogs` | Terminal stdin/stdout/stderr; **requires** `container_id` and `flow_id` |
| `vecstorelogs` | Vector-store ops (`action`, filter JSON, query/result) |
| `screenshots` | Screenshot metadata (`name`, `url`); **requires** `flow_id` |

Several log/artifact tables carry nullable `task_id` and `subtask_id` in addition to a required `flow_id`, allowing flow-, task- and subtask-level retrieval.

#### Vector knowledge and memory

| Table | Purpose |
|---|---|
| `langchain_pg_collection` | Logical pgvector collections (`name`, `cmetadata`, `uuid`) |
| `langchain_pg_embedding` | Document text, vector embedding and JSON `cmetadata` |

PentAGI uses the collection named `langchain`. Ownership and association are represented in `cmetadata` fields such as `user_id`, `flow_id`, `task_id`, `subtask_id`, `doc_type`, `question`, `description`, `guide_type`, `answer_type`, `code_lang`, chunk sizing and a `manual` flag. See `backend/pkg/database/knowledge`.

The knowledge query API excludes `doc_type = 'memory'` from user-managed knowledge listings/searches. GraphQL and REST flow-deletion paths issue an explicit best-effort deletion of memory rows (`DeleteFlowMemoryDocuments`); this is application behavior, not a database trigger or foreign-key cascade. The knowledge migration intentionally does not drop the LangChain tables on downgrade because they may contain production data managed by the vector store.

### Enums

PostgreSQL enums are migrated explicitly and generated as Go string types in `models.go`. Current values:

| Enum | Values |
|---|---|
| `FLOW_STATUS` / `TASK_STATUS` / `SUBTASK_STATUS` / `ASSISTANT_STATUS` | `created`, `running`, `waiting`, `finished`, `failed` |
| `CONTAINER_STATUS` | `starting`, `running`, `stopped`, `deleted`, `failed` |
| `CONTAINER_TYPE` | `primary`, `secondary` |
| `TOOLCALL_STATUS` | `received`, `running`, `finished`, `failed` |
| `TOKEN_STATUS` | `active`, `revoked` |
| `USER_STATUS` | `created`, `active`, `blocked` |
| `USER_TYPE` | `local`, `oauth` |
| `MSGCHAIN_TYPE` | `primary_agent`, `reporter`, `generator`, `refiner`, `reflector`, `enricher`, `adviser`, `coder`, `memorist`, `searcher`, `installer`, `pentester`, `summarizer`, `tool_call_fixer`, `assistant` |
| `MSGLOG_TYPE` | `answer`, `report`, `thoughts`, `browser`, `terminal`, `file`, `search`, `advice`, `ask`, `input`, `done` |
| `MSGLOG_RESULT_FORMAT` | `plain`, `markdown`, `terminal` |
| `TERMLOG_TYPE` | `stdin`, `stdout`, `stderr` |
| `VECSTORE_ACTION_TYPE` | `retrieve`, `store` |
| `PROVIDER_TYPE` | `openai`, `anthropic`, `gemini`, `bedrock`, `ollama`, `custom`, `deepseek`, `glm`, `kimi`, `qwen`, `minimax`, `mistral`, `xai` |
| `SEARCHENGINE_TYPE` | `google`, `tavily`, `firecrawl`, `traversaal`, `browser`, `duckduckgo`, `perplexity`, `searxng`, `sploitus` |
| `PROMPT_TYPE` | Agent/system prompt keys from `primary_agent` through `task_assignment_wrapper` (full list in `models.go`) |

Never add an enum value only in Go code. Add or replace the PostgreSQL enum in a goose migration, regenerate sqlc, and update backend validation where applicable. Provider and search-engine additions have additional project steps documented in `CLAUDE.md`.

### Data lifecycle and integrity

Primary keys use PostgreSQL identity columns. Foreign keys define ownership and use `ON DELETE CASCADE` for records that have no meaning without their parent, including user-owned settings/content and flow-owned execution data. Cascades apply only to physical deletion; ordinary flow, assistant, provider and API-token deletion paths use `deleted_at` where their schema supports soft deletion.

The shared `update_modified_column()` trigger maintains `updated_at` for mutable entities such as flows, tasks, subtasks, containers, tool calls, message chains, assistants, providers, API tokens, preferences, templates and resources. Generated models represent database-default timestamps with `sql.NullTime`, so callers should not assume a non-null Go `time.Time` before insertion/returning.

Important integrity constraints include unique user mail/hash values, one prompt type per user, one preferences row per user, active provider names per user, API-token identifiers, resource paths per user and non-empty template/resource text fields. JSON configuration and chain payloads use PostgreSQL `JSON`; preferences use `JSONB` with a GIN index; optional relationships use nullable SQL columns and generated `sql.Null*` wrappers.

## Application Scoping and Deployment Tenancy

PentAGI has two distinct isolation layers. They solve different problems and must not be confused.

### User scoping inside one PentAGI instance

Rows such as flows, providers, prompts, templates, resources and preferences carry `user_id` directly or are reached through a flow owned by a user. User-facing handlers select user-scoped sqlc methods such as `GetUserFlow` / `GetUserFlows`; admin paths may intentionally use unscoped variants such as `GetFlow` / `GetFlows`.

This access control is implemented by application queries and privilege checks. PentAGI does not rely on PostgreSQL row-level security for its own tables.

### Instance scoping with `TENANT_ID`

`TENANT_ID` isolates independent PentAGI installations that share a PostgreSQL database. It creates one PostgreSQL schema per instance (`public` when empty, otherwise the tenant name). For a tenant, the effective search path is `<tenant>,<DATABASE_EXTENSIONS_SCHEMA>`.

Tenant bootstrap (`backend/pkg/database/tenant.go`):

- validates `TENANT_ID`;
- creates the tenant schema under the `pentagi-tenant-bootstrap` advisory lock;
- ensures `vector` and `pg_trgm` exist in the configured shared extension schema;
- refuses to move provider-managed extensions automatically;
- rewrites `DATABASE_URL` once, before sqlc, GORM, goose or pgxpool consume it;
- verifies `current_schema()` and aborts on mismatch.

The goose version table is schema-qualified as `<schema>.goose_db_version`. This prevents a new tenant from reading `public.goose_db_version`, incorrectly deciding that migrations are already applied, and starting with an empty schema.

For the complete multi-instance deployment contract (non-database resources, validation regex, PgBouncer `connect_query`, Supavisor `DATABASE_SEARCH_PATH_VIA_OPTIONS`, Supabase `DATABASE_EXTENSIONS_SCHEMA=extensions`), see [config.md](config.md#multi-instance-deployment-tenant_id).

## Migrations

Migration files are stored in `backend/migrations/sql` and embedded by `backend/migrations/migrations.go`:

```go
//go:embed sql/*.sql
var EmbedMigrations embed.FS
```

At startup goose uses that filesystem and runs `goose.Up`. Migrations use goose `Up`/`Down` annotations and numeric filename prefixes. The current migration head is determined by the newest migration file; do not hard-code it in application logic.

PentAGI serializes:

1. tenant schema/extension bootstrap with the `pentagi-tenant-bootstrap` advisory lock;
2. migrations with a schema-specific `pentagi-migrations-<schema>` advisory lock.

The lock is held on a dedicated `*sql.Conn` because PostgreSQL session advisory locks belong to a physical connection, not to a `*sql.DB` pool.

### Adding a migration

1. Create a uniquely ordered SQL file in `backend/migrations/sql`.
2. Add `-- +goose Up` and, where safe, a reversible `Down`.
3. Use `-- +goose StatementBegin`/`StatementEnd` for multi-statement units.
4. Preserve tenant compatibility: unqualified application objects must be created in the active tenant schema, while extensions remain in the shared extension schema.
5. Regenerate sqlc if the schema or query types changed.
6. Test both a fresh database and an upgrade from the previous migration head.

Do not edit an already released migration. Add a new migration.

## sqlc Query Layer

### Configuration

`backend/sqlc/sqlc.yml` reads queries from `models/*.sql` and schema from `../migrations/sql/*.sql`, generating package `database` into `../pkg/database` with `emit_interface` and `emit_json_tags`. Notable overrides:

- `pg_catalog.numeric` → `float64`;
- nullable `vector` / `pg_catalog.vector` → `string` (queries cast vector literals explicitly).

Generated files start with `Code generated by sqlc. DO NOT EDIT.` Edit SQL or migrations and regenerate instead.

### Query files

| Query file | Product area | Named queries |
|---|---|---|
| `flows.sql` | Flow CRUD, soft deletion, provider/model updates, flow statistics | `GetFlows`, `GetUserFlows`, `GetFlow`, `GetUserFlow`, `CreateFlow`, `UpdateFlow*`, `DeleteFlow`, `GetFlowStats`, `GetUserTotalFlowsStats`, `GetFlowsStatsByDayLast{Week,Month,3Months}` |
| `tasks.sql` | Task lifecycle and hierarchy-scoped retrieval | `GetFlowTasks`, `GetUserFlowTasks`, `GetFlowTask`, `GetUserFlowTask`, `GetTask`, `CreateTask`, `UpdateTaskStatus/Result/FinishedResult/FailedResult` |
| `subtasks.sql` | Subtask lifecycle, context, planned/completed filters | `GetFlowSubtasks`, `GetFlowTaskSubtasks`, `GetUserFlow*`, `GetTaskSubtasks`, `GetTaskPlannedSubtasks`, `GetTaskCompletedSubtasks`, `GetSubtask`, `GetFlowSubtask`, `CreateSubtask`, `UpdateSubtask*`, `DeleteSubtask(s)` |
| `containers.sql` | Flow container lookup and status | `GetContainers`, `GetUserContainers`, `GetRunningContainers`, `GetFlowContainers`, `GetFlowPrimaryContainer`, `GetUserFlowContainers`, `CreateContainer`, `UpdateContainer*` |
| `assistants.sql` | User/admin assistant access and settings | `GetFlowAssistants`, `GetUserFlowAssistants`, `GetFlowAssistant`, `GetUserFlowAssistant`, `GetAssistant`, `GetAssistantUseAgents`, `CreateAssistant`, `UpdateAssistant*`, `DeleteAssistant` |
| `msgchains.sql` | Conversation chains plus usage analytics | Chain CRUD/lookup by hierarchy/type; `UpdateMsgChainUsage`; aggregates by flow/task/subtask/provider/model/type/day/user |
| `toolcalls.sql` | Tool-call lifecycle and analytics | Hierarchy CRUD/status updates; aggregates by flow/task/subtask/function/day/user |
| `analytics.sql` | Period-based flow selection and hierarchy batches for execution analytics | `GetFlowsForPeriodLast{Week,Month,3Months}`, `GetTasksForFlow`, `GetSubtasksForTasks`, `GetMsgchainsForFlow`, `GetToolcallsForFlow`, `GetAssistantsCountForFlow` |
| `screenshots.sql` | Screenshot retrieval/creation | Flow/user/task/subtask getters + `CreateScreenshot` |
| `agentlogs.sql` | Agent delegation logs | Flow/user/task/subtask getters + `CreateAgentLog` |
| `assistantlogs.sql` | Assistant message/result/thinking logs | Create/update/delete plus flow/user getters |
| `msglogs.sql` | General message logs | Create/update plus flow/user/task/subtask getters |
| `searchlogs.sql` | Search operation logs | Flow/user/task/subtask getters + `CreateSearchLog` |
| `termlogs.sql` | Terminal logs | Container and hierarchy-scoped getters + `CreateTermLog` |
| `vecstorelogs.sql` | Vector-store audit logs | Flow/user/task/subtask getters + `CreateVectorStoreLog` |
| `users.sql` | User identity and administration | `GetUsers`, `GetUser`, `GetUserByHash`, `CreateUser`, `UpdateUser*`, `DeleteUser` |
| `roles.sql` | Roles and privileges | `GetRoles`, `GetRole`, `GetRoleByName` |
| `api_tokens.sql` | Token lifecycle | Admin and user-scoped create/update/soft-delete/list |
| `user_preferences.sql` | Preferences and favorite flows | CRUD/upsert + `AddFavoriteFlow` / `DeleteFavoriteFlow` |
| `providers.sql` | Provider configuration | Admin and user-scoped CRUD/soft-delete, lookup by type/name |
| `prompts.sql` | Prompt templates by `PROMPT_TYPE` | Admin and user-scoped CRUD, lookup/update by type |
| `flow_templates.sql` | Flow templates | User-owned CRUD |
| `resources.sql` | `user_resources` trees (**read-only** sqlc; create/update/delete go through GORM REST services) | Root/dir/recursive/all lookups for one user or all users; lookup by ID(s) |
| `knowledge.sql` | LangChain pgvector documents | Admin/user get/list/update/delete, cosine search, insert, `DeleteFlowMemoryDocuments` |

The generated `Querier` interface in `backend/pkg/database/querier.go` currently exposes 251 methods matching the named SQL queries. Do not hand-edit it.

### Regeneration

From `backend/`, with the PentAGI PostgreSQL network available:

```bash
docker run --rm \
  -v "$(pwd):/src" \
  -w /src \
  --network pentagi-network \
  -e DATABASE_URL='postgres://postgres:postgres@pgvector:5432/pentagidb?sslmode=disable' \
  sqlc/sqlc:1.27.0 generate -f sqlc/sqlc.yml
```

Use credentials matching the target database. Then review generated changes in `pkg/database`, especially `models.go`, `querier.go` and the affected `*.sql.go` file.

### Query conventions

```sql
-- name: GetUserFlow :one
SELECT *
FROM flows
WHERE id = $1
  AND user_id = $2
  AND deleted_at IS NULL;
```

Established conventions:

- `Create*`, `Get*`, `Update*`, `Delete*` for basic operations;
- `GetUser*` for user-owned paths;
- `GetFlow*`, `GetTask*`, `GetSubtask*` for hierarchy-scoped paths;
- admin/unscoped methods only where authorization is enforced by the caller;
- filter `deleted_at IS NULL` when querying soft-deletable entities;
- use foreign keys and `ON DELETE CASCADE` for owned child records;
- use `sqlc.arg(...)` for repeated or named parameters.

Parameterized queries protect values from SQL injection. Dynamic identifiers cannot be parameterized and must be validated and quoted separately.

### Transactions

`database.Queries` accepts the `DBTX` interface, so the same generated methods work with `*sql.DB` and `*sql.Tx`:

```go
tx, err := sqlDB.BeginTx(ctx, nil)
if err != nil {
    return err
}
defer tx.Rollback()

qtx := queries.WithTx(tx)
if _, err := qtx.CreateTask(ctx, taskParams); err != nil {
    return err
}
if _, err := qtx.CreateSubtask(ctx, subtaskParams); err != nil {
    return err
}

return tx.Commit()
```

Keep transactions short and pass the caller's context through every query.

## Helpers (`database.go`)

Hand-written helpers used across controllers and GraphQL resolvers:

| Helper | Purpose |
|---|---|
| `NullStringToPtrString` / `PtrStringToNullString` / `StringToNullString` | Nullable string bridging |
| `Int64ToNullInt64` / `Uint64ToNullInt64` / `NullInt64ToInt64` | Nullable integer bridging |
| `TimeToNullTime` / `PtrTimeToNullTime` | Nullable timestamp bridging |
| `SanitizeUTF8` | Strip NUL bytes and replace invalid UTF-8 before storing untrusted tool output |
| `NewGorm` / `GormLogger` | Shared GORM init and optional SQL logging when `DEBUG=true` |

## Converter Package

`backend/pkg/database/converter` maps sqlc rows to GraphQL models and computes execution analytics:

- entity converters for flows, containers/terminals, tasks, subtasks, assistants, screenshots, terminal/message/agent/search/vector/tool-call/assistant logs, prompts, preferences, API tokens, flow templates, user resources;
- provider/model converters between database provider configs, internal `pconfig` structures and GraphQL agent configs (including reasoning mode and call options);
- usage/toolcall/flow stats converters that adapt the typed sqlc analytics rows to GraphQL stats models;
- `BuildFlowExecutionStats` and related helpers in `analytics.go`, which combine tasks, subtasks, message-chain durations, tool-call counts and assistant activity, including overlap compensation for concurrent subtasks.

Analytics calculation behavior belongs in `analytics.go` and its tests. Avoid copying entire generated result structs into documentation; sqlc changes them when query aliases change.

## Knowledge Package

`backend/pkg/database/knowledge` implements the GraphQL knowledge API on top of `knowledge.sql`, an optional LangChain `VectorStore` and an embedding provider:

- admin reads have no `user_id` filter; user-scoped reads filter `cmetadata ->> 'user_id'`;
- writes always record the acting `userID` in metadata and publish scoped subscription events;
- create/update/search require a configured embedder/store; list/get/delete still work without embeddings;
- text sent to the embedding model is truncated to `maxEmbeddingBytes` (default 8192), while the full original text is stored in the database;
- in-memory filters refine SQL results by flow/task/subtask, doc type and related metadata.

## GORM Integration

`database.NewGorm(db, debug)` wraps the existing `*sql.DB`, installs `GormLogger` and enables GORM SQL logging only when `debug` is true. GORM v1 remains in use by server models and handlers; new type-safe database operations should prefer sqlc unless they need existing GORM model behavior.

GORM `AutoMigrate` is not used. Goose migrations are the only supported schema management path. Because sqlc and GORM share one pool, do not call `gorm.Open` elsewhere with the same `DATABASE_URL` without accounting for another pool.

Representative GORM model surfaces in `backend/pkg/server/models/` include users/roles/privileges/preferences, flows/tasks/subtasks/assistants, providers/prompts, API tokens, resources, knowledge request/response DTOs, settings, flow files and the various log entity models used by REST handlers. In particular, `user_resources` mutations are GORM-only; sqlc covers hierarchy listing and ID lookup. API token GraphQL conversion also uses the hand-written `APITokenWithSecret` helper in `backend/pkg/database/api_token_with_secret.go`.

## Vector Operations

PentAGI requires:

- `vector` for embeddings and cosine-distance search;
- `pg_trgm` for GIN trigram indexes on message/log text.

Tenant bootstrap guarantees that both extensions are reachable from every tenant. On a stock deployment they live in `public`; on Supabase they commonly live in `extensions`.

`knowledge.sql` supports admin and user-scoped document retrieval, metadata-only and full document updates, insertion with precomputed embeddings, cosine-similarity search using `<=>`, ownership filtering through `cmetadata ->> 'user_id'`, and cleanup of flow memory documents. Embedding arguments are PostgreSQL vector literals (`[f1,f2,...]`); metadata arguments must be valid JSON text.

The query layer currently filters by collection and metadata. No approximate HNSW/IVFFlat vector index is created by the migrations, so evaluate an appropriate pgvector index before assuming similarity searches will scale linearly to a large corpus.

## Analytics

Usage and execution analytics are derived from relational data rather than a separate warehouse.

### LLM usage (`msgchains`)

Stored fields: input/output tokens, cache input/output tokens, input/output cost, accumulated `duration_seconds`, model, provider and chain type.

Aggregates in `msgchains.sql`: per flow/task/subtask, all flows, by provider/model/type, by type or model-agents for a flow, daily windows (week/month/3 months) and per-user totals.

### Tool-call analytics (`toolcalls`)

Aggregates in `toolcalls.sql`: per flow/task/subtask, all flows, by function (global and per flow), daily windows and per-user totals. Converters distinguish agent tools from ordinary tools where needed for GraphQL presentation.

### Flow counts (`flows`)

`GetFlowStats`, `GetUserTotalFlowsStats` and daily flow-count windows for week/month/3 months.

### Flow execution analytics

`analytics.sql` selects flows for a period and loads hierarchy batches. `converter.BuildFlowExecutionStats` turns that into execution duration models (task/subtask timing, generator/refiner contributions, finished tool-call counts, assistant message-chain time).

## Indexing and Performance

PostgreSQL automatically indexes primary keys and unique constraints. It does **not** automatically index foreign-key columns; PentAGI migrations create the needed foreign-key and query-pattern indexes explicitly.

Current migrations include:

- ownership/hierarchy indexes (`user_id`, `flow_id`, `task_id`, `subtask_id`);
- partial indexes for active soft-deletable rows;
- provider/model/type/time indexes for analytics;
- GIN indexes for JSON preferences;
- trigram GIN indexes for message, result and thinking text;
- path-prefix indexes for user resources.

Large text B-tree indexes on task input/result and subtask description/result were deliberately removed by later migrations. Use full-text or trigram indexing for a concrete query pattern instead of restoring broad B-tree indexes.

Before adding an index: capture the real query and expected cardinality, run `EXPLAIN (ANALYZE, BUFFERS)` on representative data, account for write amplification and index size, add it through a migration, and confirm the generated schema still passes sqlc.

## Observability and Troubleshooting

### Query logging

Set `DEBUG=true` to enable the custom GORM logger. sqlc/libpq queries are not automatically printed by that logger; use PostgreSQL logging, tracing around the caller or a database proxy when those queries need inspection.

Never log `DATABASE_URL`, provider configs, token values or arbitrary SQL arguments containing credentials.

### Tenant schema mismatch

```text
search_path resolved to schema "public", expected "acme"
```

The connection or pooler ignored the tenant search path. Do not bypass the check. Use a direct connection, configure PgBouncer as documented in [config.md](config.md#multi-tenant-postgresql-access-through-pgbouncer), or try `DATABASE_SEARCH_PATH_VIA_OPTIONS=true` for a compatible Supavisor version.

### Extension schema mismatch

```text
extension "vector" is installed in schema "extensions",
but multi-tenant mode requires it in "<DATABASE_EXTENSIONS_SCHEMA>"
```

Set `DATABASE_EXTENSIONS_SCHEMA` to the existing shared extension schema. Do not move provider-managed extensions unless the database operator explicitly requires it. The effective default is `public`.

### No migrations run in a new tenant

Every tenant must have its own `<schema>.goose_db_version`. If a newly created tenant reports the public migration version but has no application tables, verify that the running binary includes schema-qualified goose table handling from `cmd/pentagi/main.go`.

### Constraint and scan errors

- Foreign-key errors usually mean the parent flow/task/subtask was not created or belongs to a different scoped path.
- `sql.ErrNoRows` is expected for missing user-scoped data and should normally be translated to a not-found/access-denied result by the service layer.
- Nullable columns use `sql.Null*` values in generated models; use helpers in `pkg/database/database.go` where they make call sites clearer.
- `SanitizeUTF8` must be applied to untrusted tool output before insert into PostgreSQL text fields.

## Development Checklist

### Adding or changing a query

1. Edit the appropriate file in `backend/sqlc/models`.
2. Keep user/admin scoping explicit in the query name and SQL.
3. Regenerate sqlc.
4. Review generated diffs; never hand-edit them.
5. Add tests at the service, converter or database boundary appropriate to the behavior.
6. Run:

```bash
cd backend
go test ./pkg/database/... ./pkg/server/... ./pkg/graph/...
go vet ./pkg/database/...
```

### Adding a table or field

1. Add a new goose migration.
2. Add explicit indexes for the actual foreign-key/query patterns.
3. Add or update sqlc queries.
4. Regenerate sqlc.
5. Update GraphQL/GORM converters only if the field crosses those boundaries.
6. Test fresh install, upgrade and tenant-schema startup.

### Review points

- Does every user-facing path enforce ownership?
- Is an unscoped/admin query intentionally authorized by its caller?
- Are soft-deleted rows filtered where expected?
- Does the migration work in a non-`public` tenant schema?
- Are shared extensions referenced through the configured search path?
- Does the change fit within the two-pool connection budget?
- Are generated files free of manual edits?

## Related Documentation

- [config.md](config.md) — database environment variables, tenancy, PgBouncer and Supavisor configuration
- [README.md](../../README.md) — deployment and development commands
- `backend/sqlc/sqlc.yml` — sqlc generation settings
- `backend/migrations/sql` — authoritative schema history
- `backend/pkg/database` — generated API and database helpers

---

### Backend Docs Dependency Advisories (`backend/docs/dependency_advisories.md`)

# Dependency advisories

`govulncheck ./...` is the source of truth here. Run it from `backend/`:

```bash
go run golang.org/x/vuln/cmd/govulncheck@latest ./...
```

It reports two classes separately: vulnerabilities your code **calls**, and ones
that merely sit in the module graph. Only the first class is worth acting on, and
even there the call graph is deliberately conservative — reaching any symbol of a
module is enough to attribute every advisory filed against that module, including
advisories in code the module ships but we never execute.

Everything with a published fix is taken. Fifteen findings stay, in two classes.

## Eight in a server we only talk to

`github.com/ollama/ollama` — GO-2025-4251, 3824, 3695, 3689, 3582, 3559, 3558 and
3557, none with a fixed version: missing authentication on model management,
cross-domain token exposure, denial of service via null dereference, divide by
zero, out-of-bounds read, unthrottled allocation. Every one of them describes the
**Ollama server**. We import `github.com/ollama/ollama/api` — its HTTP client —
and nothing else, and we never run `ollama serve`. The traces govulncheck prints
say as much: `api.AuthorizationError.Error`, i.e. formatting an error we
received, and `api.Client.Chat`, i.e. making a request. The vulnerable code lives
on whatever host serves the endpoint an operator configures.

## Seven the `go` line reports and the binary does not carry

GO-2026-6218, 6091, 6090, 6089, 6088, 5972 and 5026 are filed against `net/url`,
`html/template`, `crypto/tls`, `net/http`, `encoding/xml` and `encoding/asn1`,
all fixed in go1.26.6. govulncheck attributes standard-library advisories to the
version on the `go` line of `go.mod`, which states the floor the module supports
rather than the toolchain that compiles it. The image builds on `golang:1.26`;
`go version` run against the binary inside the built image prints a patch past
those fixes. Raising the `go` line silences all seven, and it also raises the
floor for everyone building from source, so it is a release decision and not a
security fix.

## What would change this

An advisory here gaining a `Fixed in` version, our code starting to run Ollama's
server side in-process, or the `go` line moving. Re-run the command above after
any dependency work and compare the module list, not just the count: a new module
appearing is the signal.

---

### Backend Docs Docker (`backend/docs/docker.md`)

# Docker Client Package Documentation

## Table of Contents

- [Overview](#overview)
- [Architecture](#architecture)
- [Configuration](#configuration)
  - [Worker Docker Access](#worker-docker-access)
- [Core Interfaces](#core-interfaces)
- [Container Lifecycle Management](#container-lifecycle-management)
- [Security and Isolation](#security-and-isolation)
- [Integration with PentAGI](#integration-with-pentagi)
- [Usage Examples](#usage-examples)
- [Error Handling](#error-handling)
- [Best Practices](#best-practices)

## Overview

The Docker client package (`backend/pkg/docker`) provides a secure and isolated containerized environment for PentAGI's AI agents to execute penetration testing operations. This package serves as a wrapper around the official Docker SDK, offering specialized functionality for managing containers that AI agents use to perform security testing tasks.

### Key Features

- **Secure Isolation**: All operations are performed in sandboxed Docker containers with complete isolation
- **AI Agent Integration**: Specifically designed to support AI agent workflows and terminal operations
- **Container Lifecycle Management**: Comprehensive container creation, execution, and cleanup
- **Port Management**: Automatic port allocation for flow-specific containers
- **File Operations**: Safe file transfer, path metadata lookup, and non-recursive directory listing between host and containers
- **Network Isolation**: Configurable network policies for security
- **Resource Management**: Memory and CPU limits for controlled execution
- **Volume Management**: Persistent and temporary storage solutions

### Role in PentAGI Ecosystem

The Docker client is a critical component that enables PentAGI's core promise of secure, isolated penetration testing. It provides the foundation for:

- **Terminal Access**: AI agents execute commands in isolated environments
- **Tool Execution**: Professional pentesting tools run in dedicated containers
- **File Management**: Secure file operations and artifact storage
- **Environment Preparation**: Dynamic container setup based on task requirements
- **Resource Cleanup**: Automatic cleanup of completed or failed operations

## Architecture

### Core Components

The Docker client package consists of several key components:

```
backend/pkg/docker/
├── client.go          # Main Docker client implementation
└── (future files)     # Additional Docker utilities
```

### Key Constants and Configuration

```go
const WorkFolderPathInContainer = "/work"   // Standard working directory in containers
const WorkerVolumeNameSuffix   = "-data"    // Suffix of the per-flow named volume
const BaseContainerPortsNumber = 28000      // Default base for dynamic port allocation

const defaultImage              = "debian:latest"          // Fallback image if custom image fails
const defaultDockerSocketPath   = "/var/run/docker.sock"   // Mount point of a bound socket
const containerPortsNumber      = 2                        // Number of ports allocated per container
const limitContainerPortsNumber = 2000                     // Port window size per instance
const containerListWorkers      = 20                       // Parallel stat workers for directory listing
```

### Port Allocation Strategy

PentAGI uses a deterministic port allocation algorithm to ensure each flow gets unique, predictable ports:

```go
func GetPrimaryContainerPorts(portsBase int, flowID int64) []int {
    if portsBase <= 0 || portsBase > (65535-limitContainerPortsNumber) {
        portsBase = BaseContainerPortsNumber
    }
    ports := make([]int, containerPortsNumber)
    for i := range containerPortsNumber {
        delta := (int(flowID)*containerPortsNumber + i) % limitContainerPortsNumber
        ports[i] = portsBase + delta
    }
    return ports
}
```

This ensures that:
- Each flow gets consistent port numbers across restarts
- Port conflicts are avoided between different flows
- Ports stay inside a 2000-wide window starting at `DOCKER_PORTS_BASE` (default `28000`, so `28000-30000`)

The base is configurable because flow ids restart at `1` in every PentAGI instance: two instances sharing one worker node would otherwise request identical host ports for their respective flow `1`. Give each instance a disjoint window (`28000`, `30000`, …). An out-of-range base silently falls back to `28000` rather than producing unbindable ports.

## Configuration

### Environment Variables

The Docker client is configured through several environment variables defined in the main configuration:

| Variable | Default | Description |
|----------|---------|-------------|
| `DOCKER_HOST` | `unix:///var/run/docker.sock` | Docker daemon connection |
| `DOCKER_INSIDE` | `false` | Whether PentAGI communicates with host Docker daemon from containers |
| `DOCKER_NET_ADMIN` | `false` | Whether PentAGI grants the primary container NET_ADMIN capability for advanced networking. |
| `DOCKER_SOCKET` | | Explicit socket path on the worker node to bind into worker containers. Empty enables autodetection only for local legacy setups — see [Worker Docker Access](#worker-docker-access) |
| `DOCKER_INSIDE_HOST` | | Separate Docker daemon endpoint given **to worker containers**; requires a successful policy preflight before agent use |
| `DOCKER_INSIDE_TLS_VERIFY` | | TLS verification for the worker container's Docker connection |
| `DOCKER_INSIDE_CERT_PATH` | | TLS certificate directory **on the worker node**, mounted read-only into worker containers |
| `DOCKER_INSIDE_POLICY_TESTS` | `false` | Test the sandbox once at startup and log the verdict. Its only effect is whether agents get Docker: a sandbox that fails is turned off, and PentAGI still starts. Off by default, and then startup warns the isolation was never measured |
| `DOCKER_DEFAULT_IMAGE_FOR_TEST` | `vxcontrol/kali-linux:test` | Worker the startup test runs in, and the image it pulls into the sandbox to prove an agent could |
| `DOCKER_PORTS_BASE` | `28000` | First host port of this instance's per-flow allocation window |
| `DOCKER_NETWORK` | | Docker network for container communication (bridge mode) or `host` for host network mode |
| `DOCKER_PUBLIC_IP` | `0.0.0.0` | Public IP for port binding (bridge mode only) |
| `DOCKER_WORK_DIR` | | Custom work directory path on host |
| `DOCKER_DEFAULT_IMAGE` | `debian:latest` | Fallback image if AI-selected image fails |
| `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` | `vxcontrol/kali-linux` | Default Docker image for penetration testing tasks |
| `DATA_DIR` | `./data` | Local data directory for file operations |

### Configuration Structure

```go
type Config struct {
    // Docker (terminal) settings
    DockerInside   bool   `env:"DOCKER_INSIDE" envDefault:"false"`
    DockerNetAdmin bool   `env:"DOCKER_NET_ADMIN" envDefault:"false"`
    DockerSocket   string `env:"DOCKER_SOCKET"`

    // How a worker container reaches a Docker daemon (see Worker Docker Access)
    DockerInsideHost      string `env:"DOCKER_INSIDE_HOST"`
    DockerInsideTLSVerify string `env:"DOCKER_INSIDE_TLS_VERIFY"`
    DockerInsideCertPath  string `env:"DOCKER_INSIDE_CERT_PATH"`

    DockerNetwork                string `env:"DOCKER_NETWORK"`
    DockerPublicIP               string `env:"DOCKER_PUBLIC_IP" envDefault:"0.0.0.0"`
    DockerWorkDir                string `env:"DOCKER_WORK_DIR"`
    DockerPortsBase              int    `env:"DOCKER_PORTS_BASE" envDefault:"28000"`
    DockerDefaultImage           string `env:"DOCKER_DEFAULT_IMAGE" envDefault:"debian:latest"`
    DockerDefaultImageForPentest string `env:"DOCKER_DEFAULT_IMAGE_FOR_PENTEST" envDefault:"vxcontrol/kali-linux"`
    DataDir                      string `env:"DATA_DIR" envDefault:"./data"`
}
```

The three `DOCKER_INSIDE_*` values are consumed through helpers on `*config.Config` (`WorkerDockerSocket`, `WorkerDockerEnv`, `WorkerDockerCertPath`) so the decision logic lives in one place and the Docker client stays a thin consumer.

### NET_ADMIN Capability Configuration

The `DOCKER_NET_ADMIN` option controls whether PentAGI containers are granted the `NET_ADMIN` Linux capability, which provides advanced networking permissions essential for many penetration testing operations.

#### Network Administration Capabilities

When `DOCKER_NET_ADMIN=true`, containers receive the following networking capabilities:

- **Network Interface Management**: Create, modify, and delete network interfaces
- **Routing Control**: Manipulate routing tables and network routes
- **Firewall Rules**: Configure iptables, netfilter, and other firewall systems
- **Traffic Shaping**: Implement QoS (Quality of Service) and bandwidth controls
- **Bridge Operations**: Create and manage network bridges
- **VLAN Configuration**: Set up and modify VLAN configurations
- **Packet Capture**: Enhanced access to raw sockets and packet capture mechanisms

#### Security Implications

**Enabling NET_ADMIN (`DOCKER_NET_ADMIN=true`)**:
- **Benefits**: Enables full-featured network penetration testing tools
- **Risks**: Containers can potentially modify host network configuration
- **Use Cases**: Network scanning, traffic interception, custom routing setups
- **Tools Enabled**: Advanced nmap features, tcpdump, wireshark, custom networking tools

**Disabling NET_ADMIN (`DOCKER_NET_ADMIN=false`)**:
- **Benefits**: Enhanced security isolation from host networking
- **Limitations**: Some advanced networking tools may not function fully (nmap)
- **Use Cases**: Application-level testing, web security assessment
- **Recommended**: For environments where network-level testing is not required

#### Container Capability Assignment

The primary container does not rely on Docker's implicit default capability set. `tools.go` (`flowToolsExecutor.Prepare`) sets `CapDrop: ["ALL"]` and then adds back an explicit allow-list — Docker's own default 14-capability set minus `MKNOD`, plus `NET_ADMIN` when `DOCKER_NET_ADMIN=true`:

```go
// Primary containers (when DOCKER_NET_ADMIN=false)
hostConfig := &container.HostConfig{
    CapDrop: []string{"ALL"},
    CapAdd: []string{
        "CHOWN", "DAC_OVERRIDE", "FSETID", "FOWNER",
        "NET_RAW", "SETGID", "SETUID", "SETFCAP", "SETPCAP",
        "NET_BIND_SERVICE", "SYS_CHROOT", "KILL", "AUDIT_WRITE", "SYS_PTRACE",
    },
}

// Primary containers (when DOCKER_NET_ADMIN=true) — same list, plus NET_ADMIN
hostConfig := &container.HostConfig{
    CapDrop: []string{"ALL"},
    CapAdd: []string{
        "CHOWN", "DAC_OVERRIDE", "FSETID", "FOWNER",
        "NET_RAW", "SETGID", "SETUID", "SETFCAP", "SETPCAP",
        "NET_BIND_SERVICE", "SYS_CHROOT", "KILL", "AUDIT_WRITE", "SYS_PTRACE",
        "NET_ADMIN",
    },
}
```

Why the full default set (minus one) instead of just `NET_RAW`/`NET_ADMIN`: pentest workflows routinely install new tools at runtime via `apt`/`dpkg` (the Installer Agent's core job), and several common network tools' `postinst` maintainer scripts call `setcap` on their binaries instead of relying on setuid (`ping`, `traceroute`, `nmap`, `dumpcap`, `hping3`, …). That needs `SETFCAP`; `SETPCAP`/`FSETID`/`AUDIT_WRITE` round out the rest of Docker's default set that ordinary package management and privilege-dropping daemons expect. See [Capability Management](#capability-management) below for the full rationale, including the one deliberate omission (`MKNOD`).

### Worker Docker Access

Two independent questions are often confused, and PentAGI answers them with two separate sets of variables:

| Question | Variables | Consumer |
|---|---|---|
| Which daemon does **PentAGI** create worker containers on? | `DOCKER_HOST`, `DOCKER_TLS_VERIFY`, `DOCKER_CERT_PATH` | Docker SDK inside the PentAGI process |
| Which daemon may a **worker container** talk to? | `DOCKER_INSIDE`, `DOCKER_SOCKET`, `DOCKER_INSIDE_HOST`, `DOCKER_INSIDE_TLS_VERIFY`, `DOCKER_INSIDE_CERT_PATH` | The agent's sandbox |

`DOCKER_INSIDE` is the master switch for the second row. With it disabled a sandbox gets no Docker access and no Docker configuration at all.

#### Socket selection algorithm

With `DOCKER_INSIDE=true`, `NewDockerClient` resolves the socket once at startup:

```go
socket, autodetectSocket := cfg.WorkerDockerSocket()
if autodetectSocket {
    socket = getHostDockerSocket(ctx, cli)
}
```

| `DOCKER_SOCKET` | `DOCKER_INSIDE_HOST` | Result |
|---|---|---|
| set | empty | That socket is bind-mounted at `/var/run/docker.sock` inside every worker (legacy mode) |
| set | set | Startup error: the socket could bypass the designated daemon |
| empty | set | **Nothing is mounted**; autodetection is skipped and the worker must pass the Docker API preflight |
| empty | empty | Host socket is autodetected only when PentAGI uses a local daemon; a remote `DOCKER_HOST` causes a startup error |

#### Environment injection

Every non-empty `DOCKER_INSIDE_*` value is injected into the worker container with the `_INSIDE_` segment removed, so the Docker CLI inside picks it up unmodified:

| Configured on PentAGI | Seen inside the worker container |
|---|---|
| `DOCKER_INSIDE_HOST=tcp://10.0.0.5:3376` | `DOCKER_HOST=tcp://10.0.0.5:3376` |
| `DOCKER_INSIDE_TLS_VERIFY=1` | `DOCKER_TLS_VERIFY=1` |
| `DOCKER_INSIDE_CERT_PATH=/etc/docker/dind-client` | `DOCKER_CERT_PATH=/etc/docker/dind-client` |

Empty values are omitted rather than injected blank. When `DOCKER_INSIDE_CERT_PATH` is set, that directory is additionally bind-mounted **read-only at the identical path**, so the injected `DOCKER_CERT_PATH` resolves unchanged inside the container. The path is resolved on the **worker node** — the machine whose daemon creates sandboxes — which is frequently not the machine running PentAGI.

```go
if dc.inside {
    if dc.socket != "" {
        hostConfig.Binds = append(hostConfig.Binds, dc.socket+":"+defaultDockerSocketPath)
    }
    config.Env = append(config.Env, dc.insideEnv...)
    if dc.insideCertPath != "" {
        hostConfig.Binds = append(hostConfig.Binds, dc.insideCertPath+":"+dc.insideCertPath+":ro")
    }
}
```

#### Why the TCP endpoint is preferred over a bound socket

Bind-mounting a socket into worker containers has two distinct problems.

**Ordering fragility.** A bind-mount source that does not exist yet is created by Docker as a **directory**. If the worker node reboots and a worker container with `restart: on-failure` starts before the dind daemon has recreated its socket, Docker materialises a directory at `/var/run/docker-dind/docker.sock` — and dind then cannot bind its own socket at that path. The sandbox gets a useless mount and dind fails to start until the directory is removed by hand.

**Blast radius.** The mounted socket may belong to the isolated dind daemon or to the worker node's host daemon. The legacy socket path does not run the policy preflight or prove which daemon it reaches. If it is the host daemon's socket, an autonomous agent can start a privileged container, mount `/`, and take over the node — including PentAGI itself and every other flow's containers. Inspect the actual mount source and daemon ID before relying on a socket deployment.

Pointing sandboxes at a hardened dind daemon over TLS avoids both. There is no mount to race on, and the authorization policy on that daemon constrains what the agent may create. See [Worker Node Setup](../../examples/guides/worker_node.md) for a complete configuration.

#### Docker API policy preflight

When `DOCKER_INSIDE=true` and `DOCKER_INSIDE_HOST` names a separate daemon, each new worker checks that daemon **from inside the worker** before PentAGI returns it to an agent. A restored worker is checked before reuse. Successful checks are cached by worker container ID for 30 seconds, so a later preparation rechecks a changed policy. The worker image must contain `sh` and `curl`; the check uses the worker's actual `DOCKER_HOST` and TLS certificate mount, so the certificate files need not exist on the main PentAGI node. Legacy socket mode remains supported as an unverified exception: the preflight does not establish its daemon identity or policy, and an incorrectly chosen socket could grant host Docker access.

The check reads Docker `/version` and `/info`, and rejects an empty daemon ID or one equal to the daemon PentAGI uses to create workers. It then sends fifteen `POST /containers/create` requests drawn from [the full policy test suite](../../examples/guides/worker_node/policy-tests.sh): privileged mode, legacy and structured host bind mounts, `SYS_ADMIN`, device passthrough and device cgroup rules, host PID and IPC namespaces, unconfined seccomp, sysctl injection, `ReadonlyPaths:[]`, `MaskedPaths:[]`, `CgroupParent:"/"`, the combined Phase 7 request, and an API v1.24 downgrade attempt. Each supported request must receive HTTP 403 with an explicit `opa-docker-authz` authorization denial; a Docker response saying API v1.24 is below the daemon's minimum is also safe for the downgrade attempt. A generic HTTP 403, an image-not-found error, or an unreachable endpoint fails the preflight.

The requests use a valid image name with a random tag that is overwhelmingly unlikely to exist, never pull an image, and never call `ContainerStart`. One ordinary `containers/create` request must pass authorization and reach Docker's image lookup; an image-not-found response shows that the policy has not blocked all agent use. If a request is unexpectedly accepted, the preflight deletes the container by its unique probe name, checks that it is absent, and fails; failure to delete is also reported. If the Docker API times out after accepting a request, cleanup can only be best effort, so operators should inspect `pentagi-policy-probe-*` objects after a timeout. A newly launched worker that fails the preflight is discarded. This quick check tests the selected API authorization rules and daemon separation at preparation time. It does not exercise seccomp or kernel enforcement, network reachability from nested containers, or every rule in the full suite; run `policy-tests.sh` separately after changing the worker-node policy.

### Network Configuration

PentAGI supports two network modes for container isolation:

#### Bridge Network Mode (Default)

When `DOCKER_NETWORK` is set to a custom network name (e.g., `pentagi-network`), containers are connected to an isolated bridge network:
- **Isolated Communication**: Containers communicate only within the defined network
- **Port Mapping**: Container ports are mapped to host ports for external access
- **Service Discovery**: Enables internal DNS-based service discovery
- **Enhanced Security**: Network-level isolation from other containers

#### Host Network Mode

When `DOCKER_NETWORK` is set to the special value `host`, containers use the host's network stack directly:
- **Direct Network Access**: Container shares the host's network interfaces
- **No Port Mapping**: Ports are directly accessible on host interfaces (no NAT)
- **Performance**: Eliminates network virtualization overhead
- **Use Cases**: Advanced network testing, raw packet manipulation, network monitoring

**Security Consideration**: Host network mode reduces isolation. Use only when necessary for penetration testing tasks requiring direct host network access.

## Core Interfaces

### DockerClient Interface

The main interface defines all Docker operations available to PentAGI components:

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Implementation Structure

```go
type dockerClient struct {
    db        database.Querier    // Database for container state management
    logger    *logrus.Logger      // Structured logging
    dataDir   string              // Local data directory
    hostDir   string              // Host-mapped data directory
    client    *client.Client      // Docker SDK client
    inside    bool                // Worker containers may reach a Docker daemon
    defImage  string              // Default fallback image
    socket    string              // Socket to bind into workers ("" = none)
    network   string              // Docker network name
    publicIP  string              // Public IP for port binding
    portsBase int                 // First host port of this instance's window
    labels    map[string]string   // Tenant ownership labels (nil without a tenant)

    insideEnv      []string       // DOCKER_* injected into worker containers
    insideCertPath string         // TLS dir mounted read-only into workers
}
```

## Container Lifecycle Management

### Container Creation Process

The `RunContainer` method handles the complete container creation workflow:

1. **Preparation**:
   - Creates flow-specific work directory
   - Generates unique container name
   - Records container in database with "starting" status

2. **Image Management**:
   - Attempts to pull requested image
   - Falls back to default image if pull fails
   - Updates database with actual image used

3. **Container Configuration**:
   - Sets hostname based on container name hash
   - Configures working directory to `/work`
   - Sets up restart policy (`on-failure`, maximum 5 retries)
   - Configures logging (JSON driver with rotation)

4. **Storage Setup**:
   - Creates dedicated volume or bind mount
   - Mounts work directory to `/work` in container
   - When `DOCKER_INSIDE=true`: optionally mounts a Docker socket, injects the `DOCKER_*` environment derived from `DOCKER_INSIDE_*`, and mounts the TLS certificate directory read-only — see [Worker Docker Access](#worker-docker-access)

5. **Network and Ports**:
   - **Bridge Mode**: Assigns flow-specific ports using deterministic algorithm, binds to public IP
   - **Host Mode** (`DOCKER_NETWORK=host`): Uses host network stack, skips port bindings
   - Connects to specified Docker network (unless host mode)

6. **Container Startup**:
   - Creates container with all configurations
   - Starts container
   - When a separate worker Docker daemon is configured, checks its API policy and daemon identity from inside the worker; discards the worker on failure
   - Updates database status to "running"

### Example Container Configuration

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Container States and Transitions

PentAGI tracks container states in the database:

- **`Starting`**: Container creation in progress
- **`Running`**: Container is active and available
- **`Stopped`**: Container has been stopped but not removed
- **`Failed`**: Container creation or startup failed
- **`Deleted`**: Container has been removed

### Container Naming Convention

Containers follow a specific naming pattern for easy identification:

```go
func PrimaryTerminalName(tenantPrefix string, flowID int64) string {
    return fmt.Sprintf("%s%s%d", tenantPrefix, PrimaryTerminalNamePrefix, flowID)
}
```

This creates names like `pentagi-terminal-123` for flow ID 123, making it easy to:
- Identify containers belonging to specific flows
- Perform flow-based cleanup operations
- Debug container-related issues

When `TENANT_ID` is set the tenant leads the name — `acme-pentagi-terminal-123` — so that several PentAGI instances can share one worker node. The prefix goes in front deliberately: sweeps that match `pentagi-terminal-*` then reach only their own instance's containers. The per-flow volume (`<container name>-data`) and the container hostname derive from this name, so both inherit the scoping. Containers and volumes additionally carry a `pentagi.tenant` label for filter-based cleanup.

### Cleanup Operations

The `Cleanup` method performs comprehensive cleanup:

1. **Flow State Assessment**:
   - Identifies flows that should be terminated
   - Marks incomplete flows as failed
   - Preserves running flows that should continue

2. **Container Cleanup**:
   - Stops all containers for terminated flows
   - Removes stopped containers and their volumes
   - Updates database to reflect current state

3. **Parallel Processing**:
   - Uses goroutines for concurrent container deletion
   - Ensures cleanup doesn't block system operation

## Security and Isolation

### Container Security Model

PentAGI implements a multi-layered security approach for container isolation:

#### Network Isolation
- **Custom Networks**: Containers run in dedicated Docker networks
- **Port Control**: Only specific ports are exposed to the host
- **Host Protection**: Container cannot access host network by default

#### File System Isolation
- **Read-Only Root**: Base container filesystem is immutable
- **Controlled Mounts**: Only specific directories are writable
- **Volume Separation**: Each flow gets isolated storage space

#### Capability Management

The primary container uses an explicit allow-list instead of Docker's implicit defaults: `CapDrop: ["ALL"]`, then `CapAdd` back Docker's own default 14-capability set minus `MKNOD`, plus `NET_ADMIN` when `DOCKER_NET_ADMIN=true` and `SYS_PTRACE` (one deliberate addition beyond Docker's defaults, see below):

```go
hostConfig := &container.HostConfig{
    CapDrop: []string{"ALL"},
    CapAdd: []string{
        "CHOWN", "DAC_OVERRIDE", "FSETID", "FOWNER",
        "NET_RAW", "SETGID", "SETUID", "SETFCAP", "SETPCAP",
        "NET_BIND_SERVICE", "SYS_CHROOT", "KILL", "AUDIT_WRITE", "SYS_PTRACE",
        // + "NET_ADMIN" when DOCKER_NET_ADMIN=true
    },
}
```

Why this exact set, rather than a minimal `NET_RAW`-only list:

| Capability | Why it is needed |
|---|---|
| `NET_RAW` | Raw sockets — nmap, ping, packet crafting |
| `NET_BIND_SERVICE` | Bind ports below 1024 — reverse shells, Responder, rogue DNS |
| `SETUID` / `SETGID` | Daemons and tools that drop privileges after starting as root |
| `SETFCAP` / `SETPCAP` | `apt`/`dpkg` `postinst` scripts that `setcap` network tools instead of relying on setuid (`ping`, `traceroute`, `nmap`, `dumpcap`, `hping3`, …) — without these, on-the-fly package installs (the Installer Agent's core job) fail |
| `FSETID` | Preserves set-id bits when dpkg installs/modifies files as a non-owner |
| `CHOWN` / `DAC_OVERRIDE` / `FOWNER` | Root file-permission overrides needed during package installs and builds |
| `KILL` | Signal other processes inside the container |
| `SYS_CHROOT` | chroot-based isolation within the sandbox |
| `AUDIT_WRITE` | Lets `sudo`/`sshd` write audit-log entries instead of warning |
| `SYS_PTRACE` | Not a Docker default — added so `gdb`/`strace`/`ltrace`/dynamic binary analysis (`pwndbg`, `radare2`) work for the Coder Agent's exploit-development role. Without it, `ptrace()` and friends (`process_vm_readv`/`writev`, `kcmp`) stay blocked by Docker's *default seccomp profile*, which independently gates them behind `CAP_SYS_PTRACE` — no custom seccomp profile is needed to unblock them, since moby/containerd auto-extend the default profile's syscall allow-list to match added capabilities. Scope stays contained to the sandbox: `ptrace` only works within the container's own PID namespace, never against host or sibling-container processes. |
| `NET_ADMIN` (opt-in via `DOCKER_NET_ADMIN`) | Interface/routing/firewall control for advanced network pentesting |

**`MKNOD` is the one deliberate omission** from Docker's default set: creating device nodes has no legitimate use for pentest tooling or package management, and this repository's own dind-hardening research (see [Worker Node Setup](../../examples/guides/worker_node.md) and its [`authz.rego`](../../examples/guides/worker_node/authz.rego)) identifies block-device `mknod` combined with `debugfs` as "the primary confirmed escape vector" for a hostile-code container. `SYS_ADMIN`, `SYS_MODULE`, `SYS_RAWIO`, and `SYS_BOOT` are never granted — none are part of Docker's default set and none are required by any supported workflow.

Docker's default set minus `MKNOD`, plus `NET_ADMIN`, is exactly the `allowed_caps` whitelist already vetted in `authz.rego` for nested dind containers running the same kind of pentest workload — the two are kept intentionally consistent at that shared baseline. `SYS_PTRACE` is the one place the primary container's allow-list goes further than `authz.rego`'s: it is not offered to nested dind containers (an agent there can request arbitrary `containers/create` calls, and the dind threat model does not special-case debugging), but the primary worker container is created solely by PentAGI itself with a fixed capability list, so granting it here does not expand what an agent can ask for.

An earlier revision of this container also forced `no-new-privileges:true` via `SecurityOpt`, intended as defense-in-depth against setuid/file-capability escalation. It was removed: the capability bounding set above already caps what any process can ever gain regardless of setuid, so the flag added no protection beyond the allow-list while unconditionally breaking SUID/SGID privilege-escalation testing and `sudo`/`su` from a non-root shell — both routine penetration-testing workflows.

#### Resource Limits
- **PidsLimit**: Defaults to 2048 when the caller does not set one — a cheap fork-bomb / resource-exhaustion guard, generous enough for parallel scans (nmap, hydra). Mirrors the same default used for the dind daemon in the [Worker Node Setup](../../examples/guides/worker_node.md) guide.
- **Memory/CPU**: Controlled via standard `HostConfig` resource fields when set by the caller.

#### Process Isolation
- **User Namespaces**: Containers run with isolated user space
- **PID Isolation**: Container processes are isolated from host
- **Resource Limits**: Memory and CPU usage are controlled

### Security Best Practices Implemented

1. **Image Validation**: All images are pulled and verified before use
2. **Fallback Strategy**: Safe default image used if custom image fails
3. **State Tracking**: All container operations are logged and monitored
4. **Automatic Cleanup**: Failed or abandoned containers are automatically removed
5. **Socket Security**: Docker socket is only mounted when explicitly required

## Integration with PentAGI

### Tool Integration

The Docker client integrates with PentAGI's tool system to provide terminal access:

```go
type terminal struct {
    flowID       int64
    containerID  int64
    containerLID string
    dockerClient docker.DockerClient
    tlp          TermLogProvider
}
```

The terminal tool uses the Docker client for:
- **Command Execution**: Running shell commands in isolated containers
- **File Operations**: Reading and writing files safely
- **Result Capture**: Collecting command output and artifacts

### Flow File Integration

Flow files are managed by the REST API in `pkg/server/services/flow_files.go` and use Docker client file APIs for synchronization with the running primary container.

PentAGI keeps two different storage areas for flow files:

- **Local cache**: `{DATA_DIR}/flow-{id}-data/uploads` and `{DATA_DIR}/flow-{id}-data/container`
- **Container workspace**: `/work` inside the primary container

This separation is intentional. It supports both single-node deployments and remote worker-node deployments where the backend host filesystem is not the same filesystem used by Docker workers.

The current behavior is:

- User uploads are saved to the local cache under `uploads/`.
- If the primary container is running, uploaded files are pushed best-effort to `/work/uploads`.
- When the primary container starts or is reused, cached uploads are synchronized into `/work/uploads`; the cache is the source of truth.
- Files pulled from the container are stored under `container/` using their normalized full container path, for example:
  - `/etc/nginx/nginx.conf` -> `container/etc/nginx/nginx.conf`
  - `/work/test.md` -> `container/work/test.md`
- Deleting cached upload files is allowed even when the container is not running. The next container start will resynchronize `/work/uploads` from cache.

The flow files API also exposes a non-recursive live container directory listing endpoint. It uses `ContainerStatPath` to determine whether the requested path is a file or directory:

- If the path is a file, it returns that file metadata directly.
- If the path is a directory, it calls `ListContainerDir`.
- If the path is omitted, it defaults to `/work`.

### Provider Integration

The provider system uses Docker client for environment preparation:

```go
// In providers.go
type flowProvider struct {
    // ... other fields
    docker    docker.DockerClient
    publicIP  string
}
```

Providers use the Docker client to:
- **Image Selection**: AI agents choose appropriate container images
- **Environment Setup**: Prepare containers for specific tasks
- **Resource Management**: Allocate and deallocate containers as needed

### Database Integration

Container states are persisted in the PostgreSQL database:

```sql
-- Container state tracking
CREATE TABLE containers (
    id SERIAL PRIMARY KEY,
    flow_id INTEGER REFERENCES flows(id),
    name VARCHAR NOT NULL,
    image VARCHAR NOT NULL,
    status container_status NOT NULL,
    local_id VARCHAR,
    local_dir VARCHAR,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);
```

### Observability Integration

All Docker operations are instrumented with:
- **Structured Logging**: JSON logs with context and metadata
- **Error Tracking**: Comprehensive error capture and reporting
- **Performance Metrics**: Container creation and execution timing
- **Resource Monitoring**: CPU, memory, and network usage tracking

## Usage Examples

### Basic Container Creation

```go
// Initialize Docker client
dockerClient, err := docker.NewDockerClient(ctx, db, cfg)
if err != nil {
    return fmt.Errorf("failed to create docker client: %w", err)
}

// Create container for a flow
containerName := tools.PrimaryTerminalName(cfg.TenantPrefix(), flowID)
container, err := dockerClient.RunContainer(
    ctx,
    containerName,
    database.ContainerTypePrimary,
    flowID,
    &container.Config{
        Image:      "kali:latest",
        Entrypoint: []string{"tail", "-f", "/dev/null"},
    },
    &container.HostConfig{
        // See Container Capability Assignment above for the full allow-list
        // that flowToolsExecutor.Prepare actually passes in production.
        CapDrop: []string{"ALL"},
        CapAdd: []string{
            "CHOWN", "DAC_OVERRIDE", "FSETID", "FOWNER",
            "NET_RAW", "SETGID", "SETUID", "SETFCAP", "SETPCAP",
            "NET_BIND_SERVICE", "SYS_CHROOT", "KILL", "AUDIT_WRITE", "SYS_PTRACE",
            "NET_ADMIN",
        },
    },
)
```

### Command Execution

```go
// Execute command in container
createResp, err := dockerClient.ContainerExecCreate(ctx, containerName, container.ExecOptions{
    Cmd:          []string{"sh", "-c", "nmap -sS 192.168.1.1"},
    AttachStdout: true,
    AttachStderr: true,
    WorkingDir:   "/work",
    Tty:          true,
})

// Attach to execution
resp, err := dockerClient.ContainerExecAttach(ctx, createResp.ID, container.ExecAttachOptions{
    Tty: true,
})

// Read output
output, err := io.ReadAll(resp.Reader)
```

### File Operations

```go
// Write file to container
content := "#!/bin/bash\necho 'Hello from container'"
archive := createTarArchive("script.sh", content)
err := dockerClient.CopyToContainer(ctx, containerID, "/work", archive, container.CopyToContainerOptions{})

// Read file from container
reader, stats, err := dockerClient.CopyFromContainer(ctx, containerID, "/work/results.txt")
defer reader.Close()

// Extract content from tar
content := extractFromTar(reader)

// Stat a file or directory in the container
stat, err := dockerClient.ContainerStatPath(ctx, containerID, "/work/results.txt")

// List direct entries in a container directory
entries, err := dockerClient.ListContainerDir(ctx, containerID, "/work")
```

### Container Directory Listing

`ListContainerDir` performs a non-recursive directory listing inside a running container:

1. Uses `ContainerStatPath` to verify that `dirPath` exists and is a directory.
2. Executes `find <dirPath> -maxdepth 1 -mindepth 1 ! -name '.*' -print0` (no TTY) inside the container. `find -print0` emits literal, NUL-delimited entry paths, so names with spaces, newlines, or non-UTF8 bytes survive intact — the old `ls -1` parse mangled them.
3. Stats every entry concurrently through an `errgroup` bounded to `containerListWorkers = 20`, preserving input order.

The method returns a `ContainerDirListing`: `Files` are the readable entries' `container.PathStat` metadata, `Failures` carries any per-entry stat errors so a live directory degrades to a partial listing instead of failing outright, and `Truncated` is set when the directory held more than the entry cap and only the first page was listed.

If `dirPath` is empty, it defaults to `WorkFolderPathInContainer` (`/work`).

### Cleanup and Resource Management

```go
// Check if container is running
isRunning, err := dockerClient.IsContainerRunning(ctx, containerID)

// Stop container
err = dockerClient.StopContainer(ctx, containerID, dbID)

// Remove container and volumes
err = dockerClient.RemoveContainer(ctx, containerID, dbID)

// Global cleanup (usually called on startup)
err = dockerClient.Cleanup(ctx)
```

### Error Handling

```go
// The client implements comprehensive error handling
container, err := dockerClient.RunContainer(ctx, name, containerType, flowID, config, hostConfig)
if err != nil {
    // Errors include:
    // - Image pull failures (handled with fallback)
    // - Container creation failures
    // - Network configuration issues
    // - Database update failures

    // The client automatically:
    // - Updates database with failure status
    // - Cleans up partially created resources
    // - Logs detailed error information

    return fmt.Errorf("container creation failed: %w", err)
}
```

## Error Handling

### Error Categories

The Docker client handles several categories of errors:

1. **Docker Daemon Errors**:
   - Connection failures to Docker daemon
   - API version mismatches
   - Permission issues

2. **Image-Related Errors**:
   - Image pull failures (network, authentication)
   - Invalid image names or tags
   - Image compatibility issues

3. **Container Runtime Errors**:
   - Container creation failures
   - Container startup issues
   - Resource allocation problems

4. **Network and Storage Errors**:
   - Port binding conflicts
   - Volume mount failures
   - Network configuration issues

### Error Recovery Strategies

1. **Image Fallback**:
   ```go
   if err := dc.pullImage(ctx, config.Image); err != nil {
       logger.WithError(err).Warnf("failed to pull image '%s', using default", config.Image)
       config.Image = dc.defImage
       // Retry with default image
   }
   ```

2. **Container Cleanup**:
   ```go
   if containerCreationFails {
       defer updateContainerInfo(database.ContainerStatusFailed, containerID)
       // Clean up any partially created resources
   }
   ```

3. **State Synchronization**:
   - Database state always reflects actual container state
   - Failed operations are marked appropriately
   - Orphaned resources are cleaned up automatically

## Best Practices

### Resource Management
- Always use the `Cleanup()` method on application startup
- Monitor container resource usage through observability tools
- Set appropriate timeouts for long-running operations
- Use deterministic port allocation to avoid conflicts

### Security Considerations
- Regularly update base images used for containers
- Minimize capabilities granted to containers
- Use dedicated networks for container communication
- Monitor and audit all container operations

### Development and Debugging
- Use structured logging for all Docker operations
- Implement comprehensive error handling with context
- Test container operations in isolated environments
- Use the ftester utility for debugging specific operations

### Performance Optimization
- Reuse containers when possible instead of creating new ones
- Implement efficient cleanup to prevent resource leaks
- Use appropriate container restart policies
- Monitor container startup times and optimize configurations

### Integration Guidelines
- Always use the DockerClient interface instead of direct Docker SDK calls
- Integrate with PentAGI's database for state management
- Use the provided logging and observability infrastructure
- Follow the established naming conventions for containers

---

### Backend Docs Flow Execution (`backend/docs/flow_execution.md`)

# Flow Execution in PentAGI

This document describes the internal architecture and execution workflow of Flow in PentAGI, an autonomous penetration testing system that leverages AI agents to perform complex security testing workflows.

## 1. Core Concepts and Terminology

### Hierarchy
- **Flow** - Top-level workflow representing a complete penetration testing session (persistent)
- **Task** - User-defined objective within a Flow (multiple Tasks can exist in one Flow)
- **Subtask** - Auto-decomposed sequential step to complete a Task (generated and refined by system)
- **Action** - Individual operation performed by agents (commands, searches, analyses)

### Workers
- **FlowWorker** - Manages the complete lifecycle of a Flow, coordinates Tasks
- **TaskWorker** - Executes individual Tasks, manages Subtask generation and refinement
- **SubtaskWorker** - Handles execution of specific Subtasks via AI agents
- **AssistantWorker** - Manages interactive assistant mode within a Flow; optionally controls the Flow's automation via injected `FlowWorker` reference

### Providers
- **FlowProvider** - Core interface for Flow execution, agent coordination and orchestration
- **AssistantProvider** - Specialized provider for assistant mode interactions; receives `FlowWorker` via `SetFlowWorker()` after chain preparation to enable flow management tool callbacks
- **ProviderController** - Factory for creating and managing different LLM providers

### AI Agents
- **Primary Agent** - Main orchestrator that coordinates all other agents within a Subtask
- **Generator Agent** - Decomposes Tasks into ordered lists of Subtasks (max 15)
- **Refiner Agent** - Reviews and updates planned Subtask list after each Subtask completion (can add/remove/modify planned Subtasks)
- **Reporter Agent** - Creates comprehensive final reports for completed Tasks
- **Coder Agent** - Writes and maintains code for specific requirements
- **Pentester Agent** - Performs penetration testing and vulnerability assessment
- **Installer Agent** - Manages environment setup and tool installation
- **Memorist Agent** - Handles long-term memory storage and retrieval
- **Searcher Agent** - Conducts internet research and information gathering
- **Enricher Agent** - Enhances information from multiple sources
- **Adviser Agent** - Provides expert guidance and recommendations
- **Reflector Agent** - Corrects agents that return unstructured text instead of tool calls
- **Assistant Agent** - Provides interactive assistance, operates autonomously within Flow independently from Task/Subtask (UseAgents flag controls delegation); when FlowWorker is available, exposes `get_flow_status`, `stop_flow`, `submit_flow_input`, and `patch_flow_subtasks` tools for automation control

### Tools and Capabilities by Category
- **Environment Tools** - Terminal commands, file operations within Docker containers
  - `terminal` - Command execution (configurable via `TERMINAL_TOOL_TIMEOUT`, default: 1200s; hard limit: 3h/10800s; 0 or negative values are clamped to the hard limit)
  - `file` - Read/write operations with absolute path requirements
  - User-provided flow files are available under `/work/uploads` and `/work/resources`
  
- **Search Network Tools** - External information sources
  - `browser` - Web scraping with screenshot capture
  - `web_search` - Unified web search. The agent supplies a `query` and an intent `mode`
    (`links`, `answer`, `research`, `exploit`); the orchestrator selects the underlying
    engine, retries transient failures, and falls back across providers automatically.
    Engines orchestrated under the hood (when configured): Google, DuckDuckGo, Searxng,
    Tavily, Traversaal, Perplexity, Firecrawl, Sploitus, and the optional internal
    browser-analytics engine (`WEB_SEARCH_INTERNAL_ENABLED`, off by default). The
    per-mode fallback order is a single data table at the top of
    `backend/pkg/tools/web_search.go`; the engine primitives live in
    `backend/pkg/tools/searchers/`.
  
- **Vector Database Tools** - Semantic search in long-term memory
  - `search_in_memory` - General execution memory search
  - `search_guide` / `store_guide` - Installation guides (doc_type: guide)
  - `search_answer` / `store_answer` - Q&A pairs (doc_type: answer)
  - `search_code` / `store_code` - Code samples (doc_type: code)
  
- **Agent Tools** - Delegation to specialist agents
  - `search`, `maintenance`, `coder`, `pentester`, `advice`, `memorist`
  
- **Result Storage Tools** - Agent result delivery
  - `maintenance_result`, `code_result`, `hack_result`, `memorist_result`
  - `search_result`, `enricher_result`, `report_result`
  - `subtask_list` (Generator), `subtask_patch` (Refiner)
  - Future signed evidence receipt design is outlined in [evidence_chain.md](../../examples/proposals/evidence_chain.md)

- **Barrier Tools** - Control flow termination  
  - `done` - Complete subtask, `ask` - Request user input (configurable via ASK_USER env)

- **Flow Management Tools** - Assistant-only; available when FlowWorker is injected into AssistantProvider
  - `get_flow_status` - Query flow state, tasks, subtasks, planned/running subtask with optional agent messages (verbose mode: 50 messages, execution context)
  - `stop_flow` - Cancel the currently running task (15s timeout; reports actual post-stop state)
  - `submit_flow_input` - Deliver text to a waiting flow: answers an `ask` checkpoint or creates a new task
  - `patch_flow_subtasks` - Replace planned subtask list via delta operations (add/remove/modify/reorder); returns new IDs after recreation

### Execution Context
- **Message Chain** - Conversation history maintained for each agent interaction
- **Execution Context** - Comprehensive state including completed/planned Subtasks
- **Docker Environment** - Isolated container for secure tool execution
- **Vector Store** - Long-term semantic memory for knowledge retention
- **Task Materials** - Optional user uploads and resources exposed to prompts via `{{.UserFiles}}`

### Performance Results
- **PerformResultDone** - Subtask completed successfully via `done` tool
- **PerformResultWaiting** - Subtask paused for user input via `ask` tool
- **PerformResultError** - Subtask failed due to unrecoverable errors

## 2. Main Flow Execution Process

```
/* Detailed source-code truncated for AI context efficiency. */
```

## 3. AI Agent Interactions and Capabilities

```
/* Detailed source-code truncated for AI context efficiency. */
```

## 4. Supporting Workflows

### Assistant Mode Workflow

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Vector Database (RAG) Integration

```mermaid
graph LR
    subgraph "Knowledge Storage Types"
        Guides[Installation Guides<br/>doc_type: guide<br/>guide_type: install/configure/use/etc]
        Answers[Q&A Pairs<br/>doc_type: answer<br/>answer_type: guide/vulnerability/code/tool/other]
        Code[Code Samples<br/>doc_type: code<br/>code_lang: python/bash/etc]
        Memory[Execution Memory<br/>doc_type: memory<br/>tool_name + results]
    end
    
    subgraph "Vector Operations (threshold: 0.2, limit: 3)"
        SearchOps[search_guide<br/>search_answer<br/>search_code<br/>search_in_memory]
        StoreOps[store_guide<br/>store_answer<br/>store_code<br/>auto-store from 18 tools]
    end
    
    subgraph "Auto-Storage Tools"
        EnvTools[terminal, file]
        SearchEngines[web_search]
        AgentTools[search, maintenance, coder,<br/>pentester, advice]
    end
    
    SearchOps --> Guides
    SearchOps --> Answers  
    SearchOps --> Code
    SearchOps --> Memory
    
    StoreOps --> Guides
    StoreOps --> Answers
    StoreOps --> Code
    StoreOps --> Memory
    
    EnvTools --> Memory
    SearchEngines --> Memory
    AgentTools --> Memory
    
    subgraph "Vector Database"
        PostgreSQL[(PostgreSQL + pgvector<br/>Similarity Search<br/>Metadata Filtering)]
    end
    
    SearchOps --> PostgreSQL
    StoreOps --> PostgreSQL
    Memory --> PostgreSQL
```

### Multi-Provider LLM Integration

```mermaid
graph TD
    PC[ProviderController] --> OpenAI[OpenAI Provider]
    PC --> Anthropic[Anthropic Provider]
    PC --> Gemini[Gemini Provider]
    PC --> Bedrock[AWS Bedrock Provider]
    PC --> DeepSeek[DeepSeek Provider]
    PC --> GLM[Zhipu AI Provider]
    PC --> Kimi[Moonshot AI Provider]
    PC --> Qwen[Alibaba Cloud DashScope Provider]
    PC --> Ollama[Ollama Provider]
    PC --> Custom[Custom Provider]
    
    subgraph "Agent Configurations"
        Simple[Simple Agent]
        JSON[Simple JSON Agent]
        Primary[Primary Agent]
        Assistant[Assistant Agent]
        Generator[Generator Agent]
        Refiner[Refiner Agent]
        Adviser[Adviser Agent]
        Reflector[Reflector Agent]
        Searcher[Searcher Agent]
        Enricher[Enricher Agent]
        Coder[Coder Agent]
        Installer[Installer Agent]
        Pentester[Pentester Agent]
    end
    
    OpenAI --> Simple
    OpenAI --> JSON
    OpenAI --> Primary
    OpenAI --> Assistant
    OpenAI --> Generator
    OpenAI --> Refiner
    OpenAI --> Adviser
    OpenAI --> Reflector
    OpenAI --> Searcher
    OpenAI --> Enricher
    OpenAI --> Coder
    OpenAI --> Installer
    OpenAI --> Pentester
    
    Note1[Each provider supports 13 agent types:<br/>Simple, SimpleJSON, PrimaryAgent, Assistant<br/>Generator, Refiner, Adviser, Reflector<br/>Searcher, Enricher, Coder, Installer, Pentester]
```

### Tool Execution and Context Management

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Comprehensive Logging Architecture

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Docker Container Management

```
/* Detailed source-code truncated for AI context efficiency. */
```

## 5. Complex Interaction Patterns

### Message Chain Management
Each AI agent interaction is managed through typed message chains that maintain conversation context:

**Chain Types by Agent**:
- `MsgchainTypePrimaryAgent` - Primary Agent orchestration chains
- `MsgchainTypeGenerator` - Subtask generation chains  
- `MsgchainTypeRefiner` - Subtask refinement chains
- `MsgchainTypeReporter` - Final report generation chains
- `MsgchainTypeCoder` - Code development chains
- `MsgchainTypePentester` - Security testing chains
- `MsgchainTypeInstaller` - Infrastructure maintenance chains
- `MsgchainTypeMemorist` - Memory operation chains
- `MsgchainTypeSearcher` - Information retrieval chains
- `MsgchainTypeAdviser` - Expert consultation chains
- `MsgchainTypeReflector` - Response correction chains
- `MsgchainTypeAssistant` - Interactive assistance chains
- `MsgchainTypeSummarizer` - Context summarization chains
- `MsgchainTypeToolCallFixer` - Tool argument repair chains

**Chain Properties**:
- **Serialized to JSON** and stored in the database for persistence
- **Summarized periodically** to prevent context window overflow
- **Restored on system restart** to maintain continuity
- **Type-specific retrieval** for agent-specific context loading

### Agent Context Tracking
The system maintains agent execution context through the call chain:

**Agent Context Structure**:
- **ParentAgentType** - The agent that initiated the current operation
- **CurrentAgentType** - The agent currently executing  

**Context Propagation**:
- Set via `PutAgentContext(ctx, agentType)` when invoking agents
- Retrieved via `GetAgentContext(ctx)` for logging and tracing
- Used for vector store logging to track agent delegation chains
- Enables observability of inter-agent communication patterns

**Message Chain Types** (tracks agent interactions):
- `MsgchainTypePrimaryAgent`, `MsgchainTypeGenerator`, `MsgchainTypeRefiner`
- `MsgchainTypeReporter`, `MsgchainTypeCoder`, `MsgchainTypePentester` 
- `MsgchainTypeInstaller`, `MsgchainTypeMemorist`, `MsgchainTypeSearcher`
- `MsgchainTypeAdviser`, `MsgchainTypeReflector`, `MsgchainTypeEnricher`
- `MsgchainTypeAssistant`, `MsgchainTypeSummarizer`, `MsgchainTypeToolCallFixer`

### Agent Chain Execution Loop
The Primary Agent follows a sophisticated execution pattern:

1. **Context Preparation** - Loads execution context including completed/planned Subtasks
2. **Tool Call Loop** - Iteratively calls LLM with available tools until completion
3. **Function Execution** - Executes tool calls with retry logic and error handling
4. **Repeating Detection** - Prevents infinite loops by detecting repeated tool calls (threshold: 3)
5. **Reflection Mechanism** - If no tools are called, invokes Reflector Agent for guidance
6. **Barrier Functions** - Special tools (`done`, `ask`) that control execution flow

### Reflector Agent Correction Mechanism
A critical system component that handles agent errors:

- **Triggers when** - Any agent returns unstructured text instead of structured tool calls
- **Maximum iterations** - Limited to 3 reflector calls per chain to prevent loops
- **Response style** - Acts as the user providing direct, concise guidance
- **Correction process** - Analyzes the unstructured response and guides agent to proper tool usage
- **Barrier tool emphasis** - Specifically reminds agents about completion tools (`done`, `ask`)
- **Assistant exception** - Assistant agents return natural text responses to users, not tool calls
- **Final response mode** - Assistants use completion mode for user-facing communication
- **Context isolation** - Assistants use `nil` taskID/subtaskID when accessing agent handlers
- **Cross-flow operation** - Assistants can access Flow-level context without specific Task/Subtask binding

### Subtask Lifecycle States
Subtasks progress through well-defined states:
- **Created** - Initially generated by Generator Agent
- **Running** - Currently being executed by Primary Agent
- **Waiting** - Paused for user input via `ask` tool
- **Finished** - Successfully completed via `done` tool
- **Failed** - Terminated due to errors

### Error Handling and Recovery
The system implements comprehensive error handling:

**Multi-layer Error Correction**:
- **Tool Call Retries** - Failed LLM calls retried up to 3 times with 5-second delays
- **Tool Call Fixer** - Invalid JSON arguments automatically corrected using schema validation
- **Reflector Correction** - Unstructured responses redirected to proper tool call format
- **Chain Consistency** - Adds default responses to incomplete tool calls after interruptions
- **Repeating Detection** - Prevents infinite loops by limiting identical tool calls to 3 attempts

**Chain Consistency Mechanism**:
- **Triggered on errors** - System interruptions or context cancellations
- **Fallback responses** - Adds default content for unresponded tool calls
- **Message integrity** - Maintains valid conversation structure
- **AST-based processing** - Uses Chain AST for structured message analysis

**System Resilience**:
- **Graceful Degradation** - Falls back to simpler operations when complex ones fail
- **Context Preservation** - Maintains state across system restarts
- **Container cleanup** - Automatic resource cleanup on failures
- **Flow status management** - Proper state transitions on errors

### Task Refinement Process
The Refiner Agent uses a sophisticated analysis approach:
1. **Reviews completed Subtasks** and their execution results
2. **Analyzes remaining planned Subtasks** for relevance
3. **Considers overall Task progress** and user requirements
4. **Updates the Subtask plan** by removing obsolete tasks and adding necessary ones
5. **Maintains execution efficiency** by limiting total Subtasks to 15 maximum
6. **Dynamic limit calculation** - Available slots = 15 minus completed Subtasks count
7. **Completion detection** - Returns empty list when Task objectives are achieved

### Memory and Knowledge Management
The system maintains multiple types of persistent knowledge with PostgreSQL + pgvector:

**Vector Store Types**:
- **Memory Storage** (`doc_type: memory`) - Tool execution results and agent observations
- **Guide Storage** (`doc_type: guide`) - Installation and configuration procedures
- **Answer Storage** (`doc_type: answer`) - Q&A pairs for common scenarios  
- **Code Storage** (`doc_type: code`) - Programming language-specific code samples

**Lifecycle Guidance**:
- Treat `memory` as flow-scoped execution history. It is most useful for understanding what happened in a specific engagement and is commonly inspected with a `flow_id` filter.
- Treat `guide`, `answer`, and `code` as reusable knowledge. These document types exist to preserve durable procedures, reusable target notes, Q&A material, and code snippets across future runs.
- If you want a later flow to begin with known context, store the confirmed result intentionally through `store_guide`, `store_answer`, or `store_code` instead of assuming execution history alone will provide the right reusable context.
- Current prompt templates already distinguish these roles: reusable guides, answers, and code live in vector documents, while Graphiti is intended for episodic memory about what actually happened during execution.

**Technical Parameters**:
- **Similarity Threshold**: 0.2 for all vector searches
- **Result Limits**: 3 documents maximum per search
- **Memory Storage Tools**: 18 tools automatically store results (terminal, file, all search engines, all agent tools)
- **Summarization Eligible**: Only `terminal` and `browser` tools results are auto-summarized when > 16KB

### Search Tool Priority System
The Searcher Agent follows a strict hierarchy for information retrieval:

1. **Priority 1-2: Memory Tools** - Always check internal knowledge first
   - `search_answer` - Primary tool for accessing existing knowledge
   - `memorist` - Retrieves task/subtask execution history

2. **Priority 3: `web_search`** - All external web search, via one unified tool
   - The agent supplies a `query` and an intent `mode`; it does NOT choose a provider.
   - **Modes**: `links` (fast link discovery), `answer` (synthesized answer, default),
     `research` (deep multi-source synthesis), `exploit` (exploits/PoCs/offensive tools).

3. **Priority 4: `browser`** - Targeted content extraction from a specific, known URL

**web_search orchestration**: For the chosen mode, the orchestrator walks a per-mode
fallback chain, skipping engines that are not configured, retrying only *retryable*
failures (HTTP 429/5xx/transport) on the same engine, and moving on immediately for
*fatal* ones (HTTP 4xx/decode). It returns an error to the agent only after every
available engine is exhausted, and it records one search-log entry attributed to the
engine that actually served the result. The fallback table is the single source of
truth at the top of `backend/pkg/tools/web_search.go`.

**Engines orchestrated** (used only when configured):
- **Google** - Custom Search API with CX key and language restrictions (links)
- **DuckDuckGo** - Anonymous search (links)
- **Searxng** - Privacy meta search aggregating multiple engines (links)
- **Firecrawl** - Web search with main-content markdown scraped from each result
- **Tavily** - Research-grade exploration with raw content and citations (answer/research)
- **Traversaal** - Structured Q&A responses with web links (answer/research)
- **Perplexity** - AI-powered synthesis with configurable context size (answer/research)
- **Sploitus** - Security exploits and pentest tools (exploit)
- **Internal Analytics Engine** - optional, off by default (`WEB_SEARCH_INTERNAL_ENABLED`);
  answers analytic queries by scraping and summarizing pages via the browser, bounded to
  a configurable byte budget per site

**Action Economy Rules**: Maximum 3-5 search actions per query, stop immediately when sufficient information is found

### Summarization Protocol
A critical system-wide mechanism for context management:

- **Two Summary Types**:
  1. **Tool Call Summary** - AI message with only `SummarizationToolName` tool call
  2. **Prefixed Summary** - AI message starting with `SummarizedContentPrefix`
  
- **Agent Handling Rules**:
  - Must treat summaries as **historical records** of actual past events
  - Extract useful information to inform current strategy
  - **Never mimic** summary formats or use summarization tools
  - Continue using structured tool calls for all actions
  
- **System Benefits**:
  - Prevents context window overflow during long conversations
  - Maintains conversation coherence across system restarts
  - Preserves critical execution context while reducing token usage

### Real-time Communication System

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Multi-tenancy and Security
The architecture supports multiple users with isolation:
- **User-specific providers** - Each user can configure their own LLM providers
- **Flow isolation** - Docker containers provide security boundaries
- **Resource management** - Containers are automatically cleaned up
- **Access control** - Database queries are user-scoped

### Specialized System Prompts
The system uses 25+ dedicated prompt types for specific functions:

**System Function Prompts**:
- **Image Chooser** - Selects optimal Docker image, fallback to `vxcontrol/kali-linux` for pentest
- **Language Chooser** - Detects user's preferred language for responses  
- **Flow Descriptor** - Generates concise Flow titles (max 20 characters)
- **Task Descriptor** - Creates descriptive Task titles (max 150 characters)  
- **Tool Call Fixer** - Repairs invalid JSON arguments using schema validation
- **Execution Context** - Templates for full/short context summaries
- **Execution Logs** - Formats chronological action histories for summarization

**Agent System Prompts** (13 types):
- **Primary Agent** - Team orchestration with delegation capabilities
- **Assistant** - Interactive mode with UseAgents flag configuration
- **Specialist Agents** - Pentester, Coder, Installer, Searcher, Memorist, Adviser
- **Meta Agents** - Generator, Refiner, Reporter, Enricher, Reflector, Summarizer

**Question Templates** (13 types):
- Structured input templates for each agent's human interaction patterns
- Context-specific variable injection for Task/Subtask/Flow information
- Formatted data presentation for optimal agent comprehension

**Critical Prompt Features**:
- **XML Semantic Delimiters** - Structured sections like `<memory_protocol>`, `<terminal_protocol>`
- **Summarization Awareness** - Universal protocol for handling historical summaries  
- **Tool Placeholder System** - `{{.ToolPlaceholder}}` injection at prompt end
- **Task Materials Section** - `{{.UserFiles}}` lists user uploads/resources when present
- **Template Variable System** - 50+ variables for dynamic content injection

### Performance Optimizations and Limits
Several mechanisms ensure efficient execution:

**Execution Limits**:
- **Subtask Limits** - Maximum 15 Subtasks per Task (TasksNumberLimit)
- **Refiner Calculations** - Available slots = 15 minus completed Subtasks count
- **Reflector Iterations** - Maximum 3 corrections per agent chain
- **Tool Call Retries** - Maximum 3 attempts for failed executions
- **Repeating Detection** - Blocks repeated tool calls after 3 attempts
- **Search Action Economy** - Searcher limited to 3-5 actions per query

**Timeout Configuration**:
- **Terminal Operations** - Configurable via `TERMINAL_TOOL_TIMEOUT` (default: 1200s); hard limit 3h/10800s; 0 or negative values clamped to hard limit
- **LLM API Calls** - 3 retries with 5-second delays between attempts  
- **Vector Search** - Threshold 0.2, max 3 results per query
- **Tool Result Summarization** - Triggered at 16KB result size
- **Flow Input Processing** - 1 second timeout for input queueing
- **Assistant Input** - 2 second timeout for assistant input queueing
- **Flow Management Operations** - 15 second timeout for `stop_flow` and `submit_flow_input` handlers

**Container Resource Management**:
- **Port Allocation** - 2 ports per Flow starting from base 28000
- **Volume Management** - Per-flow data directories with cleanup
- **File Sync** - Incremental sync of `uploads/` and `resources/` into `/work/uploads` and `/work/resources`
- **Network Isolation** - Optional custom Docker networks
- **Image Fallback** - Automatic fallback to default Debian image

**Memory Optimization**:
- **Message summarization** - Prevents context window overflow
- **Summarizer result cache** - LRU cache per FlowProvider (1000 entries, 4h TTL); keyed by SHA-256 of input; skips LLM call for repeated identical content
- **Tool argument limits** - 1KB limit for individual argument values  
- **Connection pooling** - Database connections are reused
- **Automatic cleanup** - Containers removed after Flow completion

### User Interaction Flow (`ask` Tool)
Critical mechanism for human-in-the-loop operations:

**Ask User Workflow**:
1. **Primary Agent** calls `ask` tool with question for user
2. **PerformResultWaiting** returned to SubtaskWorker
3. **Subtask status** set to `SubtaskStatusWaiting`  
4. **Task status** propagated to `TaskStatusWaiting`
5. **Flow status** propagated to `FlowStatusWaiting`
6. **User provides input** via Flow interface
7. **Input processed** through `updateMsgChainResult` with `AskUserToolName`
8. **Subtask continues** execution with user's response

**Configuration**:
- **ASK_USER environment variable** - Controls availability of `ask` tool
- **Default**: false (disabled by default)
- **Primary Agent only** - Only available in PrimaryExecutor configuration
- **Barrier function** - Causes execution flow pause until user responds

### Browser and Screenshot System
The browser tool provides advanced web interaction capabilities:

**Browser Actions**:
- **Markdown Extraction** - Clean text content from web pages  
- **HTML Content** - Raw HTML for detailed analysis
- **Link Extraction** - Collect all URLs from pages for further navigation

**Screenshot Integration**:
- **Automatic Screenshots** - Every browser action captures page screenshot
- **Dual Scraper Support** - Private URL scraper for internal networks, public for external
- **Screenshot Storage** - Organized by Flow ID with timestamp naming
- **Minimum Content Sizes** - MD: 50 bytes, HTML: 300 bytes, Images: 2048 bytes

**Network Resolution**:
- **IP Analysis** - Automatic detection of private vs public targets
- **Scraper Selection** - Private scraper for internal IPs, public for external
- **Security Isolation** - Web scraping isolated from main execution container

## Advanced Agent Supervision

PentAGI implements a sophisticated multi-layered agent supervision system to ensure efficient task execution, prevent infinite loops, and provide intelligent recovery from stuck states.

### Execution Monitoring System

**ExecutionMonitorDetector** continuously monitors agent tool call patterns and automatically invokes the Adviser agent for progress reviews:

**Trigger Conditions**:
- **Same Tool Threshold**: Triggered after 5 consecutive calls to the same tool (configurable via `EXECUTION_MONITOR_SAME_TOOL_LIMIT`)
- **Total Tool Threshold**: Triggered after 10 total tool calls regardless of variety (configurable via `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT`)
- **Reset Behavior**: Counters reset after adviser intervention or when different tools are used

**Monitoring Process**:
1. **Pattern Detection**: `execToolCall` method checks detector before executing each tool
2. **Context Collection**: Gathers recent messages, executed tool calls, subtask description, and agent prompt
3. **Mentor Invocation**: Calls `performMentor` with comprehensive execution context
4. **Enhanced Response**: Mentor analysis is formatted as `<mentor_analysis>` alongside `<original_result>`
5. **Counter Reset**: Monitor state resets after successful intervention

**Mentor Analysis Provides**:
- **Progress Assessment**: Evaluation of whether agent is advancing toward subtask objective
- **Issue Identification**: Detection of loops, inefficiencies, or incorrect approaches
- **Alternative Strategies**: Recommendations for different approaches when current strategy fails
- **Information Retrieval Guidance**: Suggestions to search for established solutions instead of reinventing
- **Termination Guidance**: Clear indication if task is impossible or should be completed with completion function call

**Configuration**:
- `EXECUTION_MONITOR_ENABLED` (default: false) - Enable/disable automatic monitoring
- `EXECUTION_MONITOR_SAME_TOOL_LIMIT` (default: 5) - Consecutive same-tool threshold
- `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT` (default: 10) - Total tool calls threshold

For local or custom Qwen-style deployments where flows run for hours or repeat commands, see the [vLLM Qwen3.5-27B-FP8 troubleshooting guide](../../examples/guides/vllm-qwen35-27b-fp8.md#issue-automation-flow-runs-for-hours-or-repeats-commands) for prompt-bounding and log-review guidance.

### Enhanced Reflector Integration

**Automatic Reflector on Generation Failures**:

When LLM fails to generate valid tool calls after 3 attempts in `callWithRetries`, the system now automatically invokes the Reflector agent instead of failing:

**Invocation Process**:
1. **Failure Detection**: `callWithRetries` reaches `maxRetriesToCallAgentChain` (3 attempts)
2. **Context Preparation**: Builds reflector message describing all failed attempts and errors
3. **Reflector Call**: Invokes `performReflector` to analyze situation and provide guidance
4. **Recovery Options**: Reflector guides agent to either:
   - Fix the issue with specific corrective instructions
   - Use barrier tool to report completion or request assistance

**Benefits**:
- Prevents premature task termination due to transient LLM issues
- Provides contextual guidance based on specific failure patterns
- Maintains conversation flow rather than hard errors
- Enables graceful degradation and adaptive recovery

### Hard Limit Graceful Termination

**Max Tool Calls Per Agent Execution**:

To prevent runaway executions, each agent has a hard limit on tool calls. The limit varies by agent type to balance capabilities with efficiency:

**Agent Types and Limits**:
- **General Agents** (Assistant, Primary Agent, Pentester, Coder, Installer):
  - Default: 100 tool calls
  - Configurable via `MAX_GENERAL_AGENT_TOOL_CALLS`
  - Designed for complex, multi-step workflows requiring extensive tool usage
  
- **Limited Agents** (Searcher, Enricher, Memorist, Generator, Reporter, Adviser, Reflector, Planner):
  - Default: 20 tool calls
  - Configurable via `MAX_LIMITED_AGENT_TOOL_CALLS`
  - Designed for focused, specific tasks with limited scope

**Termination Process**:
1. **Limit Check**: Before each `callWithRetries` in `performAgentChain`, system checks `iteration` against agent-specific limit
2. **Reflector Invocation**: When approaching limit (within 3 iterations), reflector is called with termination context
3. **Graceful Completion**: Reflector guides agent to use barrier tool (`done` or `ask`) to:
   - Report successful completion if objective was achieved
   - Report partial progress with clear blocker explanation
   - Request user assistance if critical information is missing
4. **Forced Exit**: After reflector guidance, execution terminates gracefully

**Configuration**:
- `MAX_GENERAL_AGENT_TOOL_CALLS` (default: 100) - Maximum tool calls for general agents before forced termination
- `MAX_LIMITED_AGENT_TOOL_CALLS` (default: 20) - Maximum tool calls for limited agents before forced termination

**Why Differentiated Limits**:
- **Resource Efficiency**: Limited agents handle focused tasks and don't require extensive iteration
- **Task Complexity**: General agents need more autonomy for complex penetration testing, coding, and installation workflows
- **System Stability**: Prevents resource exhaustion while maintaining necessary capabilities for each agent type

### Intelligent Task Planning (Planner)

**Planner-Generated Execution Plans**:

When specialist agents (Pentester, Coder, Installer) are invoked, the Planner (adviser in planning mode) optionally generates a structured execution plan before task execution:

**Planning Process**:
1. **Context Analysis**: Planner analyzes full execution context via enricher agent
2. **Plan Generation**: Creates 3-7 specific, actionable steps via `PromptTypeQuestionTaskPlanner` template
3. **Scope Limitation**: Ensures plan focuses only on current subtask objective
4. **Plan Wrapping**: Original task question is wrapped in `<task_assignment>` structure with plan
5. **Agent Execution**: Specialist receives both original request and decomposed execution plan

**Plan Structure**:
```xml
<task_assignment>
  <original_request>[Original task from delegating agent]</original_request>
  <execution_plan>
  1. [First critical action/verification]
  2. [Second step with specific details]
  ...
  </execution_plan>
  <instructions>
  Follow the execution plan above to complete this task efficiently.
  You may deviate from the plan if you discover better approaches.
  </instructions>
</task_assignment>
```

**Benefits**:
- **Prevents scope creep**: Keeps agents focused on current subtask only
- **Reduces redundancy**: Leverages enriched context to avoid duplicate work
- **Improves success rate**: Breaks complex tasks into manageable steps
- **Provides guardrails**: Highlights potential pitfalls and verification points

**Configuration**:
- `AGENT_PLANNING_STEP_ENABLED` (default: false) - Enable/disable automatic task planning

### Mentor Supervision Protocol

All agents with adviser handler access (Primary, Pentester, Coder, Installer, Assistant) now include explicit awareness of mentor supervision in their system prompts:

**Enhanced Response Format**:
Agents are instructed to expect tool responses containing both:
- `<original_result>`: Actual tool execution output
- `<mentor_analysis>`: Mentor's evaluation with progress assessment, identified issues, alternative approaches, information retrieval suggestions, and next steps

**Agent Instructions**:
- Agents must read and integrate BOTH sections into decision-making
- Mentor analysis should guide next actions when provided
- Agents can explicitly request advice via `advice` tool
- Automatic mentor reviews occur at configured thresholds (not revealed to agents)

### Supervision System Integration

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Implementation Details

**Key Components**:
- `executionMonitorDetector` struct in `helpers.go` - Tracks tool call patterns
- `performMentor` method in `performer.go` - Coordinates mentor invocation for execution monitoring
- `performPlanner` method in `performers.go` - Generates execution plans via adviser
- `formatEnhancedToolResponse` function in `helpers.go` - Formats mentor analysis
- `flowStatusTool`, `stopFlowTool`, `submitFlowInputTool`, `patchFlowSubtasksTool` in `tools/flow_manager.go` - Flow management tool implementations
- `buildFlowManagerHandlers()` in `providers/assistant.go` - Builds handler closures from injected `FlowWorker`
- `patchAssistantFlowSubtasks()` in `providers/assistant.go` - Applies subtask delta operations and recreates DB entries
- Summarizer LRU cache on `flowProvider` (`providers/provider.go`) - 1000 entries, 4h TTL, SHA-256 key
- Template `question_execution_monitor.tmpl` - Question format for execution monitoring
- Template `question_task_planner.tmpl` - Question format for task planning

**Modified Methods**:
- `execToolCall`: Integrated execution monitor checks before tool execution
- `callWithRetries`: Calls reflector on max retries instead of returning error
- `performAgentChain`: Checks hard limit and invokes reflector for graceful termination
- `performPentester/Coder/Installer`: Apply task planning before execution

**Execution Limits Updated**:
- **Repeating Tool Threshold**: 3 identical calls (existing)
- **Execution Monitor Same Tool**: 5 identical calls (new)
- **Execution Monitor Total Tools**: 10 total calls (new)
- **Max Retries per Call**: 3 attempts (existing)
- **Max Retries per Chain**: 3 attempts → Reflector invocation (modified)
- **Max Tool Calls per Subtask**: 100 calls → Reflector termination (new)
- **Max Reflector Iterations**: 3 per chain (existing)
- **Max Agent Chain Iterations**: 100 total (existing)

## Summary

The Flow execution system represents a sophisticated orchestration platform that combines multiple AI agents, tools, and infrastructure components to deliver autonomous penetration testing capabilities. Key architectural highlights include:

### System Robustness
- **Triple-layer error handling** - Tool call fixing, reflector correction, and retry mechanisms
- **Context preservation** - Typed message chains with summarization for long-running operations
- **Resource limits** - Bounded execution with configurable timeouts and iteration limits
- **Isolation guarantees** - Docker containers provide security boundaries for all operations

### Agent Specialization
- **Role-based delegation** - Each agent has specific expertise and tool access
- **Memory-first approach** - All agents check vector storage before external operations  
- **Structured communication** - Exclusive use of tool calls except Assistant final responses
- **Adaptive planning** - Generator creates initial plans, Refiner optimizes based on results
- **Context propagation** - Parent/Current agent tracking for delegation chains
- **Categorized tools** - 6 tool categories with specific access patterns

### Operational Flexibility
- **Multi-provider LLM support** - Different models optimized for different agent types
- **Assistant dual modes** - UseAgents flag enables delegation or direct tool access; FlowWorker injection enables 4 flow management tools for automation control
- **Flow continuity** - System can resume operations after interruptions
- **Real-time feedback** - Streaming responses provide immediate user visibility
- **Vector knowledge system** - 4 storage types with semantic search and metadata filtering
- **Comprehensive tool ecosystem** - 44+ tools across 7 categories with automatic memory storage
- **GraphQL subscriptions** - Real-time Flow/Task/Log updates via WebSocket connections
- **Logging architecture** - 8-layer logging system with Controller/Worker pattern

### Critical Technical Details

**Assistant Streaming Architecture**:
- **Stream Cache**: LRU cache (1000 entries, 2h TTL) maps StreamID → MessageID
- **Stream Workers**: Background goroutines with 30-second timeout per stream
- **Buffer Management**: Separate buffers for thinking/content with controlled updates
- **Real-time Distribution**: Immediate GraphQL subscription updates to frontend

**Assistant Flow Management Architecture**:
- **FlowWorker injection**: `SetFlowWorker(fw FlowWorker)` called after `getFlowProviderWorkers()` in `controller/assistant.go`
- **Handler closures**: `buildFlowManagerHandlers()` produces `StopFlow`, `SendFlowInput`, `PatchSubtasks` callbacks
- **Two-layer validation**: tool handlers (`tools/flow_manager.go`) perform primary checks; provider callbacks are safety nets
- **Subtask patching**: delta operations applied via `applySubtaskOperations()`; all planned subtask IDs are recreated; new IDs returned in tool result
- **State verification**: `stop_flow` re-queries `GetFlowTasks` after handler returns and reports actual state
- **Resource handoff**: assistant inputs with resource IDs delegate file copying to `FlowWorker.PutResources()`

**Message Chain Consistency**:
- **AST Processing**: Uses Chain Abstract Syntax Tree for structured message analysis
- **Fallback Content**: Unresponded tool calls get default responses via `cast.FallbackResponseContent`
- **Chain Restoration**: Intelligent restoration of conversation context after interruptions  
- **Summarization Integration**: Seamless integration with summarization when restoring chains

**Flow Publisher Integration**:
- **Centralized Updates**: Single publisher coordinates all Flow-related real-time updates
- **Event Types**: 10 different event types (Flow, Task, Agent logs, Message logs, Tool Call logs, etc.)
- **User-scoped**: Each user gets their own publisher instance for proper isolation
- **WebSocket Distribution**: Efficient real-time delivery to frontend clients

### Implementation Architecture Summary

**User Files and Resources**:
- **User Resources** - Per-user blob storage with metadata in `user_resources`
- **Flow Files** - Per-flow filesystem cache under `flow-{id}-data/{uploads,container,resources}`
- **Container Availability** - `FlowToolsExecutor.Prepare()` incrementally syncs missing `uploads/` and `resources/` files into `/work`
- **Prompt Visibility** - Agents receive a compact `<task_files>` listing only when user files are present

**Core Flow Processing**:
- **3-layer hierarchy**: FlowWorker → TaskWorker → SubtaskWorker with proper lifecycle management
- **Agent orchestration**: 13 specialized agent types with role-specific tool access
- **Tool ecosystem**: 44+ tools across 7 categories (Environment, SearchNetwork, SearchVectorDb, Agent, StoreAgentResult, Barrier, FlowManagement)
- **Message chain types**: 14 distinct chain types for agent communication tracking

**Error Resilience & Recovery**:
- **4-level error handling**: Tool retries → Tool call fixing → Reflector correction → Chain consistency
- **Bounded execution**: Configurable limits preventing runaway operations
- **State recovery**: Complete system state restoration after interruptions
- **Graceful degradation**: Automatic fallbacks to simpler operational modes

**Real-time & Observability**:
- **8-layer logging**: Comprehensive tracking from agent interactions to terminal commands
- **GraphQL subscriptions**: Real-time updates via WebSocket connections
- **Streaming architecture**: Progressive response delivery with thinking/content separation
- **Vector observability**: Complete tracking of knowledge storage and retrieval operations

**Security & Isolation**:
- **Container isolation**: Docker-based security boundaries with capability controls
- **Multi-tenant design**: User-scoped operations with resource isolation  
- **Network segmentation**: Separate containers for web scraping vs. tool execution
- **Resource limits**: Comprehensive timeout and resource management

This architecture enables autonomous security testing while maintaining human oversight, technical precision, and operational security throughout the entire penetration testing workflow.

---

### Backend Docs Gemini (`backend/docs/gemini.md`)

# Google AI (Gemini) Provider

The Google AI provider enables PentAGI to use Google's Gemini language models through the Generative AI API. This provider supports advanced features like function calling, streaming responses, and competitive pricing.

## Features

- **Multi-model Support**: Access to Gemini 2.5 Flash, Gemini 2.5 Pro, and other Google AI models
- **Function Calling**: Full support for tool usage and function calls
- **Streaming Responses**: Real-time response streaming for better user experience
- **Competitive Pricing**: Cost-effective inference with transparent pricing
- **Proxy Support**: HTTP proxy support for enterprise environments
- **Advanced Configuration**: Fine-tuned parameters for different agent types

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `GEMINI_API_KEY` | - | Your Google AI API key (required) |
| `GEMINI_SERVER_URL` | `https://generativelanguage.googleapis.com` | Google AI API base URL |

### Getting API Key

1. Visit [Google AI Studio](https://aistudio.google.com/app/apikey)
2. Create a new API key
3. Set it as `GEMINI_API_KEY` environment variable

## Available Models

| Model | Context Window | Max Output | Input Price* | Output Price* | Best For |
|-------|----------------|------------|--------------|---------------|----------|
| gemini-2.5-flash | 1M tokens | 65K tokens | $0.15 | $0.60 | General tasks, fast responses |
| gemini-2.5-pro | 1M tokens | 65K tokens | $2.50 | $10.00 | Complex reasoning, analysis |
| gemini-2.0-flash | 1M tokens | 8K tokens | $0.15 | $0.60 | High-frequency tasks |
| gemini-1.5-flash | 1M tokens | 8K tokens | $0.075 | $0.30 | Legacy model (deprecated) |
| gemini-1.5-pro | 2M tokens | 8K tokens | $1.25 | $5.00 | Legacy model (deprecated) |

*Prices per 1M tokens (USD)

## Agent Configuration

Each agent type is optimized with specific parameters for Google AI models:

### Basic Agents
- **Simple**: General-purpose tasks with balanced settings
- **Simple JSON**: Structured output generation with JSON formatting
- **Primary Agent**: Core reasoning with moderate creativity
- **Assistant (A)**: User interaction with contextual responses

### Specialized Agents
- **Generator**: Creative content with higher temperature
- **Refiner**: Content improvement with focused parameters
- **Adviser**: Strategic guidance with extended context
- **Reflector**: Analysis and evaluation tasks
- **Searcher**: Information retrieval with precise settings
- **Enricher**: Data enhancement and augmentation
- **Coder**: Programming tasks with minimal temperature
- **Installer**: System setup with deterministic responses
- **Pentester**: Security testing with balanced creativity

## Usage Examples

### Basic Setup

```bash
# Set environment variables
export GEMINI_API_KEY="your_api_key_here"
export GEMINI_SERVER_URL="https://generativelanguage.googleapis.com"

# Test the provider
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  vxcontrol/pentagi /opt/pentagi/bin/ctester -type gemini
```

### Custom Configuration

```yaml
# gemini-custom.yml
simple:
  model: "gemini-2.5-pro"
  temperature: 0.3
  top_p: 0.4
  max_tokens: 8000
  price:
    input: 2.50
    output: 10.00

coder:
  model: "gemini-2.5-flash"
  temperature: 0.05
  top_p: 0.1
  max_tokens: 16000
  price:
    input: 0.15
    output: 0.60
```

### Docker Usage

```bash
# Using pre-configured Gemini provider
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  vxcontrol/pentagi /opt/pentagi/bin/ctester \
  -config /opt/pentagi/conf/gemini.provider.yml

# Using custom configuration
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  -v $(pwd)/gemini-custom.yml:/opt/pentagi/gemini-custom.yml \
  vxcontrol/pentagi /opt/pentagi/bin/ctester \
  -type gemini \
  -config /opt/pentagi/gemini-custom.yml
```

## Integration with PentAGI

### Environment File (.env)

```bash
# Google AI Configuration
GEMINI_API_KEY=your_api_key_here
GEMINI_SERVER_URL=https://generativelanguage.googleapis.com

# Optional: Proxy settings
PROXY_URL=http://your-proxy:port
```

### Provider Selection

The Google AI provider is automatically available when `GEMINI_API_KEY` is set. You can use it for:

- **Flow Execution**: Autonomous penetration testing workflows
- **Assistant Mode**: Interactive chat and analysis
- **Custom Tasks**: Specialized security assessments
- **API Integration**: Programmatic access to Google AI models

## Best Practices

### Model Selection
- Use **gemini-2.5-flash** for general tasks and fast responses
- Use **gemini-2.5-pro** for complex reasoning and detailed analysis
- Avoid deprecated models (1.5 series) for new projects

### Performance Optimization
- Set appropriate `max_tokens` limits based on your use case
- Use lower `temperature` values for deterministic tasks
- Configure `top_p` to balance creativity and consistency

### Cost Management
- Monitor token usage through PentAGI's cost tracking
- Use cheaper models for simple tasks
- Implement request batching where possible

### Security Considerations
- Store API keys securely (environment variables, secrets management)
- Use HTTPS for all API communications
- Implement rate limiting to prevent abuse
- Monitor API usage and costs regularly

## Troubleshooting

### Common Issues

1. **API Key Issues**
   ```
   Error: failed to create gemini provider: invalid API key
   ```
   - Verify your API key is correct
   - Check API key permissions in Google AI Studio
   - Ensure the key hasn't expired

2. **Model Not Found**
   ```
   Error: model "gemini-x.x-xxx" not found
   ```
   - Use supported model names from the table above
   - Check for typos in model names
   - Verify model availability in your region

3. **Rate Limiting**
   ```
   Error: quota exceeded
   ```
   - Implement exponential backoff
   - Reduce request frequency
   - Check your quota limits in Google AI Studio

4. **Network Issues**
   ```
   Error: connection timeout
   ```
   - Check internet connectivity
   - Verify proxy settings if applicable
   - Check firewall rules for outbound HTTPS

### Testing Provider

```bash
# Test basic functionality
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  vxcontrol/pentagi /opt/pentagi/bin/ctester \
  -type gemini \
  -agent simple \
  -prompt "Hello, world!"

# Test JSON functionality
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  vxcontrol/pentagi /opt/pentagi/bin/ctester \
  -type gemini \
  -agent simple_json \
  -prompt "Generate a JSON object with name and age fields"

# Test all agents
docker run --rm \
  -v $(pwd)/.env:/opt/pentagi/.env \
  vxcontrol/pentagi /opt/pentagi/bin/ctester \
  -type gemini
```

## Support and Resources

- [Google AI Documentation](https://ai.google.dev/docs)
- [Gemini API Reference](https://ai.google.dev/api)
- [PentAGI Documentation](https://docs.pentagi.com)
- [Issue Tracker](https://github.com/vxcontrol/pentagi/issues)

For provider-specific issues, include:
- Provider type: `gemini`
- Model name used
- Configuration snippet (without API keys)
- Error messages and logs
- Environment details (Docker, OS, etc.)

---

### Backend Docs Installer (`backend/docs/installer.md`)

# PentAGI Installer Documentation

## Overview

The PentAGI installer provides a robust Terminal User Interface (TUI) for configuring the application. Built using the [Charm](https://charm.sh/) tech stack (bubbletea, lipgloss, bubbles), it implements modern responsive design patterns optimized for terminal environments.

## ⚠️ Development Constraints & TUI Workflow

### Core Workflow Principles

1. **Build-Only Development**: NEVER run TUI apps during development - breaks terminal session
2. **Test Cycle**: Build → Run separately → Return to development session
3. **Debug Output**: All debug MUST go to `logger.Log()` (writes to `log.json`) - never `fmt.Printf`
4. **Development Monitoring**: Use `tail -f log.json` in separate terminal

## Advanced Form Patterns

### **Boolean Field Pattern with Suggestions**

```go
func (m *FormModel) addBooleanField(key, title, description string, envVar loader.EnvVar) {
    input := textinput.New()
    input.Prompt = ""
    input.PlaceholderStyle = m.styles.FormPlaceholder
    input.ShowSuggestions = true
    input.SetSuggestions([]string{"true", "false"})  // Tab completion

    // Show default in placeholder
    if envVar.Default == "true" {
        input.Placeholder = "true (default)"
    } else {
        input.Placeholder = "false (default)"
    }

    // Set value only if actually present in environment
    if envVar.Value != "" && envVar.IsPresent() {
        input.SetValue(envVar.Value)
    }
}

// Tab completion handler
func (m *FormModel) completeSuggestion() {
    if m.focusedIndex < len(m.fields) {
        suggestion := m.fields[m.focusedIndex].Input.CurrentSuggestion()
        if suggestion != "" {
            m.fields[m.focusedIndex].Input.SetValue(suggestion)
            m.fields[m.focusedIndex].Input.CursorEnd()
            m.hasChanges = true
            m.updateFormContent()
        }
    }
}
```

### **Integer Field with Range Validation**

```go
func (m *FormModel) addIntegerField(key, title, description string, envVar loader.EnvVar, min, max int) {
    input := textinput.New()
    input.Prompt = ""
    input.PlaceholderStyle = m.styles.FormPlaceholder

    // Parse and format default value
    defaultValue := 0
    if envVar.Default != "" {
        if val, err := strconv.Atoi(envVar.Default); err == nil {
            defaultValue = val
        }
    }

    // Human-readable placeholder with default
    input.Placeholder = fmt.Sprintf("%s (%s default)",
        m.formatNumber(defaultValue), m.formatBytes(defaultValue))

    // Add validation range to description
    fullDescription := fmt.Sprintf("%s (Range: %s - %s)",
        description, m.formatBytes(min), m.formatBytes(max))
}

// Real-time validation
func (m *FormModel) validateField(index int) {
    field := &m.fields[index]
    value := field.Input.Value()

    if intVal, err := strconv.Atoi(value); err != nil {
        field.Input.Placeholder = "Enter a valid number or leave empty for default"
    } else {
        // Check ranges with human-readable feedback
        if intVal < min || intVal > max {
            field.Input.Placeholder = fmt.Sprintf("Range: %s - %s",
                m.formatBytes(min), m.formatBytes(max))
        } else {
            field.Input.Placeholder = "" // Clear error
        }
    }
}
```

### **Value Formatting Helpers**

```go
// Universal byte formatting
func (m *FormModel) formatBytes(bytes int) string {
    if bytes >= 1048576 {
        return fmt.Sprintf("%.1fMB", float64(bytes)/1048576)
    } else if bytes >= 1024 {
        return fmt.Sprintf("%.1fKB", float64(bytes)/1024)
    }
    return fmt.Sprintf("%d bytes", bytes)
}

// Universal number formatting
func (m *FormModel) formatNumber(num int) string {
    if num >= 1000000 {
        return fmt.Sprintf("%.1fM", float64(num)/1000000)
    } else if num >= 1000 {
        return fmt.Sprintf("%.1fK", float64(num)/1000)
    }
    return strconv.Itoa(num)
}
```

### **EnvVar Integration Pattern**

```go
// Helper to create field from EnvVar
addFieldFromEnvVar := func(suffix, key, title, description string) {
    envVar, _ := m.controller.GetVar(m.getEnvVarName(suffix))

    // Track if field was initially set for cleanup logic
    m.initiallySetFields[key] = envVar.Value != ""

    if key == "preserve_last" || key == "use_qa" {
        m.addBooleanField(key, title, description, envVar)
    } else {
        // Determine validation ranges
        var min, max int
        switch key {
        case "last_sec_bytes", "max_qa_bytes":
            min, max = 1024, 1048576 // 1KB to 1MB
        case "max_bp_bytes":
            min, max = 1024, 524288 // 1KB to 512KB
        default:
            min, max = 0, 999999
        }
        m.addIntegerField(key, title, description, envVar, min, max)
    }
}
```

### **Field Cleanup Pattern**

```go
func (m *FormModel) saveConfiguration() (tea.Model, tea.Cmd) {
    // First pass: Handle fields that were cleared (remove from environment)
    for _, field := range m.fields {
        value := field.Input.Value()

        // If field was initially set but now empty, remove it
        if value == "" && m.initiallySetFields[field.Key] {
            envVarName := m.getEnvVarName(getEnvSuffixFromKey(field.Key))

            // Remove the environment variable
            if err := m.controller.SetVar(envVarName, ""); err != nil {
                logger.Errorf("[FormModel] SAVE: error clearing %s: %v", envVarName, err)
                return m, nil
            }
            logger.Log("[FormModel] SAVE: cleared %s", envVarName)
        }
    }

    // Second pass: Save only non-empty values
    for _, field := range m.fields {
        value := field.Input.Value()
        if value == "" {
            continue // Skip empty values - use defaults
        }

        // Validate and save
        envVarName := m.getEnvVarName(getEnvSuffixFromKey(field.Key))
        if err := m.controller.SetVar(envVarName, value); err != nil {
            logger.Errorf("[FormModel] SAVE: error setting %s: %v", envVarName, err)
            return m, nil
        }
    }

    m.hasChanges = false
    return m, nil
}
```

### **Token Estimation Pattern**

```
/* Detailed source-code truncated for AI context efficiency. */
```

### **Current Configuration Display Pattern**

```
/* Detailed source-code truncated for AI context efficiency. */
```

### **Enhanced Localization Pattern**

```go
// Field-specific descriptions with validation hints
const (
    SummarizerFormLastSecBytes     = "Last Section Size (bytes)"
    SummarizerFormLastSecBytesDesc = "Maximum byte size for each preserved conversation section"

    // Enhanced help with practical guidance
    SummarizerFormGeneralHelp = `Balance information depth vs model performance.

Reduce these settings if:
• Using models with ≤64K context (Open Source Reasoning Models)
• Getting "context too long" errors
• Responses become vague or unfocused with long conversations

Key Settings Impact:
• Last Section Size: Larger = more detail, but uses more tokens
• Keep QA Sections: More sections = better continuity, higher token usage

Recommended Adjustments:
• Open Source Reasoning Models: Reduce Last Section to 25-35KB, Keep QA to 1
• OpenAI/Anthropic/Google: Default settings work well`
)
```

### **Type-Based Dynamic Field Generation**

```go
func (m *FormModel) buildForm() {
    // Set type-specific name
    switch m.summarizerType {
    case "general":
        m.typeName = locale.SummarizerTypeGeneralName
    case "assistant":
        m.typeName = locale.SummarizerTypeAssistantName
    }

    // Common fields for all types
    addFieldFromEnvVar("PRESERVE_LAST", "preserve_last", locale.SummarizerFormPreserveLast, locale.SummarizerFormPreserveLastDesc)

    // Type-specific fields
    if m.summarizerType == "general" {
        addFieldFromEnvVar("USE_QA", "use_qa", locale.SummarizerFormUseQA, locale.SummarizerFormUseQADesc)
        addFieldFromEnvVar("SUM_MSG_HUMAN_IN_QA", "sum_human_in_qa", locale.SummarizerFormSumHumanInQA, locale.SummarizerFormSumHumanInQADesc)
    }

    // Common configuration fields
    addFieldFromEnvVar("LAST_SEC_BYTES", "last_sec_bytes", locale.SummarizerFormLastSecBytes, locale.SummarizerFormLastSecBytesDesc)
    // ... additional fields
}
```

These patterns provide a robust foundation for implementing advanced configuration forms with:
- **Type Safety**: Validation at input time
- **User Experience**: Auto-completion, formatting, real-time feedback
- **Resource Awareness**: Token estimation and optimization guidance
- **Environment Integration**: Proper handling of defaults and cleanup
- **Maintainability**: Centralized helpers and consistent patterns

### **Implementation Guidelines for Future Screens**

#### **Langfuse Integration Forms**

**Ready Patterns**: Based on locale constants, implement:
- **Deployment Type Selection**: Embedded/External/Disabled pattern (similar to summarizer types)
- **Conditional Fields**: Show admin fields only for embedded deployment
- **Connection Testing**: Validate external server connectivity
- **Environment Variables**: `LANGFUSE_*` prefix pattern with cleanup

```go
// Implementation pattern
func (m *LangfuseFormModel) buildForm() {
    // Deployment type field (radio-style selection)
    m.addDeploymentTypeField("deployment_type", locale.LangfuseDeploymentType, locale.LangfuseDeploymentTypeDesc)

    // Conditional fields based on deployment type
    if m.deploymentType == "external" {
        m.addFieldFromEnvVar("LANGFUSE_BASE_URL", "base_url", locale.LangfuseBaseURL, locale.LangfuseBaseURLDesc)
        m.addFieldFromEnvVar("LANGFUSE_PROJECT_ID", "project_id", locale.LangfuseProjectID, locale.LangfuseProjectIDDesc)
        m.addFieldFromEnvVar("LANGFUSE_PUBLIC_KEY", "public_key", locale.LangfusePublicKey, locale.LangfusePublicKeyDesc)
        m.addMaskedFieldFromEnvVar("LANGFUSE_SECRET_KEY", "secret_key", locale.LangfuseSecretKey, locale.LangfuseSecretKeyDesc)
    } else if m.deploymentType == "embedded" {
        // Admin configuration for embedded instance
        m.addFieldFromEnvVar("LANGFUSE_ADMIN_EMAIL", "admin_email", locale.LangfuseAdminEmail, locale.LangfuseAdminEmailDesc)
        m.addMaskedFieldFromEnvVar("LANGFUSE_ADMIN_PASSWORD", "admin_password", locale.LangfuseAdminPassword, locale.LangfuseAdminPasswordDesc)
        m.addFieldFromEnvVar("LANGFUSE_ADMIN_NAME", "admin_name", locale.LangfuseAdminName, locale.LangfuseAdminNameDesc)
    }
}
```

#### **Observability Integration Forms**

**Ready Patterns**: Monitoring stack configuration with similar architecture:
- **Deployment Selection**: Embedded/External/Disabled (reuse pattern)
- **External Collector**: OpenTelemetry endpoint configuration
- **Service Selection**: Enable/disable individual monitoring components
- **Resource Estimation**: Calculate monitoring overhead

```go
// Environment variables pattern
func (m *ObservabilityFormModel) getEnvVarName(suffix string) string {
    if m.deploymentType == "external" {
        return "OTEL_" + suffix
    }
    return "OBSERVABILITY_" + suffix
}
```

#### **Security Configuration**
**Potential Patterns**: Based on established architecture:
- **Certificate Management**: File path inputs with validation
- **Access Control**: Boolean enable/disable with role configuration
- **Network Security**: Port ranges, IP allowlists with validation
- **Encryption Settings**: Key generation, algorithm selection

#### **Enhanced Troubleshooting**
**AI-Powered Diagnostics**:
- **System Analysis**: Real-time health checks with recommendations
- **Log Analysis**: Parse error logs and suggest solutions
- **Configuration Validation**: Cross-check settings for conflicts
- **Interactive Fixes**: Guided repair workflows

### **Screen Development Template**

#### **Type Selection Screen Pattern**

```go
type TypesModel struct {
    controller *controllers.StateController
    types      []TypeInfo
    selectedIndex int
    args       []string
}

// Universal type info structure
type TypeInfo struct {
    ID          string
    Name        string
    Description string
}

// Current configuration display (reusable pattern)
func (m *TypesModel) renderCurrentConfiguration(selectedType TypeInfo) string {
    sections = append(sections, m.styles.Subtitle.Render("Current Configuration"))

    // Type-specific value retrieval
    getValue := func(suffix string) string {
        envVar, _ := m.controller.GetVar(m.getEnvVarName(selectedType.ID, suffix))
        if envVar.Value != "" {
            return envVar.Value
        }
        return envVar.Default + " (default)"
    }

    // Display current settings with formatting
    sections = append(sections, fmt.Sprintf("• Setting 1: %s", getValue("SETTING_1")))
    sections = append(sections, fmt.Sprintf("• Setting 2: %s", formatBytes(getIntValue("SETTING_2"))))

    return strings.Join(sections, "\n")
}
```

#### **Form Screen Pattern**

```go
type FormModel struct {
    // Standard form architecture
    controller         *controllers.StateController
    configType        string
    fields            []FormField
    initiallySetFields map[string]bool
    viewport          viewport.Model

    // Pattern-specific additions
    resourceEstimation string
    validationErrors   map[string]string
}

// Universal form building
func (m *FormModel) buildForm() {
    m.fields = []FormField{}
    m.initiallySetFields = make(map[string]bool)

    // Type-specific field generation
    switch m.configType {
    case "type1":
        m.addCommonFields()
        m.addType1SpecificFields()
    case "type2":
        m.addCommonFields()
        m.addType2SpecificFields()
    }

    // Focus and content update
    if len(m.fields) > 0 {
        m.fields[0].Input.Focus()
    }
    m.updateFormContent()
}

// Resource calculation (reusable pattern)
func (m *FormModel) calculateResourceEstimate() string {
    // Get current values from form or defaults
    setting1 := m.getIntValueOrDefault("setting1")
    setting2 := m.getBoolValueOrDefault("setting2")

    // Algorithm-specific calculation
    var estimate int
    if setting2 {
        estimate = setting1 * 2
    } else {
        estimate = setting1
    }

    return fmt.Sprintf("~%s", m.formatNumber(estimate))
}
```

These templates ensure consistency across all future configuration screens while leveraging the proven patterns from summarizer and LLM provider implementations.

## New Implementation Architecture

### **Controller Layer Design**

- **StateController**: Central bridge between TUI forms and state persistence
- **Purpose**: Abstracts environment variable management from UI components
- **Benefits**: Type-safe configuration, automatic validation, dirty state tracking
- **Integration**: All form screens use controller instead of direct state access

### **Adaptive Layout Strategy**

- **Right Panel Hiding**: Main innovation for responsive design
- **Breakpoint Logic**: `contentWidth < (MinMenuWidth + MinInfoWidth + 8)`
- **Graceful Degradation**: Information still accessible, just condensed
- **Performance**: No complex re-rendering, simple layout switching

### **Form Architecture with Bubbles**

- **textinput.Model**: Used for all form inputs with consistent styling
- **Masked Input Toggle**: Ctrl+H to show/hide sensitive values
- **Field Navigation**: Tab/Shift+Tab for keyboard-only navigation
- **Real-time Validation**: Immediate feedback and dirty state tracking
- **Provider-Specific Forms**: Dynamic field generation based on provider type
- **State Persistence**: Composite ScreenIDs with `§` separator (`llm_provider_form§openai`)

### **Composite ScreenID Architecture**

- **Format**: `"screen"` or `"screen§arg1§arg2§..."` for parameterized screens
- **Helper Methods**: `GetScreen()`, `GetArgs()`, `CreateScreenID()`
- **State Persistence**: Complete navigation stack with arguments preserved
- **Benefits**: Type-safe parameter passing, automatic state restoration

```go
// Example: LLM Provider Form with specific provider
targetScreen := CreateScreenID("llm_provider_form", "gemini")
// Results in: "llm_provider_form§gemini"

// Navigation preserves arguments
return NavigationMsg{Target: targetScreen}

// On app restart, user returns to Gemini form, not default OpenAI
```

### **File Organization Pattern**

- **One Model Per File**: `welcome.go`, `eula.go`, `main_menu.go`, etc.
- **Shared Constants**: All in `types.go` for type safety
- **Locale Centralization**: All user-visible text in `locale/locale.go`
- **Controller Separation**: Business logic isolated from presentation

## Architecture & Design Patterns

### 1. Unified App Architecture

**Central Orchestrator (`app.go`)**:
- **Navigation Management**: Stack-based navigation with step persistence
- **Screen Lifecycle**: Model creation, initialization, and cleanup
- **Unified Layout**: Header and footer rendering for all screens
- **Global Event Handling**: ESC, Ctrl+C, window resize
- **Dimension Management**: Terminal size distribution to models

```go
// UNIFIED RENDERING - All screens follow this pattern:
func (a *App) View() string {
    header := a.renderHeader()  // Screen-specific header
    footer := a.renderFooter()  // Dynamic footer with actions
    content := a.currentModel.View()  // Model provides content only

    // App.go calculates and enforces layout constraints
    contentHeight := max(height - headerHeight - footerHeight, 0)
    contentArea := a.styles.Content.Height(contentHeight).Render(content)

    return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```

### 2. Navigation & State Management

#### Navigation Rules (Universal)

- **ESC Behavior**: ALWAYS returns to Welcome screen from any screen (never nested back navigation)
- **Type Safety**: Use `ScreenID` with `CreateScreenID()` for parameterized screens
- **Composite Support**: Screens can carry arguments via `§` separator
- **State Persistence**: Complete navigation stack with arguments preserved
- **EULA Consent**: Check `GetEulaConsent()` on Welcome→EULA transition, call `SetEulaConsent()` on acceptance

```go
// Type-safe navigation structure with composite support
type NavigationMsg struct {
    Target ScreenID  // Can be simple or composite
    GoBack bool
}

type ScreenID string
const (
    WelcomeScreen         ScreenID = "welcome"
    EULAScreen            ScreenID = "eula"
    MainMenuScreen        ScreenID = "main_menu"
    LLMProviderFormScreen ScreenID = "llm_provider_form"
)

// ScreenID methods for composite support
func (s ScreenID) GetScreen() string {
    parts := strings.Split(string(s), "§")
    return parts[0]
}

func (s ScreenID) GetArgs() []string {
    parts := strings.Split(string(s), "§")
    if len(parts) <= 1 {
        return []string{}
    }
    return parts[1:]
}

// Navigation with parameters
targetScreen := CreateScreenID("llm_provider_form", "anthropic")
return NavigationMsg{Target: targetScreen}

// Universal ESC implementation
case "esc":
    if a.navigator.Current().GetScreen() != string(models.WelcomeScreen) {
        a.navigator.stack = []models.ScreenID{models.WelcomeScreen}
        a.navigator.stateManager.SetStack([]string{"welcome"})
        a.currentModel = a.createModelForScreen(models.WelcomeScreen, nil)
        return a, a.currentModel.Init()
    }
```

#### State Integration

- `state.State` remains authoritative for env variables
- Controllers translate between TUI models and state operations
- Complete state reset in `Init()` for predictable behavior

### 3. Layout & Responsive Design

#### Constants & Breakpoints

```go
// Layout Constants
const (
    SmallScreenThreshold = 30    // Height threshold for viewport mode
    MinTerminalWidth = 80        // Minimum width for horizontal layout
    MinPanelWidth = 25           // Panel width constraints
    WelcomeHeaderHeight = 8      // Fixed by ASCII Art Logo (8 lines)
    EULAHeaderHeight = 3         // Title + subtitle + spacing
    FooterHeight = 1             // Always 1 line with background approach
)
```

#### Responsive Breakpoints

- **Small screens**: < 30 rows → viewport mode for scrolling
- **Large screens**: ≥ 30 rows → normal layout mode
- **Narrow terminals**: < 80 cols → vertical stacking
- **Wide terminals**: ≥ 80 cols → horizontal panel layout

#### Height Control (CRITICAL)

```go
// ❌ WRONG - Height() sets MINIMUM height, can expand
style.Height(1).Border(lipgloss.Border{Top: true})

// ✅ CORRECT - Background approach ensures exactly 1 line
style.Background(borderColor).Foreground(textColor).Padding(0,1,0,1)
```

### 4. Footer & Header Systems

#### Unified Footer Strategy

**Background Approach (Production-Ready)**:
- Always exactly 1 line regardless of terminal size
- Modern appearance with background color
- Reliable height calculations
- Dynamic actions based on screen state

```go
// Footer pattern implementation
actions := locale.BuildCommonActions()
if specificCondition {
    actions = append(actions, locale.SpecificAction)
}
footerText := strings.Join(actions, locale.NavSeparator)

return lipgloss.NewStyle().
    Width(width).
    Background(styles.Border).
    Foreground(styles.Foreground).
    Padding(0, 1, 0, 1).
    Render(footerText)
```

#### Header Strategy

- **Welcome Screen**: ASCII Art Logo (8 lines height)
- **Other Screens**: Text title with consistent styling
- **Responsive**: Always present, managed by `app.go`

### 5. Scrolling & Input Handling

#### Modern Scroll Methods

```go
viewport.ScrollUp(1)     // Replaces deprecated LineUp()
viewport.ScrollDown(1)   // Replaces deprecated LineDown()
viewport.ScrollLeft(2)   // Horizontal scroll (2 steps for faster navigation)
viewport.ScrollRight(2)  // Horizontal scroll (2 steps for faster navigation)
```

#### Essential Key Handling

- **↑/↓**: Vertical scrolling (1 line per press)
- **←/→**: Horizontal scrolling (2 steps per press for faster navigation)
- **PgUp/PgDn**: Page-level scrolling
- **Home/End**: Jump to beginning/end

### 6. Content & Resource Management

#### Shared Renderer (Prevents Freezing)
```go
// Single renderer instance in styles.New()
type Styles struct {
    renderer *glamour.TermRenderer
    width    int
    height   int
}

// Usage pattern
rendered, err := m.styles.GetRenderer().Render(markdown)
if err != nil {
    // Fallback to plain text
    rendered = fmt.Sprintf("# Content\n\n%s\n\n*Render error: %v*", content, err)
}
```

#### Content Loading Strategy

- Single renderer instance prevents glamour freezing
- Reset model state completely in `Init()` for clean transitions
- Force view update after content loading with no-op command
- Use embedded files via `files.GetContent()` - handles working directory variations

### 7. Component Architecture

#### Component Types

1. **Welcome Screen**: ASCII art, system checks, info display
2. **EULA Screen**: Markdown viewer with scroll-to-accept
3. **Menu Screen**: Main navigation with dynamic availability
4. **Form Screens**: Configuration input with validation
5. **Status Screens**: Progress and result display

#### Key Components

- **StatusIndicator**: System check results with green checkmarks/red X's
- **MarkdownViewer**: EULA and help text display with scroll support
- **FormController**: Bridges huh forms with state package
- **MenuList**: Dynamic menu with availability checking

### 8. Localization & Styling

#### Localization Structure

```
wizard/locale/
└── locale.go    # All user-visible text constants
```

**Naming Convention**:
- `Welcome*`, `EULA*`, `Menu*`, `LLM*`, `Checks*` - Screen-specific
- `Nav*`, `Status*`, `Error*`, `UI*` - Functional prefixes

#### Styles Centralization

- Single styles instance with shared renderer and dimensions
- Prevents glamour freezing, centralizes terminal size management
- All models access dimensions via `m.styles.GetSize()`

## Development Guidelines

### Screen Model Requirements

#### Required Implementation Pattern

```go
// REQUIRED: State reset in Init()
func (m *Model) Init() tea.Cmd {
    logger.Log("[Model] INIT")
    m.content = ""
    m.ready = false
    // ... reset ALL state
    return m.loadContent
}

// REQUIRED: Dimension handling via styles
func (m *Model) updateViewport() {
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        return
    }
    // ... viewport logic
}

// REQUIRED: Adaptive layout methods
func (m *Model) isVerticalLayout() bool {
    return m.styles.GetWidth() < MinTerminalWidth
}
```

### Screen Development Checklist

**For each new screen:**
- [ ] Type-safe `ScreenID` defined in `types.go`
- [ ] State reset in `Init()` method with logger
- [ ] Dimension handling via `m.styles.GetSize()`
- [ ] Modern `Scroll*` methods for navigation
- [ ] Arrow key handling (↑/↓/←/→) with 2-step horizontal
- [ ] Background footer approach using locale helpers
- [ ] Shared renderer from `styles.GetRenderer()`
- [ ] ESC navigation to Welcome screen
- [ ] Logger integration for debug output

### Code Style Guidelines

#### Compact vs Expanded Style

```go
// ✅ Compact where appropriate:
leftWidth = max(leftWidth, MinPanelWidth)
return lipgloss.NewStyle().Width(width).Padding(0, 2, 0, 2).Render(content)

// ✅ Expanded where needed:
coreChecks := []struct {
    label string
    value bool
}{
    {locale.CheckEnvironmentFile, m.checker.EnvFileExists},
    {locale.CheckDockerAPI, m.checker.DockerApiAccessible},
}
```

#### Comment Guidelines

- Comments explain **why** and **how**, not **what**
- Place comments where code might raise questions about business logic
- Avoid redundant comments that repeat obvious code behavior

## Recent Fixes & Improvements

### ✅ **Composite ScreenID Navigation System**
**Problem**: Need to preserve selected menu items and provider selections across navigation
**Solution**: Implemented composite ScreenIDs with `§` separator for parameter passing

**Features**:
```go
// Composite ScreenID examples
"main_menu§llm_providers"           // Main menu with "llm_providers" selected
"llm_providers§gemini"              // Providers list with "gemini" selected
"llm_provider_form§anthropic"       // Form for "anthropic" provider
```

**Benefits**:
- Type-safe parameter passing via `GetScreen()`, `GetArgs()`, `CreateScreenID()`
- Automatic state restoration - user returns to exact selection after ESC
- Clean navigation stack with full context preservation
- Extensible for multiple arguments per screen

### ✅ **Complete Localization Architecture**

**Problem**: Hardcoded strings scattered throughout UI components
**Solution**: Centralized all user-visible text in `locale.go` with structured constants

**Implementation**:
```go
// Multi-line text stored as single constants
const MainMenuLLMProvidersInfo = `Configure AI language model providers for PentAGI.

Supported providers:
• OpenAI (GPT-4, GPT-3.5-turbo)
• Anthropic (Claude-3, Claude-2)
...`

// Usage in components
sections = append(sections, m.styles.Paragraph.Render(locale.MainMenuLLMProvidersInfo))
```

**Coverage**: 100% of user-facing text moved to locale constants
- Menu descriptions and help text
- Form labels and error messages
- Provider-specific documentation
- Keyboard shortcuts and hints

### ✅ **Viewport-Based Form Scrolling**
**Problem**: Forms with many fields don't fit on smaller terminals
**Solution**: Implemented auto-scrolling viewport with focus tracking

**Based on research**: [BubbleTea viewport best practices](https://pkg.go.dev/github.com/charmbracelet/bubbles/viewport) and [Perplexity guidance on form scrolling](https://www.inngest.com/blog/interactive-clis-with-bubbletea)

**Key Features**:
- **Auto-scroll**: Focused field automatically stays visible
- **Smart positioning**: Calculates field heights for precise scroll positioning
- **Seamless navigation**: Tab/Shift+Tab scroll form as needed
- **No extra hotkeys**: Uses existing navigation keys

**Technical Implementation**:
```go
// Auto-scroll on field focus change
func (m *Model) ensureFocusVisible() {
    focusY := m.calculateFieldPosition(m.focusedIndex)
    if focusY < m.viewport.YOffset {
        m.viewport.YOffset = focusY  // Scroll up
    }
    if focusY >= m.viewport.YOffset + m.viewport.Height {
        m.viewport.YOffset = focusY - m.viewport.Height + 1  // Scroll down
    }
}
```

### ✅ **Enhanced Provider Configuration**

**Problem**: Missing configuration fields for several LLM providers
**Solution**: Added complete field sets for all supported providers

**Provider Field Mapping**:
- **OpenAI/Anthropic/Gemini**: Base URL + API Key
- **AWS Bedrock**: Region + Authentication (Default Auth OR Bearer Token OR Access Key + Secret Key) + Session Token (optional) + Base URL (optional) + Config Path (optional)
- **DeepSeek/GLM/Kimi/Qwen**: Base URL + API Key + Provider Name (optional, for LiteLLM)
- **Ollama**: Base URL + API Key (optional, for cloud) + Model + Config Path + Pull/Load options
- **Custom**: Base URL + API Key + Model + Config Path + Preserve Reasoning (boolean) + Provider Name

**Dynamic Form Generation**: Forms adapt based on provider type with appropriate validation and help text.

## Error Handling & Performance

### Error Handling Strategy

**Graceful degradation with user-friendly messages**:
1. System check failures: Show specific resolution steps
2. Form validation: Real-time feedback with clear messaging
3. State persistence errors: Allow retry with explanation
4. Network issues: Offer offline/manual alternatives

### Performance Considerations

**Lazy loading approach**:
- System checks run asynchronously after welcome screen loads
- Markdown content loaded on-demand when screens are accessed
- Form validation debounced to avoid excessive state updates

## Common Pitfalls & Solutions

### Content Loading Issues

**Problem**: "Loading EULA" state persists, content doesn't appear
**Solutions**:
1. **Multiple Path Fallback**: Try embedded FS first, then direct file access
2. **State Reset**: Always reset model state in `Init()` for clean loading
3. **No ClearScreen**: Avoid `tea.ClearScreen` during navigation
4. **Force View Update**: Return no-op command after content loading

### Layout Consistency Issues

**Problem**: Layout breaks on terminal resize
**Solution**: Always account for actual footer height (1 line)

```go
// Consistent height calculation across all screens
headerHeight := 3 // Fixed based on content
footerHeight := 1 // Background approach always 1 line
contentHeight := m.height - headerHeight - footerHeight
```

### Common Mistakes to Avoid
- Using `tea.ClearScreen` in navigation
- Border-based footer (height inconsistency)
- String-based navigation messages
- Creating new glamour renderer instances
- Forgetting state reset in `Init()`
- Using `fmt.Printf` for debug output
- Deprecated `Line*` scroll methods

## Technology Stack

- **bubbletea**: Core TUI framework using Model-View-Update pattern
- **lipgloss**: Styling and layout engine for visual presentation
- **bubbles**: Component library for interactive elements (list, textinput, viewport)
- **huh**: Form builder for structured input collection (future screens)
- **glamour**: Markdown rendering with single shared instance
- **logger**: Custom file-based logging for TUI-safe development

## ✅ **Production Architecture Implementation**

### **Completed Form System Architecture**

#### **Form Model Pattern (llm_provider_form.go)**

```go
type LLMProviderFormModel struct {
    controller *controllers.StateController
    styles     *styles.Styles
    window     *window.Window

    // Form state
    providerID   string
    fields       []FormField
    focusedIndex int
    showValues   bool
    hasChanges   bool
    args         []string // From composite ScreenID

    // Permanent viewport for scroll state
    viewport     viewport.Model
    formContent  string
    fieldHeights []int
}
```

**Key Implementation Decisions**:
- **Args-based Construction**: `NewLLMProviderFormModel(controller, styles, window, args)`
- **Permanent Viewport**: Form viewport as struct property to preserve scroll state
- **Auto-completion**: Tab key triggers suggestion completion for boolean fields
- **GoBack Navigation**: `return NavigationMsg{GoBack: true}` prevents navigation loops

#### **Navigation Hotkeys (Production Pattern)**

```go
// Modern form navigation
case "down":    // ↓: Next field + auto-scroll
case "up":      // ↑: Previous field + auto-scroll
case "tab":     // Tab: Complete suggestion (true/false for booleans)
case "ctrl+h":  // Ctrl+H: Toggle show/hide masked values
case "ctrl+s":  // Ctrl+S: Save configuration
case "enter":   // Enter: Save and return via GoBack
```

**Important**: Tab navigation replaced with suggestion completion. Field navigation uses ↑/↓ only.

### **Adaptive Layout System**

#### **Layout Constants (Production Values)**

```go
const (
    MinMenuWidth  = 38  // Minimum left panel width
    MaxMenuWidth  = 66  // Maximum left panel width (prevents too wide forms)
    MinInfoWidth  = 34  // Minimum right panel width
    PaddingWidth  = 8   // Total horizontal padding
    PaddingHeight = 2   // Vertical padding
)
```

#### **Two-Column Layout Implementation**

```go
func (m *Model) renderHorizontalLayout(leftPanel, rightPanel string, width, height int) string {
    leftWidth, rightWidth := MinMenuWidth, MinInfoWidth
    extraWidth := width - leftWidth - rightWidth - PaddingWidth

    // Distribute extra space intelligently
    if extraWidth > 0 {
        leftWidth = min(leftWidth+extraWidth/2, MaxMenuWidth)  // Cap at MaxMenuWidth
        rightWidth = width - leftWidth - PaddingWidth/2
    }

    leftStyled := lipgloss.NewStyle().Width(leftWidth).Padding(0, 2, 0, 2).Render(leftPanel)
    rightStyled := lipgloss.NewStyle().Width(rightWidth).PaddingLeft(2).Render(rightPanel)

    // Final layout viewport (temporary)
    viewport := viewport.New(width, height-PaddingHeight)
    viewport.SetContent(lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled))
    return viewport.View()
}
```

#### **Content Hiding Strategy**

```go
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
    verticalStyle := lipgloss.NewStyle().Width(width).Padding(0, 4, 0, 2)

    leftStyled := verticalStyle.Render(leftPanel)
    rightStyled := verticalStyle.Render(rightPanel)

    // Show both panels if they fit
    if lipgloss.Height(leftStyled)+lipgloss.Height(rightStyled)+2 < height {
        return lipgloss.JoinVertical(lipgloss.Left,
            leftStyled,
            verticalStyle.Height(1).Render(""),
            rightStyled,
        )
    }

    // Hide right panel if insufficient space - show only essential content
    return leftStyled
}
```

### **Composite ScreenID Navigation System**

#### **ScreenID Argument Packaging**

```go
// Navigation with selection preservation
func (m *MainMenuModel) handleMenuSelection() (tea.Model, tea.Cmd) {
    selectedItem := m.getSelectedItem()

    return m, func() tea.Msg {
        return NavigationMsg{
            Target: CreateScreenID(string(targetScreen), selectedItem.ID),
        }
    }
}

// Result: "llm_providers§openai" -> llm_providers screen with "openai" pre-selected
```

#### **Args-Based Model Construction**

```go
// No SetSelected* methods needed - selection from constructor
func NewLLMProvidersModel(
    controller *controllers.StateController, styles *styles.Styles,
    window *window.Window, args []string,
) *LLMProvidersModel {
    return &LLMProvidersModel{
        controller: controller,
        args:       args,  // Selection restored from args in Init()
    }
}

func (m *LLMProvidersModel) Init() tea.Cmd {
    // Automatic selection restoration from args[1]
    if len(m.args) > 1 && m.args[1] != "" {
        for i, provider := range m.providers {
            if provider.ID == m.args[1] {
                m.selectedIndex = i
                break
            }
        }
    }
    return nil
}
```

#### **Navigation Stack Management**

**Stack Example**: `["main_menu§llm_providers", "llm_providers§openai", "llm_provider_form§openai"]`

- **Forward Navigation**: Pushes composite ScreenID with arguments
- **Back Navigation**: `GoBack: true` pops current screen, returns to previous with preserved selection
- **No Navigation Loops**: GoBack pattern prevents infinite stack growth

### **Viewport Usage Patterns**

#### **Forms: Permanent Viewport Property**

```go
// ✅ CORRECT: Form viewport as struct property
type FormModel struct {
    viewport viewport.Model  // Preserves scroll position across updates
}

func (m *FormModel) ensureFocusVisible() {
    // Auto-scroll to focused field
    focusY := m.calculateFieldPosition(m.focusedIndex)
    if focusY < m.viewport.YOffset {
        m.viewport.YOffset = focusY
    }
    if focusY+m.fieldHeights[m.focusedIndex] >= offset+visibleRows {
        m.viewport.YOffset = focusY + m.fieldHeights[m.focusedIndex] - visibleRows + 1
    }
}
```

#### **Layout: Temporary Viewport Creation**

```go
// ✅ CORRECT: Layout viewport created for rendering only
func (m *Model) renderHorizontalLayout(left, right string, width, height int) string {
    content := lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)

    vp := viewport.New(width, height-PaddingHeight)  // Temporary
    vp.SetContent(content)
    return vp.View()
}
```

### **Dynamic Form Field Architecture**

#### **Field Configuration Pattern**

```go
// Clean input setup without fixed width
func (m *FormModel) addInputField(fieldType string) {
    input := textinput.New()
    input.Prompt = ""  // Clean appearance
    input.PlaceholderStyle = m.styles.FormPlaceholder

    // Width set dynamically during updateFormContent()
    // NOT set here: input.Width = 50

    if fieldType == "boolean" {
        input.ShowSuggestions = true
        input.SetSuggestions([]string{"true", "false"})
    }
}
```

#### **Dynamic Width Calculation**

```go
func (m *FormModel) getInputWidth() int {
    viewportWidth, _ := m.getViewportSize()
    inputWidth := viewportWidth - 6  // Standard padding
    if m.isVerticalLayout() {
        inputWidth = viewportWidth - 4  // Tighter in vertical mode
    }
    return inputWidth
}

// Applied during form content update
func (m *FormModel) updateFormContent() {
    inputWidth := m.getInputWidth()

    for i, field := range m.fields {
        field.Input.Width = inputWidth - 3  // Account for border/cursor
        field.Input.SetValue(field.Input.Value())  // Trigger width update

        inputStyle := m.styles.FormInput.Width(inputWidth)
        if i == m.focusedIndex {
            inputStyle = inputStyle.BorderForeground(styles.Primary)
        }

        renderedInput := inputStyle.Render(field.Input.View())
        sections = append(sections, renderedInput)
    }
}
```

### **Provider Configuration Architecture**

#### **Simplified Status Model**

```go
// ✅ PRODUCTION: Single status field
type ProviderInfo struct {
    ID          string
    Name        string
    Description string
    Configured  bool    // Single status - has required fields
}

// Status check via controller
configs := m.controller.GetLLMProviders()
provider := ProviderInfo{
    Configured: configs["openai"].Configured,  // Controller determines status
}
```

**Removed**: Dual `Configured`/`Enabled` status - controller handles enable/disable logic internally.

#### **Provider-Specific Field Sets**
- **OpenAI/Anthropic/Gemini**: Base URL + API Key
- **AWS Bedrock**: Region + Authentication (Default Auth OR Bearer Token OR Access Key + Secret Key) + Session Token (optional) + Base URL (optional) + Config Path (optional)
  - **Default Auth**: Use AWS SDK credential chain (environment, EC2 role, ~/.aws/credentials) - highest priority
  - **Bearer Token**: Token-based authentication - priority over static credentials
  - **Static Credentials**: Access Key + Secret Key + Session Token (optional) - traditional IAM authentication
- **DeepSeek/GLM/Kimi/Qwen**: Base URL + API Key + Provider Name (optional, for LiteLLM)
- **Ollama**: Base URL + API Key (optional, for cloud) + Model + Config Path + Pull/Load options
- **Custom**: Base URL + API Key + Model + Config Path + Preserve Reasoning (boolean with suggestions) + Provider Name

### **Screen Architecture (App.go Integration)**

#### **Content Area Responsibility**

```go
// ✅ Screen models handle ONLY content area
func (m *Model) View() string {
    leftPanel := m.renderForm()
    rightPanel := m.renderHelp()

    // Adaptive layout decision
    if m.isVerticalLayout() {
        return m.renderVerticalLayout(leftPanel, rightPanel, width, height)
    }
    return m.renderHorizontalLayout(leftPanel, rightPanel, width, height)
}
```

#### **App.go Layout Management**

```go
// App.go handles complete layout structure
func (a *App) View() string {
    header := a.renderHeader()    // Screen-specific (logo or title)
    footer := a.renderFooter()    // Dynamic actions based on screen
    content := a.currentModel.View()  // Content only from model

    contentWidth, contentHeight := a.window.GetContentSize()
    contentArea := a.styles.Content.
        Width(contentWidth).
        Height(contentHeight).
        Render(content)

    return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```

### **Navigation Anti-Patterns & Solutions**

#### **❌ Common Mistakes**

```go
// ❌ WRONG: Direct navigation creates loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    m.saveConfiguration()
    return m, func() tea.Msg {
        return NavigationMsg{Target: LLMProvidersScreen}  // Loop!
    }
}

// ❌ WRONG: Separate SetSelected methods
func (m *Model) SetSelectedProvider(providerID string) {
    // Complexity - removed in favor of args-based construction
}

// ❌ WRONG: Fixed input widths
input.Width = 50  // Breaks responsive design
```

#### **✅ Correct Patterns**

```go
// ✅ CORRECT: GoBack navigation
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    if err := m.saveConfiguration(); err != nil {
        return m, nil  // Stay on form if save fails
    }
    return m, func() tea.Msg {
        return NavigationMsg{GoBack: true}  // Return to previous screen
    }
}

// ✅ CORRECT: Args-based selection
func NewModel(..., args []string) *Model {
    selectedIndex := 0
    if len(args) > 1 && args[1] != "" {
        // Set selection from args during construction
        for i, item := range items {
            if item.ID == args[1] {
                selectedIndex = i
                break
            }
        }
    }
    return &Model{selectedIndex: selectedIndex, args: args}
}

// ✅ CORRECT: Dynamic input sizing
func (m *FormModel) updateFormContent() {
    inputWidth := m.getInputWidth()  // Calculate based on available space
    field.Input.Width = inputWidth - 3
}
```

---

### Backend Docs Langfuse (`backend/docs/langfuse.md`)

# Langfuse Integration for PentAGI

This document provides a comprehensive guide to the Langfuse integration in PentAGI, covering architecture, setup, usage patterns, and best practices for developers.

## Table of Contents

- [Langfuse Integration for PentAGI](#langfuse-integration-for-pentagi)
  - [Table of Contents](#table-of-contents)
  - [Introduction](#introduction)
  - [Architecture](#architecture)
    - [Component Overview](#component-overview)
    - [Data Flow](#data-flow)
    - [Key Interfaces](#key-interfaces)
      - [Observer Interface](#observer-interface)
      - [Observation Interface](#observation-interface)
      - [Span, Event, and Generation Interfaces](#span-event-and-generation-interfaces)
  - [Setup and Configuration](#setup-and-configuration)
    - [Infrastructure Requirements](#infrastructure-requirements)
    - [Configuration Options](#configuration-options)
    - [Initialization](#initialization)
  - [Usage Guide](#usage-guide)
    - [Creating Observations](#creating-observations)
    - [Recording Spans](#recording-spans)
    - [Tracking Events](#tracking-events)
    - [Logging Generations](#logging-generations)
    - [Adding Scores](#adding-scores)
    - [Recording Agent Observations](#recording-agent-observations)
    - [Recording Tool Observations](#recording-tool-observations)
    - [Recording Chain Observations](#recording-chain-observations)
    - [Recording Retriever Observations](#recording-retriever-observations)
    - [Recording Evaluator Observations](#recording-evaluator-observations)
    - [Recording Embedding Observations](#recording-embedding-observations)
    - [Recording Guardrail Observations](#recording-guardrail-observations)
    - [Context Propagation](#context-propagation)
  - [Integration Examples](#integration-examples)
    - [Flow Controller Integration](#flow-controller-integration)
    - [Agent Execution Tracking](#agent-execution-tracking)
    - [LLM Call Monitoring](#llm-call-monitoring)
  - [Advanced Topics](#advanced-topics)
    - [Batching and Performance](#batching-and-performance)
    - [Error Handling](#error-handling)
    - [Custom Metadata](#custom-metadata)

## Introduction

Langfuse is an open-source observability platform specifically designed for LLM-powered applications. The PentAGI integration with Langfuse provides:

- **Comprehensive tracing** for AI agent flows and tasks
- **Detailed telemetry** for LLM interactions and tool calls
- **Performance metrics** for system components
- **Evaluation** capabilities for agent outputs and behaviors

This integration enables developers to:

1. Debug complex multi-step agent flows
2. Track token usage and costs across different models
3. Monitor system performance in production
4. Gather data for ongoing improvement of agents and models

## Architecture

### Component Overview

The Langfuse integration in PentAGI is built around a layered architecture that provides both high-level abstractions for simple use cases and fine-grained control for complex scenarios.

```mermaid
flowchart TD
    Application[PentAGI Application] --> Observer[Observer]
    Observer --> Client[Langfuse Client]
    Client --> API[Langfuse API]

    Application -- "Creates" --> Observation[Observation Interface]
    Observation -- "Manages" --> Spans[Spans]
    Observation -- "Manages" --> Events[Events]
    Observation -- "Manages" --> Generations[Generations]
    Observation -- "Manages" --> Scores[Scores]

    Observer -- "Batches" --> Events
    Observer -- "Batches" --> Spans
    Observer -- "Batches" --> Generations
    Observer -- "Batches" --> Scores

    subgraph "Langfuse SDK"
        Observer
        Client
        Observation
        Spans
        Events
        Generations
        Scores
    end

    subgraph "Langfuse Backend"
        API
        PostgreSQL[(PostgreSQL)]
        ClickHouse[(ClickHouse)]
        Redis[(Redis)]
        MinIO[(MinIO)]

        API --> PostgreSQL
        API --> Redis
        API --> MinIO
        PostgreSQL --> ClickHouse
    end
```

### Data Flow

The data flow through the Langfuse system follows a consistent pattern:

```mermaid
sequenceDiagram
    participant App as PentAGI Application
    participant Obs as Observer
    participant Queue as Event Queue
    participant Client as Langfuse Client
    participant API as Langfuse API
    participant DB as Storage

    App->>Obs: Create Observation
    Obs->>App: Return Observation Interface

    App->>Obs: Start Span
    Obs->>Queue: Enqueue span-start event

    App->>Obs: Record Event
    Obs->>Queue: Enqueue event

    App->>Obs: End Span
    Obs->>Queue: Enqueue span-end event

    loop Batch Processing
        Queue->>Client: Batch events
        Client->>API: Send batch
        API->>DB: Store data
        API->>Client: Confirm receipt
    end
```

### Key Interfaces

The Langfuse integration is built around several key interfaces:

#### Observer Interface

The `Observer` interface is the primary entry point for Langfuse integration:

```go
type Observer interface {
    // Creates a new observation and returns updated context
    NewObservation(
        ctx context.Context,
        opts ...ObservationContextOption,
    ) (context.Context, Observation)

    // Gracefully shuts down the observer
    Shutdown(ctx context.Context) error

    // Forces immediate flush of queued events
    ForceFlush(ctx context.Context) error
}
```

#### Observation Interface

The `Observation` interface provides methods to record different types of data:

```go
type Observation interface {
    // Returns the observation ID
    ID() string

    // Returns the trace ID
    TraceID() string

    // Records a log message
    Log(ctx context.Context, message string)

    // Records a score for evaluation
    Score(opts ...ScoreOption)

    // Creates a new event observation
    Event(opts ...EventStartOption) Event

    // Creates a new span observation
    Span(opts ...SpanStartOption) Span

    // Creates a new generation observation
    Generation(opts ...GenerationStartOption) Generation
}
```

#### Span, Event, and Generation Interfaces

These interfaces represent different observation types:

```go
type Span interface {
    // Ends the span with optional data
    End(opts ...SpanOption)

    // Creates a child observation context
    Observation(ctx context.Context) (context.Context, Observation)

    // Returns observation metadata
    ObservationInfo() ObservationInfo
}

type Event interface {
    // Ends the event with optional data
    End(opts ...EventEndOption)

    // Other methods similar to Span
    // ...
}

type Generation interface {
    // Ends the generation with optional data
    End(opts ...GenerationEndOption)

    // Other methods similar to Span
    // ...
}
```

## Setup and Configuration

### Infrastructure Requirements

Langfuse requires several backend services. For development and testing, you can use the included Docker Compose file:

```bash
# Start Langfuse infrastructure
docker-compose -f docker-compose-langfuse.yml up -d
```

The infrastructure includes:
- **PostgreSQL**: Primary data storage
- **ClickHouse**: Analytical data storage for queries
- **Redis**: Caching and queue management
- **MinIO**: S3-compatible storage for large objects
- **Langfuse Web**: Admin UI (accessible at http://localhost:4000)
- **Langfuse Worker**: Background processing

### Configuration Options

The Langfuse integration can be configured through environment variables:

| Variable | Description | Default |
|----------|-------------|---------|
| `LANGFUSE_BASE_URL` | Base URL for Langfuse API | |
| `LANGFUSE_PROJECT_ID` | Project ID in Langfuse | |
| `LANGFUSE_PUBLIC_KEY` | Public API key | |
| `LANGFUSE_SECRET_KEY` | Secret API key | |
| `LANGFUSE_INIT_USER_EMAIL` | Admin user email | admin@pentagi.com |
| `LANGFUSE_INIT_USER_PASSWORD` | Admin user password | P3nTagIsD0d |

For a complete list of configuration options, refer to the docker-compose-langfuse.yml file.

### Initialization

To initialize the Langfuse integration in your code:

```go
// Import the necessary packages
import (
    "pentagi/pkg/observability/langfuse"
    "pentagi/pkg/config"
)

// Create a Langfuse client
client, err := langfuse.NewClient(
    langfuse.WithBaseURL(cfg.LangfuseBaseURL),
    langfuse.WithPublicKey(cfg.LangfusePublicKey),
    langfuse.WithSecretKey(cfg.LangfuseSecretKey),
    langfuse.WithProjectID(cfg.LangfuseProjectID),
)
if err != nil {
    return nil, fmt.Errorf("failed to create langfuse client: %w", err)
}

// Create an observer with the client
observer := langfuse.NewObserver(client,
    langfuse.WithProject("pentagi"),
    langfuse.WithSendInterval(10 * time.Second),
    langfuse.WithQueueSize(100),
)

// Use a no-op observer when Langfuse is not configured
if errors.Is(err, ErrNotConfigured) {
    observer = langfuse.NewNoopObserver()
}
```

## Usage Guide

### Creating Observations

Observations are the fundamental tracking unit in Langfuse. Create a new observation for each logical operation or flow:

```go
// Create a new observation from context
ctx, observation := observer.NewObservation(ctx,
    langfuse.WithObservationTraceContext(
        langfuse.WithTraceName("flow-execution"),
        langfuse.WithTraceUserId(user.Email),
        langfuse.WithTraceSessionId(fmt.Sprintf("flow-%d", flowID)),
    ),
)
```

### Recording Spans

Spans track time duration and are used for operations with a distinct start and end:

```go
// Create a span for an operation
span := observation.Span(
    langfuse.WithSpanName("database-query"),
    langfuse.WithStartSpanInput(query),
)

// Execute the operation
result, err := executeQuery(query)

// End the span with result
if err != nil {
    span.End(
        langfuse.WithSpanStatus(err.Error()),
        langfuse.WithSpanLevel(langfuse.ObservationLevelError),
    )
} else {
    span.End(
        langfuse.WithSpanOutput(result),
        langfuse.WithSpanStatus("success"),
    )
}
```

### Tracking Events

Events represent point-in-time occurrences:

```go
// Record an event
observation.Event(
    langfuse.WithEventName("user-interaction"),
    langfuse.WithEventMetadata(langfuse.Metadata{
        "action": "button-click",
        "element": "submit-button",
    }),
)
```

### Logging Generations

Generations track LLM interactions with additional metadata:

```go
// Start a generation
generation := observation.Generation(
    langfuse.WithGenerationName("task-planning"),
    langfuse.WithGenerationModel("gpt-4"),
    langfuse.WithGenerationInput(prompt),
    langfuse.WithGenerationModelParameters(&langfuse.ModelParameters{
        Temperature: 0.7,
        MaxTokens: 1000,
    }),
)

// Get the response from the LLM
response, err := llmClient.Generate(prompt)

// End the generation with the result
generation.End(
    langfuse.WithGenerationOutput(response),
    langfuse.WithEndGenerationUsage(&langfuse.GenerationUsage{
        Input: promptTokens,
        Output: responseTokens,
        Unit: langfuse.GenerationUsageUnitTokens,
    }),
)
```

### Adding Scores

Scores provide evaluations for agent outputs:

```go
// Add a score to an observation
observation.Score(
    langfuse.WithScoreName("response-quality"),
    langfuse.WithScoreFloatValue(0.95),
    langfuse.WithScoreComment("High quality and relevant response"),
)
```

### Recording Agent Observations

Agents represent autonomous reasoning processes in agentic systems:

```go
// Create an agent observation
agent := observation.Agent(
    langfuse.WithAgentName("security-analyst"),
    langfuse.WithAgentInput(analysisRequest),
    langfuse.WithAgentMetadata(langfuse.Metadata{
        "agent_role": "security_researcher",
        "capabilities": []string{"vulnerability_analysis", "exploit_detection"},
    }),
)

// Perform agent work
result := performAnalysis(ctx)

// End the agent observation
agent.End(
    langfuse.WithAgentOutput(result),
    langfuse.WithAgentStatus("completed"),
)
```

### Recording Tool Observations

Tools track the execution of specific tools or functions:

```go
// Create a tool observation
tool := observation.Tool(
    langfuse.WithToolName("web-search"),
    langfuse.WithToolInput(searchQuery),
    langfuse.WithToolMetadata(langfuse.Metadata{
        "tool_type": "search",
        "provider": "duckduckgo",
    }),
)

// Execute the tool
results, err := executeSearch(ctx, searchQuery)

// End the tool observation
if err != nil {
    tool.End(
        langfuse.WithToolStatus(err.Error()),
        langfuse.WithToolLevel(langfuse.ObservationLevelError),
    )
} else {
    tool.End(
        langfuse.WithToolOutput(results),
        langfuse.WithToolStatus("success"),
    )
}
```

### Recording Chain Observations

Chains track multi-step reasoning processes:

```go
// Create a chain observation
chain := observation.Chain(
    langfuse.WithChainName("multi-step-reasoning"),
    langfuse.WithChainInput(messages),
    langfuse.WithChainMetadata(langfuse.Metadata{
        "chain_type": "sequential",
        "steps": 3,
    }),
)

// Execute the chain
finalResult := executeReasoningChain(ctx, messages)

// End the chain observation
chain.End(
    langfuse.WithChainOutput(finalResult),
    langfuse.WithChainStatus("completed"),
)
```

### Recording Retriever Observations

Retrievers track information retrieval operations, such as vector database searches:

```go
// Create a retriever observation
retriever := observation.Retriever(
    langfuse.WithRetrieverName("vector-similarity-search"),
    langfuse.WithRetrieverInput(map[string]any{
        "query": searchQuery,
        "threshold": 0.75,
        "max_results": 5,
    }),
    langfuse.WithRetrieverMetadata(langfuse.Metadata{
        "retriever_type": "vector_similarity",
        "embedding_model": "text-embedding-ada-002",
    }),
)

// Perform retrieval
docs, err := vectorStore.SimilaritySearch(ctx, searchQuery)

// End the retriever observation
retriever.End(
    langfuse.WithRetrieverOutput(docs),
    langfuse.WithRetrieverStatus("success"),
)
```

### Recording Evaluator Observations

Evaluators track quality assessment and validation operations:

```go
// Create an evaluator observation
evaluator := observation.Evaluator(
    langfuse.WithEvaluatorName("response-quality-evaluator"),
    langfuse.WithEvaluatorInput(map[string]any{
        "response": agentResponse,
        "criteria": []string{"accuracy", "completeness", "safety"},
    }),
    langfuse.WithEvaluatorMetadata(langfuse.Metadata{
        "evaluator_type": "llm_based",
        "model": "gpt-4",
    }),
)

// Perform evaluation
scores := evaluateResponse(ctx, agentResponse)

// End the evaluator observation
evaluator.End(
    langfuse.WithEvaluatorOutput(scores),
    langfuse.WithEvaluatorStatus("completed"),
)
```

### Recording Embedding Observations

Embeddings track vector embedding generation operations:

```go
// Create an embedding observation
embedding := observation.Embedding(
    langfuse.WithEmbeddingName("text-embedding-generation"),
    langfuse.WithEmbeddingInput(map[string]any{
        "text": textToEmbed,
        "model": "text-embedding-ada-002",
    }),
    langfuse.WithEmbeddingMetadata(langfuse.Metadata{
        "embedding_model": "text-embedding-ada-002",
        "dimensions": 1536,
    }),
)

// Generate embedding
vector, err := embeddingProvider.Embed(ctx, textToEmbed)

// End the embedding observation
embedding.End(
    langfuse.WithEmbeddingOutput(map[string]any{
        "vector": vector,
        "dimensions": len(vector),
    }),
    langfuse.WithEmbeddingStatus("success"),
)
```

### Recording Guardrail Observations

Guardrails track safety and policy enforcement checks:

```go
// Create a guardrail observation
guardrail := observation.Guardrail(
    langfuse.WithGuardrailName("content-safety-check"),
    langfuse.WithGuardrailInput(map[string]any{
        "text": userInput,
        "checks": []string{"content_policy", "pii_detection"},
    }),
    langfuse.WithGuardrailMetadata(langfuse.Metadata{
        "guardrail_type": "safety",
        "strictness": "high",
    }),
)

// Perform safety checks
passed, violations := performSafetyChecks(ctx, userInput)

// End the guardrail observation
guardrail.End(
    langfuse.WithGuardrailOutput(map[string]any{
        "passed": passed,
        "violations": violations,
    }),
    langfuse.WithGuardrailStatus(fmt.Sprintf("passed=%t", passed)),
)
```

### Context Propagation

Langfuse leverages Go's context package for observation propagation:

```go
// Create a parent observation
ctx, parentObs := observer.NewObservation(ctx)

// Create a span
span := parentObs.Span(langfuse.WithSpanName("parent-operation"))

// Create a child context with the span's observation
childCtx, childObs := span.Observation(ctx)

// Use the child context for further operations
result := performOperation(childCtx)

// Child observations will be linked to the parent
childObs.Log(childCtx, "Operation completed")
```

## Data Conversion

The Langfuse integration automatically converts LangChainGo data structures to OpenAI-compatible format for optimal display in the Langfuse UI.

### Automatic Conversion

All Input and Output data passed to observation types is automatically converted:

```go
// LangChainGo message format
messages := []*llms.MessageContent{
    {
        Role: llms.ChatMessageTypeHuman,
        Parts: []llms.ContentPart{
            llms.TextContent{Text: "Analyze this vulnerability"},
        },
    },
}

// Automatically converted to OpenAI format
generation := observation.Generation(
    langfuse.WithGenerationInput(messages),  // Converted automatically
)
```

### OpenAI Format Benefits

The converter transforms messages to OpenAI-compatible format providing:

1. **Standard Structure** - Compatible with OpenAI API message format
2. **Rich UI Rendering** - Tool calls, images, and complex responses display correctly
3. **Playground Support** - Messages work with Langfuse playground feature
4. **Table Rendering** - Complex tool responses shown as expandable tables

### Message Conversion

#### Role Mapping

| LangChainGo Role | OpenAI Role |
|------------------|-------------|
| `ChatMessageTypeHuman` | `"user"` |
| `ChatMessageTypeAI` | `"assistant"` |
| `ChatMessageTypeSystem` | `"system"` |
| `ChatMessageTypeTool` | `"tool"` |

#### Simple Text Message

**Input:**
```go
&llms.MessageContent{
    Role: llms.ChatMessageTypeHuman,
    Parts: []llms.ContentPart{
        llms.TextContent{Text: "Hello"},
    },
}
```

**Output (JSON):**
```json
{
  "role": "user",
  "content": "Hello"
}
```

#### Message with Tool Calls

**Input:**
```go
&llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{
        llms.TextContent{Text: "I'll search for that"},
        llms.ToolCall{
            ID: "call_001",
            FunctionCall: &llms.FunctionCall{
                Name: "search_database",
                Arguments: `{"query":"test"}`,
            },
        },
    },
}
```

**Output (JSON):**
```json
{
  "role": "assistant",
  "content": "I'll search for that",
  "tool_calls": [
    {
      "id": "call_001",
      "type": "function",
      "function": {
        "name": "search_database",
        "arguments": "{\"query\":\"test\"}"
      }
    }
  ]
}
```

#### Tool Response - Simple vs Rich

**Simple Content (1-2 keys):**
```go
llms.ToolCallResponse{
    ToolCallID: "call_001",
    Content: `{"status": "success"}`,
}
```

Rendered as plain string in UI.

**Rich Content (3+ keys or nested):**
```go
llms.ToolCallResponse{
    ToolCallID: "call_001",
    Content: `{
        "results": [...],
        "count": 10,
        "page": 1,
        "total_pages": 5
    }`,
}
```

Rendered as **expandable table** in Langfuse UI with toggle button.

#### Reasoning/Thinking Content

Messages with reasoning are converted to include thinking blocks:

**Input:**
```go
llms.TextContent{
    Text: "The answer is 42",
    Reasoning: &reasoning.ContentReasoning{
        Content: "Step-by-step analysis...",
    },
}
```

**Output (JSON):**
```json
{
  "role": "assistant",
  "content": "The answer is 42",
  "thinking": [
    {
      "type": "thinking",
      "content": "Step-by-step analysis..."
    }
  ]
}
```

#### Multimodal Messages

Images and binary content are properly converted:

**Input:**
```go
&llms.MessageContent{
    Role: llms.ChatMessageTypeHuman,
    Parts: []llms.ContentPart{
        llms.TextContent{Text: "What's in this image?"},
        llms.ImageURLContent{
            URL: "https://example.com/image.jpg",
            Detail: "high",
        },
    },
}
```

**Output (JSON):**
```json
{
  "role": "user",
  "content": [
    {"type": "text", "text": "What's in this image?"},
    {
      "type": "image_url",
      "image_url": {
        "url": "https://example.com/image.jpg",
        "detail": "high"
      }
    }
  ]
}
```

### Tool Call Linking

The converter automatically adds function names to tool responses for better UI clarity:

```go
// Message chain with tool call
messages := []*llms.MessageContent{
    {
        Role: llms.ChatMessageTypeAI,
        Parts: []llms.ContentPart{
            llms.ToolCall{
                ID: "call_001",
                FunctionCall: &llms.FunctionCall{
                    Name: "search_database",
                },
            },
        },
    },
    {
        Role: llms.ChatMessageTypeTool,
        Parts: []llms.ContentPart{
            llms.ToolCallResponse{
                ToolCallID: "call_001",
                Content: `{"results": [...]}`,
            },
        },
    },
}
```

The tool response automatically gets the `"name": "search_database"` field added, showing the function name as the title in Langfuse UI instead of just "Tool".

### ContentChoice Conversion

Output from LLM providers is also converted:

```go
output := &llms.ContentChoice{
    Content: "Based on analysis...",
    ToolCalls: []llms.ToolCall{...},
    Reasoning: &reasoning.ContentReasoning{...},
}

generation.End(
    langfuse.WithGenerationOutput(output),  // Converted to OpenAI format
)
```

Converted to assistant message with content, tool_calls, and thinking fields as appropriate.

## Integration Examples

### Flow Controller Integration

The main integration point in PentAGI is the Flow Controller, which handles the lifecycle of AI agent flows:

```go
// In flow controller initialization
ctx, observation := obs.Observer.NewObservation(ctx,
    langfuse.WithObservationTraceContext(
        langfuse.WithTraceName(fmt.Sprintf("%d flow worker", flow.ID)),
        langfuse.WithTraceUserId(user.Mail),
        langfuse.WithTraceTags([]string{"controller"}),
        langfuse.WithTraceSessionId(fmt.Sprintf("flow-%d", flow.ID)),
        langfuse.WithTraceMetadata(langfuse.Metadata{
            "flow_id": flow.ID,
            "user_id": user.ID,
            // ...additional metadata
        }),
    ),
)

// Create a span for a specific operation
flowSpan := observation.Span(langfuse.WithSpanName("prepare flow worker"))

// Propagate the context with the span
ctx, _ = flowSpan.Observation(ctx)

// End the span when the operation completes
flowSpan.End(langfuse.WithSpanStatus("flow worker started"))
```

### Agent Execution Tracking

Track individual agent executions and tool calls:

```go
// Create a span for agent execution
agentSpan := observation.Span(
    langfuse.WithSpanName("agent-execution"),
    langfuse.WithStartSpanInput(input),
)

// Track the generation
generation := agentSpan.Observation(ctx).Generation(
    langfuse.WithGenerationName("agent-thinking"),
    langfuse.WithGenerationModel(modelName),
)

// End the generation with the result
generation.End(
    langfuse.WithGenerationOutput(output),
    langfuse.WithEndGenerationUsage(&langfuse.GenerationUsage{
        Input: promptTokens,
        Output: responseTokens,
        Unit: langfuse.GenerationUsageUnitTokens,
    }),
)

// End the span
agentSpan.End(
    langfuse.WithSpanStatus("success"),
    langfuse.WithSpanOutput(result),
)
```

### LLM Call Monitoring

Track and monitor all LLM interactions:

```go
// Create a generation for an LLM call
generation := observation.Generation(
    langfuse.WithGenerationName("content-generation"),
    langfuse.WithGenerationModel(llmModel),
    langfuse.WithGenerationInput(prompt),
    langfuse.WithGenerationModelParameters(
        langfuse.GetLangchainModelParameters(options),
    ),
)

// Make the LLM call
response, err := llm.Generate(ctx, prompt, options...)

// End the generation with result
if err != nil {
    generation.End(
        langfuse.WithGenerationStatus(err.Error()),
        langfuse.WithGenerationLevel(langfuse.ObservationLevelError),
    )
} else {
    generation.End(
        langfuse.WithGenerationOutput(response),
        langfuse.WithEndGenerationUsage(&langfuse.GenerationUsage{
            Input: calculateInputTokens(prompt),
            Output: calculateOutputTokens(response),
            Unit: langfuse.GenerationUsageUnitTokens,
        }),
    )
}
```

## Advanced Topics

### Batching and Performance

The Langfuse integration uses batching to optimize performance:

```go
// Configure batch size and interval
observer := langfuse.NewObserver(client,
    langfuse.WithQueueSize(200),      // Events per batch
    langfuse.WithSendInterval(15*time.Second), // Send interval
)
```

Events are queued and sent in batches to minimize overhead. The `ForceFlush` method can be used to immediately send queued events:

```go
// Force sending of all queued events
if err := observer.ForceFlush(ctx); err != nil {
    log.Printf("Failed to flush events: %v", err)
}
```

### Error Handling

Langfuse operations are designed to be non-blocking and fail gracefully:

```go
// Create a span with try/catch pattern
span := observation.Span(langfuse.WithSpanName("risky-operation"))
defer func() {
    if r := recover(); r != nil {
        span.End(
            langfuse.WithSpanStatus(fmt.Sprintf("panic: %v", r)),
            langfuse.WithSpanLevel(langfuse.ObservationLevelError),
        )
        panic(r) // Re-panic
    }
}()

// Perform operation
result, err := performRiskyOperation()

// Handle error
if err != nil {
    span.End(
        langfuse.WithSpanStatus(err.Error()),
        langfuse.WithSpanLevel(langfuse.ObservationLevelError),
    )
    return err
}

// Success case
span.End(
    langfuse.WithSpanOutput(result),
    langfuse.WithSpanStatus("success"),
)
```

### Custom Metadata

Langfuse supports custom metadata for all observation types:

```go
// Add custom metadata to a span
span := observation.Span(
    langfuse.WithSpanName("process-file"),
    langfuse.WithStartSpanMetadata(langfuse.Metadata{
        "file_size": fileSize,
        "file_type": fileType,
        "encryption": encryptionType,
        "user_id": userID,
        // Any JSON-serializable data
    }),
)
```

This metadata is searchable and filterable in the Langfuse UI, making it easier to find and analyze specific observations.

### Data Converter Implementation

The converter is implemented in `pkg/observability/langfuse/converter.go` and provides two main functions:

```go
// Convert input data (message chains) to OpenAI format
func convertInput(input any, tools []llms.Tool) any

// Convert output data (responses, choices) to OpenAI format
func convertOutput(output any) any
```

**Type Support:**

The converter handles various data types:
- `[]*llms.MessageContent` and `[]llms.MessageContent` - Message chains
- `*llms.MessageContent` and `llms.MessageContent` - Single messages
- `*llms.ContentChoice` and `llms.ContentChoice` - LLM responses
- `[]*llms.ContentChoice` and `[]llms.ContentChoice` - Multiple choices
- Other types - Pass-through without conversion

**Conversion Features:**

1. **Role Mapping**: `human` → `user`, `ai` → `assistant`
2. **Tool Call Formatting**: Converts to OpenAI function calling format
3. **Tool Response Parsing**: Smart detection of rich vs simple content
4. **Function Name Linking**: Automatically adds function names to tool responses
5. **Reasoning Extraction**: Separates thinking content into dedicated blocks
6. **Multimodal Support**: Handles images, binary data, and text together
7. **Error Resilience**: Gracefully handles invalid JSON and edge cases

**Performance Considerations:**

- Conversion happens once at observation creation/end
- JSON parsing is cached where possible
- No additional network overhead
- Minimal memory allocation through careful type handling

**Testing:**

The converter includes comprehensive test coverage in `converter_test.go`:
- Input conversion scenarios (simple, multimodal, with tools)
- Output conversion scenarios (text, tool calls, reasoning)
- Edge cases (empty chains, invalid JSON, unknown types)
- Real-world conversation flows
- Performance benchmarks

---

### Backend Docs Llms How To (`backend/docs/llms_how_to.md`)

# LLM Provider Integration Guide

This document describes critical requirements for client code when working with LLM providers through this library. Focus is on **why** certain patterns are required, not just **what** they do.

## General Requirements

### Message Chain Construction

**Rule**: Always preserve reasoning data in multi-turn conversations.

**Why**: Models with extended thinking (Claude 3.7+, GPT-o1+, etc.) require reasoning signatures to validate message chain integrity. Missing signatures cause API errors.

**How**:
```go
// After receiving response with reasoning
if choice.Reasoning != nil {
    messages = append(messages, llms.MessageContent{
        Role: llms.ChatMessageTypeAI,
        Parts: []llms.ContentPart{
            llms.TextPartWithReasoning(choice.Content, choice.Reasoning),
            // then add tool calls if any
        },
    })
}
```

**Critical**: Use `TextPartWithReasoning()` even if `Content` is empty. The reasoning block and signature must be preserved for API validation.

---

### Tool Call ID Format Compatibility

**Rule**: Each provider uses a specific format for tool call IDs that must be validated when switching providers.

**Why**: Different providers have different validation rules for tool call IDs. For example:
- **OpenAI/Gemini**: Accept alphanumeric IDs like `call_abc123def456ghi789jkl0`
- **Anthropic**: Require base62 IDs matching pattern `^[a-zA-Z0-9_-]+$` like `toolu_A1b2C3d4E5f6G7h8I9j0K1l2`

**Problem**: When restoring a message chain that was created with one provider (e.g., Gemini) and using it with another provider (e.g., Anthropic), the API will reject tool call IDs that don't match its expected format.

**Solution**: Use `ChainAST.NormalizeToolCallIDs()` to convert incompatible tool call IDs:

```go
// Restore chain from database (may contain IDs from different provider)
var chain []llms.MessageContent
json.Unmarshal(msgChain.Chain, &chain)

// Create AST
ast, err := cast.NewChainAST(chain, true)
if err != nil {
    return err
}

// Normalize to current provider's format
err = ast.NormalizeToolCallIDs(currentProviderTemplate)
if err != nil {
    return err
}

// Use normalized chain
normalizedChain := ast.Messages()
```

**How it works**:
1. Validates each tool call ID against the new template using `ValidatePattern`
2. Generates new IDs only for those that don't match
3. Preserves IDs that already match to avoid unnecessary changes
4. Updates both tool calls and their corresponding responses

**Common Templates**:

| Provider | Template | Example |
|----------|----------|---------|
| OpenAI | `call_{r:24:x}` | `call_abc123def456ghi789jkl0` |
| Gemini | `call_{r:24:x}` | `call_xyz789abc012def345ghi6` |
| Anthropic | `toolu_{r:24:b}` | `toolu_A1b2C3d4E5f6G7h8I9j0K1l2` |

**When to use**: This is automatically handled in `restoreChain()` function, but you may need it manually when:
- Implementing custom provider logic
- Migrating between providers
- Testing with different providers

---

### Reasoning Content Cleanup

**Rule**: Clear reasoning content when switching between providers.

**Why**: Reasoning content contains provider-specific data that causes API errors with different providers:
- **Cryptographic signatures**: Anthropic's extended thinking uses signatures that other providers reject
- **Reasoning metadata**: Provider-specific formatting and validation

**Problem**: When restoring a chain created with Anthropic (with reasoning signatures) and sending to Gemini, the API will reject the request.

**Solution**: Use `ChainAST.ClearReasoning()` to remove all reasoning data:

```go
// After restoring chain and normalizing IDs
ast, err := cast.NewChainAST(chain, true)
if err != nil {
    return err
}

// Normalize IDs first
err = ast.NormalizeToolCallIDs(newTemplate)
if err != nil {
    return err
}

// Then clear reasoning signatures
err = ast.ClearReasoning()
if err != nil {
    return err
}

// Chain is now safe for the new provider
cleanedChain := ast.Messages()
```

**What gets cleared**:
- `TextContent.Reasoning` - Extended thinking signatures and content
- `ToolCall.Reasoning` - Per-tool reasoning data

**What stays preserved**:
- All text content
- Tool call IDs (after normalization)
- Function names and arguments  
- Tool responses

**Automatic handling**: Both `NormalizeToolCallIDs()` and `ClearReasoning()` are called automatically in `restoreChain()` to ensure full compatibility when switching providers.

---

## Anthropic Provider

### 1. Extended Thinking Mode

#### 1.1 Roundtrip Signature Preservation

**Why**: Anthropic API validates reasoning integrity using cryptographic signatures. Missing signature → `400 Bad Request`.

**When**: ALL responses from thinking-enabled models (Claude 3.7+, Sonnet 4+, Haiku 4.5+).

```go
// ✓ CORRECT
messages = append(messages, llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{
        llms.TextPartWithReasoning(choice.Content, choice.Reasoning),
    },
})

// ✗ WRONG - Signature lost, API will reject next request
messages = append(messages, llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{
        llms.TextPart(choice.Content), // Missing reasoning!
    },
})
```

**Components in `choice.Reasoning`**:
- `Content`: Human-readable thinking text
- `Signature`: Binary cryptographic signature (REQUIRED for roundtrip)

#### 1.2 Temperature Requirement (budget thinking)

**Rule**: Set `Temperature = 1.0` for **budget** extended thinking (`WithReasoning(effort, maxTokens)`).

**Why**: Anthropic's budget thinking mode requires temperature=1.0 to function correctly. Lower values degrade reasoning quality or cause API errors.

```go
llm.GenerateContent(ctx, messages,
    llms.WithReasoning(llms.ReasoningMedium, 2048),
    llms.WithTemperature(1.0), // REQUIRED for budget thinking
)
```

> **Adaptive thinking is the exception — do NOT set temperature for it.** Claude Opus 4.7/4.8 (and the 4.6 series when run in adaptive mode) use `thinking.type=adaptive` + `output_config.effort` and **reject every sampling parameter** (`temperature`/`top_p`/`top_k`) with HTTP 400. The forked langchaingo handles this natively: `llms.WithAdaptiveReasoning(effort)` makes the anthropic and bedrock clients emit the adaptive request shape and omit those sampling params. A model's capability is declared in its provider `models.yml` via `reasoning.mode` (`adaptive` = budget or adaptive; `adaptive-only` = adaptive required, budget returns 400); the per-agent `ReasoningConfig.Mode`/`Effort` then selects the mode at call time (PentAGI's `PrepareAdaptiveCallOptions` appends `WithAdaptiveReasoning`).

#### 1.3 Interleaved Thinking with Tools

**Auto-enable**: Library automatically adds `interleaved-thinking-2025-05-14` beta header when both `Tools` and `Reasoning` are present.

**Why**: Without this header, API cannot interleave thinking blocks between tool calls → degraded reasoning quality or errors.

**Manual override** (if needed):
```go
anthropic.WithInterleavedThinking() // Explicit beta header
```

#### 1.4 Reasoning Location

**Important**: Anthropic places reasoning in `choice.Reasoning`, NOT in individual `ToolCall.Reasoning`.

```go
// ✓ CORRECT
resp, _ := llm.GenerateContent(ctx, messages, ...)
reasoning := resp.Choices[0].Reasoning // Here!

// ✗ WRONG - Anthropic doesn't use this
for _, tc := range resp.Choices[0].ToolCalls {
    _ = tc.Reasoning // Always nil for Anthropic
}
```

**Why**: Anthropic's API design treats reasoning as conversation-level, not tool-level. One thinking block may inform multiple tool calls.

---

### 2. Prompt Caching

#### 2.1 Enable Caching

**Required**: Add `WithPromptCaching()` to enable beta header.

```go
llm.GenerateContent(ctx, messages,
    anthropic.WithPromptCaching(), // REQUIRED
    anthropic.WithCacheStrategy(anthropic.CacheStrategy{
        CacheTools:    true,
        CacheSystem:   true,
        CacheMessages: true,
    }),
)
```

**Why**: Without beta header, cache control directives are silently ignored → no cost savings.

#### 2.2 Minimum Token Threshold

**Rule**: Cached content must exceed minimum tokens:
- Claude Sonnet 4: 1024 tokens
- Claude Haiku 4.5: 4096 tokens

**Why**: API rejects cache control for content below threshold → `400 Bad Request`.

**How to ensure**:
```go
// Generate large enough content
systemPrompt := strings.Repeat("You are a helpful assistant. ", 200) // > 1024 tokens
tools := generateLargeTools() // > 1024 tokens when serialized
```

#### 2.3 Cache Hierarchy

**Critical**: Anthropic cache is hierarchical: `tools → system → messages`

**Why**: Cache works as cumulative prefix, not independent chunks. Changing `tools` invalidates `system` and `messages` cache below it.

```mermaid
graph LR
    A[Tools] --> B[System]
    B --> C[Message 1]
    C --> D[Message 2]
    D --> E[Message 3]
    
    style A fill:#90EE90
    style B fill:#90EE90
    style C fill:#90EE90
    style D fill:#FFD700
    style E fill:#FFD700
    
    A -.invalidates.-> B
    B -.invalidates.-> C
```

**Economics**:
- First turn: Write entire prefix (125% cost)
- Turn 2+: Read prefix (10% cost) + Write new messages (125% cost)

**Example**:
```go
// Turn 1: Cache creation = 1500 tokens (tools + system + msg1)
// Turn 2: Cache read = 1500, Cache write = 50 (only msg2)
// Turn 3: Cache read = 1550, Cache write = 40 (only msg3)
```

#### 2.4 Incremental Caching Pattern

**Rule**: Cache grows incrementally, NOT cumulatively re-written.

**Why**: API intelligently writes only delta between turns → massive cost savings in long conversations.

```go
// Turn 1
r1, _ := llm.GenerateContent(ctx, messages1, opts...)
// CacheCreation: 1500 tokens (tools + system + user1)
// CacheRead: 0

// Turn 2: Add AI response + new user message
messages2 := append(messages1,
    llms.MessageContent{Role: llms.ChatMessageTypeAI, Parts: ...}, // AI1
    llms.MessageContent{Role: llms.ChatMessageTypeHuman, Parts: ...}, // User2
)
r2, _ := llm.GenerateContent(ctx, messages2, opts...)
// CacheCreation: 50 tokens (AI1 + User2 ONLY)
// CacheRead: 1500 tokens (tools + system + user1)
```

**Common mistake**: Expecting `CacheCreation` to grow cumulatively (it doesn't!).

#### 2.5 Cache Invalidation

**Triggers** (order matters):
1. Tools modification → Invalidates everything
2. System prompt change → Invalidates messages
3. Message history change → Invalidates only that message onward

**Detection**: Library includes `detectThinkingModeChange()` to warn about mode switches mid-conversation (invalidates cache).

```go
// ✓ SAFE - Consistent thinking mode
Turn 1: WithReasoning() → creates cache with thinking
Turn 2: WithReasoning() → reads cache

// ✗ DANGEROUS - Mode change invalidates cache
Turn 1: no WithReasoning() → creates cache WITHOUT thinking
Turn 2: WithReasoning() → API may reject due to missing signature
```

---

### 3. Multi-turn with Tools + Thinking

#### 3.1 Message Structure

**Rule**: AI message must contain `TextPartWithReasoning` FIRST, then tool calls.

```go
// ✓ CORRECT order
aiParts := []llms.ContentPart{
    llms.TextPartWithReasoning(choice.Content, choice.Reasoning), // 1st
    toolCall1, // 2nd
    toolCall2, // 3rd
}
messages = append(messages, llms.MessageContent{
    Role:  llms.ChatMessageTypeAI,
    Parts: aiParts,
})
```

**Why**: API expects reasoning block before tool use blocks. Incorrect order → malformed request.

#### 3.2 Workflow Example

```go
// Turn 1: User request
messages := []llms.MessageContent{
    {Role: llms.ChatMessageTypeSystem, Parts: []llms.ContentPart{...}},
    {Role: llms.ChatMessageTypeHuman, Parts: []llms.ContentPart{llms.TextPart("Analyze data")}},
}

resp1, _ := llm.GenerateContent(ctx, messages,
    llms.WithTools(tools),
    llms.WithReasoning(llms.ReasoningMedium, 2048),
    anthropic.WithPromptCaching(),
    llms.WithTemperature(1.0),
)

// Turn 2: Add AI response with reasoning + tool calls
choice1 := resp1.Choices[0]
aiParts := []llms.ContentPart{
    llms.TextPartWithReasoning(choice1.Content, choice1.Reasoning), // Must include!
}
for _, tc := range choice1.ToolCalls {
    aiParts = append(aiParts, tc)
}
messages = append(messages, llms.MessageContent{
    Role:  llms.ChatMessageTypeAI,
    Parts: aiParts,
})

// Turn 3: Add tool results
for _, tc := range choice1.ToolCalls {
    messages = append(messages, llms.MessageContent{
        Role: llms.ChatMessageTypeTool,
        Parts: []llms.ContentPart{
            llms.ToolCallResponse{
                ToolCallID: tc.ID,
                Name:       tc.FunctionCall.Name,
                Content:    toolResult,
            },
        },
    })
}

// Continue conversation...
resp2, _ := llm.GenerateContent(ctx, messages, ...)
```

---

### 4. Common Pitfalls

#### 4.1 Signature Loss

```go
// ✗ WRONG - Signature discarded
messages = append(messages, llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{
        llms.TextPart(choice.Content), // Lost reasoning!
    },
})
```

**Error**: `400 Bad Request: "missing required thinking signature"`

#### 4.2 Temperature Mismatch

```go
// ✗ WRONG - Temperature too low
llms.WithReasoning(llms.ReasoningHigh, 4096),
llms.WithTemperature(0.5), // Should be 1.0!
```

**Result**: Degraded reasoning quality or API rejection.

#### 4.3 Missing Beta Header

```go
// ✗ WRONG - Cache silently ignored
llms.GenerateContent(ctx, messages,
    anthropic.WithCacheStrategy(...), // No effect without:
    // anthropic.WithPromptCaching(), // Missing!
)
```

**Result**: Full token cost, no caching savings.

#### 4.4 Cache Threshold Not Met

```go
// ✗ WRONG - System too short for caching
systemPrompt := "You are helpful." // < 1024 tokens
```

**Error**: `400 Bad Request: "cached content below minimum threshold"`

---

### 5. Performance Tips

#### 5.1 Client-Level Cache Strategy

Set default strategy at client construction:

```go
llm, _ := anthropic.New(
    anthropic.WithDefaultCacheStrategy(anthropic.CacheStrategy{
        CacheTools:    true,
        CacheSystem:   true,
        CacheMessages: true,
        TTL:           "5m", // or "1h"
    }),
)
```

**Why**: Avoids repeating strategy in every `GenerateContent()` call.

#### 5.2 Cache Metrics

Monitor cache efficiency:

```go
resp, _ := llm.GenerateContent(ctx, messages, ...)
genInfo := resp.Choices[0].GenerationInfo

cacheRead := genInfo["CacheReadInputTokens"].(int)
cacheCreation := genInfo["CacheCreationInputTokens"].(int)

// Calculate savings:
// cacheRead costs 10% of standard tokens
// cacheCreation costs 125% of standard tokens
```

---

## Google Gemini Provider

### Thought Signatures in Tool Calls

**Critical Requirement**: Gemini requires `thought_signature` to be present in all tool calls when using models with thinking capabilities.

**Why**: Gemini validates that each tool call has an associated thinking signature. Missing signatures cause API error:
```
Function call is missing a thought_signature in functionCall parts.
This is required for tools to work correctly, and missing thought_signature
may lead to degraded model performance.
```

**Where reasoning is stored**: Gemini places reasoning in `ToolCall.Reasoning`, not in a separate TextContent like Anthropic.

```go
// Gemini pattern: reasoning in ToolCall
resp, _ := llm.GenerateContent(ctx, messages, ...)
for _, choice := range resp.Choices {
    for _, toolCall := range choice.ToolCalls {
        // Gemini stores reasoning HERE
        if toolCall.Reasoning != nil {
            // Contains thinking content and signature
            signature := toolCall.Reasoning.Signature
        }
    }
}
```

**Critical for chain management**:
1. **Never summarize the last body pair** - This preserves thought_signature for the most recent tool call
2. **Never clear reasoning** when continuing with Gemini - Only clear when switching to a different provider
3. **Preserve complete AI message** including all reasoning when building message chains

**Example of preserved structure**:

```json
{
  "role": "ai",
  "parts": [
    {
      "reasoning": {
        "Content": "Thinking about the problem...",
        "Signature": "base64_encoded_signature"
      },
      "text": "",
      "type": "text"
    },
    {
      "type": "tool_call",
      "tool_call": {
        "function": {...},
        "id": "fcall_xxx",
        "reasoning": {
          "Content": "Per-tool thinking...",
          "Signature": "another_signature"
        }
      }
    }
  ]
}
```

**Automatic protection**: The summarization algorithm automatically skips the last body pair to preserve these signatures.

---

## Google Gemini Provider (Additional Details)

### 1. Extended Thinking Mode

#### 1.1 Reasoning Location (CRITICAL DIFFERENCE from Anthropic)

**Rule**: Gemini places reasoning in **DIFFERENT locations** depending on response type.

**For text-only responses**:
```go
// ✓ CORRECT - Reasoning in ContentChoice
resp, _ := llm.GenerateContent(ctx, messages, ...)
reasoning := resp.Choices[0].Reasoning // Text response
```

**For tool call responses**:
```go
// ✓ CORRECT - Reasoning in FIRST ToolCall
resp, _ := llm.GenerateContent(ctx, messages, ...)
firstToolCall := resp.Choices[0].ToolCalls[0]
reasoning := firstToolCall.Reasoning // Tool call response!

// ContentChoice.Reasoning is nil when tool calls present
assert.Nil(resp.Choices[0].Reasoning)
```

**Why**: Gemini's API design attaches reasoning to the first tool call, not to the overall response. This differs fundamentally from Anthropic.

#### 1.2 Signature Preservation Requirements

**For function calls**: MANDATORY (400 error without signature)

```go
// ✓ CORRECT - Preserve tool call with signature
messages = append(messages, llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{
        firstToolCall, // Includes signature via extractThoughtSignature
    },
})
```

**For text responses**: RECOMMENDED (enables implicit caching, see §2.2)

```go
// ✓ CORRECT - Preserve text with signature
messages = append(messages, llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{
        llms.TextPartWithReasoning(choice.Content, choice.Reasoning),
    },
})
```

#### 1.3 Signature Placement in Parallel vs Sequential Calls

**Parallel tool calls**: Signature ONLY in first tool call

```go
resp, _ := llm.GenerateContent(ctx, messages, llms.WithTools(tools), ...)

// ✓ First tool call has signature
assert.NotNil(resp.Choices[0].ToolCalls[0].Reasoning)
assert.NotEmpty(resp.Choices[0].ToolCalls[0].Reasoning.Signature)

// ✓ Subsequent parallel calls do NOT have signature
for i := 1; i < len(resp.Choices[0].ToolCalls); i++ {
    assert.Nil(resp.Choices[0].ToolCalls[i].Reasoning)
}
```

**Sequential tool calls**: DIFFERENT signature for each call

```go
// Turn 1: First tool call
resp1, _ := llm.GenerateContent(ctx, messages1, ...)
sig1 := resp1.Choices[0].ToolCalls[0].Reasoning.Signature

// Turn 2: Second tool call (after providing first tool result)
resp2, _ := llm.GenerateContent(ctx, messages2, ...)
sig2 := resp2.Choices[0].ToolCalls[0].Reasoning.Signature

// Signatures are DIFFERENT in sequential calls
assert.NotEqual(sig1, sig2)
```

**Why**: Each sequential step has unique context, requiring new signature. Parallel calls share context, requiring only one signature.

#### 1.4 Temperature Flexibility

**No special requirements**: Unlike Anthropic, Gemini works with any temperature value.

```go
// ✓ Works fine
llm.GenerateContent(ctx, messages,
    llms.WithReasoning(llms.ReasoningMedium, 1024),
    llms.WithTemperature(0.5), // Any value is OK
)
```

---

### 2. Prompt Caching

Gemini supports TWO distinct caching mechanisms with different characteristics.

#### 2.1 Explicit Caching (Manual, Pre-created)

**How**: Create cached content separately via `CachingHelper`

```go
helper, _ := googleai.NewCachingHelper(ctx, googleai.WithAPIKey(apiKey))

// Create cache (minimum 32,768 tokens required)
cached, _ := helper.CreateCachedContent(ctx, "gemini-2.0-flash",
    []llms.MessageContent{
        {Role: llms.ChatMessageTypeSystem, Parts: []llms.ContentPart{
            llms.TextPart(largeSystemPrompt), // Must be ~24,000 words
        }},
    },
    1*time.Hour, // TTL
    "my-cache-name",
)

// Use cache in requests
resp, _ := llm.GenerateContent(ctx, messages,
    googleai.WithCachedContent(cached.Name),
)
```

**Characteristics**:
- **Minimum**: 32,768 tokens (~24,000 words)
- **Storage cost**: Hourly charge based on TTL
- **Savings**: 75% discount on cached tokens
- **Control**: Full control over what gets cached
- **Best for**: Large, reusable context (knowledge bases, documentation)

#### 2.2 Implicit Caching (Automatic, Ephemeral)

**How**: Automatically enabled, no configuration needed

```go
// Request 1: Establishes cache
resp1, _ := llm.GenerateContent(ctx, messages,
    llms.WithModel("gemini-2.5-flash"),
)

// Wait 15-30 seconds for cache to establish

// Request 2: Hits cache automatically
resp2, _ := llm.GenerateContent(ctx, messages,
    llms.WithModel("gemini-2.5-flash"),
)
```

**Characteristics**:
- **Minimum**: 1024 tokens (gemini-2.5-flash), 4096 tokens (gemini-3-flash-preview)
- **Storage cost**: FREE (ephemeral, managed by Google)
- **Delay**: 15-30 seconds between requests to establish cache
- **No guarantee**: Best-effort, not guaranteed to cache

**Cache triggers**:
1. **Identical requests**: Byte-for-byte HTTP body match → always caches
2. **Conversation continuation**: Requires signature preservation (see below)

#### 2.3 Implicit Caching for Conversation Continuation

**CRITICAL**: For multi-turn conversations, signature preservation is REQUIRED for implicit caching.

**Models**: Use `gemini-3-flash-preview` with reasoning enabled

```go
// Turn 1: Initial request
messages := []llms.MessageContent{
    {Role: llms.ChatMessageTypeSystem, Parts: []llms.ContentPart{
        llms.TextPart(largeContext), // > 4096 tokens for gemini-3
    }},
    {Role: llms.ChatMessageTypeHuman, Parts: []llms.ContentPart{
        llms.TextPart("Question 1"),
    }},
}

resp1, _ := llm.GenerateContent(ctx, messages,
    llms.WithModel("gemini-3-flash-preview"),
    llms.WithReasoning(llms.ReasoningMedium, 512), // REQUIRED
)

// Turn 2: MUST preserve signature
messages = append(messages,
    llms.MessageContent{
        Role: llms.ChatMessageTypeAI,
        Parts: []llms.ContentPart{
            llms.TextPartWithReasoning(choice1.Content, choice1.Reasoning), // CRITICAL!
        },
    },
    llms.MessageContent{
        Role: llms.ChatMessageTypeHuman,
        Parts: []llms.ContentPart{llms.TextPart("Question 2")},
    },
)

resp2, _ := llm.GenerateContent(ctx, messages,
    llms.WithModel("gemini-3-flash-preview"),
    llms.WithReasoning(llms.ReasoningMedium, 512),
)

// Check cache hit
cachedTokens := resp2.Choices[0].GenerationInfo["PromptCachedTokens"].(int)
assert.Greater(cachedTokens, 0) // Cache hit! (~89% of tokens cached)
```

**Why signature is required**: Gemini uses signature to identify conversation prefix for caching. Without signature, each turn is treated as new request → no prefix caching.

**Result**: With proper signature preservation:
- Turn 1: 0 cached tokens
- Turn 2: ~89% cached tokens (prefix: system + user1)
- Turn 3: ~92% cached tokens (prefix: system + user1 + AI1 + user2)

---

### 3. Function Call ID Management

**Auto-generation**: Gemini automatically generates IDs if not provided.

```go
// Library generates ID in format: fcall_{16_hex_chars}
// IDs are cleaned (prefix removed) when sending to API
```

**Why**: Gemini API doesn't always provide IDs, but client code needs them for tracking. Library ensures consistency.

---

### 4. Signature Deduplication (Automatic)

**Problem**: Using universal pattern (Anthropic-compatible) on Gemini may create duplicate signatures.

```go
// Universal pattern (works for Anthropic)
aiParts := []llms.ContentPart{
    llms.TextPartWithReasoning(choice.Content, choice.Reasoning), // signature here
    toolCall, // signature here too (for Gemini)
}
```

**Solution**: Library automatically deduplicates.

**Rules**:
1. If `TextContent.Text == ""` AND any `ToolCall` has signature → skip empty TextContent
2. If `TextContent.Text != ""` → keep both (text content is meaningful)
3. If no ToolCalls present → keep TextContent even if empty (preserve signature)

```go
// Example 1: Empty text + tool call with signature
Parts: [TextPart("", reasoning), ToolCall(reasoning)]
// → API receives: [ToolCall with signature] ✓

// Example 2: Non-empty text + tool call with signature
Parts: [TextPart("I'll search", reasoning), ToolCall(reasoning)]
// → API receives: [Text with signature, ToolCall with signature] ✓
// (Both kept - text content is meaningful)

// Example 3: Empty text, no tool calls
Parts: [TextPart("", reasoning)]
// → API receives: [Text with signature] ✓
// (Kept - no tool call to carry signature)
```

**Why**: Prevents API errors from duplicate signatures while preserving meaningful content.

---

### 5. Common Pitfalls

#### 5.1 Wrong Reasoning Location

```go
// ✗ WRONG - Looking for reasoning in wrong place
resp, _ := llm.GenerateContent(ctx, messages, llms.WithTools(tools), ...)
reasoning := resp.Choices[0].Reasoning // nil for tool calls!

// ✓ CORRECT
reasoning := resp.Choices[0].ToolCalls[0].Reasoning // Here!
```

#### 5.2 Missing Signature in Tool Calls

```go
// ✗ WRONG - Signature lost in roundtrip
messages = append(messages, llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{
        llms.ToolCall{...}, // No signature!
    },
})
```

**Error**: `400 Bad Request: "thought signature required for function calling"`

#### 5.3 Implicit Caching Without Signature

```go
// ✗ WRONG - Conversation continuation without signature
messages = append(messages,
    llms.MessageContent{
        Role: llms.ChatMessageTypeAI,
        Parts: []llms.ContentPart{
            llms.TextPart(choice.Content), // Missing reasoning!
        },
    },
)
```

**Result**: No implicit cache hit → full token cost for entire conversation.

#### 5.4 Explicit Cache Too Small

```go
// ✗ WRONG - Below 32,768 token minimum
systemPrompt := strings.Repeat("You are helpful. ", 100) // Only ~200 tokens
```

**Error**: `400 Bad Request: "cached content below minimum size"`

---

### 6. Performance Tips

#### 6.1 Choose Right Caching Strategy

**Use Explicit Caching when**:
- Content is static and reused across many sessions
- You need guaranteed caching (not best-effort)
- Content exceeds 32,768 tokens easily

**Use Implicit Caching when**:
- Multi-turn conversations with gemini-3-flash-preview
- Content is 4096+ tokens but < 32,768 tokens
- You want zero storage costs

#### 6.2 Monitor Cache Efficiency

```go
resp, _ := llm.GenerateContent(ctx, messages, ...)
genInfo := resp.Choices[0].GenerationInfo

cachedTokens := genInfo["PromptCachedTokens"].(int)
totalTokens := genInfo["TotalTokens"].(int)

cacheHitRate := float64(cachedTokens) / float64(totalTokens) * 100
log.Printf("Cache hit rate: %.1f%%", cacheHitRate)
```

---

### 7. Thought Signature Preservation in Chain Summarization

**Critical Issue**: When using chain summarization with Gemini models that have thinking capabilities, the last body pair must NEVER be summarized.

**Why**: Summarization removes the `thought_signature` from tool calls, causing the error:
```
Function call is missing a thought_signature in functionCall parts.
This is required for tools to work correctly.
```

**Automatic Protection**: The summarization algorithm (`chain_summary.go`) automatically:
1. **Skips the last body pair** in `summarizeOversizedBodyPairs()` 
2. **Always preserves the most recent interaction** in `determineLastSectionPairs()`
3. **Maintains reasoning signatures** for continued conversation

**Example scenario where this matters**:

```go
// Scenario: Last tool call returns large response (50KB+)
// Chain before summarization:
messages = [
    {Role: "human", Parts: ["Find SQL injection"]},
    {Role: "ai", Parts: [
        {Text: "", Reasoning: {Content: "...", Signature: "..."}},  // Gemini thinking
        {ToolCall: {Name: "pentester", Reasoning: {...}}},          // With thought_signature
    ]},
    {Role: "tool", Parts: [{Response: "...50KB of results..."}]},  // Large response
]

// After summarization: Last body pair is PRESERVED with all signatures intact
// This allows the conversation to continue without API errors
```

**What gets preserved**:
- Complete AI message with all reasoning parts
- Tool call with `thought_signature`
- Large tool response (even if 50KB+)

**What gets summarized**:
- All body pairs BEFORE the last one (if they exceed size limits)
- Older sections according to QA pair rules

**Manual override**: If you need to force summarization of all pairs (not recommended):
```go
// This will cause API errors with Gemini thinking models
// Only use if switching to a different provider
err := ast.ClearReasoning() // Removes ALL reasoning including thought_signature
```

---

## OpenAI Provider (OpenAI-Compatible)

This provider handles OpenAI cloud API and **all OpenAI-compatible backends** (OpenRouter, DeepSeek, Together, Groq, etc.).

### 1. Extended Thinking Mode

#### 1.1 No Signature Support

**CRITICAL**: OpenAI does NOT use cryptographic signatures.

```go
// ✓ CORRECT - No signature in OpenAI responses
resp, _ := llm.GenerateContent(ctx, messages,
    llms.WithReasoning(llms.ReasoningMedium, 2048),
)
assert.Nil(resp.Choices[0].Reasoning.Signature) // Always nil for OpenAI
```

**Why**: OpenAI's reasoning models (o1, o3, o4) use different validation mechanism. No signature → simpler roundtrip logic.

**Implication**: Conversation continuation works WITHOUT signature preservation (unlike Anthropic/Gemini).

```go
// ✓ Works fine for OpenAI
messages = append(messages, llms.MessageContent{
    Role: llms.ChatMessageTypeAI,
    Parts: []llms.ContentPart{
        llms.TextPart(choice.Content), // No reasoning needed!
    },
})
```

**However**: For cross-provider compatibility, still use `TextPartWithReasoning()` (harmless for OpenAI, required for others).

#### 1.2 Reasoning Content Format

**Field**: `choice.Reasoning.Content` (simple string, not object)

**Location**: Always in `choice.Reasoning`, NEVER in `ToolCall.Reasoning`

```go
// ✓ CORRECT
resp, _ := llm.GenerateContent(ctx, messages, ...)
reasoning := resp.Choices[0].Reasoning.Content // Text content here

// ✗ WRONG
reasoning := resp.Choices[0].ToolCalls[0].Reasoning // Always nil for OpenAI
```

**Why**: OpenAI treats reasoning as response-level, not tool-level (same as Anthropic, different from Gemini).

#### 1.3 Temperature Auto-Override

**Auto-set**: Library automatically sets `Temperature = 0.0` and `TopP = 0.0` for reasoning models.

```go
// Your code
llm.GenerateContent(ctx, messages,
    llms.WithModel("o3-mini"),
    llms.WithReasoning(llms.ReasoningMedium, 1000),
    llms.WithTemperature(0.7), // You set this
)

// What gets sent to API
// temperature: 0.0  ← Auto-overridden!
// top_p: 0.0        ← Auto-overridden!
```

**Why**: OpenAI reasoning models IGNORE temperature/top_p (deterministic reasoning). Library prevents confusion by enforcing zeros.

**Note**: This is OPPOSITE to Anthropic which requires `Temperature = 1.0`.

#### 1.4 Two Reasoning Formats

**Legacy format** (older models):
```go
// Request uses: reasoning_effort: "high"
```

**Modern format** (newer models, o3+):
```go
llm, _ := openai.New(
    openai.WithModernReasoningFormat(), // Enable modern format
    openai.WithUsingReasoningMaxTokens(), // Use max_tokens instead of effort
)

// Request uses: reasoning: { max_tokens: 2048 }
// or: reasoning: { effort: "medium" }
```

**Why**: OpenAI API evolved from simple `reasoning_effort` string to structured `reasoning` object. Modern format provides more control.

**Recommendation**: Use modern format for new code, legacy for compatibility.

#### 1.5 XML-Tagged Reasoning Fallback

**Auto-extraction**: Library extracts reasoning from `<think>` or `<thinking>` tags if `ReasoningContent` field is empty.

```go
// Some providers return reasoning embedded in content:
// "Here's my thought process: <think>Step 1...</think> The answer is 42"

// Library automatically extracts:
choice.Reasoning.Content = "Step 1..." // Extracted from tags
choice.Content = "Here's my thought process: The answer is 42" // Cleaned
```

**Why**: Some OpenAI-compatible providers (DeepSeek, QwQ) use XML tags instead of separate `reasoning_content` field.

---

### 2. Prompt Caching

#### 2.1 Implicit Caching Only

**CRITICAL**: OpenAI supports ONLY automatic implicit caching.

- **No explicit cache creation** (unlike Gemini's `CachingHelper`)
- **No inline cache control** (unlike Anthropic's `WithCacheStrategy`)
- **No configuration needed** (completely automatic)

```go
// ✓ CORRECT - Just use it, caching works automatically
resp, _ := llm.GenerateContent(ctx, messages,
    llms.WithModel("gpt-4.1-mini"),
)

// Cache metrics available in response
cachedTokens := resp.Choices[0].GenerationInfo["PromptCachedTokens"].(int)
cacheWriteTokens := resp.Choices[0].GenerationInfo["CacheCreationInputTokens"].(int)
```

**Why**: OpenAI manages caching internally. Client has no control over cache creation/invalidation.

#### 2.2 Cache Characteristics

- **Minimum**: 1024 tokens
- **TTL**: 5-10 minutes (managed by OpenAI)
- **Cost**: Free (no storage charges)
- **Delay**: 40+ seconds to establish cache (longer than Gemini's 15s)
- **Trigger**: Identical HTTP body OR conversation prefix match

#### 2.3 Cache Metrics

**Fields available**:
```go
genInfo := resp.Choices[0].GenerationInfo

// Standard fields
cachedTokens := genInfo["PromptCachedTokens"].(int)      // Read from cache
cacheWrite := genInfo["CacheCreationInputTokens"].(int)  // Written to cache
reasoningTokens := genInfo["ReasoningTokens"].(int)       // Reasoning tokens used

// Formula (different from Anthropic!)
// PromptTokens = total - cachedTokens
// CacheCreationInputTokens = max(cache_write_tokens, total - cached)
```

**Why different from Anthropic**: OpenAI includes cached tokens in total count, then subtracts. Anthropic reports them separately.

#### 2.4 Conversation Caching (No Signature Required)

**Key difference**: OpenAI caches conversation prefix WITHOUT signature requirement.

```go
// Turn 1
msgs := []llms.MessageContent{
    {Role: llms.ChatMessageTypeSystem, Parts: []llms.ContentPart{
        llms.TextPart(largeContext), // > 1024 tokens
    }},
    {Role: llms.ChatMessageTypeHuman, Parts: []llms.ContentPart{
        llms.TextPart("Question 1"),
    }},
}
resp1, _ := llm.GenerateContent(ctx, msgs, llms.WithModel("gpt-4.1-mini"))
// CacheWrite: > 0, CachRead: 0

// Turn 2 - Simple text part works!
msgs = append(msgs,
    llms.MessageContent{
        Role: llms.ChatMessageTypeAI,
        Parts: []llms.ContentPart{
            llms.TextPart(resp1.Choices[0].Content), // No reasoning needed!
        },
    },
    llms.MessageContent{
        Role: llms.ChatMessageTypeHuman,
        Parts: []llms.ContentPart{llms.TextPart("Question 2")},
    },
)
resp2, _ := llm.GenerateContent(ctx, msgs, llms.WithModel("gpt-4.1-mini"))
// CacheRead: > 0 (prefix cached!)
```

**Why**: OpenAI doesn't validate message integrity via signatures → simpler client code.

---

### 3. OpenAI-Compatible Backends

#### 3.1 OpenRouter Specifics

**Base URL**: `https://openrouter.ai/api/v1`

**Models**: Access to 100+ models from multiple providers through single API

```go
llm, _ := openai.New(
    openai.WithBaseURL("https://openrouter.ai/api/v1"),
    openai.WithToken(openRouterAPIKey),
)

// Use models from different providers
resp, _ := llm.GenerateContent(ctx, messages,
    llms.WithModel("anthropic/claude-sonnet-4.5"),        // Anthropic via OpenRouter
    llms.WithModel("google/gemini-2.5-flash"),            // Gemini via OpenRouter
    llms.WithModel("openai/o3-mini"),                     // OpenAI via OpenRouter
    llms.WithModel("deepseek/deepseek-r1"),               // DeepSeek via OpenRouter
)
```

**Additional metrics**:
```go
genInfo := resp.Choices[0].GenerationInfo

// OpenRouter-specific cost tracking
promptCost := genInfo["UpstreamInferencePromptCost"].(float64)
completionCost := genInfo["UpstreamInferenceCompletionsCost"].(float64)
```

**Reasoning behavior**: Depends on upstream provider
- Anthropic models via OpenRouter: May or may not return reasoning (depends on `:thinking` suffix in model name)
- Gemini models via OpenRouter: May return reasoning
- OpenAI models via OpenRouter: Typically do NOT return reasoning content

#### 3.2 DeepSeek Specifics

**Base URL**: `https://api.deepseek.com`

**Reasoning extraction**: Uses XML tag fallback

```go
llm, _ := openai.New(
    openai.WithBaseURL("https://api.deepseek.com"),
    openai.WithToken(deepseekAPIKey),
)

resp, _ := llm.GenerateContent(ctx, messages,
    llms.WithModel("deepseek-v4-pro"),
)

// Reasoning extracted from <think>...</think> tags automatically
```

#### 3.3 Compatibility Requirements

**For maximum compatibility** across OpenAI-compatible backends:

1. Use modern reasoning format:
```go
llm, _ := openai.New(
    openai.WithModernReasoningFormat(),
    openai.WithUsingReasoningMaxTokens(),
)
```

2. Don't rely on reasoning content being present:
```go
// ✓ Defensive check
if choice.Reasoning != nil && choice.Reasoning.Content != "" {
    // Process reasoning
}
```

3. Check for reasoning tokens in metrics:
```go
reasoningTokens := genInfo["ReasoningTokens"].(int)
if reasoningTokens > 0 {
    // Model used reasoning, even if content not returned
}
```

---

### 4. Common Pitfalls

#### 4.1 Expecting Signatures

```go
// ✗ WRONG - OpenAI doesn't have signatures
if choice.Reasoning != nil {
    _ = choice.Reasoning.Signature // Always nil!
}
```

**Fix**: Check provider before using signatures.

#### 4.2 Assuming Reasoning Content Always Present

```go
// ✗ WRONG - Not all models/providers return reasoning content
resp, _ := llm.GenerateContent(ctx, messages,
    llms.WithModel("o3-mini"),
    llms.WithReasoning(llms.ReasoningHigh, 4096),
)
reasoning := resp.Choices[0].Reasoning.Content // May be empty!
```

**Why**: OpenAI's `/chat/completions` endpoint doesn't always return reasoning content, only `reasoning_tokens` metric.

**Fix**: Check `ReasoningTokens` metric instead:
```go
reasoningTokens := resp.Choices[0].GenerationInfo["ReasoningTokens"].(int)
if reasoningTokens > 0 {
    // Model used reasoning internally
}
```

#### 4.3 Manual Temperature for Reasoning

```go
// ✗ UNNECESSARY - Library handles this
llm.GenerateContent(ctx, messages,
    llms.WithModel("o3-mini"),
    llms.WithReasoning(llms.ReasoningMedium, 1000),
    llms.WithTemperature(0.0), // Redundant! Auto-set by library
)
```

**Why**: Library detects reasoning models and sets temperature automatically.

#### 4.4 Expecting Cache Control

```go
// ✗ WRONG - No cache control in OpenAI
llm.GenerateContent(ctx, messages,
    anthropic.WithCacheStrategy(...), // Doesn't exist for OpenAI!
)
```

**Fix**: Rely on automatic caching, monitor via metrics.

---

### 5. Performance Tips

#### 5.1 Monitor Implicit Cache

```go
resp, _ := llm.GenerateContent(ctx, messages, ...)
genInfo := resp.Choices[0].GenerationInfo

cachedTokens := genInfo["PromptCachedTokens"].(int)
totalPrompt := genInfo["PromptTokens"].(int) + cachedTokens

if cachedTokens > 0 {
    savings := float64(cachedTokens) / float64(totalPrompt) * 100
    log.Printf("Cache hit: %.1f%% of prompt cached", savings)
}
```

#### 5.2 Optimize for Cache Hits

**Strategies**:
- Keep system prompts stable across requests
- Maintain consistent message history structure
- Wait 40+ seconds between similar requests in development

**Anti-patterns**:
- Frequently changing system prompts
- Reordering message history
- Varying metadata/options

---

## Provider Conflicts & Multi-Provider Code

When building applications that switch between providers, be aware of these **CRITICAL incompatibilities**.

### 1. Signature Support Conflict ⚠️

| Provider | Signature Support | Requirement |
|----------|------------------|-------------|
| **Anthropic** | ✓ Yes (cryptographic) | MANDATORY for roundtrip |
| **Gemini** | ✓ Yes (thought signature) | MANDATORY for tool calls, RECOMMENDED for text |
| **OpenAI** | ✗ No | N/A (not supported) |

**Solution**: Universal signature preservation (safe for all)

```go
// ✓ UNIVERSAL - Works for all providers
func buildAIMessage(choice *llms.ContentChoice) llms.MessageContent {
    var parts []llms.ContentPart
    
    // Preserve reasoning if present (required for Anthropic/Gemini, harmless for OpenAI)
    if choice.Reasoning != nil {
        parts = append(parts,
            llms.TextPartWithReasoning(choice.Content, choice.Reasoning),
        )
    } else {
        parts = append(parts, llms.TextPart(choice.Content))
    }
    
    // Add tool calls
    parts = append(parts, choice.ToolCalls...)
    
    return llms.MessageContent{
        Role:  llms.ChatMessageTypeAI,
        Parts: parts,
    }
}
```

**Why**: `TextPartWithReasoning()` preserves signatures for Anthropic/Gemini while being ignored by OpenAI.

---

### 2. Reasoning Location Conflict ⚠️

| Provider | Text Response | Tool Call Response |
|----------|--------------|-------------------|
| **Anthropic** | `choice.Reasoning` | `choice.Reasoning` |
| **Gemini** | `choice.Reasoning` | `ToolCall[0].Reasoning` |
| **OpenAI** | `choice.Reasoning` | `choice.Reasoning` |

**Solution**: Provider-aware reasoning extraction

```go
func extractReasoning(resp *llms.ContentResponse, provider string) *reasoning.ContentReasoning {
    choice := resp.Choices[0]
    
    switch provider {
    case "anthropic", "openai":
        return choice.Reasoning // Always at choice level
        
    case "gemini":
        if len(choice.ToolCalls) > 0 {
            return choice.ToolCalls[0].Reasoning // First tool for tool calls
        }
        return choice.Reasoning // Choice level for text
        
    default:
        // Universal fallback: check both locations
        if choice.Reasoning != nil {
            return choice.Reasoning
        }
        if len(choice.ToolCalls) > 0 && choice.ToolCalls[0].Reasoning != nil {
            return choice.ToolCalls[0].Reasoning
        }
        return nil
    }
}
```

---

### 3. Temperature Requirements Conflict ⚠️

| Provider | Non-Reasoning Models | Reasoning Models |
|----------|---------------------|------------------|
| **Anthropic** | Any value (0.0-1.0) | MUST be 1.0 |
| **Gemini** | Any value (0.0-2.0) | Any value |
| **OpenAI** | Any value (0.0-2.0) | Auto-set to 0.0 (override ignored) |

**Solution**: Let library handle it OR provider-specific logic

```go
// Option 1: Let library auto-handle (RECOMMENDED)
llm.GenerateContent(ctx, messages,
    llms.WithReasoning(llms.ReasoningMedium, 2048),
    // Don't set temperature - library handles it per-provider
)

// Option 2: Provider-specific temperature
func buildReasoningOptions(provider string, wantedTemp float64) []llms.CallOption {
    opts := []llms.CallOption{
        llms.WithReasoning(llms.ReasoningMedium, 2048),
    }
    
    switch provider {
    case "anthropic":
        opts = append(opts, llms.WithTemperature(1.0)) // Force 1.0
    case "openai":
        // Temperature ignored (auto-set to 0.0), but set anyway for clarity
        opts = append(opts, llms.WithTemperature(0.0))
    case "gemini":
        opts = append(opts, llms.WithTemperature(wantedTemp)) // Flexible
    }
    
    return opts
}
```

**Why**: Different providers have different temperature requirements for reasoning modes.

---

### 4. Caching Strategy Conflict ⚠️

| Provider | Mechanism | Configuration | Minimum Tokens | Cost Model |
|----------|-----------|---------------|----------------|------------|
| **Anthropic** | Inline control | `WithCacheStrategy()` | 1024 (Sonnet) / 4096 (Haiku) | Read: 10%, Write: 125% |
| **Gemini** | Explicit OR Implicit | `CachingHelper` OR signatures | 32,768 (explicit) / 4096 (implicit) | 75% discount (explicit) / Free (implicit) |
| **OpenAI** | Implicit only | Automatic | 1024 | Automatic (no extra cost) |

**Solution**: Provider-specific caching setup

```go
func setupCaching(provider string, systemPrompt string) []llms.CallOption {
    var opts []llms.CallOption
    
    switch provider {
    case "anthropic":
        // Inline cache control with strategy
        opts = append(opts,
            anthropic.WithPromptCaching(),
            anthropic.WithCacheStrategy(anthropic.CacheStrategy{
                CacheTools:    true,
                CacheSystem:   true,
                CacheMessages: true,
                TTL:           "5m",
            }),
        )
        
    case "gemini":
        // Large content: use explicit caching
        if len(systemPrompt) > 24000 {
            // Requires separate CachingHelper setup
            // opts = append(opts, googleai.WithCachedContent(cachedName))
        }
        // Small content: rely on implicit caching with signature preservation
        // Just ensure llms.WithReasoning() is used with gemini-3-flash-preview
        
    case "openai":
        // Nothing to configure - fully automatic
        // Just monitor metrics to verify cache hits
    }
    
    return opts
}
```

---

### 5. Message Construction Conflict ⚠️

**Anthropic**: `TextPartWithReasoning` FIRST, then tool calls

```go
// Anthropic - specific order required
aiParts := []llms.ContentPart{
    llms.TextPartWithReasoning(choice.Content, choice.Reasoning), // 1st
    toolCall1, // 2nd
}
```

**Gemini**: Tool calls contain reasoning, may omit text part

```go
// Gemini - tool call has signature embedded
aiParts := []llms.ContentPart{
    toolCall1, // Reasoning inside first tool call
}
```

**OpenAI**: No special order, reasoning not in parts

```go
// OpenAI - simple, no reasoning in parts
aiParts := []llms.ContentPart{
    llms.TextPart(choice.Content),
    toolCall1,
}
```

**Solution**: Universal pattern (works for all)

```go
func buildUniversalAIMessage(choice *llms.ContentChoice) llms.MessageContent {
    var parts []llms.ContentPart
    
    // Add text with reasoning if present (safe for all providers)
    if choice.Content != "" || choice.Reasoning != nil {
        parts = append(parts,
            llms.TextPartWithReasoning(choice.Content, choice.Reasoning),
        )
    }
    
    // Add all tool calls
    parts = append(parts, choice.ToolCalls...)
    
    return llms.MessageContent{
        Role:  llms.ChatMessageTypeAI,
        Parts: parts,
    }
}
```

**Why this works**:
- Anthropic: Gets required `TextPartWithReasoning` + tool calls
- Gemini: Library auto-deduplicates - skips empty text if tool call has signature
- OpenAI: Ignores reasoning in parts, uses tool calls normally

**Note**: Gemini library optimization prevents duplicate signatures:
- Empty text with signature + tool call with signature → only tool call sent (signature deduplicated)
- Non-empty text with signature + tool call with signature → both sent (text content is meaningful)

---

### 6. Reasoning Content Availability

| Provider | Returns Reasoning Content | Metric Field |
|----------|--------------------------|--------------|
| **Anthropic** | ✓ Always (when enabled) | N/A (included in response) |
| **Gemini** | ✓ Always (when enabled) | `ReasoningTokens` |
| **OpenAI** | ✗ Rarely (only tokens count) | `ReasoningTokens` |

**Solution**: Defensive reasoning access

```go
func extractReasoningContent(choice *llms.ContentChoice) string {
    // Try direct access
    if choice.Reasoning != nil && choice.Reasoning.Content != "" {
        return choice.Reasoning.Content
    }
    
    // Gemini: check first tool call
    if len(choice.ToolCalls) > 0 && choice.ToolCalls[0].Reasoning != nil {
        return choice.ToolCalls[0].Reasoning.Content
    }
    
    // Check if reasoning was used (even if content not returned)
    if reasoningTokens, ok := choice.GenerationInfo["ReasoningTokens"].(int); ok && reasoningTokens > 0 {
        return "[reasoning used but content not available]"
    }
    
    return ""
}
```

**Why**: OpenAI typically doesn't return reasoning content via `/chat/completions` endpoint, only token count.

---

### 7. Quick Reference Table

| Feature | Anthropic | Gemini | OpenAI |
|---------|-----------|--------|--------|
| **Signature** | Cryptographic (required) | Thought signature (required for tools) | None |
| **Reasoning Location (text)** | `choice.Reasoning` | `choice.Reasoning` | `choice.Reasoning` |
| **Reasoning Location (tools)** | `choice.Reasoning` | `ToolCall[0].Reasoning` | `choice.Reasoning` |
| **Temperature (reasoning)** | Must be 1.0 | Any value | Auto-set to 0.0 |
| **Caching Type** | Inline (manual) | Explicit OR Implicit | Implicit only |
| **Cache Config** | `WithCacheStrategy` | `CachingHelper` OR signatures | Automatic |
| **Cache Min Tokens** | 1024-4096 | 32,768 (explicit) / 4,096 (implicit) | 1024 |
| **Cache Economics** | Read: 10%, Write: 125% | 75% discount (explicit) / Free (implicit) | Free |
| **Beta Headers** | Required for caching | Not required | Not required |
| **Reasoning Content** | ✓ Always returned | ✓ Always returned | ✗ Usually only tokens |

---

### 8. Universal Implementation Pattern

For applications supporting **all three providers**:

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### 9. Provider Detection

**Recommendation**: Explicitly track provider type rather than inferring from model name.

```go
type ProviderType string

const (
    ProviderAnthropic ProviderType = "anthropic"
    ProviderGemini    ProviderType = "gemini"
    ProviderOpenAI    ProviderType = "openai"
)

// ✓ EXPLICIT tracking
type Config struct {
    Provider ProviderType
    Model    string
}

// ✗ FRAGILE - Don't infer from model name
func detectProvider(modelName string) ProviderType {
    if strings.Contains(modelName, "claude") {
        return ProviderAnthropic // What about claude via OpenRouter?
    }
    // ... fragile logic
}
```

---

## Summary

### Critical Actions (All Providers)

1. **Always** use `TextPartWithReasoning()` when `choice.Reasoning != nil` (universal pattern)
2. **Always** check provider-specific reasoning location (Gemini differs for tool calls)
3. **Never** assume signature presence - OpenAI doesn't support signatures
4. **Never** hardcode temperature for reasoning - let library handle it OR use provider-specific logic

### Provider-Specific Requirements

**Anthropic**:
- ✓ Signatures: REQUIRED for roundtrip (400 error if missing)
- ✓ Temperature: MUST be 1.0 for reasoning
- ✓ Caching: `WithPromptCaching()` + `WithCacheStrategy()`
- ✓ Reasoning location: Always `choice.Reasoning`
- ✓ Min cache: 1024 tokens (Sonnet) / 4096 tokens (Haiku)

**Gemini**:
- ✓ Signatures: REQUIRED for tool calls, RECOMMENDED for text (enables implicit caching)
- ✓ Temperature: Any value works
- ✓ Caching: Explicit (`CachingHelper`) OR Implicit (automatic with signatures)
- ⚠️ Reasoning location: `choice.Reasoning` for text, `ToolCall[0].Reasoning` for tools
- ✓ Min cache: 32,768 tokens (explicit) / 4,096 tokens (implicit)

**OpenAI** (+ OpenAI-compatible):
- ✗ Signatures: NOT supported (always nil)
- ✓ Temperature: Auto-set to 0.0 for reasoning (override ignored)
- ✓ Caching: Implicit only (fully automatic)
- ✓ Reasoning location: Always `choice.Reasoning`
- ✓ Min cache: 1,024 tokens
- ⚠️ Reasoning content: Often unavailable (only `ReasoningTokens` metric present)

### Cost Optimization by Provider

| Provider | Caching Strategy | Savings | Notes |
|----------|-----------------|---------|-------|
| **Anthropic** | Inline incremental | 90% on cached reads | Write: 125%, Read: 10% |
| **Gemini Explicit** | Manual pre-created | 75% discount | Storage charges apply |
| **Gemini Implicit** | Auto with signatures | FREE | Requires gemini-3-flash-preview + reasoning |
| **OpenAI** | Automatic | FREE | No control, 40s delay for cache |

### Error Prevention Matrix

| Issue | Anthropic | Gemini | OpenAI |
|-------|-----------|--------|--------|
| Missing signature | ✗ 400 error | ✗ 400 error (tool calls) | ✓ OK (not used) |
| Wrong reasoning location | ✓ OK (always choice) | ⚠️ nil access if wrong | ✓ OK (always choice) |
| Wrong temperature | ✗ Degraded quality | ✓ OK (flexible) | ✓ OK (auto-override) |
| Cache below threshold | ✗ 400 error | ✗ 400 error | ✓ OK (no caching) |
| Mode change mid-conversation | ⚠️ Cache invalidation | ⚠️ Cache invalidation | ✓ OK (auto-handled) |

### Universal Best Practices

**For maximum compatibility across all providers**:

1. **Always preserve reasoning** (library handles deduplication):
```go
llms.TextPartWithReasoning(choice.Content, choice.Reasoning)
// Library automatically prevents duplicate signatures for Gemini
```

2. **Check reasoning in both locations**:
```go
reasoning := choice.Reasoning // Anthropic, OpenAI
if reasoning == nil && len(choice.ToolCalls) > 0 {
    reasoning = choice.ToolCalls[0].Reasoning // Gemini tool calls
}
```

3. **Don't set temperature for reasoning** (let library handle):
```go
llms.WithReasoning(llms.ReasoningMedium, 2048)
// Don't add WithTemperature - library sets correct value per-provider
```

4. **Monitor cache via standard metrics**:
```go
cachedTokens := genInfo["PromptCachedTokens"].(int)
reasoningTokens := genInfo["ReasoningTokens"].(int)
```

5. **Handle missing reasoning content gracefully**:
```go
if choice.Reasoning != nil && choice.Reasoning.Content != "" {
    // Process reasoning (Anthropic, Gemini)
} else if reasoningTokens, ok := genInfo["ReasoningTokens"].(int); ok && reasoningTokens > 0 {
    // Reasoning used but content unavailable (OpenAI)
}
```

### Library Automatic Handling

The library automatically handles these provider-specific edge cases:

| Feature | Automatic Handling |
|---------|-------------------|
| **Temperature** | Auto-set to 1.0 (Anthropic) or 0.0 (OpenAI) for reasoning |
| **Signature Deduplication** | Skips empty text parts when tool call has signature (Gemini) |
| **Reasoning Extraction** | Extracts from XML tags if needed (DeepSeek, QwQ) |
| **Beta Headers** | Auto-adds `interleaved-thinking` when tools + reasoning (Anthropic) |
| **Tool Call IDs** | Auto-generates if missing (Gemini) |

---

**Version**: Based on APIs as of 2025-01
**Providers**: 
- Anthropic: Claude 3.7, Sonnet 4, Haiku 4.5+
- Gemini: gemini-2.5-flash, gemini-3-flash-preview
- OpenAI: o1, o3, o4, gpt-4.1+, and OpenAI-compatible backends

---

### Backend Docs Observability (`backend/docs/observability.md`)

# PentAGI Observability Stack

## Table of Contents
- [PentAGI Observability Stack](#pentagi-observability-stack)
  - [Table of Contents](#table-of-contents)
  - [Overview](#overview)
  - [Architecture](#architecture)
    - [Component Diagram](#component-diagram)
    - [Data Flow](#data-flow)
    - [Key Interfaces](#key-interfaces)
      - [Core Interface](#core-interface)
      - [Tracing Interface](#tracing-interface)
      - [Metrics Interface](#metrics-interface)
      - [Collector Interface](#collector-interface)
      - [Langfuse Interface](#langfuse-interface)
  - [Infrastructure Requirements](#infrastructure-requirements)
    - [Components](#components)
    - [Setup](#setup)
    - [Jaeger Storage Plugin](#jaeger-storage-plugin)
  - [Configuration](#configuration)
    - [Environment Variables](#environment-variables)
    - [Initialization](#initialization)
  - [Developer Guide](#developer-guide)
    - [Logging](#logging)
      - [Logrus Integration](#logrus-integration)
      - [Context-Aware Logging](#context-aware-logging)
      - [Log Correlation with Spans](#log-correlation-with-spans)
    - [Tracing](#tracing)
      - [Span Creation and Sampling](#span-creation-and-sampling)
      - [Context Propagation in Tracing](#context-propagation-in-tracing)
    - [Metrics](#metrics)
    - [Langfuse Integration](#langfuse-integration)
      - [Advanced Observation Types](#advanced-observation-types)
    - [Profiling](#profiling)
  - [Application Instrumentation Patterns](#application-instrumentation-patterns)
    - [HTTP Server Instrumentation](#http-server-instrumentation)
    - [GraphQL Instrumentation](#graphql-instrumentation)
  - [Best Practices](#best-practices)
    - [Context Propagation](#context-propagation)
    - [Structured Logging](#structured-logging)
    - [Meaningful Spans](#meaningful-spans)
    - [Useful Metrics](#useful-metrics)
  - [Use Cases](#use-cases)
    - [Debugging Performance Issues](#debugging-performance-issues)
    - [Monitoring LLM Operations](#monitoring-llm-operations)
    - [System Resource Analysis](#system-resource-analysis)

## Overview

The PentAGI Observability Stack provides comprehensive monitoring, logging, tracing, and metrics collection for the application. It integrates multiple technologies to provide a complete view of the application's behavior, performance, and health:

- **Logging**: Enhanced logrus integration with structured logging and context propagation
- **Tracing**: Distributed tracing with OpenTelemetry and Jaeger
- **Metrics**: Application and system metrics collection
- **Langfuse**: Specialized LLM observability
- **Profiling**: Runtime profiling capabilities

This document explains how the observability stack is designed, configured, and used by developers.

## Architecture

The Observability stack is built as a set of layered interfaces that integrate multiple observability technologies. It uses OpenTelemetry as the foundation for metrics, logs, and traces, with additional integrations for Langfuse (LLM-specific observability) and Go's native profiling.

### Component Diagram

```mermaid
flowchart TD
    App[PentAGI Application] --> Observer[Observability Interface]

    subgraph ObservabilityComponents
        Observer --> Tracer[Tracer Interface]
        Observer --> Meter[Meter Interface]
        Observer --> Collector[Collector Interface]
        Observer --> LangfuseInt[Langfuse Interface]
    end

    Tracer --> OtelTracer[OpenTelemetry Tracer]
    Meter --> OtelMeter[OpenTelemetry Meter]
    Collector --> Metrics[System Metrics Collection]
    LangfuseInt --> LangfuseClient[Langfuse Client]

    OtelTracer --> OtelCollector[OpenTelemetry Collector]
    OtelMeter --> OtelCollector
    LogrusHook[Logrus Hook] --> OtelCollector

    OtelCollector --> VictoriaMetrics[VictoriaMetrics]
    OtelCollector --> Jaeger[Jaeger]
    OtelCollector --> Loki[Loki]

    Profiling[Profiling Server] --> App

    LangfuseClient --> LangfuseBackend[Langfuse Backend]

    VictoriaMetrics --> Grafana[Grafana]
    Jaeger --> Grafana
    Loki --> Grafana
```

### Data Flow

```mermaid
sequenceDiagram
    participant App as PentAGI App
    participant Obs as Observability
    participant OTEL as OpenTelemetry
    participant Lf as Langfuse
    participant Backend as Observability Backend

    App->>Obs: Log Message
    Obs->>OTEL: Format & Forward Log
    OTEL->>Backend: Export Log

    App->>Obs: Create Span
    Obs->>OTEL: Create & Configure Span
    OTEL->>Backend: Export Span

    App->>Obs: Record Metric
    Obs->>OTEL: Format & Record Metric
    OTEL->>Backend: Export Metric

    App->>Obs: New Observation
    Obs->>Lf: Create Observation
    Lf->>Backend: Export Observation

    App->>App: Access Profiling Endpoint
```

### Key Interfaces

The observability stack is designed around several interfaces that abstract the underlying implementations:

#### Core Interface

```go
// Observability is the primary interface that combines all observability features
type Observability interface {
    Flush(ctx context.Context) error
    Shutdown(ctx context.Context) error
    Meter
    Tracer
    Collector
    Langfuse
}
```

#### Tracing Interface

```go
// Tracer provides span creation and management
type Tracer interface {
    // NewSpan creates a new span with the given kind and component name
    NewSpan(
        context.Context,
        oteltrace.SpanKind,
        string,
        ...oteltrace.SpanStartOption,
    ) (context.Context, oteltrace.Span)

    // NewSpanWithParent creates a span with explicit parent trace and span IDs
    NewSpanWithParent(
        context.Context,
        oteltrace.SpanKind,
        string,
        string,
        string,
        ...oteltrace.SpanStartOption,
    ) (context.Context, oteltrace.Span)

    // SpanFromContext extracts the current span from context
    SpanFromContext(ctx context.Context) oteltrace.Span

    // SpanContextFromContext extracts span context from context
    SpanContextFromContext(ctx context.Context) oteltrace.SpanContext
}
```

#### Metrics Interface

```go
// Meter provides metric recording capabilities
type Meter interface {
    // Various counter, gauge, and histogram creation methods for
    // both synchronous and asynchronous metrics
    NewInt64Counter(string, ...otelmetric.Int64CounterOption) (otelmetric.Int64Counter, error)
    NewFloat64Counter(string, ...otelmetric.Float64CounterOption) (otelmetric.Float64Counter, error)
    // ... other metric types (removed for brevity)
}
```

#### Collector Interface

```go
// Collector provides system metric collection
type Collector interface {
    // StartProcessMetricCollect starts collecting process metrics
    StartProcessMetricCollect(attrs ...attribute.KeyValue) error

    // StartGoRuntimeMetricCollect starts collecting Go runtime metrics
    StartGoRuntimeMetricCollect(attrs ...attribute.KeyValue) error

    // StartDumperMetricCollect starts collecting metrics from a custom dumper
    StartDumperMetricCollect(stats Dumper, attrs ...attribute.KeyValue) error
}

// Dumper interface for custom metric collection
type Dumper interface {
    DumpStats() (map[string]float64, error)
}
```

#### Langfuse Interface

```go
// Langfuse provides LLM observability
type Langfuse interface {
    // NewObservation creates a new Langfuse observation
    NewObservation(
        context.Context,
        ...langfuse.ObservationContextOption,
    ) (context.Context, langfuse.Observation)
}
```

## Infrastructure Requirements

The observability stack relies on several backend services for storing and visualizing the collected data.

### Components

The infrastructure includes the following components:

- **Grafana**: Visualization and dashboarding
- **VictoriaMetrics**: Time-series database for metrics
- **ClickHouse**: Analytical database for traces and logs
- **Loki**: Log aggregation system
- **Jaeger**: Distributed tracing system
- **OpenTelemetry Collector**: Collects, processes, and exports telemetry data
- **Node Exporter**: Exposes Linux system metrics
- **cAdvisor**: Provides container resource usage metrics

### Setup

The observability stack can be deployed using Docker Compose:

```bash
# Start the observability stack
docker-compose -f docker-compose-observability.yml up -d
```

For detailed setup instructions, refer to the README.md file in the repository.

### Jaeger Storage Plugin

Jaeger keeps its traces in ClickHouse through the `jaeger-clickhouse` gRPC storage plugin. Unlike everything else in the stack, the plugin is not an image: it is a binary bind-mounted into the Jaeger container, which loads it over gRPC at start.

```yaml
# docker-compose-observability.yml, service jaeger (abridged)
entrypoint: >
  /bin/sh -c '
  if [ "$$(uname -m)" = "x86_64" ]; then ARCH="amd64"
  elif [ "$$(uname -m)" = "aarch64" ]; then ARCH="arm64"
  ... fi &&
  /go/bin/all-in-one-linux
  --grpc-storage-plugin.binary=/etc/jaeger/bin/jaeger-clickhouse-linux-$$ARCH
  --grpc-storage-plugin.configuration-file=/etc/jaeger/plugin-config.yml'
environment:
  SPAN_STORAGE_TYPE: grpc-plugin
volumes:
  - ./observability/jaeger:/etc/jaeger:rw
```

Both files — `jaeger-clickhouse-linux-amd64` and `jaeger-clickhouse-linux-arm64` — are kept in `observability/jaeger/bin/` next to the environment file regardless of the host, because which one runs is decided *inside* the container by `uname -m`: that is the architecture of the Docker VM, not of the machine the installer runs on. The entrypoint picks the file at start.

Because it is a file and not an image, no amount of `docker compose pull` brings it up to date. The installer updates it as part of an observability stack update: it sends the sha256 of each file it finds on disk with the update check, and the answer names a hash, a version and an action per file. The digest comparison is only the first word on the matter — a file whose hash differs from the offered one is marked outdated, and then the action overrides that verdict either way: `install`, `upgrade` and `downgrade` mark the file outdated whatever the hashes say, which is also how a file that is not on disk at all — and therefore was never reported, so there was no digest to compare — gets fetched, while `unknown` marks it current. Fetched, then, is every plugin component left marked outdated that also carries a target version (an empty one leaves nothing to ask the server for) and is for `linux` (the file name says `linux`, and the plugin is a linux binary whatever the host is). For each file that needs fetching:

1. `GET /api/v1/proxy/packages/info` describes the package — its size, its sha256 and the signature over it;
2. `GET /api/v1/proxy/packages/download` is streamed into `<name>.new`, a sibling of the target, created with mode `0755`;
3. the length, the sha256 and an Ed25519 signature are verified over the whole stream;
4. only then is the staging file renamed over the target.

None of those three checks can be decided before the last byte arrives, so what is on disk before step 4 is unverified — that is why the download is staged rather than written in place: an interrupted attempt would otherwise leave a truncated plugin under the name a running Jaeger loads. The staging file is a sibling of the target because `os.Rename` is atomic only within one filesystem. The mode is set at creation and not afterwards, because the rename does not preserve the destination's mode: a plugin created `0644` and moved over an executable one ends up not executable, and Jaeger then fails with an opaque gRPC plugin error rather than a permission one. A leftover `.new` from an interrupted download is removed at the start of the next observability update, by exact name — the directory is mounted into a container and may hold an operator's own files, so nothing is deleted by glob.

The download runs *before* anything else in the stack update touches the installation: the images are pulled, and the image references the server chose are written into the environment file, only afterwards — both of those happen inside the `pull` and the `up` that follow. So a plugin that cannot be fetched aborts the stack update with nothing else disturbed.

Replacing the file does not affect a running Jaeger, and the `docker compose up -d` that follows does not either: compose recreates a container only when its *spec* changes, and a different file inside a bind mount is not a spec change. The plugin is exec'd by the container's entrypoint at start, so a container left running is still serving through the plugin it loaded when it came up. The installer therefore reports whether it actually replaced anything, and when it did it restarts the `jaeger` service explicitly once the stack is up — without that, the update reports success while nothing about it is in effect.

Both plugin paths are excluded from the installer's file-integrity verification. The copy shipped inside the installer seeds a first install, and is never written over afterwards. Without that exclusion the update would undo itself: a downloaded plugin differs from the shipped one, so verification would call it modified and offer to repair it. A stack update never repairs files, but two paths do — answering yes to the integrity prompt on Apply Changes, and a factory reset, which does it without asking — and either would copy the shipped file straight back over the downloaded one.

## Configuration

### Environment Variables

The observability stack is configured through environment variables in the application's configuration:

| Variable | Description | Example Value |
|----------|-------------|---------------|
| `OTEL_HOST` | OpenTelemetry collector endpoint | `otel:4318` |
| `LANGFUSE_BASE_URL` | Langfuse API base URL | `http://langfuse-web:3000` |
| `LANGFUSE_PROJECT_ID` | Langfuse project ID | `cm47619l0000872mcd2dlbqwb` |
| `LANGFUSE_PUBLIC_KEY` | Langfuse public API key | `pk-lf-5946031c-ae6c-4451-98d2-9882a59e1707` |
| `LANGFUSE_SECRET_KEY` | Langfuse secret API key | `sk-lf-d9035680-89dd-4950-8688-7870720bf359` |

### Initialization

The observability stack is initialized in the application through the `InitObserver` function:

```go
// Initialize clients first
lfclient, err := obs.NewLangfuseClient(ctx, cfg)
if err != nil && !errors.Is(err, obs.ErrNotConfigured) {
    log.Fatalf("Unable to create langfuse client: %v\n", err)
}

otelclient, err := obs.NewTelemetryClient(ctx, cfg)
if err != nil && !errors.Is(err, obs.ErrNotConfigured) {
    log.Fatalf("Unable to create telemetry client: %v\n", err)
}

// Initialize the observer
obs.InitObserver(ctx, lfclient, otelclient, []logrus.Level{
    logrus.DebugLevel,
    logrus.InfoLevel,
    logrus.WarnLevel,
    logrus.ErrorLevel,
})

// Start metrics collection
if err := obs.Observer.StartProcessMetricCollect(); err != nil {
    log.Printf("Failed to start process metric collection: %v", err)
}

if err := obs.Observer.StartGoRuntimeMetricCollect(); err != nil {
    log.Printf("Failed to start Go runtime metric collection: %v", err)
}

// Start profiling if needed
go profiling.Start()
```

## Developer Guide

This section explains how to use the observability stack's features in application code.

### Logging

The observability stack integrates deeply with logrus for logging. Logged messages are automatically associated with the current span and exported to the observability backend.

#### Logrus Integration

The observability package implements a logrus hook that captures log entries and incorporates them into the OpenTelemetry tracing and logging systems:

```go
// InitObserver sets up the logrus hook
obs.InitObserver(ctx, lfclient, otelclient, []logrus.Level{
    logrus.DebugLevel,
    logrus.InfoLevel,
    logrus.WarnLevel,
    logrus.ErrorLevel,
})
```

The hook is implemented in the `observer` struct, which automatically:

1. Captures log entries via its `Fire` method
2. Extracts the current span from the log entry's context
3. Creates span events for all logs
4. Translates logrus entries to OpenTelemetry logs
5. Properly formats error logs to include stack traces and error details

The implementation of the logrus hook in the observer:

```go
// Fire is a logrus hook that is fired on a new log entry.
func (obs *observer) Fire(entry *logrus.Entry) error {
    // Extract context or create a new one
    ctx := entry.Context
    if ctx == nil {
        ctx = context.Background()
    }

    // Get current span from context
    span := oteltrace.SpanFromContext(ctx)
    if !span.IsRecording() {
        // Create a new span for logs without a valid span
        component := "internal"
        if op, ok := entry.Data["component"]; ok {
            component = op.(string)
        }
        _, span = obs.NewSpanWithParent(
            ctx,
            oteltrace.SpanKindInternal,
            component,
            // ... span creation details
        )
        defer span.End()
    }

    // Add log as an event to the span
    span.AddEvent("log", oteltrace.WithAttributes(obs.makeAttrs(entry, span)...))

    // Send to OpenTelemetry log pipeline
    obs.logger.Emit(ctx, obs.makeRecord(entry, span))

    return nil
}
```

#### Context-Aware Logging

For proper trace correlation, logs should include the request context. This allows the observability system to associate logs with the correct trace and span:

```go
// WithContext is critical for associating logs with the correct trace
logrus.WithContext(ctx).Info("Operation completed")

// Without context, logs may not be associated with the correct trace
logrus.Info("This log may not be properly correlated") // Avoid this

// Example with error and fields
logrus.WithContext(ctx).WithFields(logrus.Fields{
    "user_id": userID,
    "action": "login",
}).WithError(err).Error("Authentication failed")
```

When a log entry includes a context, the observability system will:

1. Extract the active span from the context
2. Associate the log with that span
3. Include trace and span IDs in the log record
4. Ensure the log appears in the trace timeline in Jaeger

If a log entry does not include a context (or the context has no active span), the system will:

1. Create a new span for the log entry
2. Associate the log with this new span
3. This creates a "span island" that isn't connected to other parts of the trace

#### Log Correlation with Spans

The observability system enriches logs with trace and span information automatically:

```go
// Create a span
ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "process-request")
defer span.End()

// All logs with this context will be associated with the span
logrus.WithContext(ctx).Info("Starting processing")

// Even errors are properly correlated and include stacktraces
if err != nil {
    logrus.WithContext(ctx).WithError(err).Error("Processing failed")
    // The error is automatically added to the span
}
```

The observer converts logrus fields to span attributes and OpenTelemetry log records:

```go
// A log with fields
logrus.WithContext(ctx).WithFields(logrus.Fields{
    "user_id": 1234,
    "request_id": "abc-123",
    "duration_ms": 42,
}).Info("Request processed")

// Results in span attributes:
// - log.severity: INFO
// - log.message: Request processed
// - log.user_id: 1234
// - log.request_id: abc-123
// - log.duration_ms: 42
```

This integration ensures that logs, traces, and metrics all share consistent context, making it easier to correlate events across the system.

### Tracing

Traces provide a way to track the flow of requests through the system.

#### Span Creation and Sampling

Spans should be created for significant operations in the code. Each span represents a unit of work:

```go
// Create a new span for a significant operation
ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "process-request")
defer span.End()

// Always end spans, preferably with defer
```

Span creation follows these principles:

1. **Span Hierarchy**: Spans created with a context containing an active span become child spans
2. **Span Kinds**: Different span kinds represent different types of operations:
   - `SpanKindInternal`: Internal operations (default)
   - `SpanKindServer`: Handling incoming requests
   - `SpanKindClient`: Making outgoing requests
   - `SpanKindProducer`: Sending messages
   - `SpanKindConsumer`: Receiving messages

3. **Component Names**: The third parameter to `NewSpan` identifies the component and becomes the span name

4. **Empty Spans**: Even spans without explicit attributes or events (empty spans) are valuable as they:
   - Show timing of operations
   - Maintain the context propagation chain
   - Provide structure to traces

```go
// Parent operation
ctx, parentSpan := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "parent-operation")
defer parentSpan.End()

// Child operation - automatically becomes a child span in the trace
ctx, childSpan := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "child-operation")
defer childSpan.End()
```

#### Context Propagation in Tracing

Context propagation is critical for maintaining trace continuity:

```go
// Create a span in function A
ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "function-a")
defer span.End()

// Pass the context to function B
resultB := functionB(ctx, param1, param2)

// Inside function B, create a child span
func functionB(ctx context.Context, param1, param2 string) Result {
    // This will be a child span of the span in function A
    ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "function-b")
    defer span.End()

    // ...function logic...
}
```

The context carries:
1. Active spans for proper parent-child relationships
2. Trace ID for continuity across the entire request
3. Span ID for linking to the parent span
4. Trace flags (like sampling decisions)

Always use the *updated context* returned from `NewSpan()`:

```go
// CORRECT: Using the updated context
ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "operation")
// Pass the updated ctx to subsequent operations

// INCORRECT: Not using the updated context
_, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "operation")
// Subsequent operations won't be part of the same trace
```

### Metrics

Metrics provide quantitative measurements of the application's behavior:

```go
// Create a counter
requestCounter, _ := obs.Observer.NewInt64Counter(
    "app.requests.total",
    otelmetric.WithDescription("Total number of requests"),
)

// Increment the counter
requestCounter.Add(ctx, 1,
    attribute.String("endpoint", "/api/users"),
    attribute.String("method", "GET"),
)

// Create a histogram for measuring latencies
latencyHistogram, _ := obs.Observer.NewFloat64Histogram(
    "app.request.duration",
    otelmetric.WithDescription("Request duration in seconds"),
)

// Record a duration
startTime := time.Now()
// ... perform operation ...
duration := time.Since(startTime).Seconds()
latencyHistogram.Record(ctx, duration,
    attribute.String("endpoint", "/api/users"),
    attribute.String("method", "GET"),
)
```

### Langfuse Integration

Langfuse provides specialized observability for LLM operations and agentic workflows with automatic data conversion to OpenAI-compatible format:

```go
// Create a new observation for an LLM operation
ctx, observation := obs.Observer.NewObservation(ctx,
    langfuse.WithObservationTraceContext(
        langfuse.WithTraceName("flow-execution"),
        langfuse.WithTraceUserId(user.Email),
    ),
)

// LangChainGo messages are automatically converted to OpenAI format
messages := []*llms.MessageContent{
    {
        Role: llms.ChatMessageTypeHuman,
        Parts: []llms.ContentPart{
            llms.TextContent{Text: "Analyze this vulnerability"},
        },
    },
}

// Create a generation for an LLM request
generation := observation.Generation(
    langfuse.WithGenerationName("content-generation"),
    langfuse.WithGenerationModel("gpt-4"),
    langfuse.WithGenerationInput(messages),  // Auto-converted to OpenAI format
)

// Complete the generation with result
output := &llms.ContentChoice{
    Content: "Based on analysis...",
    ToolCalls: []llms.ToolCall{...},
}

generation.End(
    langfuse.WithGenerationOutput(output),  // Auto-converted to OpenAI format
    langfuse.WithGenerationUsage(&langfuse.GenerationUsage{
        Input: promptTokens,
        Output: responseTokens,
        Unit: langfuse.GenerationUsageUnitTokens,
    }),
)
```

**Key Features:**

- **Automatic Conversion**: LangChainGo messages automatically convert to OpenAI format
- **Rich UI Rendering**: Tool calls, images, and reasoning display correctly in Langfuse UI
- **Tool Call Linking**: Function names automatically added to tool responses
- **Table Rendering**: Complex tool responses (3+ keys or nested) shown as expandable tables
- **Thinking Support**: Reasoning content extracted and displayed separately

#### Advanced Observation Types

Langfuse supports additional observation types for comprehensive agentic system monitoring:

**Agent Observations** for autonomous reasoning processes:

```go
agent := observation.Agent(
    langfuse.WithAgentName("task-executor"),
    langfuse.WithAgentInput(taskDescription),
    langfuse.WithAgentMetadata(langfuse.Metadata{
        "agent_type": "executor",
        "capabilities": []string{"code_execution", "file_operations"},
    }),
)
result := executeTask(ctx, taskDescription)
agent.End(
    langfuse.WithAgentOutput(result),
    langfuse.WithAgentStatus("completed"),
)
```

**Tool Observations** for tracking tool executions:

```go
tool := observation.Tool(
    langfuse.WithToolName("search-tool"),
    langfuse.WithToolInput(searchQuery),
)
results := performSearch(ctx, searchQuery)
tool.End(
    langfuse.WithToolOutput(results),
    langfuse.WithToolStatus("success"),
)
```

**Chain Observations** for multi-step reasoning:

```go
chain := observation.Chain(
    langfuse.WithChainName("reasoning-chain"),
    langfuse.WithChainInput(messages),
    langfuse.WithChainMetadata(langfuse.Metadata{
        "steps": 3,
        "model": "gpt-4",
    }),
)
finalAnswer := executeChain(ctx, messages)
chain.End(
    langfuse.WithChainOutput(finalAnswer),
    langfuse.WithChainStatus("completed"),
)
```

**Retriever Observations** for information retrieval:

```go
retriever := observation.Retriever(
    langfuse.WithRetrieverName("vector-search"),
    langfuse.WithRetrieverInput(query),
)
documents := vectorStore.Search(ctx, query)
retriever.End(
    langfuse.WithRetrieverOutput(documents),
    langfuse.WithRetrieverStatus("success"),
)
```

**Evaluator Observations** for quality assessment:

```go
evaluator := observation.Evaluator(
    langfuse.WithEvaluatorName("quality-check"),
    langfuse.WithEvaluatorInput(response),
)
score := evaluateQuality(ctx, response)
evaluator.End(
    langfuse.WithEvaluatorOutput(score),
    langfuse.WithEvaluatorStatus("completed"),
)
```

**Embedding Observations** for vector generation:

```go
embedding := observation.Embedding(
    langfuse.WithEmbeddingName("text-embedding"),
    langfuse.WithEmbeddingInput(text),
)
vector := generateEmbedding(ctx, text)
embedding.End(
    langfuse.WithEmbeddingOutput(vector),
    langfuse.WithEmbeddingStatus("success"),
)
```

**Guardrail Observations** for safety checks:

```go
guardrail := observation.Guardrail(
    langfuse.WithGuardrailName("safety-filter"),
    langfuse.WithGuardrailInput(userInput),
)
passed, violations := checkSafety(ctx, userInput)
guardrail.End(
    langfuse.WithGuardrailOutput(map[string]any{
        "passed": passed,
        "violations": violations,
    }),
    langfuse.WithGuardrailStatus(fmt.Sprintf("passed=%t", passed)),
)
```

For detailed information about Langfuse integration, data conversion, and advanced patterns, see [Langfuse Integration Documentation](langfuse.md).

### Profiling

The observability stack includes a profiling server that exposes Go's standard profiling endpoints:

```go
// The profiling server starts automatically when profiling.Start() is called
// It runs on port 7777 by default

// Access profiles using:
// - CPU profile: http://localhost:7777/profiler/profile
// - Heap profile: http://localhost:7777/profiler/heap
// - Goroutine profile: http://localhost:7777/profiler/goroutine
// - Block profile: http://localhost:7777/profiler/block
// - Mutex profile: http://localhost:7777/profiler/mutex
// - Execution trace: http://localhost:7777/profiler/trace
```

You can use standard Go tools to collect and analyze profiles:

```bash
# Collect a 30-second CPU profile
go tool pprof http://localhost:7777/profiler/profile

# Collect a heap profile
go tool pprof http://localhost:7777/profiler/heap

# Collect a 5-second execution trace
curl -o trace.out http://localhost:7777/profiler/trace?seconds=5
go tool trace trace.out
```

## Application Instrumentation Patterns

PentAGI uses several patterns for instrumenting different parts of the application. These patterns demonstrate best practices for integrating the observability stack.

### HTTP Server Instrumentation

The PentAGI application uses a Gin middleware to instrument HTTP requests, located in `pkg/server/logger/logger.go`:

```
/* Detailed source-code truncated for AI context efficiency. */
```

This middleware:
1. Creates structured logs with HTTP request and response details
2. Includes the request context for trace correlation
3. Logs errors for failed requests (status >= 400)
4. Measures and logs request duration

To use this middleware in your Gin application:

```go
// Setup the router with the logging middleware
router := gin.New()
router.Use(logger.WithGinLogger("api-service"))
```

### GraphQL Instrumentation

PentAGI also provides instrumentation for GraphQL operations:

```go
// WithGqlLogger creates middleware that instruments GraphQL operations
func WithGqlLogger(service string) func(ctx context.Context, next graphql.ResponseHandler) *graphql.Response {
    return func(ctx context.Context, next graphql.ResponseHandler) *graphql.Response {
        // Create a span for the GraphQL operation
        ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindServer, "graphql.handler")
        defer span.End()

        // Record start time
        start := time.Now()
        entry := logrus.WithContext(ctx).WithField("component", service)

        // Execute the GraphQL operation
        res := next(ctx)

        // Add operation details to the logs
        op := graphql.GetOperationContext(ctx)
        if op != nil && op.Operation != nil {
            entry = entry.WithFields(logrus.Fields{
                "operation_name": op.OperationName,
                "operation_type": op.Operation.Operation,
            })
        }

        // Add duration information
        entry = entry.WithField("duration", time.Since(start).String())

        // Log errors if present
        if res == nil {
            return res
        }

        if len(res.Errors) > 0 {
            entry = entry.WithField("gql.errors", res.Errors.Error())
            entry.Error("graphql request handled with errors")
        } else {
            entry.Debug("graphql request handled success")
        }

        return res
    }
}
```

This middleware:
1. Creates a span for each GraphQL operation
2. Attaches operation name and type to the logs
3. Records operation duration
4. Logs any GraphQL errors
5. Uses context propagation to maintain the trace

To use this middleware in your GraphQL server:

```go
// Configure the GraphQL server with the logging middleware
srv := handler.NewDefaultServer(generated.NewExecutableSchema(generated.Config{
    Resolvers: &graph.Resolver{},
}))
srv.AroundOperations(logger.WithGqlLogger("graphql-service"))
```

## Best Practices

### Context Propagation

Always propagate context through your application to maintain trace continuity:

```go
// Pass context to functions and methods
func ProcessRequest(ctx context.Context, req Request) {
    // Use the context for spans, logs, etc.
    logrus.WithContext(ctx).Info("Processing request")

    // Pass the context to downstream functions
    result, err := fetchData(ctx, req.ID)
}
```

### Structured Logging

Use structured logging with consistent field names:

```go
// Define common field names
const (
    FieldUserID     = "user_id"
    FieldRequestID  = "request_id"
    FieldComponent  = "component"
)

// Use them consistently
logrus.WithFields(logrus.Fields{
    FieldUserID:    user.ID,
    FieldRequestID: reqID,
    FieldComponent: "auth-service",
}).Info("User authenticated")
```

### Meaningful Spans

Create spans that represent logical operations:

```go
// Good: spans represent logical operations
ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "validate-user-input")
defer span.End()

// Bad: spans are too fine-grained or too coarse
ctx, span := obs.Observer.NewSpan(ctx, obs.SpanKindInternal, "process-entire-request")
defer span.End()
```

### Useful Metrics

Design metrics to answer specific questions:

```go
// Good: metrics that help troubleshoot
cacheHitCounter, _ := obs.Observer.NewInt64Counter("cache.hits")
cacheMissCounter, _ := obs.Observer.NewInt64Counter("cache.misses")

// Good: metrics with dimensions
requestCounter.Add(ctx, 1,
    attribute.String("status", status),
    attribute.String("endpoint", endpoint),
)
```

## Use Cases

### Debugging Performance Issues

When facing performance issues, you can:

1. Check traces to identify slow operations
2. Look at metrics for system resource usage
3. Examine logs for errors or warnings
4. Use the profiling tools to identify CPU, memory, or concurrency bottlenecks

Example workflow:

```bash
# 1. Look at traces in Jaeger UI to identify slow spans

# 2. Check resource metrics in Grafana

# 3. Collect and analyze profiles
go tool pprof http://localhost:7777/profiler/profile
(pprof) top10  # Show the top 10 functions by CPU usage
(pprof) web    # Generate a graph visualization
```

### Monitoring LLM Operations

For LLM-related issues:

1. Check Langfuse observations for specific flows
2. Look at trace spans to understand the context of LLM calls
3. Examine metrics for token usage, latency, and error rates

### System Resource Analysis

To understand system resource usage:

1. Check process metrics in Grafana
2. Look at Go runtime metrics to understand memory usage
3. Use CPU and memory profiles to identify resource-intensive functions

```bash
# Collect a memory profile
go tool pprof http://localhost:7777/profiler/heap
(pprof) top10  # Show the top 10 functions by memory usage
(pprof) list someFunction  # Show memory usage in a specific function
```

---

### Backend Docs Ollama (`backend/docs/ollama.md`)

# Ollama Provider

The Ollama provider enables PentAGI to use local language models through the [Ollama](https://ollama.ai/) server.

## Installation

1. Install Ollama server on your system following the [official installation guide](https://ollama.ai/download)
2. Start the Ollama server (usually runs on `http://localhost:11434`)
3. Pull required models: `ollama pull gemma3:1b`

## Configuration

Configure the Ollama provider using environment variables:

### Required Variables

```bash
# Ollama server URL (default: http://localhost:11434)
OLLAMA_SERVER_URL=http://localhost:11434
```

### Optional Variables

```bash
# Default model for inference (optional, default: llama3.1:8b-instruct-q8_0)
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0

# Path to custom config file (optional)
OLLAMA_SERVER_CONFIG_PATH=/path/to/ollama_config.yml

# Model management settings (optional)
OLLAMA_SERVER_PULL_MODELS_TIMEOUT=600        # Timeout for model downloads in seconds
OLLAMA_SERVER_PULL_MODELS_ENABLED=false      # Auto-download models on startup
OLLAMA_SERVER_LOAD_MODELS_ENABLED=false      # Load model list from server

# Proxy URL if needed
PROXY_URL=http://proxy:8080
```

### Advanced Configuration

Control how PentAGI interacts with your Ollama server:

**Model Management:**

- **Auto-pull Models** (`OLLAMA_SERVER_PULL_MODELS_ENABLED=true`): Automatically downloads models specified in config file on startup
- **Pull Timeout** (`OLLAMA_SERVER_PULL_MODELS_TIMEOUT`): Maximum time to wait for model downloads (default: 600 seconds)
- **Load Models List** (`OLLAMA_SERVER_LOAD_MODELS_ENABLED=true`): Queries Ollama server for available models via API

**Performance Note:** Enabling `OLLAMA_SERVER_LOAD_MODELS_ENABLED` adds startup latency as PentAGI queries the Ollama API. Disable if you only need specific models from config file.

**Recommended Settings:**

```bash
# Fast startup (static config)
OLLAMA_SERVER_MODEL=llama3.1:8b-instruct-q8_0
OLLAMA_SERVER_PULL_MODELS_ENABLED=false
OLLAMA_SERVER_LOAD_MODELS_ENABLED=false

# Auto-discovery (dynamic config)
OLLAMA_SERVER_PULL_MODELS_ENABLED=true
OLLAMA_SERVER_PULL_MODELS_TIMEOUT=900
OLLAMA_SERVER_LOAD_MODELS_ENABLED=true
```

## Supported Models

The provider **dynamically loads models** from your local Ollama server. Available models depend on what you have installed locally.

**Popular model families include:**

- **Gemma models**: `gemma3:1b`, `gemma3:2b`, `gemma3:7b`, `gemma3:27b`
- **Llama models**: `llama3.1:7b`, `llama3.1:8b`, `llama3.1:8b-instruct-q8_0`, `llama3.1:8b-instruct-fp16`, `llama3.1:70b`, `llama3.2:1b`, `llama3.2:3b`, `llama3.2:90b`
- **Qwen models**: `qwen2.5:1.5b`, `qwen2.5:3b`, `qwen2.5:7b`, `qwen2.5:14b`, `qwen2.5:32b`, `qwen2.5:72b`
- **DeepSeek models**: `deepseek-r1:1.5b`, `deepseek-r1:7b`, `deepseek-r1:8b`, `deepseek-r1:14b`, `deepseek-r1:32b`
- **Embedding models**: `nomic-embed-text`

To see available models on your system: `ollama list`
To download new models: `ollama pull <model-name>`

## Features

- **Dynamic model discovery**: Automatically detects models installed on your Ollama server (when enabled)
- **Model caching**: Use only configured models without API calls (when load disabled)
- **Local inference**: No API keys required, models run locally
- **Auto model pulling**: Models are automatically downloaded when needed (when enabled)
- **Agent specialization**: Different agent types (assistant, coder, pentester) with optimized settings
- **Tool support**: Supports function calling for compatible models
- **Streaming**: Real-time response streaming
- **Custom configuration**: Override default settings with YAML config files
- **Zero pricing**: Local models have no usage costs

## Agent Types

The provider supports all PentAGI agent types with optimized configurations:

- `simple`: General purpose chat (temperature: 0.2)
- `assistant`: AI assistant tasks (temperature: 0.2)
- `coder`: Code generation (temperature: 0.1, max tokens: 6000)
- `pentester`: Security testing (temperature: 0.3, max tokens: 8000)
- `generator`: Content generation (temperature: 0.4)
- `refiner`: Content refinement (temperature: 0.3)
- `searcher`: Information searching (temperature: 0.2, max tokens: 3000)
- And more...

## Custom Configuration

Create a custom config file to override default settings:

```yaml
simple:
  model: "llama3.1:8b-instruct-q8_0"
  temperature: 0.2
  top_p: 0.3
  n: 1
  max_tokens: 4000

coder:
  model: "deepseek-r1:8b"
  temperature: 0.1
  top_p: 0.2
  n: 1
  max_tokens: 8000
```

Then set `OLLAMA_SERVER_CONFIG_PATH` to the file path.

## Pricing

Ollama provides free local inference - no usage costs or API limits.

## Example Usage

```bash
# Set environment variables
export OLLAMA_SERVER_URL=http://localhost:11434

# Start PentAGI with Ollama provider
./pentagi
```

## Troubleshooting

1. **Connection errors**: Ensure Ollama server is running and accessible
2. **Model not found**: Pull the model first with `ollama pull <model-name>`
3. **Performance issues**: Use smaller models for faster inference or upgrade hardware
4. **Memory issues**: Monitor system memory usage with larger models

---

### Backend Docs Prompt Engineering Openai (`backend/docs/prompt_engineering_openai.md`)

# A Comprehensive Guide to Writing Effective Prompts for AI Agents

## Introduction

This guide provides essential principles and best practices for creating high-performing prompts for AI agent systems, with a particular focus on the latest generation models. Based on extensive research and testing, these recommendations will help you design prompts that elicit optimal AI responses across various use cases.

## Core Principles of Effective Prompt Engineering

### 1. Structure and Organization

**Clear Hierarchical Structure**
- Use meaningful sections with clear hierarchical organization (titles, subtitles)
- Start with role definition and objectives, followed by specific instructions
- Place instructions at both the beginning and end of long context prompts
- Example framework:
  ```
  # Role and Objective
  # Instructions
  ## Sub-categories for detailed instructions
  # Reasoning Steps
  # Output Format
  # Examples
  # Context
  # Final instructions
  ```

**Effective Delimiters**
- Use Markdown for general purposes (titles, code blocks, lists)
- Use XML for precise wrapping of sections and nested content
- Use JSON for highly structured data, especially in coding contexts
- Avoid JSON format for large document collections

### 2. Instruction Clarity and Specificity

**Be Explicit and Unambiguous**
- Modern AI models follow instructions more literally than previous generations
- Make instructions specific, clear, and unequivocal
- Use active voice and directive language
- If behavior deviates from expectations, a single clear clarifying instruction is usually sufficient

**Provide Complete Context**
- Include all necessary information for the agent to understand the task
- Clearly define the scope and boundaries of what the agent should and should not do
- Specify any constraints or requirements for the output

### 3. Agent Workflow Guidance

**Enable Persistence and Autonomy**
- Instruct the agent to continue until the task is fully resolved
- Include explicit instructions to prevent premature termination of the process
- Example: "You are an agent - please keep going until the user's query is completely resolved, before ending your turn and yielding back to the user."

**Encourage Tool Usage**
- Direct the agent to use available tools rather than guessing or hallucinating
- Provide clear descriptions of each tool and its parameters
- Example: "If you are not sure about information pertaining to the user's request, use your tools to gather the relevant information: do NOT guess or make up an answer."

**Induce Planning**
- Prompt the agent to plan and reflect before and after each action
- Encourage step-by-step thinking and analysis
- Example: "You MUST plan extensively before each function call, and reflect extensively on the outcomes of the previous function calls."

### 4. Reasoning and Problem-Solving

**Chain-of-Thought Prompting**
- Instruct the agent to think step-by-step for complex problems
- Request explicit reasoning before arriving at conclusions
- Use phrases like "think through this carefully" or "break this down"
- Basic instruction example: "First, think carefully step by step about what is needed to answer the query."

**Structured Problem-Solving Approach**
- Guide the agent through a specific methodology:
  1. Analysis: Understanding the problem and requirements
  2. Planning: Creating a strategy to approach the problem
  3. Execution: Performing the necessary steps
  4. Verification: Checking the solution for correctness
  5. Iteration: Improving the solution if needed

### 5. Output Control and Formatting

**Define Expected Output Format**
- Provide clear instructions on how the output should be structured
- Use examples to demonstrate desired formatting
- Specify any required sections, headers, or organizational elements

**Set Response Parameters**
- Define tone, style, and level of detail expected
- Specify any technical requirements (e.g., code formatting, citation style)
- Indicate whether to include explanations, summaries, or step-by-step breakdowns

## Special Considerations for Specific Use Cases

### 1. Coding and Technical Tasks

**Precise Tool Definitions**
- Use API-parsed tool descriptions rather than manual injection
- Name tools clearly to indicate their purpose
- Provide detailed descriptions in the tool's "description" field
- Keep parameter descriptions thorough but concise
- Place usage examples in a dedicated examples section

**Working with Code**
- Provide clear context about the codebase structure
- Specify the programming language and any framework requirements
- For file operations, use relative paths and specify the expected format
- For code changes, explain both what to change and why
- For diffs and patches, use context-based formats rather than line numbers

**Diff Generation Best Practices**
- Use formats that include both original and replacement code
- Provide sufficient context (3 lines before/after) to locate code precisely
- Use clear delimiters between old and new code
- For complex files, include class/method identifiers with @@ operator

### 2. Long Context Handling

**Context Size Management**
- Optimize for best performance at 1M token context window
- Be aware that performance may degrade as more items need to be retrieved
- For complex reasoning across large contexts, break tasks into smaller chunks

**Context Reliance Settings**
- Specify whether to use only provided context or blend with model knowledge
- For strict adherence to provided information: "Only use the documents in the provided External Context to answer. If you don't know the answer based on this context, respond 'I don't have the information needed to answer that'"
- For flexible approach: "By default, use the provided external context, but if other basic knowledge is needed, and you're confident in the answer, you can use some of your own knowledge"

### 3. Customer-Facing Applications

**Voice and Tone Control**
- Define the personality and communication style
- Provide sample phrases to guide tone while avoiding repetition
- Include instructions for handling difficult or prohibited topics

**Interaction Flow**
- Specify greeting and closing formats
- Detail how to maintain conversation continuity
- Include instructions for when to ask follow-up questions vs. ending the interaction

## Troubleshooting and Optimization

### Common Issues and Solutions

**Instruction Conflicts**
- Check for contradictory instructions in your prompt
- Remember that instructions placed later in the prompt may take precedence
- Ensure examples align with written rules

**Over-Compliance**
- If the agent follows instructions too rigidly, add flexibility clauses
- Include conditional statements: "If you don't have enough information, ask the user"
- Add permission to use judgment: "Use your best judgment when..."

**Repetitive Outputs**
- Instruct the agent to vary phrases and expressions
- Avoid providing exact quotes the agent might repeat
- Include diversity instructions: "Ensure responses are varied and not repetitive"

### Iterative Improvement Process

1. Start with a basic prompt following the structure guidelines
2. Test with representative examples of your use case
3. Identify patterns in suboptimal responses
4. Address specific issues with targeted instructions
5. Validate improvements through testing
6. Continue refining based on performance

## Implementation Example

Below is a sample prompt template for an AI agent tasked with prompt engineering:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Conclusion

Effective prompt engineering is both an art and a science. By following these guidelines and continuously refining your approach based on results, you can create prompts that consistently produce high-quality outputs from AI agent systems. Remember that the field is evolving rapidly, and staying current with best practices will help you maximize the capabilities of the latest AI models.

---

### Backend Docs Prompt Engineering Pentagi (`backend/docs/prompt_engineering_pentagi.md`)

# PentAGI Prompt Engineering Guide

A comprehensive framework for designing high-performance prompts within the PentAGI penetration testing system. This guide provides specialized principles for creating prompts that leverage the multi-agent architecture, memory systems, security tools, and specific operational context of PentAGI.

## Understanding Cognitive Aspects of Language Models

**Model Processing Fundamentals**
- Language models process information via attention mechanisms, giving higher weight to specific parts of the input.
- Position matters: Content at the beginning and end of prompts receives more attention and is processed more thoroughly.
- LLMs follow instructions more literally than humans expect; be explicit rather than implicit.
- Task decomposition improves performance: Break complex tasks into simpler, sequential steps.
- Models have no actual memory or consciousness; simulate these through explicit context and instructions.

**Priming and Contextual Influence**
- Information provided early shapes how later information is interpreted and processed.
- Set expectations clearly at the beginning to guide the model's approach to the entire task.
- Use consistent terminology throughout to avoid confusing the model with synonym switching.
- Brief examples often provide clearer guidance than lengthy explanations.
- Be aware that unintended priming can occur through choice of words, examples, or framing.

## Core Principles for PentAGI Prompts

### 1. Structure and Organization

**Clear Hierarchical Structure**
- Use Markdown headings (`#`, `##`, `###`) for clear visual hierarchy and logical grouping of instructions. Ensure a logical flow from high-level role definition to specific protocols and requirements.
- Begin with a clear definition of the agent's specific **role** (e.g., Orchestrator, Pentester, Searcher), its primary **objective** within the PentAGI workflow, and any overarching **security focus**.
- Place critical **operational constraints** (security, environment) early in the prompt for high visibility.
- Use separate, clearly marked sections for key areas:
    - `CORE CAPABILITIES / KNOWLEDGE BASE`
    - `OPERATIONAL ENVIRONMENT` (including `<container_constraints>`)
    - `COMMAND & TOOL EXECUTION RULES` (including `<terminal_protocol>`, `<tool_usage_rules>`)
    - `MEMORY SYSTEM INTEGRATION` (including `<memory_protocol>`)
    - `TEAM COLLABORATION & DELEGATION` (including `<team_specialists>`, `<delegation_rules>`)
    - `SUMMARIZATION AWARENESS PROTOCOL` (including `<summarized_content_handling>`)
    - `EXECUTION CONTEXT` (detailing use of `{{.ExecutionContext}}`)
    - `COMPLETION REQUIREMENTS`
- Ensure instructions are **specific**, **unambiguous**, use **active voice**, and are directly relevant to the agent's function within PentAGI.

**Semantic XML Delimiters**
- Use descriptive XML tags (e.g., `<container_constraints>`, `<terminal_protocol>`, `<memory_protocol>`, `<team_specialists>`, `<summarized_content_handling>`) to logically group related instructions, especially for complex protocols and constraints requiring precise adherence by the LLM.
- Maintain **consistent tag naming and structure** across all agent prompts for shared concepts (like summarization handling or team specialists) to ensure predictability.
- Use nesting appropriately (e.g., defining individual `<specialist>` tags within `<team_specialists>`). Refer to existing templates like `primary_agent.tmpl` for examples.

**Context Window Optimization**
- Prioritize information based on importance; place critical instructions at the beginning and end.
- Use compression techniques for lengthy information: summarize when possible, link to references instead of full inclusion.
- Break down extremely complex prompts into logical, manageable sections with clear transitions.
- For recurring boilerplate sections, consider using shorter references to standardized protocols.
- Use consistent formatting and avoid redundant information that consumes token space.

*Example Structure:*
```markdown
# [AGENT SPECIALIST TITLE]

[Role definition, primary objective, and security focus relevant to PentAGI]

## CORE CAPABILITIES / KNOWLEDGE BASE
[Agent-specific skills, knowledge areas relevant to PentAGI tasks]

## OPERATIONAL ENVIRONMENT
<container_constraints>...</container_constraints>

## COMMAND & TOOL EXECUTION RULES
<terminal_protocol>...</terminal_protocol>
<tool_usage_rules>...</tool_usage_rules>

## MEMORY SYSTEM INTEGRATION
<memory_protocol>...</memory_protocol>

## TEAM COLLABORATION & DELEGATION
<team_specialists>...</team_specialists>
<delegation_rules>...</delegation_rules>

## SUMMARIZATION AWARENESS PROTOCOL
<summarized_content_handling>...</summarized_content_handling>

## EXECUTION CONTEXT
[Explain how to use {{.ExecutionContext}} for Flow/Task/SubTask details]

## COMPLETION REQUIREMENTS
[Numbered list: Output format, final tool usage, language, reporting needs]

{{.ToolPlaceholder}}
```

### 2. Agent-Specific Instructions

**Role-Based Customization**
- Tailor instructions, tone, knowledge references, and complexity directly to the agent's specialized role within the PentAGI system (Orchestrator, Pentester, Searcher, Developer, Adviser, Memorist, Installer). Explicitly reference `ai-concepts.mdc` for role definitions.
- Enforce stricter command protocols and safety measures for agents with direct system/tool access (Pentester, Maintenance/Installer).
- Include references to specialized knowledge bases or toolsets relevant to the agent's function (e.g., specific security tools from `security-tools.mdc` for Pentester; search strategies and tool priorities for Searcher).
- Clearly define inter-agent communication protocols, especially delegation criteria and the expected format/content of information exchange between agents.

**Security and Operational Boundaries**
- Explicitly state the **scope** of permitted actions and **security constraints**. Reference `security-tools.mdc` for general tool security context.
- For engagement-level boundaries, start from the reusable [scope-of-work pentest prompt template](../../examples/prompts/scope_of_work_pentest.md) and adapt the allowed targets, out-of-scope targets, stop conditions, and evidence expectations before the flow starts.
- Define **Docker container limitations** within `<container_constraints>`, populated by template variables like `{{.DockerImage}}`, `{{.Cwd}}`, `{{.ContainerPorts}}`. Specify restrictions clearly (e.g., "No direct host access," "No GUI applications," "No UDP scanning").
- Specify **forbidden actions** clearly. Use **ALL CAPS** for critical security warnings, permissions, or prohibitions (e.g., "DO NOT attempt to install new software packages," "ONLY execute commands related to the current SubTask").
- Emphasize working **strictly within the scope of the current `SubTask`**. The agent must understand its current objective based on `{{.ExecutionContext}}` and not attempt actions related to other SubTasks or the overall Flow goal unless explicitly instructed within the current SubTask. Reference `data-models.mdc` and `controller.md` for task/subtask relationships.

**Ethical Boundaries and Safety**
- Explicitly include ethics guidance relevant to penetration testing context: legal compliance, responsible disclosure, data protection.
- Specify techniques for identifying and mitigating potential risks in generated prompts.
- Establish explicit guidelines for avoiding harmful outputs, jailbreaking, or prompt injection vulnerabilities.
- Include a verification step requiring agents to review outputs for potentially harmful consequences.
- Create clear escalation paths for handling edge cases requiring human judgment.

### 3. Agentic Capabilities and Persistence

**Agent Persistence Protocol**
- Include **explicit instructions** about persistence: "You are an agent - continue working until the subtask is fully completed. Do not prematurely end your turn or yield control back to the user/orchestrator until you have achieved the specific objective of your current subtask."
- Emphasize the agent's responsibility to **drive the interaction forward** autonomously and maintain momentum until a definitive result (success or failure with clear explanation) is achieved.
- Provide clear termination criteria so the agent knows precisely when its work on the subtask is considered complete.

**Planning and Reasoning**
- Instruct agents to **explicitly plan before acting**, especially for complex security operations or tool usage: "Before executing commands or invoking tools, develop a clear step-by-step plan. Think through each stage of execution, potential failure points, and contingency approaches."
- Encourage **chain-of-thought reasoning**: "When analyzing complex security issues or ambiguous results, think step-by-step through your reasoning process. Break down problems into components, consider alternatives, and justify your approach before moving to execution."
- For critical security tasks, mandate a **validation step**: "After obtaining results, verify they are correct and complete before proceeding. Cross-check findings using alternative methods when possible."

**Chain-of-Thought Engineering**
- Structure reasoning processes explicitly: problem analysis → decomposition → solution of subproblems → synthesis.
- Encourage splitting complex reasoning into discrete, traceable steps with clear transitions.
- Implement verification checkpoints throughout reasoning chains to validate intermediate conclusions.
- For complex decisions, instruct the model to evaluate multiple approaches before selecting one.
- Include prompts for explicit reflection on assumptions made during reasoning processes.

**Error Handling and Adaptation**
- Provide explicit guidance on **handling unexpected errors**: "If a command fails, do not simply repeat the same exact command. Analyze the error message, modify your approach based on the specific error, and try an alternative method if necessary."
- Define a **maximum retry threshold** (typically 3 attempts) for similar approaches before pivoting to a completely different strategy.
- Include instructions for **graceful degradation**: "If the optimal approach fails, fall back to simpler or more reliable alternatives rather than abandoning the task entirely."

**Metacognitive Processes**
- Instruct agents to periodically evaluate their own reasoning and progress toward goals.
- Include explicit steps for identifying and questioning assumptions made during problem-solving.
- Implement self-verification protocols: "After formulating a solution, critically review it for flaws or edge cases."
- Encourage steelmanning opposing viewpoints to strengthen reasoning and avoid blind spots.
- Provide mechanisms for agents to express confidence levels in their conclusions or recommendations.

### 4. Memory System Integration

**Memory Operations Protocol (`<memory_protocol>`)**
- Provide explicit, actionable instructions on *when* and *how* to interact with PentAGI's vector memory system. Reference `ai-concepts.mdc` (Memory section).
- **Crucially, specify the primary action:** Agents MUST **always attempt to retrieve relevant information from memory first** using retrieval tools (e.g., `{{.SearchGuideToolName}}`, `{{.SearchAnswerToolName}}`) *before* performing external actions like web searches or running discovery tools.
- Define clear criteria for *storing* new information: Only store valuable, novel, and reusable knowledge (e.g., confirmed vulnerabilities, successful complex command sequences, effective troubleshooting steps, reusable code snippets) using storage tools (e.g., `{{.StoreGuideToolName}}`, `{{.StoreAnswerToolName}}`). Avoid cluttering memory with trivial or intermediate results.
- Specify the exact tool names (`{{.ToolName}}`) for memory interaction.

**Vector Database Awareness**
- Guide agents on formulating effective **semantic search queries** for memory retrieval, leveraging keywords and concepts relevant to the current task context.
- If applicable, define knowledge categorization or metadata usage for more precise memory storage and retrieval (e.g., types like 'guide', 'vulnerability', 'tool_usage', 'code_snippet').

### 5. Multi-Agent Team Collaboration

**Team Specialist Definition (`<team_specialists>`)**
- Include a complete, accurate roster of **all available specialist agents** within PentAGI (searcher, pentester, developer, adviser, memorist, installer).
- For each specialist, clearly define:
    - `skills`: Core competencies.
    - `use_cases`: Specific situations or types of problems they should be delegated.
    - `tools`: General categories of tools they utilize (not the specific invocation tool name).
    - `tool_name`: The **exact tool name variable** (e.g., `{{.SearchToolName}}`, `{{.PentesterToolName}}`) used to invoke/delegate to this specialist.
- Ensure this section is consistently defined, especially in the Orchestrator prompt and any other agent prompts that allow delegation.

**Delegation Rules (`<delegation_rules>`)**
- Define clear, unambiguous criteria for *when* an agent should delegate versus attempting a task independently. A common rule is: "Attempt independent solution using your own tools/knowledge first. Delegate ONLY if the task clearly falls outside your core skills OR if a specialist agent is demonstrably better equipped to handle it efficiently and accurately."
- Mandate that **COMPREHENSIVE context** MUST be provided with every delegation request. This includes: background information, the specific objective of the delegated task, relevant data/findings gathered so far, constraints, and the expected format/content of the specialist's output.
- Instruct the delegating agent on how to handle, verify, and integrate the results received from specialists into its own workflow.

### 6. Tool-Specific Execution Rules

**Terminal Command Protocol (`<terminal_protocol>`)**
- Reinforce that commands execute within an isolated Docker container (`{{.DockerImage}}`) and that the **working directory (`{{.Cwd}}`) is NOT persistent between tool calls**.
- Mandate **explicit directory changes (`cd /path/to/dir && command`)** within a single tool call if a specific path context is required for `command`.
- Require **absolute paths** for file operations (reading, writing, listing) whenever possible to avoid ambiguity.
- Specify **timeout handling** (if controllable via parameters) and output redirection (`> file.log 2>&1`) for potentially long-running commands.
- **Limit repetition of *identical* failed commands** (e.g., maximum 3 attempts). Encourage trying variations or different approaches upon failure.
- Encourage the use of non-interactive flags (e.g., `-y`, `--assume-yes`, `--non-interactive`) where safe and appropriate to avoid hangs.
- Define when to use `detach` mode if available/applicable for background tasks.

**Tool Definition and Invocation Best Practices**
- Name tools clearly to indicate their purpose and function (e.g., `SearchGuide`, not just `Search`)
- Provide detailed yet concise descriptions in the tool's documentation
- For complex tools, include parameter examples showing proper usage
- Emphasize that **all actions MUST use structured tool calls** - the system operates exclusively through proper tool invocation
- Explicitly prohibit "simulating" or "describing" tool usage

**Search Tool Prioritization (`<search_tools>`)**
- Define an explicit **hierarchy or selection logic** for using different search tools (Internal Memory first, then potentially Browser for specific URLs, Google/DuckDuckGo for general discovery, Tavily/Perplexity/Traversaal for complex research/synthesis). Refer to `searcher.tmpl` for a good example matrix structure.
- Include tool-specific guidance (e.g., "Use `browser` tool only for accessing specific known URLs, not for general web searching," "Use `tavily` for in-depth technical research questions").
- Define **action economy rules:** Limit the total number of search tool calls per query/subtask (e.g., 3-5 max). Instruct the agent to **stop searching as soon as sufficient information is found** to fulfill the request or subtask objective. Do not exhaust all search tools unnecessarily.

**Mandatory Result Delivery Tools**
- Clearly specify the **exact final tool** (e.g., `{{.HackResultToolName}}` for Pentester, `{{.SearchResultToolName}}` for Searcher, `{{.FinalyToolName}}` for Orchestrator) that an agent **MUST** use to deliver its final output, report success/failure, and signify the completion of its current subtask.
- Define the expected structure of the output within this final tool call (e.g., "result" field contains the detailed findings/answer, "message" field contains a concise summary or status update). This signals completion to the controlling system (`controller.md`).

### 7. Context Preservation and Summarization

**Summarization Awareness Protocol (`<summarized_content_handling>`)**
- **This entire protocol section, as defined in `primary_agent.tmpl`, `pentester.tmpl`, etc., MUST be included verbatim in *all* agent prompts.**
- **Emphasize Key Points:**
    - Clearly define the two forms of system-generated summaries (Tool Call Summary via `{{.SummarizationToolName}}`, Prefixed Summary via `{{.SummarizedContentPrefix}}`).
    - Instruct agents to treat summaries *strictly* as **historical records of actual past events, tool executions, and their results**. They are *not* examples to be copied.
    - Mandate extracting useful information from summaries (past commands, successes, failures, errors, findings) to inform current strategy and **avoid redundant actions**.
    - **Strictly prohibit** agents from: mimicking summary formats, using the `{{.SummarizedContentPrefix}}`, or calling the `{{.SummarizationToolName}}` tool.
    - **Reinforce:** The PentAGI system operates **exclusively via structured tool calls.** Any attempt to simulate actions or results in plain text will fail.

**Execution Context Awareness**
- Instruct agents to **actively utilize the information provided in the `{{.ExecutionContext}}` variable.**
- Explain that this variable contains structured details about the current **Flow, Task, and SubTask** (IDs, Status, Titles, Descriptions), as managed by the `controller` package (`backend/docs/controller.md`).
- Agents *must* use this context to understand their precise current objective, operational scope, relationship to parent tasks/flows, and potentially relevant history within the current operational branch.

### 8. Environment Awareness

**Container Constraints (`<container_constraints>`)**
- Clearly define the **Docker runtime environment** using template variables: `{{.DockerImage}}` (image name), `{{.Cwd}}` (working directory), `{{.ContainerPorts}}` (available ports).
- Specify **resource limitations** (e.g., default command timeouts) and **operational restrictions** derived from PentAGI's secure execution model (No GUI, No host access, No UDP scanning, No arbitrary software installation). Reference `security-tools.mdc`.

**Available Tools (`<tools>`)**
- For agents like the Pentester, explicitly **list the specific security testing tools** confirmed to be available within their container environment. Reference the list in `pentester.tmpl` and cross-check with `security-tools.mdc`.
- Provide version-specific guidance or known limitations if necessary.

## Effective Few-Shot Learning

**Example Selection and Structure**
- Include diverse, representative examples that demonstrate expected behavior across different scenarios.
- Structure examples consistently: input conditions → reasoning process → output format.
- Order examples from simple to complex to establish foundational patterns before edge cases.
- When space is limited, prioritize examples that demonstrate difficult or non-obvious aspects of the task.
- Ensure examples demonstrate all critical behaviors mentioned in the instructions.

**Example Implementation**
- Format examples using clear delimiters like XML tags, markdown blocks, or consistent headings.
- For each example, explicitly show both the process (reasoning, planning) and the outcome.
- Include examples of both successful operations and appropriate error handling.
- If possible, annotate examples with brief explanations of why specific approaches were taken.
- Ensure examples reflect the exact output format requirements.

## Handling Ambiguity and Uncertainty

**Ambiguity Resolution Strategies**
- Establish clear protocols for handling incomplete or ambiguous information.
- Define a hierarchy of information sources to consult when clarification is needed.
- Include explicit instructions for requesting additional information when necessary.
- Specify how to present multiple interpretations when a definitive answer isn't possible.
- Mandate expression of confidence levels for conclusions based on uncertain data.

**Conflict Resolution**
- Define a clear hierarchy of priorities for resolving conflicting requirements.
- Establish explicit rules for handling contradictory information from different sources.
- Include a protocol for identifying and surfacing contradictions rather than making assumptions.
- Specify when to defer to specific authorities (documentation, security policies) in case of conflicts.
- Provide a framework for transparently documenting resolution decisions when conflicts are encountered.

## Language Model Optimization

**Structured Tool Invocation is Mandatory**
- **Reiterate:** *All* actions, queries, commands, memory operations, delegations, and final result reporting **MUST** be performed via **structured tool calls** using the correct tool name variable (e.g., `{{.ToolName}}`).
- **Explicitly state:** Plain text descriptions or simulations of actions (e.g., writing "Running command `nmap -sV target.com`") **will not be executed** by the system.
- Use consistent template variables for tool names (see list below).
- Ensure prompts clearly specify expected parameters for critical tool calls.

**Completion Requirements Section**
- Always end prompts with a clearly marked section (e.g., `## COMPLETION REQUIREMENTS`) containing a **numbered list** of final instructions.
- Include a reminder about language: Respond/report in the user's/manager's preferred language (`{{.Lang}}`).
- Specify the required **final output format** and the **mandatory final tool** to use for delivery (e.g., `MUST use "{{.HackResultToolName}}" to deliver the final report`).
- **Crucially, place the `{{.ToolPlaceholder}}` variable at the very end of the prompt.** This allows the system backend to correctly inject tool definitions for the LLM.

### LLM Instruction Following Characteristics

**Modern LLM Instruction Following**
- Understand that newer LLMs (like those used in PentAGI) follow instructions **more literally and precisely** than previous generations. Make instructions explicit and unambiguous, avoiding indirect or implied guidance.
- Use **directive language** rather than suggestions: "DO X" instead of "You might want to do X" when the action is truly required.
- For critical behaviors, use **clear, unequivocal instructions** rather than lengthy explanations. A single direct statement is often more effective than paragraphs of background.
- When creating prompts, remember that if agent behavior deviates from expectations, a single clear corrective instruction is usually sufficient to guide it back on track.

**Literal Adherence vs. Intent Inference**
- Design prompts with the understanding that PentAGI agents will **follow the letter of instructions** rather than attempting to infer unstated intent.
- Make all critical behaviors explicit rather than relying on the agent to infer them from context or examples.
- If you need the agent to reason through problems rather than following a rigid process, explicitly instruct it to "think step-by-step" or "consider alternatives before deciding."

### Prompt Template Variables

**Essential Context Variables**
- Ensure prompts utilize essential context variables provided by the PentAGI backend:
    - `{{.ExecutionContext}}`: **Critical.** Provides structured details (IDs, status, titles, descriptions) about the current `Flow`, `Task`, and `SubTask`. Essential for scope and objective understanding.
    - `{{.Lang}}`: Specifies the preferred language for agent responses and reports.
    - `{{.CurrentTime}}`: Provides the execution timestamp for context.
    - `{{.DockerImage}}`: Name of the Docker image the agent operates within.
    - `{{.Cwd}}`: Default working directory inside the Docker container.
    - `{{.ContainerPorts}}`: Available/mapped ports within the container environment.

**Standardized Tool Name Variables**
- Use the consistent naming pattern for all tool invocation variables:
    - *Specialist Invocation:*
        - `{{.SearchToolName}}`
        - `{{.PentesterToolName}}`
        - `{{.CoderToolName}}`
        - `{{.AdviceToolName}}`
        - `{{.MemoristToolName}}`
        - `{{.MaintenanceToolName}}`
    - *Memory Operations:*
        - `{{.SearchGuideToolName}}` (Retrieve Guide)
        - `{{.StoreGuideToolName}}` (Store Guide)
        - `{{.SearchAnswerToolName}}` (Retrieve Answer/General)
        - `{{.StoreAnswerToolName}}` (Store Answer/General)
        - `{{.SearchCodeToolName}}` (*Likely needed*) (Retrieve Code Snippet)
        - `{{.StoreCodeToolName}}` (*Likely needed*) (Store Code Snippet)
    - *Result Delivery:*
        - `{{.HackResultToolName}}` (Pentester Final Report)
        - `{{.SearchResultToolName}}` (Searcher Final Report)
        - `{{.FinalyToolName}}` (Orchestrator Subtask Completion Report)
    - *System & Environment Tools:*
        - `{{.SummarizationToolName}}` (**System Use Only** - Marker for historical summaries)
        - `{{.TerminalToolName}}` (*Assumed name for terminal function*)
        - `{{.FileToolName}}` (*Assumed name for file operations function*)
        - `{{.BrowserToolName}}` (*Assumed name for browser/scraping function*)
        - *Ensure this list is kept synchronized with the actual tool names defined and passed by the backend.*

## Prompt Patterns and Anti-Patterns

**Effective Patterns**
- **Progressive Disclosure**: Introduce concepts in layers of increasing complexity.
- **Explicit Ordering**: Number steps or use clear sequence markers for sequential operations.
- **Task Decomposition**: Break complex tasks into clearly defined subtasks with their own guidelines.
- **Parameter Validation**: Include instructions for validating inputs before proceeding with operations.
- **Fallback Chains**: Define explicit alternatives when primary approaches fail.

**Common Anti-Patterns**
- **Overspecification**: Providing too many constraints that paralyze decision-making.
- **Conflicting Priorities**: Giving contradictory guidance without clear hierarchy.
- **Vague Success Criteria**: Failing to define when a task is considered complete.
- **Implicit Assumptions**: Relying on unstated knowledge or context.
- **Tool Ambiguity**: Unclear guidance on which tools to use for specific situations.

## Iterative Prompt Improvement

**Systematic Diagnosis**
- When prompts underperform, systematically isolate the issue: is it in task definition, reasoning guidance, tool usage, or output formatting?
- Document specific patterns of failure to address in revisions.
- Use controlled testing with identical inputs to validate improvements.
- Maintain version history with clear annotations about changes and their effects.
- Focus on targeted, minimal changes rather than wholesale rewrites when refining.

**Improvement Metrics**
- Define objective success criteria for prompt performance before making changes.
- Measure improvements across specific dimensions: accuracy, completeness, efficiency, robustness.
- Test prompts against edge cases and unusual inputs to ensure generalizability.
- Compare performance across different LLM providers to ensure consistency.
- Document both successful and unsuccessful prompt modifications to build institutional knowledge.

## Multimodal Integration

**Text-Visual Integration**
- When referencing visual elements, use precise descriptive language and spatial relationships.
- Define protocols for describing and referencing images, diagrams, or visualizations.
- For security-relevant visual information, instruct agents to extract and document specific details systematically.
- Establish clear formats for describing visual evidence in reports and documentation.
- Include guidance on when to request visual confirmation versus relying on textual descriptions.

## Agent-Specific Guidelines Summary

### Primary Agent (Orchestrator)
- **Focus**: Task decomposition, delegation orchestration, context management across subtasks, final subtask result aggregation.
- **Key Sections**: `TEAM CAPABILITIES`, `OPERATIONAL PROTOCOLS` (esp. Task Analysis, Boundaries, Delegation Efficiency), `DELEGATION PROTOCOL`, `SUMMARIZATION AWARENESS PROTOCOL`, `COMPLETION REQUIREMENTS` (using `{{.FinalyToolName}}`).
- **Critical Instructions**: Gather context *before* delegating, strictly enforce current subtask scope, provide *full* context upon delegation, manage execution attempts/failures, report subtask completion status and comprehensive results using `{{.FinalyToolName}}`.

### Pentester Agent
- **Focus**: Hands-on security testing, execution of tools (`nmap`, `sqlmap`, etc.), vulnerability exploitation, evidence collection and documentation.
- **Key Sections**: `KNOWLEDGE MANAGEMENT` (Memory Protocol), `OPERATIONAL ENVIRONMENT` (Container Constraints), `COMMAND EXECUTION RULES` (Terminal Protocol), `PENETRATION TESTING TOOLS` (list available), `TEAM COLLABORATION`, `DELEGATION PROTOCOL`, `SUMMARIZATION AWARENESS PROTOCOL`, `COMPLETION REQUIREMENTS` (using `{{.HackResultToolName}}`).
- **Critical Instructions**: Check memory first, strictly adhere to terminal rules & container constraints, use only listed available tools, delegate appropriately (e.g., exploit development to Coder), provide detailed, evidence-backed exploitation reports using `{{.HackResultToolName}}`.

#### Pentesting Methodology Checklist for Prompt Authors
- Encode authorization boundaries explicitly. Prompts should remind the agent to test only approved targets, respect engagement scope, and avoid destructive actions unless the task requires them.
- Start with coverage before exploitation. Instruct the agent to map routes, roles, inputs, file handling, integrations, and trust boundaries before choosing attack paths.
- Organize testing by attack surface. Good prompts group checks around authentication, access control, injection, cross-site scripting, server-side request forgery, file processing, and business logic instead of presenting a random payload dump.
- Prefer low-risk validation first. Reflection markers, controlled payloads, timing checks, and out-of-band verification should be used deliberately to confirm hypotheses before deeper exploitation.
- Require evidence at every stage. Prompts should ask for captured requests, responses, tool output, prerequisites, and impact notes so confirmed findings can move directly into a report.
- Use memory and iteration intentionally. The agent should record confirmed dead ends, revisit promising leads with new context, and avoid repeating the same failed checks.
- End with actionable reporting. A strong pentesting prompt tells the agent to summarize what was confirmed, what remains unverified, how the issue can be reproduced, and which follow-up actions are justified.

#### Recommended Reference Material
- Use public methodology resources such as [HackTricks](https://book.hacktricks.wiki/en/index.html) and [Pentest Book](https://pentestbook.six2dez.com/) as inspiration for attack-surface coverage and testing depth.
- Translate those references into concise phases, priorities, and verification rules for the agent instead of copying long checklists into the system prompt verbatim.
- Keep prompt examples aligned with live PentAGI assets such as [`backend/pkg/templates/prompts/pentester.tmpl`](../pkg/templates/prompts/pentester.tmpl) and [`examples/prompts/base_web_pentest.md`](../../examples/prompts/base_web_pentest.md).

### Searcher Agent
- **Focus**: Highly efficient information retrieval (internal memory & external sources), source evaluation and prioritization, synthesis of findings.
- **Key Sections**: `CORE CAPABILITIES` (Action Economy, Search Optimization), `SEARCH TOOL DEPLOYMENT MATRIX`, `OPERATIONAL PROTOCOLS` (Search Efficiency, Query Engineering), `SUMMARIZATION AWARENESS PROTOCOL`, `SEARCH RESULT DELIVERY` (using `{{.SearchResultToolName}}`).
- **Critical Instructions**: **Always prioritize memory search** (`{{.SearchAnswerToolName}}`), strictly limit the number of search actions, use the right tool for the query complexity (Matrix), **stop searching once sufficient information is gathered**, deliver concise yet comprehensive synthesized results via `{{.SearchResultToolName}}`.

*(Guidelines for Developer, Adviser, Memorist, Installer agents should be developed following this structure, focusing on their unique roles, tools, and interactions based on their specific implementations and prompt templates).*

## Prompt Maintenance and Evolution

### Version Control and Documentation
- Store all prompt templates consistently within the `backend/pkg/templates/prompts/` directory.
- Use a clear and consistent naming pattern: `<agent_role>[_optional_specifier].tmpl`.
- Include version information or brief changelog comments within the templates themselves or in associated documentation.
- Document the purpose, expected template variables (`{{.Variable}}`), and the general input/output behavior for each prompt template. Ensure this documentation stays synchronized with the backend code that populates the variables.

### Testing and Refinement
- Utilize the `ctester` utility (`backend/cmd/ctester/`) for validating LLM provider compatibility and basic prompt adherence (e.g., JSON formatting, function calling capabilities) for different agent types. Reference `development-workflow.mdc` / `README.md`.
- Employ the `ftester` utility (`backend/cmd/ftester/`) for **in-depth testing** of specific agent functions and prompt behaviors within realistic contexts (Flow/Task/SubTask). This is crucial for debugging complex interactions and prompt logic.
- Actively analyze agent performance, errors, and interaction traces using observability tools like **Langfuse**. Identify patterns where prompts are misunderstood, lead to inefficient actions, or violate protocols.
- Refine prompts iteratively based on `ctester`, `ftester`, and Langfuse analysis. Test changes thoroughly before deployment.
- Verify prompt changes across different supported LLM providers to ensure consistent behavior.
- Regularly validate that XML structures are well-formed and consistently applied across prompts.

### Prompt Evolution Workflow
- Document successful vs. unsuccessful prompt patterns to build institutional knowledge
- Identify areas where agents commonly misunderstand instructions or violate protocols
- Focus refinement efforts on critical sections with highest impact on performance
- Test prompt changes systematically with controlled variables
- When adding new agent types or specializations, adapt existing templates rather than creating entirely new structures

### Prompt Debugging Guide
- When agents act incorrectly, first check: Are instructions contradictory? Are priorities clear? Is context sufficient?
- For reasoning failures, examine if the problem has been properly decomposed and if verification steps exist.
- For tool usage errors, verify tool descriptions and examples are clear and parameters well-defined.
- When memory usage is suboptimal, check memory protocol clarity and retrieval/storage guidance.
- Document common failure modes to address in future prompt revisions.

## Implementation Examples

*(Refer to the actual, up-to-date files in `backend/pkg/templates/prompts/` such as `primary_agent.tmpl`, `pentester.tmpl`, and `searcher.tmpl` for concrete implementation patterns that follow these guidelines.)*

---

### Frontend Docs E2e (`frontend/docs/e2e.md`)

# Frontend E2E tests (Playwright)

End-to-end tests for the PentAGI UI. The default tier runs fully offline against
network mocks — **no backend, no secrets, no VPN, no LLM keys** — so anyone
(including fork-PR authors) can run it, and CI runs it on every pull request.

Driving a real stack by hand — through the browser and through raw requests, to
prove a fix or to reach a surface no cassette covers — is a different job with
its own traps: [`live-testing.md`](./live-testing.md).

## One-time bootstrap

```bash
cd frontend
nvm use                  # Node from .nvmrc
corepack enable          # once per Node install, activates the pinned pnpm
pnpm install
pnpm e2e:setup           # downloads the Chromium build (~300MB)
```

On Linux, browser system dependencies may be needed once:
`pnpm exec playwright install --with-deps chromium` (requires sudo).
On Windows, use WSL — native Windows is untested.

## Running

```bash
pnpm e2e                 # mock tier (default): builds the production bundle,
                         # serves it via `vite preview`, mocks the whole API
pnpm e2e:ui              # same, in Playwright UI mode (watch/debug)
CI=1 pnpm e2e            # byte-identical reproduction of a CI run
                         # (same retries and trace policy; CI-style reporters)

./e2e/tools/run-local-tier.sh          # Tier 2: branch image + isolated docker
                                       # stack + mock LLM; runs specs/real/**
E2E_TIER=stand E2E_BASE_URL=https://… pnpm e2e   # live stand: @stand smoke only
                                       # (flow-run drives a real paid agent run
                                       # and is scoped to the local tier)

pnpm e2e:visual                        # visual snapshots (pinned container)
pnpm e2e:visual:update                 # regenerate baselines after a UI change
```

Every mock-tier run rebuilds the bundle and starts its own `vite preview` —
deliberately: reusing a server already listening on the port would silently
test a previous commit's dist. A stray listener on 8100 fails the run loudly;
kill it and rerun.

## Tiers

| Tier             | Backend                                                                                                                                                                      | Needs                  | Used for                                                                 |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | ------------------------------------------------------------------------ |
| `mock` (default) | Playwright route/WS mocks replaying cassettes against the **production bundle**                                                                                              | nothing                | the PR gate; every cassette spec                                         |
| `local`          | branch-built image in an isolated compose stack (`pentagi-e2e`, ports 8444/5433 — coexists with a dev stack) + an OpenAI-compatible **mock LLM** driving the real agent loop | docker                 | `specs/real/**`: the fidelity check — real GraphQL/WS/pub-sub end to end |
| `stand`          | a live stand                                                                                                                                                                 | `E2E_BASE_URL` + creds | deploy smoke, version-skew checks                                        |

Tier-2 notes: the runner isolates the stack from your `.env` (`--env-file
/dev/null`), seeds flow ids from 90001 so sandbox containers
(`pentagi-terminal-<id>`) never collide with a developer stack on the same
docker daemon, and removes those sandboxes on exit. The deterministic agent
transcript lives in `e2e/mock-llm/scenario.mjs`. Iterating: `E2E_SKIP_BUILD=1`
reuses the image, `E2E_KEEP_STACK=1` leaves the stack up.

Retries differ by tier, and that changes what a green means. The mock tier runs
`retries: 0` — a pinned clock, locale and network make a retry-recovered failure a
real race rather than infrastructure noise. The `local` and `stand` tiers run
`retries: 1`, so a spec that fails once and passes on retry is reported **flaky**
and the run still exits `0`. That is an unfinished question, not a pass:
`specs/real/flow-run.spec.ts` allows 30s for the redirect off `/flows/new` after
**Submit**, which a host busy with a docker build misses while the retry passes in
seconds. Re-run the tier on an idle host before treating a flaky result as a
defect — and if it survives that, it is a race worth fixing, not worth tagging.

**A wall of timeouts is the host, not the diff.** The mock tier runs `retries: 0`
and every worker carries its own jsdom/browser context, so a machine that is busy
with something else turns the 5s `expect` deadline into mass failure. Measured on
one box while a game held ~5 cores: the unit suite reported 130 failures across 39
files, `--maxWorkers=3` on the same tree reported 15, and each of those files
passed alone. The signature is unmistakable — every failure reads
`Test timed out` / `Hook timed out`, most of them in a setup step (waiting for a
header button to enable) rather than in the assertion the spec is about, and the
failing files have nothing to do with what changed. Check `uptime` and re-run one
failing file by itself before reading a single trace; the same suite went
`1472 passed` on the idle box minutes later.

**A repeat on the retry is the diff; a timeout is the host.** The paragraph above
sends you to the host first, and that is right for a wall of `Test timed out`.
It is wrong for a single spec that fails twice with the same assertion message:
measured on 10.09.2026 at load average 26.8, `specs/real/flow-run.spec.ts` — the
one this file names as the usual suspect — passed in 16s while
`specs/real/account-password.spec.ts` failed identically on the run and on its
retry, and that was a real regression in the branch. Read the failure text
before reading `uptime`.

Read the outcome from `e2e/test-results/.last-run.json`, not from the exit status
of a pipeline: `pnpm e2e | tail` reports `tail`'s status, so a run whose summary
you piped away says `0` while `.last-run.json` says `"status": "failed"`. Both
tiers write that file, and it is the one artifact a pipe cannot rewrite.

A finished Tier-2 run leaves the backend and mock-LLM logs in
`e2e/test-results/compose.log`, captured before `down -v` destroys the containers;
CI publishes that directory as `e2e-local-report` (`e2e-report` is the mock job's).
`E2E_KEEP_STACK=1` writes no such file — the stack is still up, so read the logs
from it.

**`specs/real/flow-run.spec.ts` passes, so a red run is a signal, not noise to wave
through.** Its failures show as a 30s wait for the redirect off `/flows/new` after
**Submit**, or a 90s wait for `Linux` in the terminal buffer. Before blaming the
branch under test, build its base commit in a worktree and run the spec there: the
comparison says whether the branch broke it. Everything else in the tier passes.

**The mock tier cannot test the order a collection arrives in.** The cassette's
array *is* the order — `createdAt` on a fixture changes nothing, because no
server sorts it. A spec written against timestamps there passes whatever you do
to them. What the tier can hold is the client half: that nothing re-sorts what it
was handed (see `flows/tabs.spec.ts`). Server ordering is held by
`e2e/list-order-gate.unit.test.ts` for every collection the cache mirrors, and by
`backend/pkg/server/services/resources_order_gate_test.go` for the one REST
collection that has its own gate. It
reads `../../backend/sqlc/models` directly (so it throws in a frontend-only
checkout), compares each collection's direction with what `src/lib/list-order.ts`
declares, and separately requires every `created_at` order inside
`screenshots.sql` and `subtasks.sql` to be ASC. Three collections are declared
`unmirrored` — `flowFiles`, `knowledgeDocuments`, `resources` — because the cache
does not follow the server's order for them. Two of the three are checked by
nothing; `resources` is the exception, held by the Go gate named above.

**A page that refetches instead of writing the cache needs two answers.**
`/settings/api-tokens` answers each of its three subscription frames by refetching
`apiTokens` rather than writing the collection, so a cassette that lists one answer
serves the same list back and the case proves nothing. List the state before the
event and the state after it, in that order — entries are consumed in turn.

Note also that two mechanisms deliver such an event, and either one alone is enough:
the central `subscriptionToCacheFieldMap` in `src/lib/apollo.ts` writes the
collection, and the page refetches on top of it. A mutation test that silences one
of them will not redden the spec; silence both.

Three more structural gates live in `src/lib/apollo.test.ts`: every collection a
subscription writes to declares an order, every subscription in the operations
document either routes to a cache field or is named as writing no list, and every
membership subscription selects at least what its collection's query reads.

## Visual snapshots

Baselines live in the repo (`e2e/specs/visual/*-snapshots/`, linux-suffixed)
and are generated ONLY inside the pinned `mcr.microsoft.com/playwright`
container — never run the visual project on the host: macOS pixels produce
parallel baselines that will never match CI. `pnpm e2e:visual` derives the
image tag from the installed `@playwright/test` version, builds `dist` on the
host, and compares inside the container; `pnpm e2e:visual:update` regenerates
baselines (commit them with the UI change that caused the diff). The xterm
canvas is masked — WebGL rendering is driver-dependent. The CI `e2e-visual`
job is advisory and never a required check.

A red visual run on a busy machine is usually the machine. The specs wait 5s for
their landmark and then shoot the page, so a starved container yields both kinds of
noise: "element(s) not found" after 30s and a 6%-of-pixels diff on a page that had
not finished painting — and it picks different specs each run, which is the tell.
Measured on a host carrying dozens of other containers: 4 then 6 failures at the
default parallelism, 20 of 20 with `--workers=1`. Pass it through:
`E2E_SKIP_BUILD=1 ./e2e/tools/run-visual.sh --workers=1`.

## Debugging a red CI run

1. Open the failed run (link in the PR comment) and download the `e2e-report`
   artifact.
2. `pnpm exec playwright show-trace <path-to>/trace.zip` — full timeline,
   network, console, and DOM snapshots for each failed test.
3. Reproduce locally with `CI=1 pnpm e2e`.

A red run with `0 failed` and a large skipped count is not a test failure: the CI
mock job is capped at ten minutes of wall clock (`globalTimeout`, one worker), and
Playwright reports the specs it never reached as skipped, so `results.json` keeps
`unexpected` at zero and only the job's own conclusion carries the abort. That is
duration creep — compare the `e2e-trend` artifact against earlier runs instead of
opening traces or re-running the job.

A spec that fails once in ten runs, with a failure snapshot showing the page it
came *from*, is usually asserting a state the product only passes through. The
clone spec did exactly that: the providers fixture ships `custom` disabled, the
create form bounces a clone of a disabled type straight back to the list, and the
assertion won whenever it caught the form in the ~50 ms before the redirect. The
trace tells this apart from a slow render — its frame snapshots carry the address,
and there the address had already gone back. The fix is to assert something that
survives: the value a field was seeded with, the toast, the address the app
settles on. `--repeat-each=10 --workers=4` reproduces this class of flake in under
a minute.

The same trap wears a second face: a role query — and `toBeHidden()` with it — answers
"gone" for the whole page while a modal dialog stands, because Radix marks the rest of the
tree inert. A spec that reads a row straight after confirming a delete therefore passes
whether or not the product removed anything. Wait for the dialog to leave
(`await expect(page.getByRole('dialog')).toHaveCount(0)`) before asserting on what is behind
it — proven here by unwiring `flowFileDeleted` from `subscriptionToCacheFieldMap`, which the
spec only noticed once that wait was in place.

A test failing with `every API call the app made must have a cassette entry`
means the app issued a request the cassette does not cover — the failure lists
the exact method/operation. Add the missing entry to the spec's cassette.

## Layout

```
e2e/
  playwright.config.ts   # tiers via E2E_TIER; mock tier builds+serves the prod bundle
  fixtures/              # merged `test`: backend/cassette options, auth seeding, error log
  mocks/                 # cassette-driven GraphQL + REST + graphql-ws mock engine
    cassettes/           # typed cassettes (checked by `pnpm typescript`)
  helpers/               # shared asserts (page errors, …)
  mock-llm/              # the deterministic agent transcript Tier 2 answers with
  tools/                 # run-local-tier.sh, run-visual.sh, review-sandbox.sh, trend.mjs,
                         # schema-compat.mjs, serve-dist.mjs, affected.ts and its
                         # default-base-ref.ts
  routes.ts              # the route manifest `affected.ts` reads
  specs/                 # *.spec.ts, tagged (@smoke, …)
  *.unit.test.ts         # structural gates — vitest, NOT playwright. They live at the
                         # e2e root AND under helpers/, mocks/ and tools/, so a glob
                         # anchored at the root alone misses most of them
```

The `e2e/**/*.unit.test.ts` files run under `pnpm test` (`vitest.config.ts`
includes them), not under `pnpm e2e`: they read source and SQL rather than
driving a browser.

Key conventions:

- **Selectors:** `getByRole`/`getByLabel` first; `data-testid` only where no
  accessible name exists (add it to the component in the same PR).
- **Cassettes are TypeScript modules** typed against the generated GraphQL
  types — schema or operation drift fails `pnpm typescript`, not the runtime.
- **The clock is pinned** (UTC, fixed epoch) on the mock tier: keep cassette
  timestamps on the `CASSETTE_EPOCH` day or date renders change under you.
- **Unmatched HTTP calls fail the test** — every GraphQL POST and REST call needs
  a cassette entry or teardown fails. An unmatched _subscription_ is legal (a flow
  page opens ~15 and a cassette mocks only the ones it cares about): it gets ack'd
  silence, so a typo'd subscription key surfaces as a UI timeout, with the missed
  operations in the `unmatched-subscriptions` report attachment. Either way nothing
  reaches a real backend: the WebSocket is routed too, and the HTTP route is re-armed
  on every page the context opens, so a popup (the report tabs) is mocked like the
  page that opened it — Playwright does not inherit a page's routes into its popups.
- **Downloads are the one hole in that.** The browser performs an `<a download>`
  transfer outside the page, so no route — page- or context-scoped — is ever offered
  it: it goes to the `vite preview` proxy and out to whatever `VITE_API_URL` names. A
  mock-tier spec must therefore never start one — assert the anchor's `href`/`download`
  (see `specs/crud/resources.spec.ts`) and leave the bytes to `specs/real/**`.
- **The clipboard is the second hole.** `navigator.clipboard.writeText` needs a
  permission this context does not hold, and the app turns the rejection into a
  "failed to copy" toast, so a spec asserting the happy path would fail for a reason
  that has nothing to do with the product. `helpers/clipboard.ts` records the writes
  from an init script instead; install it before the first navigation and assert on
  `clipboardWrites(page)`.
- **Drag and drop needs no synthetic events.** The file manager moves rows through
  HTML5 DnD with its own MIME type, and `locator.dragTo()` drives it as the browser's
  own drag — the drop lands, `dataTransfer` carries the paths, and the move request
  goes out.
- Assert errors via `pageerror`/`unhandledrejection` (`expectCleanPage`): the
  production bundle strips app console output, so console-based asserts are
  meaningless on the mock tier.
- **Scope a panel query by the panel's name.** The flow page keeps **two**
  tabpanels marked `data-state="active"` at once, so
  `locator('[role="tabpanel"][data-state="active"]')` reads out of both — measured,
  with the second holding no links, which is the only reason such a query ever
  matched what its spec expected. Radix labels each panel with its trigger, so
  `getByRole('tabpanel', { name: 'Screenshots' })` names exactly one.
- **The screenshots panel is bottom-anchored and mounts its images on
  intersection.** It opens scrolled to the newest shot, and a card that has not
  been in the viewport renders no `<img>` at all — a fixture with enough rows to
  overflow leaves the topmost ones imageless, so `getByRole('img')` never resolves
  there. Assert order off each card's source link, which renders regardless, and
  point an image-decoding assertion at the shot the panel actually opens on.

## Tags

Specs are tagged (`test.describe(..., { tag: '@x' })`) so runs can be filtered
with `--grep` / `--grep-invert` (e.g. `pnpm e2e --grep @smoke`):

| Tag         | Meaning                                                               |
| ----------- | --------------------------------------------------------------------- |
| `@smoke`    | Sanity subset — auth, nav, the load-bearing happy paths               |
| `@flows`    | Flow list / detail / subscription / terminal specs                    |
| `@crud`     | Create-read-update-delete journeys (knowledge, api tokens, templates) |
| `@coverage` | Surface coverage (dashboard, settings, resources)                     |
| `@settings` | The account page: identity, password validation, submit and rejection |
| `@cross`    | Cross-cutting: themes, responsive, a11y, contrast, route sweep        |
| `@visual`   | Screenshot baselines — runs only in the visual project                |
| `@real`     | Tier 2 — real backend + mock LLM (`specs/real/**`)                    |
| `@stand`    | Tier 3 — LLM-independent smoke against a live stand                   |

Two conventions the gate reserves:

- **`@quarantine`** — tag a newly-flaky spec to drop it from the gate (leave a
  tracking note); fix and untag rather than let it rot. The exclusion is wired
  into the mock project's `grepInvert`, not a CLI flag, and the real-tier projects
  do not read it: tagging a `specs/real/**` spec isolates it from nothing.
  Nothing is quarantined today.
- **`@generated`** — a spec whose cassette was recorder-derived and passed a
  semantic-assertion review; see the LLM recipe below.

## Stand tier (Tier 3)

Runs the LLM-independent `@stand` smoke against a real deployment. It lives in its
own workflow (`e2e-stand.yml`) so the PR gate never subscribes to `labeled`. Trigger
it by labelling a PR `e2e:stand`, or from **Actions → E2E Stand → Run workflow**
(`workflow_dispatch` takes no inputs). The job's `if` is the first gate: it runs only
for a `workflow_dispatch`, or a labelled PR whose head is **not** a fork — a
mislabeled or fork-PR run skips the job entirely and never reaches the secrets. For a
run that clears that gate, the protected `e2e-stand` Environment is the second gate:
its required reviewers approve before any secret is exposed. The stand's URL and login
come from the `E2E_STAND_URL` / `E2E_STAND_USER` / `E2E_STAND_PASSWORD` secrets
(exposed to the tools as `E2E_BASE_URL` / `E2E_USER` / `E2E_PASSWORD`).

Before the browser specs, a **schema-compat pre-flight**
(`e2e/tools/schema-compat.mjs`) introspects the stand's live GraphQL schema and
validates every frontend operation against it — a renamed or missing field
fails once, readably, instead of as dozens of red specs (the deploy-skew class
we hit manually). Run it anywhere: `E2E_BASE_URL=https://… E2E_USER=… E2E_PASSWORD=…
node e2e/tools/schema-compat.mjs` (against the local self-signed Tier-2 stack,
prefix `NODE_TLS_REJECT_UNAUTHORIZED=0`; a real stand has a valid cert).

## Trends and selective runs

- **Trend:** `node e2e/tools/trend.mjs` turns a run's `results.json` (written
  under `CI=1`) into one JSONL record (p50/p95 spec duration, slowest three,
  pass/flaky/fail counts). Each CI run uploads its own `e2e-trend` artifact;
  aggregate across runs offline (`gh run download`) to see duration creep and
  flake, not just green/red.
- **Affected routes:** `pnpm exec tsx e2e/tools/affected.ts <base>` prints the
  manifest routes a diff touches (backed by each route's owning `sources` in
  `e2e/routes.ts`). Empty output = no frontend route changed. Without a base
  argument it diffs against the branch the remote calls default, and with no
  remote branch to compare with it exits 2 rather than print nothing. This is the
  substrate for scoping runs and the exploratory agent to the changed surface;
  the mapping logic is unit-tested (`e2e/affected-routes.unit.test.ts`).

## Reviewing with agents

Verifying a claim about a gate usually means breaking something on purpose — deleting a
guard, injecting the regression it should catch, patching a fixture. Do that in a sandbox,
never in your checkout:

```bash
SANDBOX=$(./e2e/tools/review-sandbox.sh create --dirty --with-deps)
# …mutate and run anything inside $SANDBOX…
./e2e/tools/review-sandbox.sh clean "$SANDBOX"
```

`--dirty` carries uncommitted work across (usually what you are reviewing); `--with-deps`
hardlinks `node_modules` so pnpm, vitest and playwright run there — skip it for read-only
work, it costs ~30s. `clean` takes a path under the sandbox root and refuses anything else,
including the root itself; `clean --all` sweeps sandboxes untouched for two hours, so it
cannot take out a concurrent agent's live one. A bare `clean` is an error, not a sweep.

The sandbox is a `git worktree` under `$TMPDIR`, so a deleted file or an edited spec inside
it cannot reach your tree. `--with-deps` needs the root to be on the repo's own filesystem —
hardlinks cannot cross mounts — so where `$TMPDIR` is its own mount (tmpfs `/tmp`, a separate
`/home`) the tool says so and falls back to `.pentagi-review-sandboxes` beside the repo;
`PENTAGI_SANDBOX_ROOT` overrides both. Two caveats: an absolute path still escapes it, and a tool that
rewrites a dependency **in place** would reach the shared inode — don't hand `--with-deps`
to an agent whose job is patching libraries. Check `git status` in your own tree when a run
finishes; that is the only proof nothing leaked.

## LLM advisory layer (Phase 3, opt-in)

Deterministic specs are the gate; an LLM is only ever an advisory second
reviewer, **never** a merge gate. The intended stack is first-party, not a
bespoke bot:

1. `npx playwright init-agents --loop=claude` generates the planner / generator
   / healer agent definitions; the connected **playwright-mcp** drives the
   browser in accessibility-snapshot mode (a smaller prompt-injection surface
   than raw DOM or vision).
2. Scope the agent to the changed surface with `affected.ts` (the routes) — do
   not hand it the whole app.
3. Guardrails are mandatory because PentAGI renders adversarial content by
   design (tool output, target responses): treat all page text as untrusted
   data, never instructions; an action allowlist (no settings/token mutations,
   no off-origin navigation) on real stands; throwaway scoped credentials; the
   PR report renders page-derived strings as escaped, length-capped quotes;
   screenshots posted publicly come only from mock/scripted tiers (a real
   stand's api-tokens dialog shows a live secret); a hard per-run token cap.
4. A generated spec merges only with its cassette (so it runs on the Tier-1
   gate), a semantic-assertion review, and an `@generated` tag.

This section is the recipe, not yet wired — the deterministic tiers above are
the foundation it plugs into.

---

### Frontend Docs List Detail Pages (`frontend/docs/list-detail-pages.md`)

# List & Detail Page Building Blocks

Reusable surface for **list-and-detail** pages: a filterable/sortable table, a
Prev/Next/Sheet toolbar that walks the _same_ filtered subset on detail pages,
inline-rename inputs, and the URL-vs-storage state model that keeps them in
lockstep. Every list page in the app (`/flows`, `/knowledges`, `/templates`,
`/settings/prompts`, `/settings/providers`, `/settings/api-tokens`) renders a
`<DataTable>`. The detail half applies to `/flows/:id`, `/knowledges/:id` and
`/templates/:id` only. `/settings/api-tokens` has no detail route at all (it edits
rows in place, through its own state rather than the inline-edit surface below),
and `/settings/prompts/:name` and `/settings/providers/:id` are single-entity
forms with no sibling navigation, so neither has a feature nav hook.

> **Scope / how to trust this doc.** It describes code under `frontend/src`.
> Signatures below are copied verbatim from source. If a signature and the code
> disagree, the code wins — re-grep the named file and update this doc. Symbols
> flagged **removed / does not exist** are called out explicitly (see
> [Removed / renamed](#removed--renamed--do-not-use)) so nothing resurrects them.

## Table of Contents

- [Overview](#overview)
- [Mental model](#mental-model)
- [Where the code lives](#where-the-code-lives)
- [Core concepts](#core-concepts)
- [Hooks](#hooks)
- [Components](#components)
- [Library helpers (`@/lib`)](#library-helpers-lib)
- [Recipe: add a new list + detail pair](#recipe-add-a-new-list--detail-pair)
- [Design rationale](#design-rationale)
- [Gotchas](#gotchas)
- [Testing](#testing)
- [Removed / renamed — do not use](#removed--renamed--do-not-use)

## Overview

The surface splits cleanly into two responsibilities:

- **List page** — renders a `<DataTable>` driven by `useTableState()`. The
  free-text filter (`?q=`) and page number (`?page=`) live in the **URL**; view
  preferences (sorting, column visibility, page size, search columns) persist in
  **localStorage** under a single `table_4_<path>` slot.
- **Detail page** — renders a `<DetailNavigationToolbar>` driven by a
  feature-scoped hook built on `useDetailNavigation()`. It **reads** the list's
  `?q=` filter (never writes it) and offers Prev / Next / a searchable listbox
  over exactly the subset the user filtered on the list.

Two rules make the whole thing predictable:

1. **URL is authoritative** for filter + page, so a shared link reproduces the
   exact view.
2. **Prev/Next walks the same subset** — the detail toolbar filters on the same
   `?q=` the table does, so siblings stay in step with the list. The two
   matchers are close but not identical; see the caveat below.

## Mental model

```
                ┌──────────────────────────────────────────────┐
                │  URL   ?q=foo   ?page=3   ?qs=bar             │
                │  (source of truth — bookmarkable, shareable)  │
                └────────┬──────────────────────────┬──────────┘
                         │ read + write             │ read-only
                         ▼                          ▼
                   useTableState            useTableQueryFilterReader
                   (list pages)                     │
                         │                          ▼
                         ▼                   useDetailNavigation
                    <DataTable>                     │
                         │                          ▼
                         │              <DetailNavigationToolbar>
                         ▼                   (detail pages)
                 table_4_<path>
                 in localStorage
              (sorting / columns / pageSize;
               NOT filter, NOT page index)
```

- **URL** carries the ad-hoc query (`?q=`), the page index (`?page=`, 1-based),
  and — on knowledge/semantic screens — a server-side search (`?qs=`).
- **Storage** carries only durable view preferences. A detail page **never**
  writes storage and never replays storage into the URL, so opening a shared
  `/flows/:id?q=foo` link shows exactly what the link says.

## Where the code lives

```
/* Detailed source-code truncated for AI context efficiency. */
```

| Area            | File(s)                                             | Public surface                                                                                                                          |
| --------------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| List URL state  | `hooks/use-table-state.ts`                          | `useTableState`                                                                                                                         |
| Detail URL read | `hooks/use-table-query-filter.ts`                   | `useTableQueryFilterReader`                                                                                                             |
| Storage keys    | `hooks/use-page-storage-keys.ts`                    | `usePageStorageKeys`                                                                                                                    |
| Table           | `components/ui/data-table.tsx`                      | `DataTable`, `DataTableColumnHeader`, `cycleColumnSort`                                                                                 |
| Detail nav      | `components/shared/detail-navigation/`              | `DetailNavigationToolbar`, `DetailNavigationButtons`, `DetailNavigationSheet`, `useDetailNavigation`, type `DetailNavigationController` |
| Inline edit     | `components/shared/inline-edit/`                    | `InlineEditInput`, `useInlineEdit`                                                                                                      |
| Storage slots   | `lib/table-state.ts`, `lib/view-options-storage.ts` | `loadTableState`/`updateTableState`/`migrateLegacyTableState`, `loadViewOptions`/`saveViewOptions`/`migrateLegacyViewOptions`           |
| Key + URL utils | `lib/storage-keys.ts`, `lib/url-params.ts`          | `getTableStorageKey`/`getTopLevelPath`, `URL_PARAMS`/`mergeHrefWithSearchParams`                                                        |

## Core concepts

### URL is authoritative

`useTableState` reads and writes three-ish query params through
`react-router`'s `useSearchParams`:

| Param    | Meaning                                                         | Owner                                                                    |
| -------- | --------------------------------------------------------------- | ------------------------------------------------------------------------ |
| `?q=`    | client-side free-text filter                                    | `useTableState` (list) / `useTableQueryFilterReader` (detail, read-only) |
| `?page=` | **1-based** page number; absent ≡ page 1                        | `useTableState`                                                          |
| `?qs=`   | server-side semantic search (vector store); orthogonal to `?q=` | page-local `useSearchParams` (e.g. knowledges)                           |

`pageIndex` is exposed **0-based** in JS; the URL param is **1-based**.
`useTableState` converts on read (`parsed - 1`) and write (`index + 1`) and
canonicalizes `?page=1` away so `/flows` and `/flows?page=1` don't split the
history stack.

### Storage is a warm-restart bag

Each list page persists **at most one** localStorage key — `table_4_<path>` — a
single JSON object bundling `filter`, `sorting`, `columnVisibility`, `pageSize`,
and `searchColumns`. `DataTable` owns these writes via `usePageStorageKeys().table`.
The current page index and the ad-hoc `?q=` filter stay URL-only.

### Prev/Next walks the same subset

`useDetailNavigation` subscribes to the URL filter via
`useTableQueryFilterReader`, applies a pure matcher over the same `?q=`, and
resolves the Prev / Next siblings + a `"3/10"` position label over that filtered
subset. It also threads the current `?q=` through every prev/next/select
destination via `mergeHrefWithSearchParams`, so navigating siblings never drops
the filter.

The two matchers are not the same function, and they disagree on diacritics:
the toolbar folds them (`createTextMatcher` normalizes NFKD and strips combining
marks), while `DataTable`'s `globalFilterFn` only lowercases. Searching `cafe`
therefore hides a `café` row from the table while Prev/Next still steps onto it.
Align the two before relying on the subsets being identical.

## Hooks

| Hook                        | File                                   | Source of truth      | Writes URL?         | Used by                                            |
| --------------------------- | -------------------------------------- | -------------------- | ------------------- | -------------------------------------------------- |
| `useTableState`             | `hooks/use-table-state.ts`             | URL `?q=` + `?page=` | **yes**             | List pages                                         |
| `useTableQueryFilterReader` | `hooks/use-table-query-filter.ts`      | URL `?q=`            | no                  | Detail pages (indirect, via `useDetailNavigation`) |
| `usePageStorageKeys`        | `hooks/use-page-storage-keys.ts`       | router               | no                  | `DataTable`, dashboards                            |
| `useDetailNavigation`       | `components/shared/detail-navigation/` | URL + props          | no (navigates only) | Feature-scoped nav hooks                           |
| `useInlineEdit`             | `components/shared/inline-edit/`       | local state          | no                  | List cells, detail breadcrumbs                     |

### `useTableState` — list-page URL state

```ts
function useTableState(options?: {
    clearPageOnFilterChange?: boolean; // default true — setFilter also drops ?page=
    debounceMs?: number; // default 200
    filterParamName?: string; // default 'q'  (URL_PARAMS.QUERY)
    pageParamName?: string; // default 'page' (URL_PARAMS.PAGE)
}): {
    debouncedFilter: string;
    filter: string;
    pageIndex: number; // 0-based
    resetFilter: () => void;
    setFilter: (value: string) => void;
    setPage: (pageIndex: number, options?: { replace?: boolean }) => void;
    update: (patch: { filter?: null | string; pageIndex?: number; replace?: boolean }) => void;
};
```

- **`setFilter`** always navigates with `replace: true` (keystrokes don't clutter
  history) and, when `clearPageOnFilterChange` is `true`, bundles `pageIndex: 0`
  into the **same atomic update** so narrowing the results also resets the page
  (no "page 5 of nothing").
- **`setPage`** pushes a history entry by default; pass `{ replace: true }` for
  out-of-range clamping so a bad URL isn't one back-press away.
- **`update`** is the atomic multi-field primitive. Every call in the same
  microtask is coalesced into **one** `setSearchParams`, which is what makes a
  simultaneous filter+page change race-free by construction. Reach for it when
  changing both fields from one event.

### `useTableQueryFilterReader` — detail-page read-only filter

```ts
function useTableQueryFilterReader(options?: {
    debounceMs?: number; // default 200
    paramName?: string; // default 'q'
}): { debouncedFilter: string; filter: string };
```

Observes `?q=` but **never** writes the URL. This is the piece that lets a fresh
detail-page mount avoid injecting a previous tab's `?q=` into the URL. It is the
only export of `use-table-query-filter.ts` — there is no writer counterpart (see
[Removed / renamed](#removed--renamed--do-not-use)). Most code doesn't call it
directly; `useDetailNavigation` uses it internally.

### `usePageStorageKeys` — per-route storage keys

```ts
function usePageStorageKeys(options?: {
    pathname?: string; // override the path used to build keys
    useTopLevel?: boolean; // default false; /flows/abc-123 → /flows when true
}): { period: string; table: string; viewOptions: string };
```

Resolves the three per-route localStorage keys reactively via `useLocation()`
(not a non-reactive global `location.pathname` read). A detail page that wants
to share its parent list's storage bucket passes `{ useTopLevel: true }`.

### `useDetailNavigation` — headless detail-nav controller

```ts
function useDetailNavigation<T extends { id: string }>(options: {
    currentId: null | string | undefined; // pass null while creating/new
    getHref: (item: T) => string;
    getLabel: (item: T) => string;
    items: readonly T[];
    getId?: (item: T) => string; // default item.id
    getSearchableText?: (item: T) => null | string | undefined; // default getLabel
    sortFn?: (a: T, b: T) => number;
    // controlled sheet (optional):
    open?: boolean;
    defaultOpen?: boolean;
    onOpenChange?: (open: boolean) => void;
    // controlled in-sheet local search (optional):
    searchQuery?: string;
    defaultSearchQuery?: string;
    onSearchQueryChange?: (query: string) => void;
    searchDebounceMs?: number; // default 150
}): DetailNavigationController<T>;
```

Returns a large, identity-stable `DetailNavigationController<T>` (exported type):
`currentItem`, `currentIndex`, `prevId`, `nextId`, `positionLabel` (`"3/10"` or
`"–/0"`), `goToPrev` / `goToNext` / `handleItemSelect`, `filteredItems`, `total`,
sheet state (`isSheetOpen` / `openSheet` / `closeSheet` / `setSheetOpen`), local
search state (`searchQuery` / `setSearchQuery` / `clearSearchQuery` /
`debouncedSearchQuery` / `isSearchActive`), and the guard `hasEntries`. The
filtered subset is narrowed by the URL `?q=` **AND** the
in-sheet local search, combined with AND.

> Hoist `getLabel`/`getHref`/`getId`/`getSearchableText` to **module scope** (or
> wrap in `useCallback`). Inline arrows still work but forfeit the controller's
> internal memoization on `getSearchableText`.

### `useInlineEdit` — inline rename state machine

```ts
function useInlineEdit<TElement extends HTMLElement = HTMLInputElement>(options?: {
    resetKey?: null | string | undefined; // changing it closes any open editor (pass the entity id)
}): {
    handleDropdownCloseAutoFocus: (event: Event) => void; // spread onto Radix <DropdownMenuContent>
    inputRef: React.RefObject<null | TElement>;
    isEditing: boolean;
    startEdit: () => void;
    stopEdit: () => void;
};
```

Owns the edit-mode toggle, a ref for the input, and **deferred focus** on the
next animation frame so focus lands _after_ Radix's dropdown close-focus-restore
(otherwise Radix wins the race and the input never focuses). Pair with
`<InlineEditInput>` and pass `resetKey={entity.id}` so navigating items drops a
stale draft.

## Components

### `<DataTable>` — the table primitive

Generic TanStack-backed table (filter, sort, pagination, column visibility,
optional window virtualization). Controlled or uncontrolled per axis.

```ts
// exported: DataTable, DataTableColumnHeader, cycleColumnSort
function DataTable<TData, TValue = unknown>(props: {
    columns: ColumnDef<TData, TValue>[]; // required
    data: TData[]; // required
    empty?: { entityName?: string }; // plural lowercase, e.g. "flows"
    filterColumn?: string | string[]; // omit → zero-config: columns w/ meta.searchable === true
    filterPlaceholder?: string; // default "Filter..."
    filterValue?: string; // controlled filter (pair with onFilterChange)
    onFilterChange?: (value: string) => void;
    pageIndex?: number; // controlled page (0-based)
    onPageChange?: (pageIndex: number, options?: { replace?: boolean }) => void;
    initialPageSize?: number; // default 10
    initialSorting?: SortingState; // default []
    columnVisibility?: VisibilityState;
    onColumnVisibilityChange?: (visibility: VisibilityState) => void;
    onRowClick?: (row: TData) => void;
    renderRowContextMenu?: (row: TData) => ReactNode;
    renderSubComponent?: (props: { row: Row<TData> }) => ReactElement;
    isVirtualized?: boolean; // default false; ignored when renderSubComponent set; activates > 50 rows
    storageKey?: string; // default `table_4_<pathname>` (usePageStorageKeys().table)
}): JSX.Element;

function DataTableColumnHeader<TData, TValue = unknown>(props: {
    column: Column<TData, TValue>; // TanStack column — sorting + cycle action target it
    title: ReactNode;
}): JSX.Element;
```

Key rules:

- **Controlled needs the pair.** The filter is controlled only when **both**
  `filterValue` and `onFilterChange` are set; passing `filterValue` alone
  silently reverts to internal uncontrolled state and your value is ignored. A
  controlled page needs `pageIndex` **and** `onPageChange` (the latter is what
  lets the table push `pageIndex → 0` into the URL when the filter narrows).
- **Search columns are declared per column** via `meta: { searchable: true }` on
  the `ColumnDef` (zero-config multi-column mode). If no column is `searchable`
  and no `filterColumn` is passed, the search input isn't rendered at all.
- **Empty state** text comes from `empty={{ entityName: 'flows' }}`.
- **Storage aliasing:** two `<DataTable>`s on the same route share the default
  `table_4_<path>` key and overwrite each other — give each a distinct
  `storageKey` (e.g. `` `${base}:agents` ``).

### `<DetailNavigationToolbar>` and leaves

```ts
function DetailNavigationToolbar<T extends { id: string }>(props: {
    controller: DetailNavigationController<T>; // required — from the feature nav hook
    sheetTitle: string; // required
    sheetIcon?: ReactNode;
    renderItem?: (item: T, isCurrent: boolean) => ReactNode;
    hasSearch?: boolean; // default true — in-sheet search input
    searchPlaceholder?: string;
}): JSX.Element;
```

The toolbar composes `<DetailNavigationButtons>` (Prev / position / Next) and
`<DetailNavigationSheet>` (the searchable listbox) against one controller. Most
desktop call sites use the toolbar directly; pages with non-standard chrome
(e.g. mobile prev/next inside a `<DropdownMenuItem>`) compose the two leaves
themselves and read from the same controller. All three accept `controller` +
their own presentation props — **there is no `toolbarProps` object to spread.**

### `<InlineEditInput>`

```ts
function InlineEditInput(props: {
    onSave: () => void; // required — read latest text from the bound inputRef
    onCancel: () => void; // required
    autoFocus?: boolean; // default false
    busy?: boolean; // default false — disables the Save + Cancel buttons while saving (the input stays editable)
    className?: string;
    defaultValue?: string; // uncontrolled input
    inputRef?: Ref<HTMLInputElement>; // pair with useInlineEdit().inputRef
    maxLength?: number; // default 200 (native maxLength guard, not a validation gate)
    placeholder?: string;
}): JSX.Element;
```

`Enter` saves, `Escape` cancels. The input is **uncontrolled** — callers read
the value from `inputRef.current` at save time, avoiding a re-render per
keystroke.

## Library helpers (`@/lib`)

### `lib/table-state.ts` — the unified table slot

```ts
type TableState = {
    columnVisibility?: Record<string, boolean>;
    filter?: string;
    pageSize?: number;
    searchColumns?: string[];
    sorting?: { desc: boolean; id: string }[];
};

const loadTableState: (key: string) => TableState; // {} when missing/invalid
const updateTableState: (key: string, patch: Partial<TableState>) => TableState; // undefined/empty clears a field; empty object removes the key
const migrateLegacyTableState: (path: string, unifiedKey: string) => TableState;
```

`migrateLegacyTableState` is a one-shot, idempotent reader of the pre-unification
layout: it folds the four legacy keys (`column_4_`, `sorting_4_`, `filter_4_`,
`page_4_`) into the unified `table_4_<path>` slot and deletes them. From the
`page_4_` slot it carries over **only** `pageSize` (the page index now lives in
`?page=`). Safe to call on every mount.

### `lib/view-options-storage.ts` — FileManager view options

```ts
type ViewOptionsRecord = Record<string, boolean>;
const loadViewOptions: (key: string) => ViewOptionsRecord;
const saveViewOptions: (key: string, value: ViewOptionsRecord) => void; // removes key on empty
const migrateLegacyViewOptions: (path: string, unifiedKey: string) => ViewOptionsRecord;
```

Separate `viewOptions_4_<path>` slot for FileManager-style screens (folders-first
toggle, expanded dirs) that aren't backed by `DataTable`.

### `lib/storage-keys.ts` — key conventions

```ts
const STORAGE_KEY_SEPARATOR = '_4_'; // reads as "for": table_4_/flows = "table for /flows"
type LocalStorageKeyType = 'period' | 'table' | 'viewOptions';
function getStorageKey(type: LocalStorageKeyType, urlPath: string): string; // `${type}_4_${urlPath}`
function getTableStorageKey(urlPath: string): string; // table_4_<path>
function getPeriodStorageKey(urlPath: string): string; // period_4_<path> (dashboard time window)
function getViewOptionsStorageKey(urlPath: string): string; // viewOptions_4_<path>
function getTopLevelPath(pathname: string): string; // /flows/abc → /flows ; / → ''
```

The key embeds the full `urlPath` **including** its leading slash — the literal
is `table_4_/flows`, not `table_4_flows`. `getTopLevelPath` only walks one level
deep, so a nested list like `/admin/flows/:id` would resolve to `/admin`; such a
detail page must hardcode its parent path instead.

### `lib/url-params.ts` — URL constants + href merge

```ts
const URL_PARAMS = { PAGE: 'page', QUERY: 'q', SEARCH: 'qs' } as const;
function mergeHrefWithSearchParams(base: string, incoming: Iterable<[string, string]> | URLSearchParams): string;
```

`mergeHrefWithSearchParams` adds every incoming key to `base` unless `base`
already specifies it (the href wins), and preserves any `#hash`. This is how the
detail toolbar forwards the current `?q=` into prev/next/select destinations.

## Recipe: add a new list + detail pair

For a new entity at `/<entities>` + `/<entities>/:id`:

### 1. List page — `pages/<entities>/<entities>.tsx`

```
/* Detailed source-code truncated for AI context efficiency. */
```

### 2. Feature nav hook — `features/<entity>/use-<entity>-detail-navigation.ts`

```ts
import { useDetailNavigation } from '@/components/shared/detail-navigation';
import { routes } from '@/lib/routes';
import { type Entity, useEntities } from '@/providers/entities-provider';

// Module scope → stable identity → controller memoization holds.
const getLabel = (item: Entity) => item.title || `Entity #${item.id}`;
const getSearchableText = (item: Entity) => item.title;
const getId = (item: Entity) => String(item.id);
const getHref = (item: Entity) => routes.entity(item.id);

export function useEntityDetailNavigation(currentId: null | string | undefined) {
    const { entities } = useEntities();

    return useDetailNavigation<Entity>({
        currentId,
        getHref,
        getId,
        getLabel,
        getSearchableText,
        items: entities,
    });
}
```

### 3. Detail page — `pages/<entities>/<entity>.tsx`

```tsx
const entityNav = useEntityDetailNavigation(isNew ? null : entityId); // null while creating

// Desktop: inside the header actions.
{
    !isMobile && (
        <DetailNavigationToolbar<Entity>
            controller={entityNav}
            renderItem={(item, isCurrent) => (
                <span className={cn('min-w-0 flex-1 truncate', isCurrent && 'font-medium')}>{item.title}</span>
            )}
            sheetIcon={<EntityIcon className="size-4" />}
            sheetTitle="Entities"
        />
    );
}

// Mobile: leaves composed by hand — <DetailNavigationButtons> inside a
// <DropdownMenuItem>, and a standalone <DetailNavigationSheet> outside the header,
// both reading `controller={entityNav}`.
```

#### Ordering the header actions

`<AppHeaderContent>` takes the remaining width (`flex-1`), so `<AppHeaderActions>` ends up
against the right edge and grows leftward. Two rules follow:

1. **Controls that appear on a data condition go first** in the children. Everything after
   them keeps its position when they arrive late — the flow header's Report button loads with
   the task list, and the pager beside it must not move under a cursor that is clicking Next.
2. **Controls that are always meaningful for the route render always**, taking `disabled` from
   an explicit loading flag rather than being unmounted while the entity is null. Unmounting
   collapses the cluster for the length of every fetch, which costs a step per pager click.

The flow header is the reference: `Report · favourite · pager · actions menu`. A page whose
entity can be genuinely absent (a not-found card) is the exception — it renders no actions at
all, because none of them are meaningful there.

## Design rationale

### Why URL > storage

A user opens `/flows?q=alpha` and clicks "Next" in the detail toolbar to reach
flow B, then shares `/flows/b?q=alpha` with a teammate. The teammate opens it
cold: the detail page reads `q=alpha` from the URL and renders Prev/Next over the
filtered subset; hitting "Next" lands on `/flows/c?q=alpha` — still inside the
filter they never typed. `useTableQueryFilterReader` is the key: it observes the
URL but never writes, so a fresh detail mount can't inject a previous tab's
`?q=` into the URL.

### Why one storage key per page

Before unification every list page wrote four keys (`column_4_`, `sorting_4_`,
`filter_4_`, `page_4_`) across two write paths (sync + debounced); refreshing
mid-typing could land inconsistent state. The unified `table_4_<path>` slot is
one JSON object all preferences live in; `migrateLegacyTableState` folds the four
legacy keys in on first mount and deletes them.

## Gotchas

- **`pageIndex` is 0-based; `?page=` is 1-based.** Never hand `<DataTable>` or
  `useTableState` a 1-based index — the hook does the ±1 conversion.
- **Controlled axes need the prop pair.** `filterValue` without `onFilterChange`
  (or `pageIndex` without `onPageChange`) silently reverts that axis to the
  table's internal uncontrolled state, and the URL stops updating.
- **No `toolbarProps`.** Detail pages pass `controller={nav}` plus discrete
  `renderItem` / `sheetIcon` / `sheetTitle` props. There is no bundle to spread.
- **`storageKey` aliasing.** Multiple `<DataTable>`s on one route share the
  default `table_4_<path>` key; give each a distinct `storageKey`.
- **Two search mechanisms coexist on knowledge screens.** The `DataTable` client
  filter (`?q=`, via `useTableState`) and a server-side semantic search (`?qs=`,
  via page-local `useSearchParams`) are independent — don't conflate them. A
  copied list recipe only needs the `?q=` one.
- **`DetailNavigationToolbar` always renders.** It has no empty-list branch, so
  it mounts before the provider's list arrives and shows the buttons' own
  disabled/"–/0" state until items land.

## Testing

`pnpm run test` (Vitest, `jsdom`, `globals: false` — every test imports
`describe`/`it`/`expect`/`vi` explicitly). Coverage across the surface:

| File                                                     | Focus                                                                                                                                           |
| -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `lib/table-state.test.ts`                                | `load`/`update`/`migrate` — partial patch, empty-collapse, key removal, legacy fold, path variants                                              |
| `lib/view-options-storage.test.ts`                       | load/save/migrate for the view-options slot                                                                                                     |
| `lib/url-params.test.ts`                                 | `mergeHrefWithSearchParams` — append, href-wins, hash preserved, encoding                                                                       |
| `lib/storage-keys.test.ts`                               | `getTopLevelPath` edge cases + `_4_` key join                                                                                                   |
| `hooks/use-table-state.test.tsx`                         | URL read/write, **atomic `update` race regression** (two updaters in one tick keep both params), `?page=1` canonicalization, filter-is-URL-only |
| `hooks/use-table-query-filter.test.tsx`                  | read-only filter subscription                                                                                                                   |
| `hooks/use-page-storage-keys.test.tsx`                   | reactive key resolution (incl. `table_4_` for empty path)                                                                                       |
| `components/ui/data-table.test.tsx`                      | filter/sort/pagination + sorting persistence into `table_4_<path>`                                                                              |
| `components/shared/detail-navigation/*.test.*`           | pure `use-navigation` core (headless), toolbar + sheet behavior                                                                                 |
| `components/shared/inline-edit/use-inline-edit.test.tsx` | edit toggle + deferred focus                                                                                                                    |

`frontend/vitest.setup.ts` polyfills jsdom gaps used by these components and
wires RTL cleanup:

- `Element.prototype.scrollIntoView` (no-op) — roving-focus / scroll-into-view effects.
- `globalThis.ResizeObserver` (no-op class) — Radix ScrollArea in the nav sheet.
- `Element.prototype.{has,set,release}PointerCapture` (no-ops) — Radix Select gating.
- `afterEach(() => cleanup())` — RTL DOM cleanup after every test.

## Removed / renamed — do not use

These names appear in older docs/comments but **do not exist** in the code. Do
not reintroduce them:

| Old / phantom name                                     | Reality                                                                                                                                 |
| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| `usePagination`                                        | Never existed as a hook. Page state lives in `useTableState` (`pageIndex` in `?page=`) and `<DataTable>` (`pageIndex`/`onPageChange`).  |
| `useTableQueryFilter` (writer)                         | Removed. `use-table-query-filter.ts` exports **only** `useTableQueryFilterReader`. List pages write the filter through `useTableState`. |
| `toolbarProps` spread into `<DetailNavigationToolbar>` | No such object. Pass `controller={nav}` + discrete `renderItem`/`sheetIcon`/`sheetTitle`.                                               |
| `hooks/use-inline-edit`                                | Wrong path. It lives at `components/shared/inline-edit/use-inline-edit.ts`.                                                             |
| `components/shared/data-table`                         | Wrong path. It lives at `components/ui/data-table.tsx`.                                                                                 |
| `features/providers/use-provider-detail-navigation.ts` | Never existed. `/settings/providers/:id` is a form with no sibling nav — only `flows`, `knowledges` and `templates` have nav hooks.     |
| `features/prompts/use-prompt-detail-navigation.ts`     | Never existed. Same for `/settings/prompts/:name`.                                                                                      |

---

### Frontend Docs Live Testing (`frontend/docs/live-testing.md`)

# Live testing (chrome-devtools MCP)

The Playwright tiers in [`e2e.md`](./e2e.md) are the gate: deterministic, offline,
run on every PR. This document covers the other half — driving a **real stack**
(real Postgres, real GraphQL, real agent loop) through a browser and through raw
HTTP, which is what proves a fix on real data and what covers surfaces no
cassette reaches.

The two are not interchangeable:

|                      | deterministic tiers                   | live pass                                               |
| -------------------- | ------------------------------------- | ------------------------------------------------------- |
| answers              | "does the code still do what it did?" | "does this actually work against a real stack?"         |
| runs                 | every PR, minutes                     | by hand, tens of minutes                                |
| evidence it produces | pass/fail                             | before/after on a named input                           |
| catches              | regressions                           | wrong assumptions, environment truths, missing coverage |

Some paths exist only here, but fewer than it looks. The mock cassettes carry
`answer`, `input`, `report` and `terminal` messages and a `thinking` field on
them; a local-tier run adds `thoughts` and `done` through the mock LLM, and the
subscription plumbing that delivers all of them is covered by both tiers — so
`pnpm e2e` and `run-local-tier.sh` already answer for those render paths. What
neither tier produces is `advice`, `ask`, `browser`, `file` and `search`: those
are exercised by a live agent run and nothing else.

A live green proves nothing about your branch unless the stack **contains** your
branch. That check comes first, every time.

## Which stack

|                                                               | what it costs | what it proves                      |
| ------------------------------------------------------------- | ------------- | ----------------------------------- |
| `cd frontend && pnpm dev` (:8000) against an existing backend | seconds       | frontend changes only               |
| full local compose stack, image built from your branch        | ~10 min       | anything, including backend changes |

The dev server is the trap: it proxies `/api` to whatever backend you point it
at, so a backend fix you just wrote is **not** in the picture — a green result
there says nothing about it. Use it for UI work; the moment the change touches Go,
build an image.

You point it with `VITE_API_URL` in `frontend/.env`, which is gitignored and has
no example file — a fresh checkout has none, the proxy target resolves to
`https://undefined`, and every request fails before it reaches any backend. Write
it before the first `pnpm dev`:

```bash
printf 'VITE_API_URL=localhost:8443\nVITE_USE_HTTPS=true\n' > frontend/.env
```

The value carries no scheme: `VITE_USE_HTTPS` picks it, for the proxy, for the
subscription socket's `wss://`, and for TLS on the dev server itself — which is
why the UI is then at `https://localhost:8000`. Against a stack brought up with
`SERVER_USE_SSL=false` it must be `false` too.

## Bringing up the stack

```bash
docker build -t local/pentagi:live .                       # from the repo root
PENTAGI_IMAGE=local/pentagi:live \
  docker compose up -d --wait --wait-timeout 300 pentagi
```

The app answers on `https://localhost:${PENTAGI_LISTEN_PORT:-8443}` with a
self-signed certificate — every `curl` below therefore uses `-k`.

**Prove the container runs the image you just built.** `--wait` returns when the
container is healthy, not when it is the right one; compose reuses a running
container whose config did not change:

```bash
docker inspect -f '{{.Config.Image}} {{.Image}}' pentagi
docker images --no-trunc --format '{{.Repository}}:{{.Tag}} {{.ID}}' | grep live
```

The two ids must match. A stale container from an earlier session is the single
most common way a live result lies — it is why "I tested it and it works" is
worth nothing without this line.

The image is built from what is committed plus whatever is in your working tree
at build time, so check `git status` before you build: an uncommitted experiment
gets baked in, and a colleague's half-finished edit in the same checkout gets
baked in with it.

Readiness, before touching the UI:

```bash
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:8443/api/v1/info
```

Backend startup replays flows and can take a minute; poll rather than sleep.

### Environment truths that decide what you can test

- **`COOKIE_SIGNING_SALT`** — while it is empty or the literal `salt`, the server
  refuses both to mint and to validate API tokens ("token creation is disabled
  with default salt"). Anything touching tokens or the bearer tier needs a real
  value: `COOKIE_SIGNING_SALT=<throwaway> docker compose up -d …`.
- **Provider credentials** — a provider type with no server-side credentials
  cannot be created at all, so a stack with no LLM keys cannot host a provider,
  and without a provider you cannot create a flow. The repository's own `.env` is
  deliberately keyless; start the test stand from a credentialed env file of
  your own (`docker compose --env-file .env.local-test up -d …`). That file is
  yours to create — every `.env.*` is gitignored, so no checkout carries one:
  copy `.env.example`, put one provider's real key in it, and give it a salt
  (see above). Then check what it actually enables — `{ settingsProviders
  { enabled { … } } }` answers in one request. A type with a placeholder key
  still shows as enabled and fails at the first call, so run the flow against
  the one type whose credentials are real.
- **`/settings/providers` is not that list.** The page renders
  `settingsProviders.userDefined` — providers a user created — while the
  env-configured types live in `enabled`, which only fills that page's **Add
  Provider** menu, and in the `providers` query behind the composer's picker. A
  credentialed stack with no user-created rows therefore reads "No providers
  configured" while its flows run on `anthropic`: the empty page is correct, and
  it is not a reason to skip the flow walk.
- **OAuth is not testable locally** — the Google and GitHub flows need a real
  callback origin. Cover the local-account paths live and stub the rest.
- **`PENTAGI_LISTEN_PORT`** — an `.env` carried over from another checkout can
  publish the stack on `443`, and everything you then run against `:8443` quietly
  hits nothing. Keep a dedicated env file for the test stand
  (`docker compose --env-file .env.local-test …`) rather than editing the one you
  develop with.
- **No flow ⇒ no container** — paths behind a running sandbox (container file
  listing, terminal input) answer `FlowFiles.ContainerNotRunning` and are simply
  unreachable on a keyless stack. Cover those at the handler level instead of
  pretending the live 400 is the case under test.
- **The compose file names an image, it does not build one.** The `pentagi`
  service is declared `image: ${PENTAGI_IMAGE:-vxcontrol/pentagi:latest}`, so a
  plain `docker compose up -d` starts a published build and every green you then
  collect belongs to somebody else's code. Build first and name it:
  `docker build -t local/pentagi:branch . && PENTAGI_IMAGE=local/pentagi:branch docker compose up -d`.
  Prove the running stack is yours before trusting it — the served bundle carries
  the operations it was built from
  (`docker exec pentagi grep -o 'value:"<yourNewField>"' /opt/pentagi/fe/assets/index-*.js`),
  and the schema answers for the backend half
  (`{ __type(name: "FlowFile") { fields { name } } }`).
- **GraphQL is at `/api/v1/graphql`.** `/graphql` is not registered at all, so it
  falls to the router's `NoRoute`, which redirects anything that is neither a
  frontend route nor an asset with `301` to `/`. That redirect looks exactly like
  an auth failure and will send you hunting for a session you already have — the
  real path answers `403` when the session is missing, and `/api/v1/info` names
  your user when it is not.
- **Sandbox ports and names** — a flow's sandbox is `pentagi-terminal-<flowID>`,
  and its host ports come out of a 2000-wide window from `DOCKER_PORTS_BASE`
  (see [`backend/docs/docker.md`](../../backend/docs/docker.md)), so flow ids
  1000 apart land on the same ports. A Tier-2 run seeds ids from 90001 and owns
  `[30000, 32000)`: its overlay sets `DOCKER_PORTS_BASE=30000` so that its flows
  cannot ask for the 28002/28003 a developer stack is already publishing. A stack
  you bring up by hand inherits the default window and will collide — give it a
  base of its own. Containers left behind by a previous database are orphans —
  their flow ids no longer exist, and they still hold ports.

  The name collides before the port does, and it fails the whole mutation:
  `createFlow` answers `Conflict. The container name "/pentagi-terminal-<id>" is
  already in use`, so a stack whose database you recreated cannot start a flow
  until the old sandboxes are removed. `run-local-tier.sh` handles both halves —
  it removes `pentagi-terminal-9xxxx` and their `-data` volumes, then runs
  `ALTER SEQUENCE flows_id_seq RESTART WITH 90001`. Bringing the same compose
  project up by hand does neither, and ids restart at 1 straight into a developer
  stack's names.

## Driving the UI

Use the **chrome-devtools MCP** tools. Not computer-use, not screenshots read by
eye: the accessibility tree is the contract the app promises, and it is what the
Playwright specs assert against too.

The loop is: `navigate_page` → `take_snapshot` → act on a `uid` → assert with
`evaluate_script` → `list_console_messages`.

```
navigate_page  {type: "url", url: "https://localhost:8443/flows"}
take_snapshot                       # a11y tree; every control carries a uid
click / fill   {uid: "…"}           # act through the tree, not coordinates
evaluate_script                     # read state back: text, computed styles, DOM
list_console_messages {types:["error"], includePreservedMessages:true}
```

Prefer a snapshot over a screenshot — it is smaller, diffable, and names things
the way a user's screen reader does. Take screenshots only when the question is
genuinely visual.

**Zero console errors is part of the pass**, on every route you visit.

### Traps that make a live check lie

- **The router ignores `history.pushState`.** Pushing a path and dispatching
  `popstate` changes the URL without re-rendering, and the assertions then read
  the previous page. Navigate for real, one page per `navigate_page`.
- **A first login lands on "Update Password".** The migrated admin carries
  `password_change_required`, so the SPA holds you there; click **Skip for now**
  or every later step runs against the login route. (The Playwright tiers clear
  the flag in setup instead — see [`e2e.md`](./e2e.md).)
- **Logging out, or changing a password, ends that user's other sessions.** The
  server keeps a generation counter per user; a cookie carries the value it was
  issued under and the middleware refuses one that lags, answering `403` with
  `session revoked` in the log. So the parallel `curl` jar goes dead the moment
  you log out or change the password in the browser — mint it again rather than
  reading the refusal as a broken probe. A self-service password change is the
  one case that keeps its own session: it re-stamps the caller's cookie. And the
  first run of a build that introduced the counter refuses every cookie issued
  before it, which is a one-time re-login, not a fault.
- **Controlled inputs ignore `el.value = …`.** React does not see it. Use the
  `fill` tool, or the native setter plus an `input` event:
  `Object.getOwnPropertyDescriptor(HTMLInputElement.prototype,'value').set.call(el, v)`.
  That covers filling, not clearing: `fill` with an empty string empties the DOM
  value while React keeps the old one, so the list stays filtered and the `?q=`
  stays in the address — which reads exactly like "clearing the filter does not
  restore the list". Clear with the keyboard instead: focus, select all, type or
  press Backspace.
- **Radix does not answer a scripted click.** Menus, tabs, selects and popovers
  open on pointer events, so `el.click()` from `evaluate_script` silently does
  nothing — the menu stays shut, the tab stays unselected, and the assertion that
  follows reads the previous state. Act on those through the `click` tool with a
  uid from a fresh snapshot. Plain buttons and links do respond to a scripted
  click, which is what makes the failure look arbitrary.
- **An inactive tab panel is unmounted.** Measuring a panel you have not switched
  to returns zero characters, which reads exactly like an empty state. Switch the
  tab, then measure the panel whose `data-state` is `active`.
- **Validation is reported in the field, not in a toast.** A submit that fails
  client-side leaves the page silent: no toast, no navigation, and the reason
  sits in `[data-slot="form-message"]` with `aria-invalid="true"` on the control.
  Read those before concluding the button did nothing.
- **Filling a field is not typing in it.** The flow composer takes 140
  characters through the fill tool and keeps **Submit** disabled: the value is
  in the textarea, the form state never saw it. One real keystroke into the same
  field enables the button. Assert on the control you are about to press, not
  on the value you just wrote, and type at least one character before concluding
  that a submit path is broken.
- **Forms guard navigation.** Leaving a dirty create/edit form opens **Unsaved
  changes** (Cancel / Discard / Save) and the click that "did not navigate" is
  waiting on it.
- **The self-signed interstitial has to be cleared once per tab.** Click through
  it (**Advanced → Proceed**) from the snapshot; you cannot type `thisisunsafe`,
  because that needs real keystrokes on a `chrome-error://` page where
  `evaluate_script` does not run. If the tab refuses to keep the exception, bring
  the stack up with `SERVER_USE_SSL=false` and drive it over `http://`.
- **Names in the tree are not unique.** A breadcrumb renders `role="link"` with
  the same accessible name as the sidebar item that led there; match on the uid
  from the snapshot, not on a name you assume is unambiguous.
- **The terminal is a WebGL canvas.** Pixels tell you nothing; read the xterm
  buffer through its handle.
- **A response shape is not the UI's shape.** The resources list returns
  `data.items`; asserting on `data.resources` produces a confident empty list and
  a green check that means nothing. Print the payload once before trusting a key.
- **Long calendar walks time out.** Paging a date picker three years forward is
  ~36 re-renders; drive it once and assert both boundary days from the same grid.
- **DevTools "Offline" does not cut an open WebSocket to localhost.** It blocks
  fetches — they fail immediately — while an established socket keeps delivering,
  so a tab that looks dead for a minute is still subscribed. Read that as "the
  emulation did not reproduce a drop", never as "the client failed to notice one":
  the cheap disproof is a mutation over `curl` while the tab sits offline, and a
  sidebar that updates says the channel is alive. Cut the connection from the
  server side instead — `docker restart pentagi` produces a real `close 1006`, a
  reconnect within a couple of seconds, and the `ws:reconnected` event the app
  hangs its reconcile on.
- **A GraphQL error's wording is not the resolver's — unless nothing classified
  it.** On any built image the presenter (`services/graphql_errors.go`) collapses
  a refusal, a missing row, a driver error and an unreachable dependency into
  seven fixed strings behind six codes; the original is attached only in develop
  mode, which every `docker build` defeats. What it does not recognise — a
  resolver's own validation, for one — reaches the client verbatim and carries no
  `extensions.code` on any build, so waiting for a code on `token creation is
  disabled with default salt` waits forever. Match the code where there is one,
  never a redacted sentence.
- **Everything is gzipped, except a ranged request.** The compression middleware
  sits on the root engine, so GraphQL, every REST route and the static bundle all
  answer `content-encoding: gzip` to a client that advertises it. Byte-exact
  ranged downloads are kept by a predicate instead of by scope: a request carrying
  `Range` is served uncompressed, and so is anything whose extension is already
  compressed. A client advertising `Accept-Encoding: gzip` and not decoding now
  reads every route as broken, not just the endpoint.
- **Forcing a logout through GraphQL needs a code.** The client dispatches its
  session re-check on `extensions.code` being `UNAUTHENTICATED` or `FORBIDDEN`
  and no longer reads the message, so a refusal from one of the resolvers that
  still writes bare `unauthorized: …` prose leaves the session exactly as it was.
- **An open Radix menu hides the rest of the page.** While a dropdown, popover or
  sheet is open, everything behind it carries `aria-hidden`, so every role query
  outside it finds nothing and reads as "the control disappeared". Two things
  follow: close the menu before looking elsewhere, and do not count right after
  pressing Escape — the close is animated, and a count taken in the same tick
  still returns zero. Wait for `[role=menu]` to detach.
- **Settings is a second shell.** `/settings/*` replaces the main sidebar with its
  own (Account, Providers, Prompts, API Tokens), so a walk that clicks "Flows"
  from a settings route waits forever. Leave through **Back to App** in the
  settings sidebar footer, and remember that a probe which ends inside settings
  starts the next one there.
- **The theme switcher is a tab group, not menu items.** It lives inside the user
  dropdown as Radix `Tabs` labelled `System theme` / `Light theme` / `Dark theme`,
  so `getByRole('menuitem', …)` waits forever. Open the menu, then act on
  `role=tab`.
- **Table sorting is a button inside the header cell.** Clicking the `th` itself
  does nothing and leaves `aria-sort="none"`, which reads exactly like sorting
  being broken. Click the button; the cycle is `none → ascending → descending`,
  and it is not written to the query string — it goes to the page's
  `table_4_<path>` slot instead, so a reload keeps it and a case that changes the
  sort leaks into the next one.
- **The empty state is a table row.** A search with no matches renders one `tr`
  carrying "No matches", so counting `tbody tr` reports one result where there
  are none. Assert on the text, not the row count.
- **`git checkout -- <file>` discards uncommitted work, not the last edit.** It is
  a tempting way to undo a mutation during a mutation test and it reverts the fix
  you are testing along with it — the tell is a suite that goes redder than the
  single mutation should explain. Copy the file aside and restore from the copy.

## Logging in

The first login is not a single step:

1. `navigate_page` to `https://localhost:8443/` — an unauthenticated session
   lands on `/login`.
2. `take_snapshot`; fill **Login** and **Password** through the `fill` tool (a
   raw `el.value = …` is invisible to React), then click **Sign in**.
3. The seeded admin carries `password_change_required`, so the next screen is
   **Update Password**. Click **Skip for now**. Skipping is the point: changing
   the password here invalidates the credentials every later session and every
   API probe uses.
4. You land on the `returnUrl` you came from, or `/flows/new` when there is
   none. From here on, **never type another URL** — see below.

Keep a parallel `curl` session (`-c jar.txt`) logged in as the same user. The
browser proves what a user sees; the cookie jar proves what the server stored,
and you will need both on every write.

## Navigate the way a user does

Typing a route into `navigate_page` is a different code path from clicking a
link: it remounts the app, refetches everything, and skips the transition that
actually breaks — the one where a provider keeps stale state, a subscription is
not torn down, or a list does not refresh after a mutation. Half the bugs worth
finding live in that transition.

So: reach every page by clicking the sidebar, a breadcrumb, a row, or a tab.
Direct navigation is for exactly two things — the first login, and re-entering
after a deliberate full reload (which is itself a check: the page must survive
F5 with its state).

Two corollaries:

- **After every mutation, look at the list you came from** without reloading.
  A row that only appears after F5 is a broken subscription or a missing cache
  update, and no unit test will tell you.
- **`history.pushState` is not navigation.** It changes the URL without
  re-rendering; assertions afterwards read the previous page.

## Sweeping the controls of a page

The rule is _every_ control, not the happy path: buttons, tabs, dropdown items,
right-click context menus, dialogs and their confirm buttons, toggles, selects.
Enumerate them mechanically rather than from memory — the snapshot is the
inventory:

```js
// list everything actionable on the current page, with its accessible name
[...document.querySelectorAll('button,[role="tab"],[role="menuitem"],a[href],[role="switch"]')]
    .map((el) => ({
        role: el.getAttribute('role') || el.tagName.toLowerCase(),
        name: (el.getAttribute('aria-label') || el.textContent || '').trim().slice(0, 40),
        disabled: el.disabled || el.getAttribute('aria-disabled') === 'true',
    }))
    .filter((c) => c.name);
```

Work that list top to bottom. Three kinds of control hide from a naive pass and
have to be sought out:

- **Hover-only** — row action cells are `opacity-0` until the row is hovered;
  they are in the tree the whole time, so click them by name.
- **Menu-only** — a control that exists only after its trigger is opened
  (**Open menu**, **Flow actions**, **Templates and resources**). Open the
  trigger, snapshot again, then act: uids from the previous snapshot are stale.
- **State-gated** — controls that render only for a given status. **Finish**
  disappears once a flow is Finished or Failed; the composer shows **Stop**
  instead of **Submit** while a flow is running; every assistant row's delete
  button hides once the flow **or the selected assistant** is terminal — a
  Finished assistant hides the button on all the rows, not only its own. If you
  never saw the alternative state, you did not test the control — seed one.

  On the Tier-2 stack a whole run settles in about 300 ms, which is shorter than
  the state you are trying to look at. Bring the mock up with a per-completion
  delay and **Stop**, the disabled composer and the working placeholder stay on
  screen long enough to click and read:

  ```bash
  MOCK_LLM_DELAY_MS=4000 docker compose -p pentagi-e2e --env-file /dev/null \
    -f docker-compose.yml -f docker-compose.e2e.yml up -d --force-recreate mock-llm
  ```

- **Roleless** — a `div` with `onClick` and nothing else. The message bubble's
  **Show thinking** / **Hide thinking** and **Show details** / **Hide details**
  are exactly that, so the snippet above never lists them and a role-based sweep
  reports the bubble as having no controls at all. Reach them by text:

  ```js
  [...document.querySelectorAll('div.cursor-pointer')].map((el) => el.textContent.trim());
  ```

A control with no accessible name is itself a finding: it is unreachable for a
screen reader and for every spec that matches by role and name. The four roleless
toggles above are that finding, already known — do not re-file them, but do click
them, because the content they reveal is tested nowhere else.

Two more sweeps come back empty for a reason that is not a defect. A **roving
tab strip** — the editor toolbar, the file tree — gives `tabIndex 0` to exactly
one child and `-1` to the rest, so tabbing finds one control and the arrows find
the others. And a **horizontally scrolled strip** with a hidden scrollbar keeps
its overflow in the DOM: at a narrow viewport the controls are present, clickable
by name, and invisible on a screenshot.

## The sidebar header carries the version, and it is a control

The header of both sidebars — the app's and the settings one — is a button, not a
label. It opens a panel that states which build is running and what the update
service thinks of it. Three things are worth a pass of their own.

**What it must say.** The trigger shows the shell's title — PentAGI in the app,
Settings in the settings — and the version under it;
the panel repeats the version, adds a `build` line with the revision, a channel
badge (`preview`, `stable` or `nightly` — the server cannot answer `disabled`), the
verdict with its own tone, the facts as pairs (`Installed`, `Available` when an
update is offered, `Checked`), and the release links. The `build` line is
unconditional: when the pipeline stamped no git revision, the server answers with
the sha256 prefix of the running binary instead, so an empty build line is a defect,
not a state.

**The version string depends on how the image was built, not on the code.** A stand
built without build args reports a bare `develop`: the builder stage defaults
`PACKAGE_VER` to that word and stamps it through ldflags, and the runtime reports it
unchanged. Build it the way the pipeline does — `source ./scripts/version.sh`, then
pass `--build-arg PACKAGE_VER --build-arg PACKAGE_REV` — and the binary reports
`2.1.0-ce.h<rev>`, which the panel shows as `v2.1.0-ce` with a `build <rev>`
line. Check which you are looking at before filing "the version is wrong":
`docker inspect <container> --format '{{index .Config.Labels "com.pentagi.version"}}'`.
The label is composed from the same two args and reads exactly what the binary
reports — `2.1.0-ce.h<rev>`, without the `v` the panel adds — so a disagreement
between the label and the binary is itself the defect.

**Version skew removes the header silently.** The panel asks for a field the older
backend does not have, the query fails validation, and the provider hands the panel
nothing — so the sidebar header loses its version line and the trigger disappears
from the tree, with no error on screen. The only sign is one console line,
`Failed to load resource: 422`. A frontend dev server pointed at a stale stand is
the usual way to see it, and it is worth reproducing deliberately after any change
to `versionInfoFragment`.

The indicator follows the same tones as `FlowStatusIcon` — green for up to date,
yellow for an update, muted for anything unknown — and appears on the trigger only
when an update is offered. Every other state says its piece inside the panel and
leaves the trigger quiet. The collapsed rail keeps only the mark and, for an
update, a dot in its corner.

A development stand is not a build the update service published, so it never reads
up to date and never offers an update. What it can show live: **Checking for
updates** until the first check completes — it starts at a random moment within two
minutes of the backend starting and may take up to 30 seconds — then **Update status
unknown**, whether the service answered or could not be reached; and **Update checks
are disabled** when `UPDATE_CHECK_INTERVAL` is `0` or the service could not be set
up. The rest are pinned by
`src/components/shared/version-panel.test.tsx`, which renders the panel against
every answer the service can give — read the verdicts there rather than trying to
provoke them on a stand.

## CRUD, entity by entity

Do the full cycle on each entity, and close every one at the trust boundary —
the toast is the UI's opinion, the API read-back is the fact.

Two rows depend on the stack, not on the code: a **flow** cannot be created
without a configured provider, and a **knowledge document** cannot be created
without a reachable embedder — the mutation answers "embedding provider is not
available" and nothing is written. On a stack without those, walk the read,
update and delete halves and say plainly which creates you could not exercise.

| entity             | create                                                               | read                         | update                                                               | delete                                  |
| ------------------ | -------------------------------------------------------------------- | ---------------------------- | -------------------------------------------------------------------- | --------------------------------------- |
| flow               | composer on `/flows/new`, both **Automation** and **Assistant** tabs | row in the list, detail page | inline rename (row menu and breadcrumb), favorite toggle, **Finish** | row menu → dialog, header menu → dialog |
| assistant          | first message with no assistant selected                             | picker popover               | follow-up message, **Stop** while running                            | trash in the picker row → dialog        |
| flow template      | `/templates/new`                                                     | list, detail                 | **Edit** opens the detail; **Rename** edits the title in place, in the row itself | row menu → dialog                       |
| knowledge document | `/knowledges/new`                                                    | list, detail                 | edit, rename                                                         | row menu → dialog                       |
| resource           | upload, create folder                                                | file manager list            | rename, move, copy                                                   | delete, bulk delete                     |
| provider           | create by type, clone                                                | settings list, detail        | edit fields, per-agent **Test**                                      | delete → dialog                         |
| prompt             | —                                                                    | list, detail                 | edit, **Reset System**, **Reset All**                                | —                                       |
| API token          | inline create row                                                    | list                         | rename, revoke/activate                                              | row menu → dialog                       |

Three creates are gated by something other than the obvious field, and each looks
like a dead button until you find it: a **knowledge document** needs an **answer
type** picked in the select (the message "Answer type is required" appears in the
field, not in a toast), an **API token** keeps **Submit** disabled until an expiry
date is picked in the row's date popover, and an **assistant** keeps its own
**Submit** disabled until a provider is chosen in the composer's "Select
Provider" dropdown. A knowledge create also goes through the embedder, so it
answers in seconds, not milliseconds — read the list back after the mutation
resolves, not after a fixed wait.

Two rows in the table read differently from the rest. A **prompt** row does not
open on click, only through its row menu → **Edit**; its reset entries are named
**Reset System**, **Reset Human** and **Reset All**, each shown only when that
half of the prompt actually differs from the default, so a prompt with a custom
system text offers two of the three and never the third. A **resource** row
carries the same menu on the button and on right-click, and a move onto an
occupied path opens a second dialog, **Replace existing item?** — cancelling it
must leave the target file untouched.

For each: create it, find it in the list **without a reload**, open it, change
it, watch the change land in the list, delete it, confirm the row is gone and a
direct read returns not-found. Then check the same object through the API — the
stored value is what matters, not the rendered one.

Cancel paths count too: open every confirmation dialog and dismiss it, and check
nothing was written.

### The furniture every table carries

Three controls sit around the filter box and none of them appear in the CRUD
table above. **Search in** picks which columns the filter looks at — narrow it
and a row that matched a second ago disappears, which reads as lost data.
**Columns** hides columns; a hidden column takes its sort header with it. The
filter input itself stops at 200 characters, so a paste longer than that is
truncated silently at the DOM and the query you think you ran is not the query
that ran.

All three, plus sorting and page size, persist per page under a localStorage key
shaped `table_4_<path>` — the `_4_` is the separator from `src/lib/storage-keys.ts`
and reads as "table *for* /flows", not as a user id, so two accounts sharing a
browser share the state. That is the mechanism behind "it behaved differently
yesterday": clear the key before claiming a table is broken, and clear it between
cases that change columns.

### Knowledge has two searches, and they are not the same search

The box in the page header is **semantic** and writes `?qs=`; the box below it is
a plain text filter over the rows already on screen and writes `?q=`. They
compose — `?q=` narrows what `?qs=` returned — and only the first one asks the
server. Two things follow. The semantic query is capped at 100 results, and the
detail page's Prev/Next walks that same capped array, so navigation inside a
search is navigation inside those 100. And a semantic search that matches nothing
renders the empty state of an empty library — same heading, same **New
Knowledge** button, no table at all — so "search returned nothing" and "there is
no data" look identical on screen; check the address bar before believing either.

`?page=` is inert here. The table's own pager keeps its position in component
state and the page passes it no page props, so a page number carried over from
another screen changes nothing — and `?page=1` is stripped from the address bar
everywhere by the shared table hook, since the first page is the canonical URL.

Creation gates on more than the answer type. **Document type** — `answer`,
`guide`, `code` — decides which second field appears, and switching it wipes the
other two subtype values while marking the form dirty, so a tester who flips the
type to look around leaves a form that warns about unsaved changes.

### The API token you can only read once

The token is created from an inline row on the settings page, and the secret
appears exactly once, in a dialog titled **API Token Created** with a **Copy
Token** button. Dismiss it and the value is gone for good — there is no second
view, and the row in the table offers only **Copy Token ID**, which is not the
secret. Any case that needs a working token must copy it in that dialog or start
over.

**The expiry picker offers tomorrow onwards** — today and every past day are
disabled — and the day you pick is the moment the token dies, not its last
working day: picking 16 Sep issues a token whose `exp` is 15 Sep 23:59:59, and
the row prints `00:00, 16 Sep`. A case that needs the token to work on a given
day has to pick the day after it.

The same page's header carries a **Developer tools** menu that opens **GraphQL
Playground** and **Swagger UI**. Both are live surfaces served by this stand and
neither is named anywhere else in this document.

## The markdown editor, which is the body of three entities

**What leaves the editor is markdown, not what was typed.** An address or a URL is
autolinked as it is entered, so a body typed as `Contact admin@example.com` reaches
the server as a link. Anything comparing the two — the anonymizer decides "nothing
was found" by comparing its answer with the current body — is comparing against the
marked-up form. A case built on the typed text instead of the stored one tests a
branch that cannot be reached.

**What was not touched is saved as it was written.** A body that was loaded keeps its
own layout through a save: the blank lines between its blocks and how many there were,
the spaces on a blank line, the bullet character, the numbers and the leading spaces of
its lists, every row of a table as the line it was (its padding, its pipes — bare or
escaped inside a code span — and a line of text written right under the table),
a code block as it stood (its fence, its indentation, a block written by indentation and
not by a fence), the fence of a code span, a bare URL, the link around an image,
the underline of a setext heading, the spaces a heading or a list ended with, a line break
that is not a hard break. A case that types one character and compares the stored body
with the original should find that character and nothing else outside the block
it was typed into; inside that block the editor writes its own form — a table row
with one space around each cell where the table was written that way and padded
to its column otherwise, `\|` for a pipe in a code span of that row, a code block between
fences or four spaces in. What is kept gives way wherever it would be read as something else:
under a list, a table or a fenced block written a few spaces in goes to the margin,
text goes there when the list would take it for its last item's, and
inside a list item a code block is written between fences at the item's column.
The body is stored as it was sent, the line break it ends with included:
neither the forms nor the backend trim it, and only the single-line fields
(title, question, description, code language) are trimmed.

A Go template action `{{ … }}` is literal text to the editor — nothing inside
it is read as markdown and a pipe in it is never escaped — and a line that
opens with a control action (`{{ range }}`, `{{ end }}`) right under a table is
the template's line, not a row. What still differs is not a defect of such a case:
a link typed into the editor, as opposed to one that was loaded,
is stored as `[url](url)`; a named HTML entity outside code and
outside an action is decoded (`&amp;` becomes `&`);
a list item that opens with indented code is written with a fence;
and a document with CRLF line endings is saved with LF.

A template's body, a knowledge document's content and both halves of a prompt are
the same component. The document tells you to "edit" those entities; this is what
you are editing, and none of it is reachable from the entity's own controls.

**Getting to raw and back.** The mode switch is not in the editor. It lives in
each page's header menu — **Template actions**, **Knowledge actions**,
**Prompt actions** — on a row labelled **View**, which holds two icon triggers:
**Rich editor** and **Raw source**. The row keeps its menu open while you switch.
The mode is component state: it resets on reload, on remount, and when you move
to another prompt. In raw mode the toolbar, the table grips and the popovers do
not exist — `role="toolbar"` is absent from the page — so a sweep run in raw
reports the whole group missing.

**The toolbar** is `role="toolbar"`, accessible name **Formatting**,
`data-slot="markdown-editor-toolbar"`. Left to right: a **Text style** dropdown
(Heading 1…6 and Text), then **Bold**, **Italic**, **Strikethrough**,
**Inline code**, **Link**, then a **List** dropdown (bullet, ordered, task), a
**Table** dropdown, then **Blockquote**, **Code block**, **Insert image**,
**Horizontal rule**, **Clear formatting**, and pinned to the right **Undo** and
**Redo**. Every control is icon-only: its visible text appears in a tooltip after
a hover delay, so search by accessible name, never by text.

Three things about it will make a check lie.

- The two dropdown triggers name their current value — **Text style: Heading 2**,
  **List: Bullet list**, **List: None**. A locator captured by name goes stale as
  soon as the caret moves.
- Below the `md` breakpoint the strip renders *under* the text area, and it
  scrolls horizontally with the scrollbar hidden: controls exist and answer to a
  click by name while being invisible on a screenshot.
- Inside a code block the five inline marks are disabled; inside a table cell
  every heading except **Text** is disabled. A disabled control is not a missing
  control.

**Insertion is one level down.** There is no toolbar button that inserts a table:
open the **Table** dropdown and the single item **Insert table** is there while
the caret is outside a table. Move the caret inside and the same trigger opens a
ten-item menu — **Header row**, insert row/column in four directions, an **Align
column** submenu, and the three deletions.

**The popovers.** **Link** opens a popover with a **Link URL** field and the icon
buttons **Apply link**, **Open link in new tab** and — only over an existing link
— **Remove link**. The same popover appears on its own when the caret lands
inside a link; clicking a link seats the caret instead of navigating. Its URL
validation is authoring-only: pasted markdown never passes through it, so a
security check written against this field proves nothing about what the viewer
renders. **Insert image** works the same way and has no file picker anywhere —
images are URLs, and "upload an image" has no surface to test.

**Table grips do not answer to a driver.** Hovering a cell with a real mouse
summons two buttons, **Column actions** and **Row actions** (`data-table-grip`),
portalled into `document.body`. They are driven by a document-level `mousemove`,
so a synthetic click or a programmatic focus never brings them up; their menus
are the only way to align a column or delete a row without the toolbar.

**What the editor never shows you.** An invalid body renders its message
`sr-only` — the only visible signal is a red border from `aria-invalid` on the
contenteditable. A check that looks for error text finds none and reads the
field as valid.

## The file manager, which the document used to cover in six words

Two routes mount it: `/resources`, and a flow's **Files** tab. It is
`role="tree"`, accessible name **File tree**; rows are `role="treeitem"` carrying
`data-path`. On a flow the rows sit under three synthetic groups — **Uploads**,
**Resources**, **Container** — which are not themselves selectable.

**It has a keyboard model, and the document never said so.** One Tab lands on the
active row; Arrow Up/Down move between visible rows without wrapping, Home/End
jump to the ends, Arrow Right expands a directory and Arrow Left collapses it,
Enter opens a file or toggles a directory. The handler is bound to the wrapper
rather than to the tree, so those keys also fire while focus sits on the
select-all checkbox or a sort header.

**Selection is not Finder's.** A plain click replaces the selection with the row
— and a directory brings its whole subtree, collapsed or not. Cmd/Ctrl+click
toggles a row, a directory all-or-nothing. Shift+click is *additive* over the
visible range, deliberately unlike Finder: a check asserting that a range click
replaces the selection is asserting the wrong contract.

**The header row** carries a three-state **Select all** checkbox
(`data-state="indeterminate"` in the middle state) that is nonetheless a two-way
toggle: clicking it while indeterminate selects everything, and it only clears
when everything was already selected. Its universe is the *visible* tree, so with
a search active it selects what the filter left. Beside it sits **Expand all** /
**Collapse all**, whose accessible name flips to name the next action — and which
operates on the full tree, expanding directories the filter is hiding.

**Sorting is three-position** — ascending, descending, cleared — on **Name**,
**Size** and **Modified**. Two traps: the accessible name states the *next*
action (**Sort by name (ascending)** → **… (descending)** → **Clear sorting on
name**), so a locator captured by name is stale after one click; and the arrow
glyph is inverted from intuition, ascending drawing a down arrow. Judge the order
by the rows, never by the arrow. The order is not persisted anywhere: a reload
returns to insertion order.

**Row actions live in two places with identical items.** Hover a row and an
ellipsis button named **Row actions** appears — it is `opacity-0` until hover,
so it is invisible to a screenshot and immediately available to a query by name.
Right-clicking the row opens the same items as a context menu. On `/resources` a
directory row has seven items (**Download**, **Copy path**, **New folder**,
**Upload files**, **Rename or move**, **Copy to…**, **Delete**) and a file row
five; right-clicking the empty padding under the last row opens a two-item menu
targeting the library root.

**The bulk bar** appears once anything is ticked: `<N> selected · <size>` on the
left, then **Download**, **Move to…**, **Delete** with **Copy to…** and **Copy
paths** under **More actions** — on a flow's Files tab, **Save as resources**
instead. Two things to know. The N counts *affected* rows including descendants,
so ticking one folder of three files reads "4 selected". And below the `sm`
breakpoint the buttons keep only their icons, so a text query for **Download**
fails while the button is right there.

**Column settings** on `/resources` toggles the Size and Modified columns and the
folders-first order. It persists per user and per path under a localStorage key
shaped `viewOptions_4_<path>` — keyed on the path, not on the user — which means a
case that changes columns leaks into the next one until you clear that key, and
into the next account on that browser too.

**Move, copy and pull all end in the same second stage.** The dialogs are
**Rename or move resource**, **Copy resource** and **Pull from container**;
dropping a row onto a folder takes the same path. When the target path is
occupied, a second dialog, **Replace existing item?**, is what actually
overwrites — cancelling it must leave the target untouched, and that is the case
worth running, because the first dialog looks like it already did the work.

## The settings surfaces that no case in this document opens

### Presets write over what you typed

A template detail page and `/templates/new` both carry a **Preset templates**
panel — a card in the left split panel at 1280px and up, a button that opens a
sheet below that. It holds eleven presets, and each row is two buttons: the title
applies the preset, and a second, nameless one expands a preview. A sweep by role
alone counts twenty-two controls where a sweep by role and name counts eleven.

Applying a preset onto a form that already has a title or a body opens a dialog
titled **Replace content?** — so on an existing template, where the form is
pre-filled from the server, the dialog always appears, and on `/templates/new` it
only appears the second time. Either way the apply marks the form dirty, which
arms the unsaved-changes blocker: the next navigation, including the pagination
arrows described below, stops on a confirmation the case did not plan for.

The same **Replace content?** dialog also guards the flow form, so a case that
learns the string on one page will find it on the other.

### Validate, and the counter that is not the counter next to it

The prompt editor's header carries a **Validate** button that checks the text
currently in the editor, not what is saved. While the mutation is in flight both
the label and the accessible name become **Validating...**, so a lookup by the name
**Validate** finds nothing for the duration — wait for the name to come back
rather than treating the miss as a missing button. Below 768px only the icon is
rendered; the name still resolves, the visible text does not.

Beside it sits **Available variables**: a card at 1280px and up, a popover behind
a button below that, and nothing at all when the active tab declares no variables
— the System and Human tabs have different variable sets, so the panel appearing
and disappearing as you switch tabs is correct. Two numbers there are easy to
swap. The badge in the heading counts variables *declared* for the tab; the `×N`
on a chip counts *uses* in the text, per `{{ … }}` block rather than per name, so
`{{.Foo}}{{.Foo}}` counts two and `{{ .Foo .Foo }}` counts one. A chip's
accessible name changes with that count: unused, it offers to insert at the
cursor; used, it offers to go to the next occurrence.

### The pagination arrows are not pagination

Detail pages for templates, knowledge and flows share one three-button cluster:
**Previous**, **Next**, and between them a button whose accessible name carries
the position — `Open templates list (3/11)` — opening a sheet of siblings. Four
things about it will make a case lie.

They walk the filtered sibling list, not pages: there is no `?page=` on these
screens, and a `?q=` left in the address bar from an earlier case shrinks the set
until both arrows go disabled, which reads as "navigation is broken". They
navigate with `replace: true`, so Back does not undo a step — it leaves the
detail page entirely. Below 768px the cluster is not in the header at all but
inside the **…** actions menu. And on a dirty form the unsaved-changes blocker
catches the step, so the arrow appears to do nothing.

### The indicator that stays silent while the socket is down

A pill at the bottom of the screen, `role="status"` with
`data-slot="connection-status"`, reads **Not receiving updates — reconnecting**
(that is an em dash). It is mounted above the router, so it can appear over any
page, and it is absent from the DOM the rest of the time.

It is not an online/offline badge. Apollo raises the event behind it only when at
least one GraphQL subscription is live at the moment the socket closes, so
killing the network on a page that subscribes to nothing shows nothing — and its
absence is not evidence that the connection is up. In the other direction,
leaving a subscribing page while still disconnected clears the pill, because
tearing the subscription down reports the socket as recovered. Assert on it only
on a page you know holds a live subscription, and treat the flows list, which
does, as the reliable place to see it.

## The flow page in depth

This is where most of the surface lives, and where a shallow pass is worst.

**Header** — favorite toggle (`aria-pressed` flips and the list agrees), the
**Flow actions** menu (rename, finish, delete), the **Report** menu, and the
previous/next navigation between flows.

**Both tab strips** — `Automation`, `Assistant`, `Dashboard` on the left at wide
viewports; `Terminal`, `Tasks`, `Agents`, `Searches`, `Vector Store`, `Files`,
`Screenshots` on the right. Below 1280 px they collapse into one strip of ten;
check both layouts. The two strips are two panes side by side, both mounted at
once, so `[role=tabpanel]:visible` resolves to the left one whatever you clicked
on the right: take the tab's `aria-controls` and read that element by id. An
empty state is a pass only when the database agrees the collection is empty, so
confirm the zero with the API.

Two tabs answer to an address that does not match their label: **Searches** is
`tools` and **Vector Store** is `vectorStores`. The rest follow their
label in lower case — `automation`, `assistant`, `dashboard`, `terminal`, `tasks`,
`agents`, `files`, `screenshots`.

**Which parameter carries them depends on the width.** At 1280 px and above the
right-hand strip has its own: `?side=terminal`, `?side=tasks`, `?side=agents`,
`?side=tools`, `?side=vectorStores`, `?side=files`, `?side=screenshots`, while
`?tab=` keeps the three central ones. Below 1280 the strips merge and everything
goes back to `?tab=`. A deep link copied from a wide window therefore opens a
different tab on a narrow one.

Open a tab by address when you need it open
before the page settles: clicking the trigger through a tool that synthesises
events does not move a Radix strip.

**Composer states** — with the flow running, the composer must show **Stop**
(which stops generation) and the textarea carries `disabled`, not `readOnly`, with
the placeholder "PentAGI is working... Click Stop to interrupt"; with the flow
finished or failed the form stays disabled and the placeholder becomes "This flow
has ended. Create a new one to continue." Both strings end in three dots or a full
sentence exactly as quoted — a check written against a typographic ellipsis matches
nothing. Assert `el.disabled` — `el.readOnly` is false in every state, and a check
written against it reports a working composer as broken. Send a message into a
running flow and watch it appear in the log without a reload.

The assistant composer is a separate form with its own provider picker, and its
**Submit** stays disabled until a provider is chosen there — the automation
composer arrives with one preselected, so a tester who only knows that side reads
the disabled button as a bug.

It also keys off a different status. The automation composer follows the flow; the
assistant form disables on the flow **or** the selected assistant being Finished or
Failed, and shows its working state on the assistant being Created or Running. Its
placeholder names which one it is reading ("Assistant is running... Click Stop to
interrupt", "This assistant session has ended. Create a new one to continue.").
Finishing the flow therefore exercises one branch only — seed a terminal assistant
under a live flow for the other.

**Terminal** — a WebGL canvas. Screenshots and DOM text tell you nothing; read
the xterm buffer through its handle and assert on the text you find there.

**Files** — the flow's own file manager, driven over REST rather than GraphQL:
upload, pull from the container, promote to resources, download, delete, plus
the bulk selection. This is the surface where path handling bugs live; pair the
UI walk with raw requests that send an absolute path or `..`.

The panel lists the **server's** data directory, not the sandbox: the resolver
calls `flowfiles.List(DataDir, flowID)`, which reads
`<DATA_DIR>/flow-<id>-data/{uploads,container,resources}` inside the backend
container. Writing into the sandbox (`docker exec pentagi-terminal-<id> …`) or
into a `flow-<id>` directory leaves the panel empty and looks like a broken
query. To seed one file for a read path, without going through an upload:

```bash
docker exec pentagi sh -c 'mkdir -p /opt/pentagi/data/flow-<id>-data/uploads &&
  echo body > /opt/pentagi/data/flow-<id>-data/uploads/probe.txt'
```

Each listed file carries the flow it belongs to, and it has to: the id is an md5
of the path relative to the flow, so `{ flowFiles { id flowId } }` returning
`flowId: 0` is the cache-collision regression, not a cosmetic gap.

### Which runs to start

A pass needs both modes and the seams between them, and one run of each is not
enough — the interesting failures live in the mixes. Six starts cover it, and
they can overlap:

1. **Automation only** — the ordinary path: composer on `/flows/new`, Automation
   tab, a provider already selected.
2. **A second automation, started while the first runs** — this is what makes
   cross-flow leakage visible at all.
3. **Assistant only** — note that this start does not call `createFlow`; the
   composer calls `createAssistant` and the flow appears as a side effect.
4. **An assistant inside a running automation flow** — switch to the Assistant tab
   on the flow you just started and send a message there.
5. **An automation task inside an assistant flow** — the mirror of 4, and the one
   that exercises `putUserInput` on a flow that has no task yet.
6. **A second assistant session in a flow that already has one** — through
   **Select assistant** → the new-session entry. Then read `assistantLogs` for
   both ids: each session must hold only its own turns.

Keep every prompt inside the sandbox container — this is a pentest product, and a
prompt that names an outside host starts a real scan of it. "Print the kernel
version and the current user", "list the top-level directories with their sizes",
"write a small script to /tmp and run it" are enough to produce a full run.

Watch which message types the run actually emits. A sandbox-only automation
produces `input`, `thoughts`, `terminal`, `file`, `report` and `done` — and never
`search`, `browser` or `advice`. Those three appear only when the prompt sends
the agent to the web, so a pass built from local prompts leaves their rendering
untested, and the cassettes do not cover them either.

## Subscriptions and concurrency

Live data is pushed, and a subscription that silently died looks exactly like a
quiet system. Test it deliberately:

1. **Mutate outside the tab.** With the flow page open, make the change through
   `curl` — rename the flow, add a resource, finish it. The open page must
   update on its own. Nothing changing is the bug.
2. **Two tabs on one flow.** Act in one, watch the other. Both must converge
   without a reload.
3. **Run several flows at once.** Start at least two automation flows and one
   assistant, each with a different prompt, and leave them running side by side.
   Then check that:
    - each flow's messages, tasks and terminal output land only in that flow's
      panels — cross-flow leakage is the failure this catches, and it is invisible
      with a single flow;
    - the flows list shows every one of them in a live status, updating as they
      move Created → Running → Finished;
    - the message types your prompts actually produce all render. A sandbox-only
      run gives `input`, `thoughts`, `terminal`, `file`, `report` and `done`; a
      prompt that sends the agent to the web adds `search` and `browser`. The
      cassettes carry `answer`, `input`, `report` and `terminal` and a local-tier
      run adds `thoughts` and `done` — so a sandbox pass exercises `file` here,
      and `search` and `browser` only if you asked for the web.
4. **Reload mid-run.** The page must rebuild from the query and keep receiving
   updates; a flow that stops updating after F5 has a resubscription bug.
5. **Count the frames, not just the pixels.** Wrap `window.WebSocket` in an
   `initScript` and record every `payload.data` — the key, the row id and its
   `flowId`. Isolation then reads as a number instead of an impression: while
   another flow ran an assistant, the open flow held 305 messages before and
   after, its panel stayed at 42600 characters, and zero frames carried the other
   flow's id, while the rename of the open flow arrived as exactly one
   `flowUpdated`. A panel that merely "looks unchanged" cannot tell you a delta
   was dropped rather than correctly filtered.
6. **Kill the backend mid-generation.** `docker restart pentagi` while an
   assistant streams is the cheapest test of two separate things: the socket must
   close `1006`, reconnect and fire `ws:reconnected`; and the reply the user was
   already reading must survive in `assistantlogs`. A row left at `length 0` means
   the running message never reached storage.
7. **Verify the payload, not the pixels.** For any panel that looks wrong, ask
   the API for the same data (`flow`, `messages`, `tasks`, `usageStatsBy…`) and
   compare. That single query separates a rendering bug from a data bug.

## What the page opens, and what hides on it

Two things a tester cannot infer from the screen: which subscriptions a route
holds open, and which controls are in the tree but invisible until something
else happens. Both are counted from the source; re-count them when the UI
changes.

**Which subscriptions a page holds** — every one of them must be shown to
deliver, and the cheap proof is the same for all: cause the event from `curl`,
watch the open page react without a reload. They are opened by nested providers,
not by the route, so a page holds its own plus everything above it — which is why
the same event has to keep arriving on pages that look unrelated to it.

| held by                                                 | subscriptions                                                                                                                                                                                                                                                                                |
| ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| the app shell — every authenticated route **except** `/flows/:flowId/report`, which is protected but mounts its own layout | `settingsUserUpdated`, `flowTemplateCreated/Updated/Deleted`, `resourceAdded/Updated/Deleted`, `providerCreated/Updated/Deleted`                                                                                                                                                             |
| the flows layout — `/flows`, `/flows/new`, `/flows/:id` | `flowCreated`, `flowDeleted`, `flowUpdated`                                                                                                                                                                                                                                                  |
| `/flows/:id` itself                                     | `flowUpdated`, `taskCreated`, `taskUpdated`, `messageLogAdded`, `messageLogUpdated`, `agentLogAdded`, `terminalLogAdded`, `searchLogAdded`, `vectorStoreLogAdded`, `screenshotAdded`, `assistantCreated`, `assistantUpdated`, `assistantDeleted`, `assistantLogAdded`, `assistantLogUpdated` |
| the flow's **Files** panel, while it is mounted         | `flowFileAdded`, `flowFileUpdated`, `flowFileDeleted`                                                                                                                                                                                                                                        |
| `/knowledges*`                                          | `knowledgeDocumentCreated/Updated/Deleted`                                                                                                                                                                                                                                                   |
| `/settings/api-tokens`                                  | `apiTokenCreated/Updated/Deleted`                                                                                                                                                                                                                                                            |

So the flow page holds twenty-eight, thirty-one with the Files panel open, and
`flowUpdated` is genuinely open twice — the layout's copy refreshes the list behind
you, the page's copy refreshes the header. Ten of them ride on every other route
too: a resource added by `curl` must land on the settings page and on a flow page
alike, and a pass that only watches messages arrive has tested one subscription of
twenty-eight. The three `flowFile*` are the trap in the other direction — leave the
Files panel closed and they were never open to fail.

**Controls that hide.** Roughly three quarters of the app's controls are
conditional; these are the categories, with the ones most often missed:

- **Hover-only** — the sidebar's per-group actions (**New flow**, **New
  template**, **New knowledge**, **Upload file**) and every row action cell
  (**Toggle favorite**, **Open menu**) are `opacity-0` until their row or group
  is hovered. They are in the accessibility tree the whole time: click them by
  name, do not chase the hover.
- **Menu-only** — **Rename**, **Finish**, **Delete** on a flow live inside
  **Open menu** (list) and **Flow actions** (detail); the report's four exports
  live inside **Report**; the composer's attachments live inside **Templates and
  resources**. Open the trigger, snapshot again, then act — uids from the
  previous snapshot are stale.
- **Right-click twins** — every list repeats its whole row menu as a context menu
  (flows, templates, knowledges, providers, prompts, API tokens), and the file
  manager adds one on its rows and one on its empty area. Same writes, different
  path: test both.
- **State-gated** — **Finish** exists only while the flow is neither Failed nor
  Finished; the composer shows **Stop** instead of **Submit** exactly while it
  is Created or Running; the assistant row's delete hides once the flow or the
  selected assistant is terminal; **Report** appears only after the flow has at
  least one task. Seed
  the state or you have not tested the control.
- **Empty-state-only** — the **New Flow** / **New Knowledge** buttons inside an
  empty list, and **Try again**, which renders only when a query errored _and_
  the list is empty. A failed refetch with data on screen shows nothing.
- **Breakpoint-gated** — the flow header's favorite toggle and the
  previous/next navigation exist only at ≥768 px; on mobile the same writes move
  into the **Flow actions** menu. Below 1280 px the two tab strips merge into
  one strip of ten.
- **Account-type-gated** — only the password card on the account page is local-
  account-only; the email card renders for an OAuth account too, saying which
  provider it is linked from and dropping just its **Change** button. Asserting
  the card's absence therefore fails; assert the missing control.

**Table state lives in three places** — the URL (`?q=`, `?page=`), localStorage
(sorting, column visibility, page size, per-column search under a `table_*` key)
and the query itself. Check that a reload restores all three, and that clearing
the search does not quietly drop a filter you set elsewhere.

## Three widths, not two, and the routes that catch a miss

The app has three layouts, and the middle one is the reason "tested on mobile and
desktop" misses things. `useBreakpoint` calls under 768 px **mobile**, under
1280 px **tablet**, and the rest **desktop**. Between 768 and 1279 the sheet
navigation is gone but the split-pane layouts are still off — a case written for
"mobile" never visits it.

**Under 768** the sidebar is a drawer: a **Toggle Sidebar** button opens a dialog
titled **Sidebar** with the links **Dashboard**, **Flows**, **Templates**,
**Resources**, **Knowledges**, **Settings**. Its own close X is hidden by CSS, so
a role sweep finds no close control — Escape and the overlay are the way out.
Tapping any link except **Settings** leaves the drawer open on top of the new
page, and nothing resets it when the window widens: widen past 768, come back,
and the drawer is open without a click.

Also under 768, header actions keep their icons and drop their text. A check by
visible text reports **Save** as missing while a check by accessible name finds
it — the label is `display: none`, not absent. Detail-page prev/next leaves the
header and reappears inside the **…** menu as a row that opens a list sheet.

**At 768 and above** the sidebar is an icon rail. Its state is written to the
cookie `sidebar:state` for seven days and read only at mount, so a collapse in
one case silently changes the layout of every later one — and of the next
session on that browser. Ctrl/Cmd+B toggles it from anywhere with no visible
control; the shortcut is skipped while focus is in an input, a textarea or the
editor, which is why it does not steal Bold. On the main routes **two** controls
answer to the name **Toggle Sidebar** — the header trigger and an invisible rail
strip on the sidebar's edge — so a strict by-name query throws there and passes
on `/settings/*`, where the rail does not exist.

**The flow tabs change shape with the window.** Below 1280 the three central
tabs — **Automation**, **Assistant**, **Dashboard** — join the single strip and
the active one is written to `?tab=`. At 1280 and above they are a separate
pane, and a deep link like `?tab=terminal` lands on a different tab than it does
narrow. Test a deep link at the width you claim to have tested.

**A wrong address never shows a 404.** The root catch-all redirects anything
unknown to `/dashboard`, and `/settings/*` has its own; a typo therefore looks
like a working page, and a test that asserts "the app rejects a bad URL" cannot
pass. What does exist is a route error boundary with a **Reload** card, and
per-page not-found branches that differ from each other: a template shows a
**Template not found** card, a prompt shows a card that is a dead end, a
knowledge document shows a toast and bounces back to the list, and a provider
redirects with nothing said at all. Seed a bad id per entity — the four
behaviours are four different tests.

## The rest of a full pass

- **Search and filters** — case-insensitive across mixed case, the empty state,
  clearing restores the list, and the URL carries the query.
- **Pagination and sorting** — page size, "Page 1 of N", first/previous disabled
  on page 1, `aria-sort` correct before and after clicking a header.
- **Theme** — toggles and survives a reload.
- **Narrow viewport** — the tab strips collapse, nothing overflows horizontally
  on any route.
- **Console** — zero errors, everywhere, all pass.
- **The API matrix** for whatever validation the branch touched.

Budget: about an hour for the walk, plus whatever the parallel flow runs take to
finish on their own.

## The API matrix

Validation and authorization are **never** proven through the UI. A disabled
button is UX; the boundary is the endpoint, and an attacker sends the request
directly. Every rule gets the same five cells:

| cell                 | what it proves                                           |
| -------------------- | -------------------------------------------------------- |
| valid                | the value is accepted **and stored**                     |
| invalid              | blank / malformed / over-limit / unauthorized is refused |
| exactly at the limit | the boundary is where it claims to be                    |
| limit + 1            | it is not off by one                                     |
| no side effect       | after a reject, nothing was written                      |

A session and a bearer token, side by side, is the whole harness:

```bash
curl -sk -c jar.txt -X POST https://localhost:8443/api/v1/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"mail":"…","password":"…"}'

curl -sk -b jar.txt -X POST https://localhost:8443/api/v1/graphql \
  -H 'Content-Type: application/json' -d @query.json     # session

curl -sk -H "Authorization: Bearer $TOKEN" \
  "https://localhost:8443/api/v1/flows/?page=1&pageSize=5&type=init"   # token tier
```

`/users`, `/roles` and `/tokens` are session-only and answer `403` to any bearer
token by design — one that reached them could mint an administrator and escape its
own scope. The split is per-route, not per-prefix: `GET /user/` is on the bearer
tier, while `PUT /user/password`, `/user/email` and `/user/name` are session-only. Mint the token through the session, then probe the token tier on a route
it owns (`/flows`, `/settings`, `/graphql`, the log tables): a `403` from `/users`
is the boundary working, not a bad token.

`page` and `type` are both required on every list endpoint that binds the shared
table query, and the binder answers before the handler ever runs —
`…/knowledge/?page=1&pageSize=100` is `400 Knowledge.InvalidRequest`, the same URL
with `&type=init` is the list. `pageSize` is the one that is optional: leave it
out and the page comes back at its documented default of five, and `-1` asks for
the whole table. Match on the `code` in the body rather than the status, or a
binder 400 scores the "invalid is refused" cell for a request that never reached
the rule under test.

The same requests isolate a broken panel: ask the API for the data the panel
renders. If the payload is there and the card is empty, the bug is in the UI; if
the payload is empty, stop looking at React. That one query is usually shorter
than the argument about where the bug lives.

Two habits keep the results honest:

- **Read the artifact, not the tail of a pipe.** `cmd | tail && echo $?` reports
  `tail`'s exit code. Redirect to a file and check `$?`, or verify the result
  itself (the row exists, the image id changed).
- **A refusal is only half the check.** Follow every rejected request with a read
  that proves nothing was written — the list does not contain the row, the file
  is still a file, the blob was not orphaned.

## Proving a fix

The standard is a measured difference, not a passing check:

1. Run the exact input against the **previous** image and record the status and
   body. Keep the old tag around for this — `local/pentagi:<prev>`.
2. Rebuild, restart, re-verify the container id, run the same input.
3. Report both. "409 with the file intact, where the previous build answered 200
   and turned it into a directory" is evidence; "works now" is not.

The same discipline applies to a test: revert the fix, watch the new test fail
with the symptom that started it, restore. A test that passes either way is not
coverage.

Two ways that mutation lies, both seen here:

- **A deadline derived from the constant under test stretches instead of
  failing.** A test that waits `interval * 4` for a write to land, mutated by
  setting the interval to an hour, waits four hours — the run dies on the Go test
  timeout with no assertion message, and it reads like a hang rather than the
  failure you were looking for. Bound the wait in wall-clock seconds.
- **A mutation that breaks the build is not a mutation.** Deleting the call site
  leaves the helper unused, Go refuses to compile, and a red package tells you
  nothing about the assertion. Change the value, not the shape: an interval, a
  flag, one entry in a table.

Where the stack cannot produce the failure at all — an LLM key you do not have,
a process kill no unit test can stage — the fake of the interface under test is
the honest instrument. `flowMsgLogWorker` takes a `Querier` and a `FlowPublisher`,
so two structs embedding those interfaces answer "does any path publish an empty
result over a set one" in seconds, with no database and no credentials.

**A failed write is one of those.** Stopping the database mid-generation does not
stage it: the mutation that would start the generation fails first, answering
`a required service is unavailable`, so nothing is streaming when the writes
begin to fail. Timing an outage into an already-running generation is worse than
unreliable — the whole reply lands inside ten to twenty seconds, and the write
that must fail is one call inside that window. Three attempts here produced no
frame carrying the defect. The fake fails the exact call by ordinal
(`failContentOn: 1`) and reproduces it every run, which is why the assistant
worker's failure paths are pinned in `aslog_persist_test.go` and not on a stand.

What the stand *is* good for on that path is the healthy half: watching the
periodic write land. Poll the stored row while a long answer streams and it grows
in steps a few seconds apart — 1659, 2634, 3274, 4134 characters on one run —
which is the ticker doing its job, visible nowhere else.

## Leaving the stand clean

Every probe is a row someone else will trip over. Delete the tokens, providers,
templates, knowledge documents, resources and uploaded files you created, and
restart the stack without the environment overrides you added
(`COOKIE_SIGNING_SALT`, `OLLAMA_SERVER_URL`) so the next session inherits the
stand's own configuration. Then list what remains and check it against the seeds.

A shared stand is stricter still: it has no isolation, several people use the
same accounts, and anything you change is visible to everyone. Roll back what
you touch, and never put its URL or credentials in the repository — they belong
with the stand-tier secrets described in [`e2e.md`](./e2e.md#stand-tier-tier-3).

---

### Backend Docs Installer Charm Architecture Patterns (`backend/docs/installer/charm-architecture-patterns.md`)

# Charm.sh Production Architecture Patterns

> Essential patterns for building robust, scalable TUI applications with the Charm ecosystem.

## 🏗️ **Centralized Styles & Dimensions**

### **Styles Singleton Pattern**
**CRITICAL**: Never store width/height in models - use styles singleton

```
/* Detailed source-code truncated for AI context efficiency. */
```

## 🏗️ **Unified Header/Footer Management**

### **App-Level Layout Control**
```go
// app.go - Central layout control
func (a *App) View() string {
    header := a.renderHeader()
    footer := a.renderFooter()
    content := a.currentModel.View()

    contentHeight := max(height - headerHeight - footerHeight, 0)
    contentArea := a.styles.Content.Height(contentHeight).Render(content)

    return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}

func (a *App) renderHeader() string {
    switch a.navigator.Current() {
    case WelcomeScreen:
        return a.styles.RenderASCIILogo()
    default:
        title := a.getScreenTitle(a.navigator.Current())
        return a.styles.Header.Render(title)
    }
}

func (a *App) renderFooter() string {
    actions := a.buildFooterActions()
    footerText := strings.Join(actions, " • ")

    return a.styles.Footer.Render(footerText)
}

func (a *App) buildFooterActions() []string {
    actions := []string{"Esc: Back", "Ctrl+C: Exit"}

    // Screen-specific actions
    switch a.navigator.Current().GetScreen() {
    case "eula":
        if a.isEULAScrolledToEnd() {
            actions = append(actions, "Y: Accept", "N: Reject")
        } else {
            actions = append(actions, "↑↓: Scroll", "PgUp/PgDn: Page")
        }
    case "form":
        actions = append(actions, "Tab: Complete", "Ctrl+S: Save", "Enter: Save & Return")
    case "menu":
        actions = append(actions, "Enter: Select")
    }

    return actions
}
```

### **Background Footer Approach (Production)**
```go
// ✅ PRODUCTION READY: Background approach (always 1 line)
func (s *Styles) createFooter(width int, text string) string {
    return lipgloss.NewStyle().
        Width(width).
        Background(lipgloss.Color("240")).
        Foreground(lipgloss.Color("255")).
        Padding(0, 1, 0, 1).
        Render(text)
}

// ❌ WRONG: Border approach (height inconsistency)
func createFooterWrong(width int, text string) string {
    return lipgloss.NewStyle().
        Width(width).
        Height(1).
        Border(lipgloss.Border{Top: true}).
        Render(text)
}
```

## 🏗️ **Component Initialization Pattern**

### **Standard Component Architecture**
```go
// Standard component initialization
func NewModel(styles *styles.Styles, window *window.Window, controller *controllers.StateController) *Model {
    viewport := viewport.New(window.GetContentSize())
    viewport.Style = lipgloss.NewStyle() // Clean style prevents conflicts

    return &Model{
        styles:     styles,
        window:     window,
        controller: controller,
        viewport:   viewport,
        initialized: false,
    }
}

func (m *Model) Init() tea.Cmd {
    // ALWAYS reset ALL state
    m.content = ""
    m.ready = false
    m.error = nil
    m.initialized = false

    logger.Log("[Model] INIT: starting initialization")
    return m.loadContent
}

// Window size handling
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.WindowSizeMsg:
        // Update through styles, not directly
        // m.styles.SetSize() called by app.go
        m.updateViewport()

    case ContentLoadedMsg:
        m.content = msg.Content
        m.ready = true
        m.initialized = true
        m.viewport.SetContent(m.content)
        return m, nil

    case tea.KeyMsg:
        return m.handleKeyMsg(msg)
    }

    return m, nil
}
```

## 🏗️ **Model State Management**

### **Complete State Reset Pattern**
```go
// Model interface implementation
type Model struct {
    styles *styles.Styles  // ALWAYS use shared styles
    window *window.Window

    // Core state
    content     string
    ready       bool
    error       error
    initialized bool

    // Component state
    viewport viewport.Model

    // Navigation state
    args []string
}

func (m *Model) Init() tea.Cmd {
    logger.Log("[Model] INIT")

    // ALWAYS reset ALL state completely
    m.content = ""
    m.ready = false
    m.error = nil
    m.initialized = false

    // Reset component state
    m.viewport.GotoTop()
    m.viewport.SetContent("")

    return m.loadContent
}

// Force re-render after async operations
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case ContentLoadedMsg:
        m.content = msg.Content
        m.ready = true
        m.initialized = true
        m.viewport.SetContent(m.content)

        // Force view update with no-op command
        return m, func() tea.Msg { return nil }
    }
    return m, nil
}
```

### **Graceful Error Handling**
```go
func (m *Model) View() string {
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        return "Loading..." // Graceful fallback
    }

    if m.error != nil {
        return m.styles.Error.Render("Error: " + m.error.Error())
    }

    if !m.ready {
        return m.styles.Info.Render("Loading content...")
    }

    return m.viewport.View()
}
```

## 🏗️ **Responsive Layout Architecture**

### **Breakpoint-Based Design**
```
/* Detailed source-code truncated for AI context efficiency. */
```

## 🏗️ **Content Loading Architecture**

### **Async Content Loading**
```
/* Detailed source-code truncated for AI context efficiency. */
```

## 🏗️ **Window Management Pattern**

### **Window Integration**
```go
type Window struct {
    width  int
    height int
}

func NewWindow() *Window {
    return &Window{width: 80, height: 24}
}

func (w *Window) SetSize(width, height int) {
    w.width = width
    w.height = height
}

func (w *Window) GetSize() (int, int) {
    return w.width, w.height
}

func (w *Window) GetContentSize() (int, int) {
    // Account for padding and borders
    contentWidth := max(w.width-4, 0)
    contentHeight := max(w.height-6, 0) // Header + footer + padding
    return contentWidth, contentHeight
}

func (w *Window) GetContentWidth() int {
    width, _ := w.GetContentSize()
    return width
}

func (w *Window) GetContentHeight() int {
    _, height := w.GetContentSize()
    return height
}

// Integration with models
func (m *Model) updateDimensions() {
    width, height := m.window.GetContentSize()
    if width <= 0 || height <= 0 {
        return
    }

    m.viewport.Width = width
    m.viewport.Height = height
}
```

## 🏗️ **Controller Integration Pattern**

### **StateController Bridge**
```go
type StateController struct {
    state *state.State
}

func NewStateController(state *state.State) *StateController {
    return &StateController{state: state}
}

// Environment variable management
func (c *StateController) GetVar(name string) (loader.EnvVar, error) {
    return c.state.GetVar(name)
}

func (c *StateController) SetVar(name, value string) error {
    return c.state.SetVar(name, value)
}

// Configuration management
func (c *StateController) GetLLMProviders() map[string]ProviderConfig {
    // Implementation specific to controller
    return c.state.GetLLMProviders()
}

func (c *StateController) SaveConfiguration() error {
    return c.state.Save()
}

// Model integration
type Model struct {
    controller *StateController
    // ... other fields
}

func (m *Model) loadConfiguration() {
    configs := m.controller.GetLLMProviders()
    // Use configs to populate model state
}
```

## 🏗️ **Essential Key Handling**

### **Universal Key Patterns**
```go
func (m *Model) handleKeyMsg(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
    switch msg.String() {
    // Universal scroll controls
    case "up":
        m.viewport.ScrollUp(1)
    case "down":
        m.viewport.ScrollDown(1)
    case "left", "h":
        m.viewport.ScrollLeft(2)  // 2 steps for faster horizontal
    case "right", "l":
        m.viewport.ScrollRight(2)

    // Page controls
    case "pgup":
        m.viewport.ScrollUp(m.viewport.Height)
    case "pgdown":
        m.viewport.ScrollDown(m.viewport.Height)
    case "home":
        m.viewport.GotoTop()
    case "end":
        m.viewport.GotoBottom()

    // Universal actions (handled at app level)
    case "esc":
        // Handled by app.go - returns to welcome
        return m, nil
    case "ctrl+c":
        // Handled by app.go - quits application
        return m, nil

    default:
        // Screen-specific handling
        return m.handleScreenSpecificKeys(msg)
    }

    return m, nil
}

func (m *Model) handleScreenSpecificKeys(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
    // Override in specific models
    return m, nil
}
```

## 🎯 **Architecture Best Practices**

### **Separation of Concerns**
```go
// ✅ CORRECT: Clean separation
// app.go        - Navigation, layout, global state
// models/       - Screen-specific logic, local state
// styles/       - Styling, dimensions, rendering
// controller/   - Business logic, state management
// window/       - Terminal size management

// Model responsibilities
type Model struct {
    // ONLY screen-specific state
    content string
    ready   bool

    // Dependencies (injected)
    styles     *styles.Styles    // Shared styling
    window     *window.Window    // Size management
    controller *Controller       // Business logic
}

// App responsibilities
type App struct {
    // Global state
    navigator    *Navigator
    currentModel tea.Model

    // Shared resources
    styles     *styles.Styles
    window     *window.Window
    controller *Controller
}
```

### **Resource Management**
```go
// ✅ CORRECT: Shared resources
func NewApp() *App {
    styles := styles.New()
    window := window.New()
    controller := controller.New()

    return &App{
        styles:     styles,     // Single instance
        window:     window,     // Single instance
        controller: controller, // Single instance
        navigator:  navigator.New(),
    }
}

// ❌ WRONG: Resource duplication
func NewModelWrong() *Model {
    styles := styles.New()       // Multiple instances!
    renderer, _ := glamour.New() // Multiple renderers!
    return &Model{styles: styles}
}
```

This architecture provides:
- **Scalability**: Clean separation enables easy feature additions
- **Maintainability**: Centralized resources reduce coupling
- **Performance**: Shared instances prevent resource waste
- **Consistency**: Unified patterns across all components
- **Reliability**: Proper error handling and state management

---

### Backend Docs Installer Charm Best Practices (`backend/docs/installer/charm-best-practices.md`)

# Charm.sh Best Practices & Performance Guide

> Proven patterns and anti-patterns for building high-performance TUI applications.

## 🚀 **Performance Best Practices**

### **Single Glamour Renderer (Critical)**
**Problem**: Multiple renderer instances cause freezing and memory leaks
**Solution**: Create once, reuse everywhere

```go
// ✅ CORRECT: Single renderer instance
type Styles struct {
    renderer *glamour.TermRenderer
}

func New() *Styles {
    renderer, _ := glamour.NewTermRenderer(
        glamour.WithAutoStyle(),
        glamour.WithWordWrap(80),
    )
    return &Styles{renderer: renderer}
}

func (s *Styles) GetRenderer() *glamour.TermRenderer {
    return s.renderer
}

// Usage: Always use shared renderer
rendered, _ := m.styles.GetRenderer().Render(content)

// ❌ WRONG: Creating new renderer each time
func renderContent(content string) string {
    renderer, _ := glamour.NewTermRenderer(...) // Memory leak + freeze risk!
    return renderer.Render(content)
}
```

### **Centralized Dimension Management**
**Problem**: Dimensions stored in models cause sync issues
**Solution**: Single source of truth in styles

```go
// ✅ CORRECT: Centralized dimensions
type Styles struct {
    width  int
    height int
}

func (s *Styles) SetSize(width, height int) {
    s.width = width
    s.height = height
    s.updateStyles() // Recalculate responsive styles
}

func (s *Styles) GetSize() (int, int) {
    return s.width, s.height
}

// Models use styles for dimensions
func (m *Model) updateViewport() {
    width, height := m.styles.GetSize()
    // ... safe dimension usage
}

// ❌ WRONG: Models managing dimensions
type Model struct {
    width, height int // Will get out of sync!
}
```

### **Efficient Viewport Usage**

#### **Permanent vs Temporary Viewports**
```go
// ✅ CORRECT: Permanent viewport for forms (preserves scroll state)
type FormModel struct {
    viewport viewport.Model // Permanent - keeps scroll position
}

func (m *FormModel) ensureFocusVisible() {
    // Scroll calculations use permanent viewport state
    if focusY < m.viewport.YOffset {
        m.viewport.YOffset = focusY
    }
}

// ✅ CORRECT: Temporary viewport for layout rendering
func (m *Model) renderHorizontalLayout(left, right string, width, height int) string {
    content := lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)

    // Create viewport just for rendering
    vp := viewport.New(width, height-PaddingHeight)
    vp.SetContent(content)
    return vp.View()
}

// ❌ WRONG: Creating viewport in View() - loses scroll state
func (m *FormModel) View() string {
    vp := viewport.New(width, height) // State lost on re-render!
    return vp.View()
}
```

### **Content Loading Optimization**
```go
// ✅ CORRECT: Lazy loading with caching
type Model struct {
    contentCache map[string]string
    loadOnce     sync.Once
}

func (m *Model) loadContent() tea.Cmd {
    return func() tea.Msg {
        // Check cache first
        if cached, exists := m.contentCache[m.contentKey]; exists {
            return ContentLoadedMsg{cached}
        }

        // Load and cache
        content, err := m.loadFromSource()
        if err != nil {
            return ErrorMsg{err}
        }

        m.contentCache[m.contentKey] = content
        return ContentLoadedMsg{content}
    }
}

// ❌ WRONG: Loading on every Init()
func (m *Model) Init() tea.Cmd {
    return m.loadContent // Reloads every time!
}
```

## 🎯 **Architecture Best Practices**

### **Clean Separation of Concerns**
```go
// ✅ CORRECT: Clear responsibilities
// app.go - Global navigation, layout management, shared resources
type App struct {
    navigator    *Navigator      // Navigation state
    currentModel tea.Model       // Current screen
    styles       *Styles         // Shared styling
    window       *Window         // Dimension management
    controller   *Controller     // Business logic
}

// models/ - Screen-specific logic only
type ScreenModel struct {
    // Screen-specific state only
    content string
    ready   bool

    // Injected dependencies
    styles     *Styles
    window     *Window
    controller *Controller
}

// controller/ - Business logic, no UI concerns
type Controller struct {
    state *State
}

func (c *Controller) GetConfiguration() Config {
    // Pure business logic, no UI dependencies
}

// ❌ WRONG: Mixed responsibilities
type Model struct {
    // UI state
    viewport viewport.Model

    // Business logic (should be in controller)
    database *sql.DB
    apiClient *http.Client

    // Global state (should be in app)
    allScreens map[string]tea.Model
}
```

### **Resource Management**
```go
// ✅ CORRECT: Dependency injection
func NewApp() *App {
    // Create shared resources once
    styles := styles.New()
    window := window.New()
    controller := controller.New()

    return &App{
        styles:     styles,
        window:     window,
        controller: controller,
    }
}

func (a *App) createModelForScreen(screenID ScreenID) tea.Model {
    // Inject shared dependencies
    return NewScreenModel(a.styles, a.window, a.controller)
}

// ❌ WRONG: Resource duplication
func NewScreenModel() *ScreenModel {
    return &ScreenModel{
        styles:     styles.New(),     // Multiple instances!
        controller: controller.New(), // Multiple instances!
    }
}
```

## 🎯 **State Management Best Practices**

### **Complete State Reset Pattern**
```go
// ✅ CORRECT: Reset ALL state in Init()
func (m *Model) Init() tea.Cmd {
    // Reset UI state
    m.content = ""
    m.ready = false
    m.error = nil
    m.initialized = false

    // Reset component state
    m.viewport.GotoTop()
    m.viewport.SetContent("")

    // Reset form state
    m.focusedIndex = 0
    m.hasChanges = false
    for i := range m.fields {
        m.fields[i].Input.Blur()
    }

    // Reset navigation args
    m.selectedIndex = m.getSelectedIndexFromArgs()

    return m.loadContent
}

// ❌ WRONG: Partial state reset
func (m *Model) Init() tea.Cmd {
    m.content = "" // Only resetting some fields!
    return m.loadContent
}
```

### **Args-Based Construction**
```go
// ✅ CORRECT: Selection from constructor args
func NewModel(args []string) *Model {
    selectedIndex := 0
    if len(args) > 0 && args[0] != "" {
        for i, item := range items {
            if item.ID == args[0] {
                selectedIndex = i
                break
            }
        }
    }

    return &Model{
        selectedIndex: selectedIndex,
        args:          args,
    }
}

// No separate SetSelected methods needed
func (m *Model) Init() tea.Cmd {
    // Selection already set in constructor
    return m.loadData
}

// ❌ WRONG: Separate setter methods
func (m *Model) SetSelectedItem(itemID string) {
    // Adds complexity, sync issues
    for i, item := range m.items {
        if item.ID == itemID {
            m.selectedIndex = i
            break
        }
    }
}
```

## 🎯 **Navigation Best Practices**

### **Type-Safe Navigation**
```go
// ✅ CORRECT: Type-safe constants and helpers
type ScreenID string
const (
    WelcomeScreen ScreenID = "welcome"
    MenuScreen    ScreenID = "menu"
)

func CreateScreenID(screen string, args ...string) ScreenID {
    if len(args) == 0 {
        return ScreenID(screen)
    }
    return ScreenID(screen + "§" + strings.Join(args, "§"))
}

// Usage
return NavigationMsg{Target: CreateScreenID("form", "provider", "openai")}

// ❌ WRONG: String-based navigation
return NavigationMsg{Target: "form/provider/openai"} // Typo-prone!
```

### **GoBack Navigation Pattern**
```go
// ✅ CORRECT: Use GoBack to prevent loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    if err := m.saveConfiguration(); err != nil {
        return m, nil // Stay on form if save fails
    }

    return m, func() tea.Msg {
        return NavigationMsg{GoBack: true} // Return to previous screen
    }
}

// ❌ WRONG: Direct navigation creates loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    m.saveConfiguration()
    return m, func() tea.Msg {
        return NavigationMsg{Target: ProvidersScreen} // Creates navigation loop!
    }
}
```

## 🎯 **Form Best Practices**

### **Dynamic Width Management**
```go
// ✅ CORRECT: Calculate width dynamically
func (m *FormModel) updateFormContent() {
    inputWidth := m.getInputWidth()

    for i, field := range m.fields {
        // Apply width during rendering, not initialization
        field.Input.Width = inputWidth - 3
        field.Input.SetValue(field.Input.Value()) // Trigger width update
    }
}

func (m *FormModel) getInputWidth() int {
    viewportWidth, _ := m.getViewportSize()
    inputWidth := viewportWidth - 6 // Account for padding
    if m.isVerticalLayout() {
        inputWidth = viewportWidth - 4 // Less padding in vertical
    }
    return inputWidth
}

// ❌ WRONG: Fixed width at initialization
func (m *FormModel) createField() {
    input := textinput.New()
    input.Width = 50 // Breaks responsive design!
}
```

### **Environment Variable Integration**
```go
// ✅ CORRECT: Track initial state for cleanup
func (m *FormModel) buildForm() {
    m.initiallySetFields = make(map[string]bool)

    for _, fieldConfig := range m.fieldConfigs {
        envVar, _ := m.controller.GetVar(fieldConfig.EnvVarName)

        // Track if field was initially set
        m.initiallySetFields[fieldConfig.Key] = envVar.IsPresent()

        field := m.createFieldFromEnvVar(fieldConfig, envVar)
        m.fields = append(m.fields, field)
    }
}

func (m *FormModel) saveConfiguration() error {
    // First pass: Remove cleared fields
    for _, field := range m.fields {
        value := strings.TrimSpace(field.Input.Value())

        if value == "" && m.initiallySetFields[field.Key] {
            // Field was set but now empty - remove from environment
            m.controller.SetVar(field.EnvVarName, "")
        }
    }

    // Second pass: Save non-empty values
    for _, field := range m.fields {
        value := strings.TrimSpace(field.Input.Value())
        if value != "" {
            m.controller.SetVar(field.EnvVarName, value)
        }
    }

    return nil
}

// ❌ WRONG: No cleanup tracking
func (m *FormModel) saveConfiguration() error {
    for _, field := range m.fields {
        // Always sets value, even if it should be removed
        m.controller.SetVar(field.EnvVarName, field.Input.Value())
    }
}
```

## 🎯 **Layout Best Practices**

### **Responsive Breakpoints**
```go
// ✅ CORRECT: Consistent breakpoint logic
const (
    MinTerminalWidth = 80
    MinMenuWidth     = 38
    MaxMenuWidth     = 66
    MinInfoWidth     = 34
    PaddingWidth     = 8
)

func (m *Model) isVerticalLayout() bool {
    contentWidth := m.window.GetContentWidth()
    return contentWidth < (MinMenuWidth + MinInfoWidth + PaddingWidth)
}

func (m *Model) renderHorizontalLayout(leftPanel, rightPanel string, width, height int) string {
    leftWidth, rightWidth := MinMenuWidth, MinInfoWidth
    extraWidth := width - leftWidth - rightWidth - PaddingWidth

    if extraWidth > 0 {
        leftWidth = min(leftWidth+extraWidth/2, MaxMenuWidth) // Cap at max
        rightWidth = width - leftWidth - PaddingWidth/2
    }

    // ... render with calculated widths
}

// ❌ WRONG: Arbitrary breakpoints
func (m *Model) isVerticalLayout() bool {
    return m.width < 85 // Magic number!
}
```

### **Content Hiding Strategy**
```go
// ✅ CORRECT: Graceful content hiding
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
    leftStyled := verticalStyle.Render(leftPanel)
    rightStyled := verticalStyle.Render(rightPanel)

    // Show both panels if they fit
    if lipgloss.Height(leftStyled)+lipgloss.Height(rightStyled)+2 < height {
        return lipgloss.JoinVertical(lipgloss.Left, leftStyled, "", rightStyled)
    }

    // Hide right panel if insufficient space - show only essential content
    return leftStyled
}

// ❌ WRONG: Always showing all content
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
    // Forces both panels even if they don't fit
    return lipgloss.JoinVertical(lipgloss.Left, leftPanel, rightPanel)
}
```

## 🚀 **Performance Optimization**

### **Memory Management**
```go
// ✅ CORRECT: Efficient string building
func (m *Model) buildLargeContent() string {
    var builder strings.Builder
    builder.Grow(1024) // Pre-allocate capacity

    for _, section := range m.sections {
        builder.WriteString(section)
        builder.WriteString("\n")
    }

    return builder.String()
}

// ❌ WRONG: String concatenation in loop
func (m *Model) buildLargeContent() string {
    content := ""
    for _, section := range m.sections {
        content += section + "\n" // Creates new string each iteration!
    }
    return content
}
```

### **Viewport Content Updates**
```go
// ✅ CORRECT: Update content only when changed
func (m *Model) updateViewportContent() {
    newContent := m.buildContent()

    // Only update if content changed
    if newContent != m.lastContent {
        m.viewport.SetContent(newContent)
        m.lastContent = newContent
    }
}

// ❌ WRONG: Always updating content
func (m *Model) View() string {
    content := m.buildContent()
    m.viewport.SetContent(content) // Updates every render!
    return m.viewport.View()
}
```

## 🚀 **Error Handling Best Practices**

### **Graceful Degradation**
```go
// ✅ CORRECT: Multiple fallback levels
func (m *Model) View() string {
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        return "Loading..." // Dimension fallback
    }

    if m.error != nil {
        return m.styles.Error.Render("Error: " + m.error.Error()) // Error fallback
    }

    if !m.ready {
        return m.styles.Info.Render("Loading content...") // Loading fallback
    }

    return m.viewport.View() // Normal rendering
}

// ❌ WRONG: No fallbacks
func (m *Model) View() string {
    return m.viewport.View() // Crashes if viewport not initialized!
}
```

### **Safe Async Operations**
```go
// ✅ CORRECT: Safe async with error handling
func (m *Model) loadContent() tea.Cmd {
    return func() tea.Msg {
        defer func() {
            if r := recover(); r != nil {
                return ErrorMsg{fmt.Errorf("panic in loadContent: %v", r)}
            }
        }()

        content, err := m.loadFromSource()
        if err != nil {
            return ErrorMsg{err}
        }

        return ContentLoadedMsg{content}
    }
}

// ❌ WRONG: No error handling
func (m *Model) loadContent() tea.Cmd {
    return func() tea.Msg {
        content, _ := m.loadFromSource() // Ignores errors!
        return ContentLoadedMsg{content}
    }
}
```

## 🎯 **Key Anti-Patterns to Avoid**

### **❌ Don't Do These**
```go
// ❌ NEVER: Console output in TUI
fmt.Println("debug")
log.Println("debug")

// ❌ NEVER: Multiple glamour renderers
renderer1 := glamour.NewTermRenderer(...)
renderer2 := glamour.NewTermRenderer(...)

// ❌ NEVER: Dimensions in models
type Model struct {
    width, height int
}

// ❌ NEVER: Direct navigation creating loops
return NavigationMsg{Target: PreviousScreen}

// ❌ NEVER: Fixed input widths
input.Width = 50

// ❌ NEVER: Partial state reset
func (m *Model) Init() tea.Cmd {
    m.content = "" // Missing other state!
}

// ❌ NEVER: ClearScreen during navigation
return tea.Batch(cmd, tea.ClearScreen)

// ❌ NEVER: String-based navigation
return NavigationMsg{Target: "screen_name"}
```

### **✅ Always Do These**
```go
// ✅ ALWAYS: File-based logging
logger.Log("[Component] EVENT: %v", msg)

// ✅ ALWAYS: Single shared renderer
rendered := m.styles.GetRenderer().Render(content)

// ✅ ALWAYS: Centralized dimensions
width, height := m.styles.GetSize()

// ✅ ALWAYS: GoBack navigation
return NavigationMsg{GoBack: true}

// ✅ ALWAYS: Dynamic input sizing
input.Width = m.getInputWidth()

// ✅ ALWAYS: Complete state reset
func (m *Model) Init() tea.Cmd {
    m.resetAllState()
    return m.loadContent
}

// ✅ ALWAYS: Clean model initialization
return a, a.currentModel.Init()

// ✅ ALWAYS: Type-safe navigation
return NavigationMsg{Target: CreateScreenID("screen", "arg")}
```

This guide ensures:
- **Performance**: Efficient resource usage and rendering
- **Reliability**: Robust error handling and state management
- **Maintainability**: Clean architecture and consistent patterns
- **User Experience**: Responsive design and graceful degradation

---

### Backend Docs Installer Charm Core Libraries (`backend/docs/installer/charm-core-libraries.md`)

# Charm.sh Core Libraries Reference

> Comprehensive guide to the core libraries in the Charm ecosystem for building TUI applications.

## 📦 **Core Libraries Overview**

### Core Packages
- **`bubbletea`**: Event-driven TUI framework (MVU pattern)
- **`lipgloss`**: Styling and layout engine
- **`bubbles`**: Pre-built components (viewport, textinput, etc.)
- **`huh`**: Advanced form builder
- **`glamour`**: Markdown renderer

## 🫧 **BubbleTea (MVU Pattern)**

### Model-View-Update Lifecycle
```go
// Model holds all state
type Model struct {
    content string
    ready   bool
}

// Update handles events and returns new state
func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.WindowSizeMsg:
        // Handle resize
        return m, nil
    case tea.KeyMsg:
        // Handle keyboard input
        return m, nil
    }
    return m, nil
}

// View renders current state
func (m Model) View() string {
    return "content"
}

// Init returns initial command
func (m Model) Init() tea.Cmd {
    return nil
}
```

### Commands and Messages
```go
// Commands return future messages
func loadDataCmd() tea.Msg {
    return DataLoadedMsg{data: "loaded"}
}

// Async operations
return m, tea.Cmd(func() tea.Msg {
    time.Sleep(time.Second)
    return TimerMsg{}
})
```

### Critical Patterns
```go
// Model interface implementation
type Model struct {
    styles *styles.Styles  // ALWAYS use shared styles
}

func (m Model) Init() tea.Cmd {
    // ALWAYS reset state completely
    m.content = ""
    m.ready = false
    return m.loadContent
}

func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.WindowSizeMsg:
        // NEVER store dimensions in model - use styles.SetSize()
        // Model gets dimensions via m.styles.GetSize()
    case tea.KeyMsg:
        switch msg.String() {
        case "enter": return m, navigateCmd
        }
    }
}
```

## 🎨 **Lipgloss (Styling & Layout)**

**Purpose**: CSS-like styling for terminal interfaces
**Key Insight**: Height() vs MaxHeight() behavior difference!

### Critical Height Control
```go
// ❌ WRONG: Height() sets MINIMUM height (can expand!)
style := lipgloss.NewStyle().Height(1).Border(lipgloss.NormalBorder())

// ✅ CORRECT: MaxHeight() + Inline() for EXACT height
style := lipgloss.NewStyle().MaxHeight(1).Inline(true)

// ✅ PRODUCTION: Background approach for consistent 1-line footers
footer := lipgloss.NewStyle().
    Width(width).
    Background(borderColor).
    Foreground(textColor).
    Padding(0, 1, 0, 1).  // Only horizontal padding
    Render(text)

// FOOTER APPROACH - PRODUCTION READY (✅ PROVEN SOLUTION)
// ❌ WRONG: Border approach (inconsistent height)
style.BorderTop(true).Height(1)

// ✅ CORRECT: Background approach (always 1 line)
style.Background(color).Foreground(textColor).Padding(0,1,0,1)
```

### Layout Patterns
```go
// LAYOUT COMPOSITION
lipgloss.JoinVertical(lipgloss.Left, header, content, footer)
lipgloss.JoinHorizontal(lipgloss.Top, left, right)
lipgloss.Place(width, height, lipgloss.Center, lipgloss.Top, content)

// Horizontal layout
left := lipgloss.NewStyle().Width(leftWidth).Render(leftContent)
right := lipgloss.NewStyle().Width(rightWidth).Render(rightContent)
combined := lipgloss.JoinHorizontal(lipgloss.Top, left, right)

// Vertical layout with consistent spacing
sections := []string{header, content, footer}
combined := lipgloss.JoinVertical(lipgloss.Left, sections...)

// Centering content
centered := lipgloss.Place(width, height, lipgloss.Center, lipgloss.Center, content)

// Responsive design
verticalStyle := lipgloss.NewStyle().Width(width).Padding(0, 2, 0, 2)
if width < 80 {
    // Vertical layout for narrow screens
}
```

### Responsive Patterns
```go
// Breakpoint-based layout
width, height := m.styles.GetSize()  // ALWAYS from styles
if width < 80 {
    return lipgloss.JoinVertical(lipgloss.Left, panels...)
} else {
    return lipgloss.JoinHorizontal(lipgloss.Top, panels...)
}

// Dynamic width allocation
leftWidth := width / 3
rightWidth := width - leftWidth - 4
```

## 📺 **Bubbles (Interactive Components)**

**Purpose**: Pre-built interactive components
**Key Components**: viewport, textinput, list, table

### Viewport - Critical for Scrolling
```go
import "github.com/charmbracelet/bubbles/viewport"

// Setup
viewport := viewport.New(width, height)
viewport.Style = lipgloss.NewStyle() // Clean style prevents conflicts

// Modern scroll methods (use these!)
viewport.ScrollUp(1)     // Replaces LineUp()
viewport.ScrollDown(1)   // Replaces LineDown()
viewport.ScrollLeft(2)   // Horizontal, 2 steps for forms
viewport.ScrollRight(2)

// Deprecated (avoid)
vp.LineUp(lines)       // ❌ Deprecated
vp.LineDown(lines)     // ❌ Deprecated

// Status tracking
viewport.ScrollPercent() // 0.0 to 1.0
viewport.AtBottom()      // bool
viewport.AtTop()         // bool

// State checking
isScrollable := !(vp.AtTop() && vp.AtBottom())
progress := vp.ScrollPercent()

// Content management
viewport.SetContent(content)
viewport.View() // Renders visible portion

// Update in message handling
var cmd tea.Cmd
m.viewport, cmd = m.viewport.Update(msg)
```

### TextInput
```go
import "github.com/charmbracelet/bubbles/textinput"

ti := textinput.New()
ti.Placeholder = "Enter text..."
ti.Focus()
ti.EchoMode = textinput.EchoPassword  // For masked input
ti.CharLimit = 100
```

## 📝 **Huh (Forms)**

**Purpose**: Advanced form builder for complex user input
```go
import "github.com/charmbracelet/huh"

form := huh.NewForm(
    huh.NewGroup(
        huh.NewInput().
            Key("api_key").
            Title("API Key").
            Password().  // Masked input
            Validate(func(s string) error {
                if len(s) < 10 {
                    return errors.New("API key too short")
                }
                return nil
            }),

        huh.NewSelect[string]().
            Key("provider").
            Title("Provider").
            Options(
                huh.NewOption("OpenAI", "openai"),
                huh.NewOption("Anthropic", "anthropic"),
            ),
    ),
)

// Integration with bubbletea
func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    var cmd tea.Cmd
    m.form, cmd = m.form.Update(msg)

    if m.form.State == huh.StateCompleted {
        // Form submitted - access values
        apiKey := m.form.GetString("api_key")
        provider := m.form.GetString("provider")
    }

    return m, cmd
}
```

## ✨ **Glamour (Markdown Rendering)**

**Purpose**: Beautiful markdown rendering in terminal
**CRITICAL**: Create renderer ONCE in styles.New(), reuse everywhere

```go
// ✅ CORRECT: Single renderer instance (prevents freezing)
// styles.go
type Styles struct {
    renderer *glamour.TermRenderer
}

func New() *Styles {
    renderer, _ := glamour.NewTermRenderer(
        glamour.WithAutoStyle(),
        glamour.WithWordWrap(80),
    )
    return &Styles{renderer: renderer}
}

func (s *Styles) GetRenderer() *glamour.TermRenderer {
    return s.renderer
}

// Usage in models
rendered, err := m.styles.GetRenderer().Render(markdown)

// ❌ WRONG: Creating new renderer each time (can freeze!)
renderer, _ := glamour.NewTermRenderer(...)
```

### Safe Rendering with Fallback
```go
// Safe rendering with fallback
rendered, err := renderer.Render(content)
if err != nil {
    // Fallback to plain text
    rendered = fmt.Sprintf("# Content\n\n%s\n\n*Render error: %v*", content, err)
}
```

## 🏗️ **Core Integration Patterns**

### Centralized Styles & Dimensions

**CRITICAL**: Never store width/height in models - use styles singleton

```go
// ✅ CORRECT: Centralized in styles
type Styles struct {
    width    int
    height   int
    renderer *glamour.TermRenderer
    // ... all styles
}

func (s *Styles) SetSize(width, height int) {
    s.width = width
    s.height = height
    s.updateStyles()  // Recalculate responsive styles
}

func (s *Styles) GetSize() (int, int) {
    return s.width, s.height
}

// Models use styles for dimensions
func (m *Model) updateViewport() {
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        return  // Graceful handling
    }
    // ... viewport setup
}
```

### Component Initialization Pattern
```go
// Standard component initialization
func NewModel(styles *styles.Styles, window *window.Window) *Model {
    viewport := viewport.New(window.GetContentSize())
    viewport.Style = lipgloss.NewStyle() // Clean style

    return &Model{
        styles:   styles,
        window:   window,
        viewport: viewport,
    }
}

func (m *Model) Init() tea.Cmd {
    // ALWAYS reset ALL state
    m.content = ""
    m.ready = false
    m.error = nil
    return m.loadContent
}
```

### Essential Key Handling
```go
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.WindowSizeMsg:
        // Update through styles, not directly
        // m.styles.SetSize() called by app.go
        m.updateViewport()

    case tea.KeyMsg:
        switch msg.String() {
        case "up":
            m.viewport.ScrollUp(1)
        case "down":
            m.viewport.ScrollDown(1)
        case "left":
            m.viewport.ScrollLeft(2)
        case "right":
            m.viewport.ScrollRight(2)
        case "pgup":
            m.viewport.ScrollUp(m.viewport.Height)
        case "pgdown":
            m.viewport.ScrollDown(m.viewport.Height)
        case "home":
            m.viewport.GotoTop()
        case "end":
            m.viewport.GotoBottom()
        }
    }
    return m, nil
}
```

## 🔧 **Common Integration Patterns**

### Content Loading
```go
// Load content asynchronously
func (m *Model) loadContent() tea.Cmd {
    return func() tea.Msg {
        content, err := loadFromSource()
        if err != nil {
            return ErrorMsg{err}
        }
        return ContentLoadedMsg{content}
    }
}

// Handle loading in Update
case ContentLoadedMsg:
    m.content = msg.Content
    m.ready = true
    m.viewport.SetContent(m.content)
    return m, nil
```

### Error Handling
```go
// Graceful error handling
func (m *Model) View() string {
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        return "Loading..." // Graceful fallback
    }

    if m.error != nil {
        return m.styles.Error.Render("Error: " + m.error.Error())
    }

    if !m.ready {
        return m.styles.Info.Render("Loading content...")
    }

    return m.viewport.View()
}
```

This reference provides the foundation for building robust TUI applications with the Charm ecosystem. Each library serves a specific purpose and when combined correctly, creates powerful terminal interfaces.

---

### Backend Docs Installer Charm Debugging Guide (`backend/docs/installer/charm-debugging-guide.md`)

# Charm.sh Debugging & Troubleshooting Guide

> Comprehensive guide to debugging TUI applications and avoiding common pitfalls.

## 🔧 **TUI-Safe Logging System**

### **File-Based Logger Pattern**
**Problem**: fmt.Printf breaks TUI rendering
**Solution**: File-based logger with structured output

```go
// logger.go - TUI-safe logging implementation
package logger

import (
    "encoding/json"
    "os"
    "time"
)

type LogEntry struct {
    Timestamp string `json:"timestamp"`
    Level     string `json:"level"`
    Component string `json:"component"`
    Message   string `json:"message"`
    Data      any    `json:"data,omitempty"`
}

var logFile *os.File

func init() {
    var err error
    logFile, err = os.OpenFile("log.json", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
    if err != nil {
        panic(err)
    }
}

func Log(format string, args ...any) {
    writeLog("INFO", fmt.Sprintf(format, args...))
}

func Errorf(format string, args ...any) {
    writeLog("ERROR", fmt.Sprintf(format, args...))
}

func Debugf(format string, args ...any) {
    writeLog("DEBUG", fmt.Sprintf(format, args...))
}

func LogWithData(message string, data any) {
    entry := LogEntry{
        Timestamp: time.Now().Format(time.RFC3339),
        Level:     "INFO",
        Message:   message,
        Data:      data,
    }

    jsonData, _ := json.Marshal(entry)
    logFile.Write(append(jsonData, '\n'))
}

func writeLog(level, message string) {
    entry := LogEntry{
        Timestamp: time.Now().Format(time.RFC3339),
        Level:     level,
        Message:   message,
    }

    jsonData, _ := json.Marshal(entry)
    logFile.Write(append(jsonData, '\n'))
}
```

### **Development Monitoring**
```bash
# Monitor logs in separate terminal during development
tail -f log.json | jq '.'

# Filter by component
tail -f log.json | jq 'select(.component == "FormModel")'

# Filter by level
tail -f log.json | jq 'select(.level == "ERROR")'

# Real-time pretty printing
tail -f log.json | jq -r '"\(.timestamp) [\(.level)] \(.message)"'
```

### **Safe Debug Output**
```go
// ❌ NEVER: Breaks TUI rendering
fmt.Println("debug")
log.Println("debug")
os.Stdout.WriteString("debug")

// ✅ ALWAYS: File-based logging
logger.Log("[Component] Event: %v", msg)
logger.Log("[Model] UPDATE: key=%s", msg.String())
logger.Log("[Model] VIEWPORT: %dx%d ready=%v", width, height, m.ready)
logger.Errorf("[Model] ERROR: %v", err)

// Development pattern
func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    logger.Log("[%s] UPDATE: %T", m.componentName, msg)

    switch msg := msg.(type) {
    case tea.KeyMsg:
        logger.Log("[%s] KEY: %s", m.componentName, msg.String())
        return m.handleKeyMsg(msg)
    }

    return m, nil
}
```

## 🔧 **Key Debugging Techniques**

### **Dimension Debugging**
```go
func (m *Model) debugDimensions() {
    width, height := m.styles.GetSize()
    contentWidth, contentHeight := m.window.GetContentSize()

    logger.LogWithData("Dimensions Debug", map[string]interface{}{
        "terminal_size":    fmt.Sprintf("%dx%d", width, height),
        "content_size":     fmt.Sprintf("%dx%d", contentWidth, contentHeight),
        "viewport_size":    fmt.Sprintf("%dx%d", m.viewport.Width, m.viewport.Height),
        "viewport_offset":  m.viewport.YOffset,
        "viewport_percent": m.viewport.ScrollPercent(),
        "is_vertical":      m.isVerticalLayout(),
    })
}

func (m *Model) View() string {
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        logger.Log("[%s] VIEW: invalid dimensions %dx%d", m.componentName, width, height)
        return "Loading..." // Graceful fallback
    }

    // Debug dimensions on resize
    if m.lastWidth != width || m.lastHeight != height {
        m.debugDimensions()
        m.lastWidth, m.lastHeight = width, height
    }

    return m.viewport.View()
}
```

### **Navigation Stack Debugging**
```go
func (n *Navigator) debugStack() {
    stackInfo := make([]string, len(n.stack))
    for i, screenID := range n.stack {
        stackInfo[i] = string(screenID)
    }

    logger.LogWithData("Navigation Stack", map[string]interface{}{
        "stack":   stackInfo,
        "current": string(n.Current()),
        "depth":   len(n.stack),
    })
}

func (n *Navigator) Push(screenID ScreenID) {
    logger.Log("[Navigator] PUSH: %s", string(screenID))
    n.stack = append(n.stack, screenID)
    n.debugStack()
    n.persistState()
}

func (n *Navigator) Pop() ScreenID {
    if len(n.stack) <= 1 {
        logger.Log("[Navigator] POP: cannot pop last screen")
        return n.stack[0]
    }

    popped := n.stack[len(n.stack)-1]
    n.stack = n.stack[:len(n.stack)-1]
    logger.Log("[Navigator] POP: %s -> %s", string(popped), string(n.Current()))
    n.debugStack()
    n.persistState()
    return popped
}
```

### **Form State Debugging**
```go
func (m *FormModel) debugFormState() {
    fields := make([]map[string]interface{}, len(m.fields))
    for i, field := range m.fields {
        fields[i] = map[string]interface{}{
            "key":         field.Key,
            "value":       field.Input.Value(),
            "placeholder": field.Input.Placeholder,
            "focused":     i == m.focusedIndex,
            "width":       field.Input.Width,
        }
    }

    logger.LogWithData("Form State", map[string]interface{}{
        "focused_index": m.focusedIndex,
        "has_changes":   m.hasChanges,
        "show_values":   m.showValues,
        "field_count":   len(m.fields),
        "fields":        fields,
    })
}

func (m *FormModel) validateField(index int) {
    logger.Log("[FormModel] VALIDATE: field %d (%s)", index, m.fields[index].Key)

    // ... validation logic ...

    if hasError {
        logger.Log("[FormModel] VALIDATE: field %s failed - %s",
            m.fields[index].Key, errorMsg)
    }
}
```

### **Content Loading Debugging**
```go
func (m *Model) loadContent() tea.Cmd {
    return func() tea.Msg {
        logger.Log("[%s] LOAD: starting content load", m.componentName)

        // Try multiple sources with detailed logging
        sources := []func() (string, error){
            m.loadFromEmbedded,
            m.loadFromFile,
            m.loadFromFallback,
        }

        for i, loadFunc := range sources {
            logger.Log("[%s] LOAD: trying source %d", m.componentName, i+1)

            content, err := loadFunc()
            if err != nil {
                logger.Errorf("[%s] LOAD: source %d failed: %v", m.componentName, i+1, err)
                continue
            }

            logger.Log("[%s] LOAD: source %d success (%d chars)",
                m.componentName, i+1, len(content))
            return ContentLoadedMsg{content}
        }

        logger.Errorf("[%s] LOAD: all sources failed", m.componentName)
        return ErrorMsg{fmt.Errorf("failed to load content")}
    }
}
```

## 🔧 **Common Pitfalls & Solutions**

### **1. Glamour Renderer Freezing**
**Problem**: Creating new renderer instances can freeze
**Solution**: Single shared renderer in styles.New()

```go
// ❌ WRONG: New renderer each time
func (m *Model) renderMarkdown(content string) string {
    renderer, _ := glamour.NewTermRenderer(...)  // Can freeze!
    return renderer.Render(content)
}

// ✅ CORRECT: Shared renderer instance
func (m *Model) renderMarkdown(content string) string {
    rendered, err := m.styles.GetRenderer().Render(content)
    if err != nil {
        logger.Errorf("[%s] RENDER: glamour error: %v", m.componentName, err)
        // Fallback to plain text
        return fmt.Sprintf("# Content\n\n%s\n\n*Render error: %v*", content, err)
    }
    return rendered
}

// Debug renderer creation
func NewStyles() *Styles {
    logger.Log("[Styles] Creating glamour renderer")
    renderer, err := glamour.NewTermRenderer(
        glamour.WithAutoStyle(),
        glamour.WithWordWrap(80),
    )
    if err != nil {
        logger.Errorf("[Styles] Failed to create renderer: %v", err)
        panic(err)
    }
    logger.Log("[Styles] Glamour renderer created successfully")
    return &Styles{renderer: renderer}
}
```

### **2. Footer Height Inconsistency**
**Problem**: Border-based footers vary in height
**Solution**: Background approach with padding

```go
// ❌ WRONG: Border approach (height varies)
func createFooterWrong(width int, text string) string {
    logger.Log("[Footer] Using border approach - height may vary")
    return lipgloss.NewStyle().
        Height(1).
        Border(lipgloss.Border{Top: true}).
        Render(text)
}

// ✅ CORRECT: Background approach (exactly 1 line)
func createFooter(width int, text string) string {
    logger.Log("[Footer] Using background approach - consistent height")
    return lipgloss.NewStyle().
        Background(lipgloss.Color("240")).
        Foreground(lipgloss.Color("255")).
        Padding(0, 1, 0, 1).
        Render(text)
}

// Debug footer height
func (a *App) View() string {
    header := a.renderHeader()
    footer := a.renderFooter()
    content := a.currentModel.View()

    headerHeight := lipgloss.Height(header)
    footerHeight := lipgloss.Height(footer)

    logger.LogWithData("Layout Heights", map[string]interface{}{
        "header_height": headerHeight,
        "footer_height": footerHeight,
        "total_height":  a.styles.GetHeight(),
    })

    contentHeight := max(a.styles.GetHeight() - headerHeight - footerHeight, 0)
    contentArea := a.styles.Content.Height(contentHeight).Render(content)

    return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```

### **3. Dimension Synchronization Issues**
**Problem**: Models store their own width/height, get out of sync
**Solution**: Centralize dimensions in styles singleton

```go
// ❌ WRONG: Models managing their own dimensions
type ModelWrong struct {
    width, height int  // Will get out of sync!
}

func (m *ModelWrong) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.WindowSizeMsg:
        m.width, m.height = msg.Width, msg.Height  // Inconsistent!
        logger.Log("[Model] Direct dimension update: %dx%d", m.width, m.height)
    }
}

// ✅ CORRECT: Centralized dimension management
type Model struct {
    styles *styles.Styles  // Access via styles.GetSize()
}

func (m *Model) updateViewport() {
    width, height := m.styles.GetSize()
    logger.Log("[%s] Using centralized dimensions: %dx%d", m.componentName, width, height)

    if width <= 0 || height <= 0 {
        logger.Log("[%s] Invalid dimensions, skipping update", m.componentName)
        return
    }

    // ... safe viewport update
}
```

### **4. TUI Rendering Corruption**
**Problem**: Console output breaks rendering
**Solution**: File-based logger, never fmt.Printf

```go
// ❌ NEVER: Use tea.ClearScreen during navigation
func (a *App) handleNavigation() (tea.Model, tea.Cmd) {
    logger.Log("[App] Navigation: using ClearScreen")
    return a, tea.Batch(cmd, tea.ClearScreen)  // Corrupts rendering!
}

// ✅ CORRECT: Let model Init() handle clean state
func (a *App) handleNavigation() (tea.Model, tea.Cmd) {
    logger.Log("[App] Navigation: clean model initialization")
    return a, a.currentModel.Init()
}

// Debug rendering corruption
func (m *Model) View() string {
    view := m.viewport.View()

    // Debug view corruption
    if strings.Contains(view, "\x1b[2J") || strings.Contains(view, "\x1b[H") {
        logger.Errorf("[%s] VIEW: detected ANSI clear sequences", m.componentName)
    }

    logger.Log("[%s] VIEW: rendered %d chars", m.componentName, len(view))
    return view
}
```

### **5. Navigation State Issues**
**Problem**: Models retain state between visits
**Solution**: Complete state reset in Init()

```go
// ❌ WRONG: Partial state reset
func (m *Model) Init() tea.Cmd {
    logger.Log("[%s] INIT: partial reset", m.componentName)
    m.content = ""  // Only resetting some fields!
    return m.loadContent
}

// ✅ CORRECT: Complete state reset
func (m *Model) Init() tea.Cmd {
    logger.Log("[%s] INIT: complete state reset", m.componentName)

    // Reset ALL state fields
    m.content = ""
    m.ready = false
    m.error = nil
    m.initialized = false
    m.scrolled = false
    m.focusedIndex = 0
    m.hasChanges = false

    // Reset component state
    m.viewport.GotoTop()
    m.viewport.SetContent("")

    // Reset form state if applicable
    for i := range m.fields {
        m.fields[i].Input.Blur()
    }

    logger.Log("[%s] INIT: state reset complete", m.componentName)
    return m.loadContent
}
```

## 🔧 **Performance Debugging**

### **Viewport Performance**
```go
func (m *Model) debugViewportPerformance() {
    start := time.Now()

    // Measure viewport operations
    m.viewport.SetContent(m.content)
    setContentDuration := time.Since(start)

    start = time.Now()
    view := m.viewport.View()
    viewDuration := time.Since(start)

    logger.LogWithData("Viewport Performance", map[string]interface{}{
        "content_size":       len(m.content),
        "rendered_size":      len(view),
        "set_content_ms":     setContentDuration.Milliseconds(),
        "view_render_ms":     viewDuration.Milliseconds(),
        "viewport_height":    m.viewport.Height,
        "total_lines":        strings.Count(m.content, "\n"),
        "scroll_percent":     m.viewport.ScrollPercent(),
    })
}
```

### **Memory Usage Tracking**
```go
import "runtime"

func (m *Model) debugMemoryUsage(operation string) {
    var memStats runtime.MemStats
    runtime.ReadMemStats(&memStats)

    logger.LogWithData("Memory Usage", map[string]interface{}{
        "operation":      operation,
        "alloc_mb":       memStats.Alloc / 1024 / 1024,
        "total_alloc_mb": memStats.TotalAlloc / 1024 / 1024,
        "sys_mb":         memStats.Sys / 1024 / 1024,
        "num_gc":         memStats.NumGC,
    })
}

// Usage in critical operations
func (m *Model) updateFormContent() {
    m.debugMemoryUsage("form_update_start")

    // ... form update logic ...

    m.debugMemoryUsage("form_update_end")
}
```

## 🔧 **Error Recovery Patterns**

### **Graceful Degradation**
```go
func (m *Model) View() string {
    defer func() {
        if r := recover(); r != nil {
            logger.Errorf("[%s] VIEW: panic recovered: %v", m.componentName, r)
        }
    }()

    // Multi-level fallbacks
    width, height := m.styles.GetSize()
    if width <= 0 || height <= 0 {
        logger.Log("[%s] VIEW: invalid dimensions, using fallback", m.componentName)
        return "Loading..."
    }

    if m.error != nil {
        logger.Log("[%s] VIEW: error state, showing error message", m.componentName)
        return m.styles.Error.Render("Error: " + m.error.Error())
    }

    if !m.ready {
        logger.Log("[%s] VIEW: not ready, showing loading", m.componentName)
        return m.styles.Info.Render("Loading content...")
    }

    return m.viewport.View()
}
```

### **State Recovery**
```go
func (m *Model) recoverFromError(err error) tea.Cmd {
    logger.Errorf("[%s] ERROR: %v", m.componentName, err)

    // Try to recover state
    m.error = err
    m.ready = true

    // Attempt graceful recovery
    return func() tea.Msg {
        logger.Log("[%s] RECOVERY: attempting state recovery", m.componentName)

        // Try to reload content
        if content, loadErr := m.loadFallbackContent(); loadErr == nil {
            logger.Log("[%s] RECOVERY: fallback content loaded", m.componentName)
            return ContentLoadedMsg{content}
        }

        logger.Log("[%s] RECOVERY: using minimal content", m.componentName)
        return ContentLoadedMsg{"# Error\n\nContent temporarily unavailable."}
    }
}
```

## 🔧 **Testing Strategies**

### **Manual Testing Checklist**
```go
// Test dimensions
// 1. Resize terminal to various sizes
// 2. Test minimum dimensions (80x24)
// 3. Test very narrow terminals (< 80 cols)
// 4. Test very short terminals (< 24 rows)

func (m *Model) testDimensions() {
    testSizes := []struct{ width, height int }{
        {80, 24},   // Standard
        {40, 12},   // Small
        {120, 40},  // Large
        {20, 10},   // Tiny
    }

    for _, size := range testSizes {
        m.styles.SetSize(size.width, size.height)
        view := m.View()

        logger.LogWithData("Dimension Test", map[string]interface{}{
            "test_size":   fmt.Sprintf("%dx%d", size.width, size.height),
            "view_length": len(view),
            "has_ansi":    strings.Contains(view, "\x1b["),
            "line_count":  strings.Count(view, "\n"),
        })
    }
}
```

### **Navigation Testing**
```go
func testNavigationFlow() {
    // Test complete navigation flow
    testSteps := []struct {
        action   string
        expected string
    }{
        {"start", "welcome"},
        {"continue", "main_menu"},
        {"select_providers", "llm_providers"},
        {"select_openai", "llm_provider_form§openai"},
        {"go_back", "llm_providers"},
        {"esc", "welcome"},
    }

    for _, step := range testSteps {
        logger.LogWithData("Navigation Test", map[string]interface{}{
            "action":   step.action,
            "expected": step.expected,
            "actual":   string(navigator.Current()),
        })
    }
}
```

This debugging guide provides comprehensive tools for:
- **Safe Development**: TUI-compatible logging without rendering corruption
- **State Inspection**: Real-time monitoring of component state
- **Performance Analysis**: Memory and viewport performance tracking
- **Error Recovery**: Graceful degradation and state recovery patterns
- **Testing Strategies**: Systematic approaches to manual testing

---

### Backend Docs Installer Charm Form Patterns (`backend/docs/installer/charm-form-patterns.md`)

# Charm.sh Advanced Form Patterns

> Comprehensive guide to building sophisticated forms using Charm ecosystem libraries.

## 🎯 **Advanced Form Field Patterns**

### **Boolean Fields with Tab Completion**
**Innovation**: Auto-completion for boolean values with suggestions

```go
import "github.com/charmbracelet/bubbles/textinput"

func createBooleanField() textinput.Model {
    input := textinput.New()
    input.Prompt = ""
    input.ShowSuggestions = true
    input.SetSuggestions([]string{"true", "false"})  // Enable tab completion

    // Show default value in placeholder
    input.Placeholder = "true (default)"  // Or "false (default)"

    return input
}

// Tab completion handler in Update()
func (m *FormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.KeyMsg:
        switch msg.String() {
        case "tab":
            // Complete boolean suggestion
            if m.focusedField.Input.ShowSuggestions {
                suggestion := m.focusedField.Input.CurrentSuggestion()
                if suggestion != "" {
                    m.focusedField.Input.SetValue(suggestion)
                    m.focusedField.Input.CursorEnd()
                    return m, nil
                }
            }
        }
    }
    return m, nil
}
```

### **Integer Fields with Range Validation**
**Innovation**: Real-time validation with human-readable formatting

```
/* Detailed source-code truncated for AI context efficiency. */
```

### **Value Formatting Utilities**
**Critical**: Consistent formatting across all forms

```go
// Universal byte formatting for configuration values
func formatBytes(bytes int) string {
    if bytes >= 1048576 {
        return fmt.Sprintf("%.1fMB", float64(bytes)/1048576)
    } else if bytes >= 1024 {
        return fmt.Sprintf("%.1fKB", float64(bytes)/1024)
    }
    return fmt.Sprintf("%d bytes", bytes)
}

// Universal number formatting for display
func formatNumber(num int) string {
    if num >= 1000000 {
        return fmt.Sprintf("%.1fM", float64(num)/1000000)
    } else if num >= 1000 {
        return fmt.Sprintf("%.1fK", float64(num)/1000)
    }
    return strconv.Itoa(num)
}

// Usage in forms and info panels
sections = append(sections, fmt.Sprintf("• Memory Limit: %s", formatBytes(memoryLimit)))
sections = append(sections, fmt.Sprintf("• Estimated tokens: ~%s", formatNumber(tokenCount)))
```

## 🎯 **Advanced Form Scrolling with Viewport**

### Auto-Scrolling Forms Pattern
**Problem**: Forms with many fields don't fit on smaller terminals, focused fields go off-screen
**Solution**: Viewport component with automatic scroll-to-focus behavior

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Key Benefits of Viewport Forms
- **Automatic Clipping**: Viewport handles content that exceeds available space
- **Smooth Scrolling**: Fields slide into view without jarring jumps
- **Focus Preservation**: Focused field always remains visible
- **No Extra Hotkeys**: Uses standard navigation (Tab, arrows)
- **Terminal Friendly**: Works on any terminal size

### Critical Implementation Details
1. **Field Height Tracking**: Must calculate actual rendered height of each field
2. **Scroll Timing**: Call `ensureFocusVisible()` after every focus change
3. **Content Updates**: Re-render form content when input values change
4. **Viewport Sizing**: Account for padding, headers, footers in size calculation

## 🎯 **Environment Variable Integration Pattern**

**Innovation**: Direct EnvVar integration with presence detection

```go
// EnvVar wrapper (from loader package)
type EnvVar struct {
    Key     string
    Value   string  // Current value in environment
    Default string  // Default value from config
}

func (e EnvVar) IsPresent() bool {
    return e.Value != "" // Check if actually set in environment
}

// Form field creation from EnvVar
func (m *FormModel) addFieldFromEnvVar(envVarName, fieldKey, title, description string) {
    envVar, _ := m.controller.GetVar(envVarName)

    // Track initially set fields for cleanup logic
    m.initiallySetFields[fieldKey] = envVar.IsPresent()

    input := textinput.New()
    input.Prompt = ""

    // Show default in placeholder if not set
    if !envVar.IsPresent() {
        input.Placeholder = fmt.Sprintf("%s (default)", envVar.Default)
    } else {
        input.SetValue(envVar.Value) // Set current value
    }

    field := FormField{
        Key:         fieldKey,
        Title:       title,
        Description: description,
        Input:       input,
        EnvVarName:  envVarName,
    }

    m.fields = append(m.fields, field)
}
```

### **Smart Field Cleanup Pattern**
**Innovation**: Environment variable cleanup for empty values

```go
func (m *FormModel) saveConfiguration() error {
    // First pass: Remove cleared fields from environment
    for _, field := range m.fields {
        value := strings.TrimSpace(field.Input.Value())

        // If field was initially set but now empty, remove it
        if value == "" && m.initiallySetFields[field.Key] {
            if err := m.controller.SetVar(field.EnvVarName, ""); err != nil {
                return fmt.Errorf("failed to clear %s: %w", field.EnvVarName, err)
            }
            logger.Log("[FormModel] SAVE: cleared %s", field.EnvVarName)
        }
    }

    // Second pass: Save only non-empty values
    for _, field := range m.fields {
        value := strings.TrimSpace(field.Input.Value())
        if value == "" {
            continue // Skip empty - use defaults
        }

        // Validate before saving
        if err := m.validateFieldValue(field, value); err != nil {
            return fmt.Errorf("validation failed for %s: %w", field.Key, err)
        }

        // Save validated value
        if err := m.controller.SetVar(field.EnvVarName, value); err != nil {
            return fmt.Errorf("failed to set %s: %w", field.EnvVarName, err)
        }
        logger.Log("[FormModel] SAVE: set %s=%s", field.EnvVarName, value)
    }

    return nil
}
```

## 🎯 **Resource Estimation Pattern**

**Innovation**: Real-time calculation of resource usage

```
/* Detailed source-code truncated for AI context efficiency. */
```

## 🎯 **Current Configuration Preview Pattern**

**Innovation**: Live display of current settings in info panel

```
/* Detailed source-code truncated for AI context efficiency. */
```

## 🎯 **Type-Based Dynamic Forms**

**Innovation**: Conditional field generation based on selection

```go
func (m *FormModel) buildDynamicForm() {
    m.fields = []FormField{} // Reset

    // Common fields for all types
    m.addFieldFromEnvVar("ENABLED", "enabled", "Enable Service", "Enable or disable this service")
    m.addFieldFromEnvVar("MAX_MEMORY", "max_memory", "Memory Limit", "Maximum memory usage in bytes")

    // Type-specific fields
    switch m.configType {
    case "database":
        m.addFieldFromEnvVar("MAX_CONNECTIONS", "max_connections", "Max Connections", "Maximum database connections")
        m.addFieldFromEnvVar("CACHE_SIZE", "cache_size", "Cache Size", "Database cache size in bytes")

    case "worker":
        m.addFieldFromEnvVar("WORKER_COUNT", "worker_count", "Worker Count", "Number of worker processes")
        m.addFieldFromEnvVar("QUEUE_SIZE", "queue_size", "Queue Size", "Maximum queue size")

    case "api":
        m.addFieldFromEnvVar("RATE_LIMIT", "rate_limit", "Rate Limit", "API requests per minute")
        m.addFieldFromEnvVar("TIMEOUT", "timeout", "Request Timeout", "Request timeout in seconds")
    }

    // Set focus on first field
    if len(m.fields) > 0 {
        m.fields[0].Input.Focus()
    }
}

// Environment variable naming helper
func (m *FormModel) getEnvVarName(configType, suffix string) string {
    prefix := strings.ToUpper(configType) + "_"
    return prefix + suffix
}
```

## 🏗️ **Form Architecture Best Practices**

### Viewport Usage Patterns

#### **Forms: Permanent Viewport Property**
```go
// ✅ For forms with user interaction and scroll state
type FormModel struct {
    viewport viewport.Model  // Permanent - preserves scroll position
}

func (m *FormModel) ensureFocusVisible() {
    // Auto-scroll to focused field
    focusY := m.calculateFieldPosition(m.focusedIndex)
    if focusY < m.viewport.YOffset {
        m.viewport.YOffset = focusY
    }
    // ... scroll logic
}
```

#### **Layout: Temporary Viewport Creation**
```go
// ✅ For final layout rendering only
func (m *Model) renderHorizontalLayout(left, right string, width, height int) string {
    content := lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)

    // Create viewport just for layout rendering
    vp := viewport.New(width, height-PaddingHeight)
    vp.SetContent(content)
    return vp.View()
}
```

### Form Field State Management

```go
type FormField struct {
    Key         string
    Title       string
    Description string
    Input       textinput.Model
    Value       string
    Required    bool
    Masked      bool
    Min         int // For integer validation
    Max         int // For integer validation
    EnvVarName  string
}

// Dynamic width application
func (m *FormModel) updateFormContent() {
    inputWidth := m.getInputWidth()

    for i, field := range m.fields {
        // Apply dynamic width to input
        field.Input.Width = inputWidth - 3  // Account for borders
        field.Input.SetValue(field.Input.Value())  // Trigger width update

        // Render with consistent styling
        inputStyle := m.styles.FormInput.Width(inputWidth)
        if i == m.focusedIndex {
            inputStyle = inputStyle.BorderForeground(styles.Primary)
        }

        renderedInput := inputStyle.Render(field.Input.View())
        sections = append(sections, renderedInput)
    }
}
```

These advanced patterns enable:
- **Smart Validation**: Real-time feedback with user-friendly error messages
- **Resource Awareness**: Live estimation of memory, CPU, or token usage
- **Environment Integration**: Proper handling of defaults, presence detection, and cleanup
- **Type Safety**: Compile-time validation and runtime error handling
- **User Experience**: Auto-completion, formatting, and intuitive navigation

---

### Backend Docs Installer Charm Navigation Patterns (`backend/docs/installer/charm-navigation-patterns.md`)

# Charm.sh Navigation Patterns

> Comprehensive guide to implementing robust navigation systems in TUI applications.

## 🎯 **Type-Safe Navigation with Composite ScreenIDs**

### **Composite ScreenID Pattern**
**Problem**: Need to pass parameters to screens (e.g., which provider to configure)
**Solution**: Composite ScreenIDs with `§` separator

```go
// Format: "screen§arg1§arg2§..."
type ScreenID string

// Methods for parsing composite IDs
func (s ScreenID) GetScreen() string {
    parts := strings.Split(string(s), "§")
    return parts[0]
}

func (s ScreenID) GetArgs() []string {
    parts := strings.Split(string(s), "§")
    if len(parts) <= 1 {
        return []string{}
    }
    return parts[1:]
}

// Helper for creating composite IDs
func CreateScreenID(screen string, args ...string) ScreenID {
    if len(args) == 0 {
        return ScreenID(screen)
    }
    parts := append([]string{screen}, args...)
    return ScreenID(strings.Join(parts, "§"))
}
```

### **Usage Examples**
```go
// Simple screen (no arguments)
welcome := WelcomeScreen  // "welcome"

// Composite screen (with arguments)
providerForm := CreateScreenID("llm_provider_form", "openai")  // "llm_provider_form§openai"

// Navigation with arguments
return m, func() tea.Msg {
    return NavigationMsg{
        Target: CreateScreenID("llm_provider_form", "anthropic"),
        Data:   FormData{ProviderID: "anthropic"},
    }
}

// In createModelForScreen - extract arguments
func (a *App) createModelForScreen(screenID ScreenID, data any) tea.Model {
    baseScreen := screenID.GetScreen()
    args := screenID.GetArgs()

    switch ScreenID(baseScreen) {
    case LLMProviderFormScreen:
        providerID := "openai" // default
        if len(args) > 0 {
            providerID = args[0]
        }
        return NewLLMProviderFormModel(providerID, ...)
    }
}
```

### **State Persistence**
```go
// Stack automatically preserves composite IDs
navigator.Push(CreateScreenID("llm_provider_form", "gemini"))

// State contains: ["welcome", "main_menu", "llm_providers", "llm_provider_form§gemini"]
// On restore: user returns to Gemini provider form, not default OpenAI
```

## 🎯 **Navigation Message Pattern**

### **NavigationMsg Structure**
```go
type NavigationMsg struct {
    Target ScreenID  // Can be simple or composite
    GoBack bool      // Return to previous screen
    Data   any       // Optional data to pass
}

// Type-safe constants
type ScreenID string
const (
    WelcomeScreen         ScreenID = "welcome"
    EULAScreen           ScreenID = "eula"
    MainMenuScreen       ScreenID = "main_menu"
    LLMProviderFormScreen ScreenID = "llm_provider_form"
)
```

### **Navigation Commands**
```go
// Simple navigation
return m, func() tea.Msg {
    return NavigationMsg{Target: EULAScreen}
}

// Navigation with parameters
return m, func() tea.Msg {
    return NavigationMsg{Target: CreateScreenID("llm_provider_form", "openai")}
}

// Go back to previous screen
return m, func() tea.Msg {
    return NavigationMsg{GoBack: true}
}

// Navigation with data passing
return m, func() tea.Msg {
    return NavigationMsg{
        Target: CreateScreenID("config_form", "database"),
        Data:   ConfigData{Type: "database", Settings: currentSettings},
    }
}
```

## 🎯 **Navigator Implementation**

### **Navigation Stack Management**
```go
type Navigator struct {
    stack        []ScreenID
    stateManager StateManager
}

func NewNavigator(stateManager StateManager) *Navigator {
    return &Navigator{
        stack:        []ScreenID{WelcomeScreen},
        stateManager: stateManager,
    }
}

func (n *Navigator) Push(screenID ScreenID) {
    n.stack = append(n.stack, screenID)
    n.persistState()
}

func (n *Navigator) Pop() ScreenID {
    if len(n.stack) <= 1 {
        return n.stack[0] // Can't pop last screen
    }

    popped := n.stack[len(n.stack)-1]
    n.stack = n.stack[:len(n.stack)-1]
    n.persistState()
    return popped
}

func (n *Navigator) Current() ScreenID {
    if len(n.stack) == 0 {
        return WelcomeScreen
    }
    return n.stack[len(n.stack)-1]
}

func (n *Navigator) Replace(screenID ScreenID) {
    if len(n.stack) == 0 {
        n.stack = []ScreenID{screenID}
    } else {
        n.stack[len(n.stack)-1] = screenID
    }
    n.persistState()
}

func (n *Navigator) persistState() {
    stringStack := make([]string, len(n.stack))
    for i, screenID := range n.stack {
        stringStack[i] = string(screenID)
    }
    n.stateManager.SetStack(stringStack)
}

func (n *Navigator) RestoreState() {
    stringStack := n.stateManager.GetStack()
    if len(stringStack) == 0 {
        n.stack = []ScreenID{WelcomeScreen}
        return
    }

    n.stack = make([]ScreenID, len(stringStack))
    for i, s := range stringStack {
        n.stack[i] = ScreenID(s)
    }
}
```

## 🎯 **Universal ESC Behavior**

### **Global Navigation Handling**
```go
func (a *App) handleGlobalNavigation(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
    switch msg.String() {
    case "esc":
        // Universal ESC: ALWAYS returns to Welcome screen
        if a.navigator.Current().GetScreen() != string(WelcomeScreen) {
            a.navigator.stack = []ScreenID{WelcomeScreen}
            a.navigator.persistState()
            a.currentModel = a.createModelForScreen(WelcomeScreen, nil)
            return a, a.currentModel.Init()
        }

    case "ctrl+c":
        // Global quit
        return a, tea.Quit
    }
    return a, nil
}

// In main Update loop
func (a *App) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.KeyMsg:
        // Handle global navigation first
        if newModel, cmd := a.handleGlobalNavigation(msg); cmd != nil {
            return newModel, cmd
        }

        // Then pass to current model
        var cmd tea.Cmd
        a.currentModel, cmd = a.currentModel.Update(msg)
        return a, cmd

    case NavigationMsg:
        return a.handleNavigationMsg(msg)
    }

    // Delegate to current model
    var cmd tea.Cmd
    a.currentModel, cmd = a.currentModel.Update(msg)
    return a, cmd
}
```

## 🎯 **Navigation Message Handling**

### **App-Level Navigation**
```
/* Detailed source-code truncated for AI context efficiency. */
```

## 🎯 **Args-Based Model Construction**

### **Model Constructor Pattern**
```go
// Model constructor receives args from composite ScreenID
func NewModel(
    controller *controllers.StateController, styles *styles.Styles,
    window *window.Window, args []string,
) *Model {
    // Initialize with selection from args
    selectedIndex := 0
    if len(args) > 1 && args[1] != "" {
        // Find matching item and set selectedIndex
        for i, item := range items {
            if item.ID == args[1] {
                selectedIndex = i
                break
            }
        }
    }

    return &Model{
        controller:    controller,
        selectedIndex: selectedIndex,
        args:          args,
    }
}

// No separate SetSelected* methods needed
func (m *Model) Init() tea.Cmd {
    logger.Log("[Model] INIT: args=%s", strings.Join(m.args, " § "))

    // Selection already set in constructor from args
    m.loadData()
    return nil
}
```

### **Selection Preservation Pattern**
```go
// Navigation from menu with argument preservation
func (m *MenuModel) handleSelection() (tea.Model, tea.Cmd) {
    selectedItem := m.getSelectedItem()

    // Create composite ScreenID with current selection for stack preservation
    return m, func() tea.Msg {
        return NavigationMsg{
            Target: CreateScreenID(string(targetScreen), selectedItem.ID),
        }
    }
}

// Form navigation back - use GoBack to avoid stack loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    model, cmd := m.saveConfiguration()
    if cmd != nil {
        return model, cmd
    }

    // ✅ CORRECT: Use GoBack to return to previous screen
    return m, func() tea.Msg {
        return NavigationMsg{GoBack: true}
    }
}

// ❌ WRONG: Direct navigation creates stack loops
return m, func() tea.Msg {
    return NavigationMsg{Target: LLMProvidersScreen} // Creates loop!
}
```

## 🎯 **Data Passing Pattern**

### **Structured Data Transfer**
```go
// Define data structures for navigation
type FormData struct {
    ProviderID string
    Settings   map[string]string
}

type ConfigData struct {
    Type     string
    Settings map[string]interface{}
}

// Pass data through navigation
func (m *MenuModel) openConfiguration() (tea.Model, tea.Cmd) {
    return m, func() tea.Msg {
        return NavigationMsg{
            Target: CreateScreenID("config_form", "database"),
            Data: ConfigData{
                Type: "database",
                Settings: m.getCurrentSettings(),
            },
        }
    }
}

// Receive data in target model
func NewConfigFormModel(
    controller *controllers.StateController, styles *styles.Styles,
    window *window.Window, args []string, data any,
) *ConfigFormModel {
    configType := "default"
    if len(args) > 0 {
        configType = args[0]
    }

    var settings map[string]interface{}
    if configData, ok := data.(ConfigData); ok {
        settings = configData.Settings
    }

    return &ConfigFormModel{
        configType: configType,
        settings:   settings,
        // ...
    }
}
```

## 🎯 **Navigation Anti-Patterns & Solutions**

### **❌ Common Mistakes**
```go
// ❌ WRONG: Direct navigation creates loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    m.saveConfiguration()
    return m, func() tea.Msg {
        return NavigationMsg{Target: LLMProvidersScreen}  // Loop!
    }
}

// ❌ WRONG: Separate SetSelected methods
func (m *Model) SetSelectedProvider(providerID string) {
    // Complexity - removed in favor of args-based construction
}

// ❌ WRONG: String-based navigation (typo-prone)
return NavigationMsg{Target: "main_menu"}

// ❌ WRONG: Manual string concatenation for arguments
return NavigationMsg{Target: ScreenID("llm_provider_form/openai")}
```

### **✅ Correct Patterns**
```go
// ✅ CORRECT: GoBack navigation
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    if err := m.saveConfiguration(); err != nil {
        return m, nil  // Stay on form if save fails
    }
    return m, func() tea.Msg {
        return NavigationMsg{GoBack: true}  // Return to previous screen
    }
}

// ✅ CORRECT: Args-based selection
func NewModel(..., args []string) *Model {
    selectedIndex := 0
    if len(args) > 1 && args[1] != "" {
        // Set selection from args during construction
        for i, item := range items {
            if item.ID == args[1] {
                selectedIndex = i
                break
            }
        }
    }
    return &Model{selectedIndex: selectedIndex, args: args}
}

// ✅ CORRECT: Type-safe constants
return NavigationMsg{Target: MainMenuScreen}

// ✅ CORRECT: Composite ScreenID with helper
return NavigationMsg{Target: CreateScreenID("llm_provider_form", "openai")}
```

## 🎯 **Navigation Stack Examples**

### **Typical Navigation Flow**
```go
// Stack progression example:
// 1. Start: ["welcome"]
// 2. Continue: ["welcome", "main_menu"]
// 3. LLM Providers: ["welcome", "main_menu§llm_providers", "llm_providers"]
// 4. OpenAI Form: ["welcome", "main_menu§llm_providers", "llm_providers§openai", "llm_provider_form§openai"]
// 5. GoBack: ["welcome", "main_menu§llm_providers", "llm_providers§openai"]
// 6. ESC: ["welcome"]

func demonstrateNavigation() {
    nav := NewNavigator(stateManager)

    // Initial state
    current := nav.Current() // "welcome"

    // Navigate to main menu
    nav.Push(CreateScreenID("main_menu", "llm_providers"))
    current = nav.Current() // "main_menu§llm_providers"

    // Navigate to providers list
    nav.Push(CreateScreenID("llm_providers", "openai"))
    current = nav.Current() // "llm_providers§openai"

    // Navigate to form
    nav.Push(CreateScreenID("llm_provider_form", "openai"))
    current = nav.Current() // "llm_provider_form§openai"

    // Go back
    nav.Pop()
    current = nav.Current() // "llm_providers§openai"

    // ESC to home (clear stack)
    nav.stack = []ScreenID{WelcomeScreen}
    current = nav.Current() // "welcome"
}
```

### **State Restoration**
```go
// On app restart, navigation stack is restored with all parameters
func (a *App) initializeNavigation() {
    a.navigator.RestoreState()

    // User returns to exact screen with preserved selection
    // e.g., "llm_provider_form§anthropic" restores Anthropic form
    currentScreen := a.navigator.Current()
    a.currentModel = a.createModelForScreen(currentScreen, nil)
}
```

This navigation system provides:
- **Type Safety**: Compile-time validation of screen IDs
- **Parameter Preservation**: Arguments maintained across navigation
- **Stack Management**: Proper back navigation without loops
- **State Persistence**: Complete navigation state restoration
- **Universal Behavior**: Consistent ESC and global navigation

---

### Backend Docs Installer Checker Test Scenarios (`backend/docs/installer/checker-test-scenarios.md`)

# Checker Test Scenarios

This document outlines test scenarios for the installer's system checking functionality, focusing on failure modes and their detection.

The scenarios fall into two groups. The first is what the checker observes about this machine — Docker, disk, memory, local connectivity. The second is what the PentAGI Cloud API answers, and there the interesting part is that a failed call is not one condition: "the server could not be reached", "the daily allowance is spent" and "this license tier has no access" need different things from the user, and each has its own reason code and its own test.

## Test Scenarios

### 1. Docker Not Installed
**Setup**: Remove Docker from the system
**Expected**:
- `DockerErrorType`: "not_installed"
- `DockerApiAccessible`: false
- `DockerInstalled`: false
- UI shows: "Docker Not Installed" with installation instructions

### 2. Docker Daemon Not Running
**Setup**: Install Docker but stop the daemon (e.g., quit Docker Desktop on macOS)
**Expected**:
- `DockerErrorType`: "not_running"
- `DockerApiAccessible`: false
- `DockerInstalled`: true
- UI shows: "Docker Daemon Not Running" with start instructions

### 3. Docker Permission Denied
**Setup**: Run installer as non-docker user on Linux
**Expected**:
- `DockerErrorType`: "permission"
- `DockerApiAccessible`: false
- `DockerInstalled`: true
- UI shows: "Docker Permission Denied" with usermod instructions

### 4. Remote Docker Connection Failed
**Setup**: Set DOCKER_HOST to invalid address
**Expected**:
- `DockerErrorType`: "api_error"
- `DockerApiAccessible`: false
- UI shows: "Docker API Connection Failed" with DOCKER_HOST troubleshooting

### 5. Write Permissions Denied
**Setup**: Run installer in read-only directory
**Expected**:
- `EnvDirWritable`: false
- UI shows: "Write Permissions Required" with chmod instructions

### 6. Network Issues - DNS Failure
**Setup**: Break name resolution itself — firewall port 53, or point the resolver at a dead server. Adding `127.0.0.1 docker.io` to /etc/hosts does **not** trigger this: the name resolves, and what fails is scenario 7.
**Expected**:
- `SysNetworkFailures`: ["• DNS resolution failed for docker.io"]
- `SysNetworkOK`: false
- UI shows specific DNS failure with resolution steps

The probe is a plain host lookup and it never goes through `PROXY_URL`. On a network where only the proxy can resolve names, this line appears while everything the installer actually does works — treat it as a hint, not a verdict.

### 7. Network Issues - HTTPS Blocked
**Setup**: Block outbound HTTPS (port 443), or point docker.io at a closed port
**Expected**:
- `SysNetworkFailures`: ["• Cannot reach external services via HTTPS"]
- UI shows HTTPS failure with proxy configuration info

The probe is a 5-second GET to https://docker.io through `PROXY_URL` when one is set, and **any answer below 500 counts as reachable** — an interception proxy replying 403 to the probe passes it. It says the path is open, not that the registry is usable.

### 8. Network Issues - Docker Registry Blocked
**Setup**: Block docker.io for the Docker daemon
**Expected**:
- `SysNetworkFailures`: ["• Cannot pull Docker images from registry"]
- UI shows registry access failure

The probe pulls `debian:latest` and succeeds if **either** the main daemon or the worker daemon can pull it, so a broken remote worker alone does not raise this line. It is skipped entirely when either client is absent — an unreachable daemon is scenarios 2-4, not this one.

### 9. Behind Corporate Proxy
**Setup**: Network requires a proxy; `PROXY_URL` unset, or set while the daemon has no proxy of its own
**Expected**:
- Multiple network failures
- UI shows proxy configuration instructions for HTTP_PROXY/HTTPS_PROXY

Only the HTTPS probe uses `PROXY_URL`. The lookup resolves locally, and the pull goes through the Docker daemon with the daemon's own proxy configuration — so a correctly configured `PROXY_URL` can still leave two of the three lines standing.

**None of these three probes touches update.pentagi.com.** `SysNetworkOK` is not evidence that the update check will work, and a failed update check is not evidence that the registry is blocked. The update server has its own scenarios below.

### 10. Low Memory
**Setup**: System with < 2GB available RAM
**Expected**:
- `SysMemoryOK`: false
- `SysMemoryAvailable`: < 2.0
- UI shows memory requirements with specific numbers

### 11. Low Disk Space
**Setup**: System with < 25GB free space
**Expected**:
- `SysDiskFreeSpaceOK`: false
- `SysDiskAvailable`: < 25.0
- UI shows disk requirements with cleanup suggestions

### 12. Worker Docker Environment Issues
**Setup**: Configure DOCKER_HOST for remote, but remote unavailable
**Expected**:
- `WorkerEnvApiAccessible`: false
- UI shows worker environment troubleshooting

## Update Server Scenarios

The installer talks to the PentAGI Cloud API over three calls — `POST /api/v1/proxy/updates/check`, `GET /api/v1/proxy/packages/info` and `GET /api/v1/proxy/packages/download`. Every one of them negotiates TLS and solves a proof-of-work challenge before the request is sent, so there is no local server to point the installer at: these scenarios are exercised in unit tests by substituting the call function with the error the transport would have produced (`cmd/installer/cloud/updates_test.go`, `cmd/installer/cloud/packages_test.go`, `cmd/installer/cloud/errors_test.go`), and observed in the field the way a support case arrives.

A call that produces no answer is classified into one of eight reasons. The distinction is the whole point — a spent allowance and a broken proxy call for opposite reactions, and collapsing them into one "update server unavailable" flag tells the user nothing they can act on.

| `reason` | What it means | `retryable` | `retry_after` |
|---|---|---|---|
| `unreachable` | never reached a verdict: no network, no DNS, refused proxy, connection died mid-flight | true | — |
| `timeout` | a deadline: the caller's, or the proof-of-work budget | true | — |
| `rate_limited` | too many requests in a rolling window | true | when the server sent one |
| `quota_exceeded` | the allowance for the period is spent | true | when the server sent one |
| `forbidden` | the license tier has no access to this call, or the client is blocked | false | — |
| `not_found` | no such package or version — an answer, not an outage | false | — |
| `rejected` | the server refused the request as malformed | false | — |
| `server_error` | the server failed on its side | true | — |

**What holds for every failed update check**, and is worth asserting in each scenario below:

- `UpdateServerAccessible`: false, `StackUpdates`: nil, and **all six** per-stack verdicts (`PentagiIsUpToDate`, `WorkerIsUpToDate`, `InstallerIsUpToDate`, `GraphitiIsUpToDate`, `LangfuseIsUpToDate`, `ObservabilityIsUpToDate`) reset to false. Leaving one behind is how a verdict from an earlier successful check gets presented as a current one, with nothing on screen to suggest it is old — the worker flag was the one that used to be missed.
- `update_failure`: `{reason, message, retry_after, retryable}` on the check result, plus a warning line `update check failed (<reason>): <error>` in the log.
- The gather continues. A check that could not be made is a fact about the installation, not a reason to abandon everything else the checker learned, so no other field is affected.
- The update overview screen says the check did not complete and that applying an update will pull whatever the compose files currently point at. It must **never** print "Everything is up to date" — that sentence belongs to an answer that arrived and listed nothing.
- The "Update Installer" entry disappears from Maintenance: it requires the server, and without a package description there is nothing that could be verified. The "Update PentAGI" entry stays **for any stack that is installed**, because none of them is known to be current — it leads to the overview, which says exactly that. On a machine with no compose stack installed the entry is absent whatever the check did: the offer is gated on a stack being installed *and* not up to date, so resetting the verdicts is only half of what makes it appear.
- The reason and the cooldown are what separate these scenarios; the overview's failure text is currently the same sentence for all of them. Assert the reason on the check result and in the log, not a per-reason wording that does not exist.

### 1. Update Server Unreachable
**Setup**: point `UPDATE_SERVER_HOST` at a name that does not resolve or a host that drops 443, or set `PROXY_URL` to a port nothing listens on
**Expected**:
- `update_failure.reason`: "unreachable", `retryable`: true, `retry_after`: 0
- `message` carries the transport error (`dial tcp ...`, proxy refused, TLS failure)
**Must not**:
- claim any stack is up to date, or keep the previous answer on screen
- be read as evidence about the Docker registry — that is scenarios 6-8, a different probe against a different host

A configuration the client refuses to build with never becomes a call at all: `UPDATE_SERVER_HOST=http://...` is an error rather than a silent upgrade to https (a user who wrote `http` meant it), as is a `PROXY_URL` with no host. It is still recorded as a failed check, but as an *unclassified* error — `reason` falls back to "unreachable" and `retryable` stays false, while the message names the setting. Read the message before blaming the network.

### 2. Proof-of-Work Challenge Times Out
**Setup**: in tests, return the SDK's challenge-timeout error from the substituted call. In the field it appears on CPU-limited VMs and throttled laptops — **do not manufacture load to reproduce it**; record the host it happened on instead.
**Expected**:
- `update_failure.reason`: "timeout", `retryable`: true
- The budget is 60 seconds per call, deliberately double the SDK default, because slow hardware genuinely needs the extra seconds
**Must not**:
- be presented as a network failure. The network was fine and the client ran out of budget; the two lead to opposite investigations, which is the entire reason this reason exists.
- be assumed to have been retried. A timed-out challenge is not retried by the transport — if a retry is wanted, the installer has to make it.

### 3. Request Rate Limited
**Setup**: several checks inside one rolling window. Note how easily this happens: each update step refreshes the answer in a deferred call, and updating everything recurses over the stacks — one press of "Update PentAGI" is therefore one check per stack touched, not one check. A limit sized for a single request breaks the client instead of throttling it.
**Expected**:
- `update_failure.reason`: "rate_limited", `retryable`: true
- `retry_after`: the server-advertised cooldown when it sent one, 0 otherwise. It is recorded on the check result and rendered nowhere — the overview prints its single failure sentence and the operation panel prints `failed to check updates: <message>`. Assert the value on the result, not a screen that tells the user when to come back: that screen does not exist yet.
**Must not**:
- retry in a tight loop. The per-minute and general limits are already retried by the transport (up to three attempts, waiting the advertised cooldown capped at 10 seconds); the per-hour and per-day ones are not retried at all, because that wait is too long to hold a screen — so what the installer sees for those is the final answer.
- report anything as up to date.

### 4. Daily Allowance Exhausted
**Setup**: spend the account's daily allowance (in tests: a quota error with the daily scope and a `Retry-After`)
**Expected**:
- `update_failure.reason`: "quota_exceeded", `retryable`: true
- `retry_after`: time until the allowance resets, taken from the server's answer — again recorded, not displayed
- On screen this is indistinguishable from any other failed check: the same overview sentence, the same `failed to check updates: <message>` line. What separates it is the reason and the cooldown on the check result and the warning in the log, so that is what the test asserts
**Must not**:
- be shown as a permission problem — the license is fine, the period's allowance is not
- be retried before the reset; the allowance does not return sooner for being asked
- report anything as up to date

Every check costs a slice of that allowance, which is why the answer is gathered once and shared, and why the package description is held rather than requested twice for the same build.

### 5. Access Forbidden for the License Tier
**Setup**: call an endpoint the tier does not carry, or run as a blocked client
**Expected**:
- `update_failure.reason`: "forbidden", `retryable`: false, `retry_after`: 0
- This includes a quota answer whose scope is "blocked": a tier that never had access is not an allowance that is spent, and there is nothing to wait for
- Nothing on screen names the license: the failure text is the same sentence as for every other reason. The distinction lives in `reason` and in the log line — and it is the one failure where waiting cannot help, because the license, not time, is what would have to change
**Must not**:
- offer "try again later", schedule a retry, or count down a cooldown that does not exist
- report anything as up to date

Related: a license key the client cannot introspect is dropped **before** the call and the request goes anonymously, with `license key is not usable, continuing without it` in the log. The client is still built — the update check is the one thing that works without a license, and refusing to run would leave the user with no way to learn that the key is the problem. The symptom of ignoring that warning is precisely this scenario: a tier nobody expected.

### 6. Package Not Published for This Platform (404)
**Setup**: ask `packages/info` for a component, version, os and arch combination that has no published package — a plugin version the server never published for one of the two plugin architectures, or an installer build that exists for another platform only
**Expected**:
- Reason "not_found", `retryable`: false. This arrives on the package calls, so the check result is untouched — the update check itself may well have succeeded.
- Installer self-update: the screen shows `Could not get the package description: ...` and Enter does nothing. Without the description there is no length, sha256 or signature to check a file against, so there is nothing that could be fetched safely.
- Jaeger plugin: the observability stack update aborts **before any image is pulled** — the plugin step runs first for exactly this reason — and the installation keeps the plugin it had.
**Must not**:
- retry. A 404 is an answer; asking again produces the same one.
- download anything without a description to verify it against.
- leave a staged `.new` file behind. Staging files are removed on failure, and again at the start of the next attempt, because nothing else ever looks for them.

An unfamiliar platform never produces this 404 on the plugin path: a component whose OS is not linux is dropped before anything is fetched, and an architecture with no plugin file defined for it fails locally with `no jaeger plugin file is defined for architecture <a>` before the call is made. A "not_found" here means the server publishes no package for a version and platform the installer legitimately asked about.

### 7. Malformed Request Refused (400)
**Setup**: in a test, hand the client a request the contract does not accept — an unknown update strategy, more than 30 reported artefacts, an image digest carrying its `sha256:` prefix, a file component reported under `images`, a version that is not semver
**Expected** — two distinct paths:
- **Caught locally.** The client validates before marshalling and does not send. Only the update check leaves a mark on the check result: `update check request is malformed: <field>` is recorded as a failed check with the fallback reason "unreachable" and `retryable` false, because nothing classified it. The package calls produce `package info request is malformed: ...` and `package download request is malformed: ...`, which are returned to their caller and surface as an operation error or as `Could not get the package description: ...` — the check result is untouched, exactly as in scenario 6. Either way the message, which names the offending field, is the useful half, and catching it here costs no call and no slice of the allowance.
- **Refused by the server.** Reason "rejected", `retryable`: false.
**Must not**:
- resend the identical request. It cannot start succeeding, and each attempt costs a challenge and a slice of the allowance.
- be presented to the user as a server outage. This is the installer's own bug; the field name in the message is what makes it fixable.

### 8. Download Does Not Match Its Description
**Setup**: serve a package that is truncated, padded, altered while keeping its length, or signed with a key that is not the release key
**Expected** — three checks, all describing the plain file (transport encryption is transparent and changes none of them), all decided only after the last byte has arrived:
- Length ≠ `size` → `package is <n> bytes, expected <m>`. The encrypted response carries no `Content-Length`, so the size from `packages/info` is the only length the client ever learns — this is a real check, not a formality, and it is also what the progress percentage is computed against.
- sha256 ≠ `hash` → `package sha256 is <x>, expected <y>`
- Ed25519 signature over the file's SHA-512 does not verify → `package signature does not verify: ...`
**Must not**:
- treat the destination as usable before the call returns nil. Bytes are written as they arrive, so a failed download leaves a partial or unverified file: the installer build is deleted, and the Jaeger plugin's `.new` is deleted with the target left untouched — which is the entire reason it is staged next to the target rather than written in place.
- keep using the description the failed download was checked against. A build republished under the same version since it was described fails all three checks identically forever; the held description is dropped so the next attempt asks again.
- report success on a matching length and hash alone. The signature is what makes the package ours rather than merely intact.

### 9. Server Failed on Its Side
**Setup**: the server answers an internal error or a bad gateway
**Expected**:
- `update_failure.reason`: "server_error", `retryable`: true
- This is one of the few failures the client retries on its own — up to three attempts, a few seconds apart — so what the installer sees is the state after them, not the first failure
**Must not**: report anything as up to date; suggest the user reconfigure anything — nothing on this machine caused it.

### 10. The Confirming Check After an Update Fails
**Setup**: apply an update, then make the check that follows it fail — any of scenarios 1-5 and 9 at that moment
**Expected**:
- The local comparison still runs and reports per component: `<component>: expected <a>, got <b>`, or `no published digest to compare against` when the answer named none
- The menus keep offering the update, because no stack is known to be current
**Must not**:
- report the stack as current on the strength of the local comparison alone. Only the server knows whether something newer appeared while the update was being applied.
- report a component that differs from its target as "newer than the offered build". That verdict — the registry legitimately moving on between the answer and the pull — is only available when a fresh answer says there is nothing left to apply; without one, a difference is a mismatch and is reported as such.

## Environment Variable Tests

### 1. HTTP_PROXY Auto-Detection
**Setup**: Set HTTP_PROXY before running installer
**Expected**: PROXY_URL in .env automatically populated

### 2. DOCKER_HOST Inheritance
**Setup**: Set DOCKER_HOST, DOCKER_TLS_VERIFY, DOCKER_CERT_PATH
**Expected**: 
- Values synchronized to .env on first run via DoSyncNetworkSettings()
- DOCKER_CERT_PATH migrated to PENTAGI_DOCKER_CERT_PATH (host path) + DOCKER_CERT_PATH set to /opt/pentagi/docker/ssl (container path)

## Edge Cases

### 1. Docker Version Too Old
**Setup**: Docker 19.x installed
**Expected**:
- `DockerVersionOK`: false
- UI shows version upgrade instructions

### 2. Docker Compose Missing
**Setup**: Docker installed without Compose
**Expected**:
- `DockerComposeInstalled`: false
- UI shows Compose installation instructions

### 3. Multiple Failures
**Setup**: No Docker + network issues + low resources
**Expected**: All issues shown in priority order:
1. Environment file
2. Write permissions
3. Docker issues
4. Resource issues
5. Network issues

## Testing Commands

```bash
# Simulate Docker not running (macOS)
osascript -e 'quit app "Docker"'

# Simulate permission issues (Linux)
sudo gpasswd -d $USER docker

# Simulate network issues
sudo iptables -A OUTPUT -p tcp --dport 443 -j DROP

# Simulate an unreachable registry — NOT a DNS failure: the name still resolves,
# so this trips the HTTPS probe (scenario 7), not the lookup (scenario 6)
echo "127.0.0.1 docker.io" | sudo tee -a /etc/hosts

# Test with proxy
export HTTP_PROXY=http://proxy:3128
export HTTPS_PROXY=http://proxy:3128

# Test remote Docker
export DOCKER_HOST=tcp://remote:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=/path/to/certs  # auto-migrated to PENTAGI_DOCKER_CERT_PATH on startup

# Update server unreachable: a name that does not resolve
export UPDATE_SERVER_HOST=update.invalid

# Update server unreachable: a proxy nothing listens on
export PROXY_URL=http://127.0.0.1:1

# Refused before a call is ever made — http is an error, not a silent upgrade to https.
# Recorded as a failed check whose message names the setting.
export UPDATE_SERVER_HOST=http://update.pentagi.com
```

`UPDATE_STRATEGY` cannot be used to trigger the malformed-request path: an unrecognised value is validated, logged and replaced with `preview`, because a typo must not leave an installation unable to learn about security updates.

Everything the environment cannot produce — rate limits, exhausted allowances, a forbidden tier, a timed-out challenge, a 404, a tampered download — is driven in unit tests by substituting the call function with the error the transport would have returned:

```bash
go test ./cmd/installer/cloud/...    # classification, envelope, download verification
go test ./cmd/installer/checker/...  # what a failed check does to the check result
```

## Verification

Each scenario should:
1. Be detected correctly by the checker
2. Show appropriate error message in UI
3. Provide actionable fix instructions
4. Not block other checks unnecessarily
5. Work under both privileged and unprivileged users

For the update server scenarios, two more, and they are the ones that matter:

6. Never claim an installation is up to date on the strength of a check that produced no answer — "nothing to update" and "we never found out" are different sentences, and only the first may be printed as reassurance
7. Never retry what cannot succeed. `retryable` is the property to branch on: telling a user to wait out a forbidden endpoint wastes their time, and discarding a cooldown the server did advertise wastes the license. Both are carried on the check result for whoever branches on them — no screen reads either one today, so assert them there.

---

### Backend Docs Installer Checker (`backend/docs/installer/checker.md`)

# Checker Package Documentation

## Overview

The `checker` package is responsible for gathering system facts and verifying installation prerequisites for PentAGI. It performs comprehensive system analysis to determine the current state of the installation and what operations are available.

## Architecture

### Core Design Principles

1. **Delegation Pattern**: Uses a `CheckHandler` interface to delegate information gathering logic, allowing for flexible implementations and testing
2. **Parallel Information Gathering**: Collects information from multiple sources (Docker, filesystem, network) concurrently
3. **Fail-Safe Approach**: Returns sensible defaults when checks cannot be performed, avoiding false negatives
4. **Context-Aware**: All operations support context for cancellation and timeouts

### Key Components

#### CheckResult Structure
Central data structure that holds all system check results:
- Installation status for each component (PentAGI, Graphiti, Langfuse, Observability)
- System resource availability (CPU, memory, disk)
- Docker environment status
- Network connectivity status
- The update check: the answer per stack (`StackUpdates`), why there is no answer (`UpdateFailure`), and what the check observed on this machine (`InstalledComponents`), plus the per-stack `*IsUpToDate` booleans the menus read
- Computed values for UI display:
  - CPU count
  - Required and available memory in GB
  - Required and available disk space in GB
  - Detailed network failure messages
  - Docker error type (not_installed, not_running, permission, api_error)
  - Write permissions for configuration directory

#### CheckHandler Interface
```go
type CheckHandler interface {
    GatherAllInfo(ctx context.Context, c *CheckResult) error
    GatherDockerInfo(ctx context.Context, c *CheckResult) error
    GatherWorkerInfo(ctx context.Context, c *CheckResult) error
    GatherPentagiInfo(ctx context.Context, c *CheckResult) error
    GatherGraphitiInfo(ctx context.Context, c *CheckResult) error
    GatherLangfuseInfo(ctx context.Context, c *CheckResult) error
    GatherObservabilityInfo(ctx context.Context, c *CheckResult) error
    GatherSystemInfo(ctx context.Context, c *CheckResult) error
    GatherUpdatesInfo(ctx context.Context, c *CheckResult) error
}
```

## Check Categories

### 1. Docker Environment Checks
- **Docker API Accessibility**: Verifies connection to Docker daemon
- **Docker Error Detection**: Identifies specific Docker issues (not installed, not running, permission denied)
- **Docker Version**: Ensures Docker version >= 20.0.0
- **Docker Compose Version**: Ensures Docker Compose version >= 1.25.0
- **Worker Environment**: Checks separate Docker environment for pentesting tools (supports remote Docker hosts)

### 2. Component Installation Checks
- **File Existence**: Verifies presence of docker-compose files
- **Container Status**: Checks if containers exist and their running state
- **Script Installation**: Verifies PentAGI CLI script in /usr/local/bin

### 3. System Resource Checks
- **Write Permissions**: Verifies write access to configuration directory
- **CPU**: Minimum 2 CPU cores required
- **Memory**: Dynamic calculation based on components to be installed
  - Base: 0.5GB free
  - PentAGI: +0.5GB
  - Graphiti: +2GB
  - Langfuse: +1.5GB
  - Observability: +1.5GB
- **Disk Space**: Context-aware requirements
  - Worker images not present: 25GB (for large pentesting images)
  - Components to install: 10GB + 2GB per component
  - Already installed: 5GB minimum

### 4. Network Connectivity Checks
Three-tier verification process:
1. **DNS Resolution**: Tests ability to resolve docker.io
2. **HTTP Connectivity**: Verifies HTTPS access (proxy-aware)
3. **Docker Pull Test**: Attempts to pull `debian:latest` — a small probe image, the default of `DOCKER_DEFAULT_IMAGE`, not the pentest worker image — when both Docker clients are available

#### Restricted Network Troubleshooting

The current checker validates Docker Hub reachability by resolving `docker.io`, making an HTTPS connectivity check, and — when both Docker clients are available — attempting a pull of `debian:latest` (the default of `DOCKER_DEFAULT_IMAGE`, used here only as a small reachability probe — the pentest worker image `vxcontrol/kali-linux` is not pulled by this check). This means the installer can fail network validation even when the host has general internet access but Docker itself is not configured for the target network.

Recommended remediation order:

1. Confirm general internet access and DNS resolution for `docker.io`
2. If your environment requires an outbound proxy for installer or PentAGI HTTP traffic, set the `PROXY_URL` environment variable. To route Docker image pulls through a proxy, configure the Docker daemon or Docker Desktop proxy separately — Docker does not use `PROXY_URL` for registry access.
3. If Docker Hub is blocked or rate-limited, configure an organization-approved Docker registry mirror or registry proxy at the Docker daemon / Docker Desktop level
4. Restart Docker and rerun the installer checks

PentAGI variables such as `PENTAGI_IMAGE`, `DOCKER_DEFAULT_IMAGE`, and `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` do not replace Docker daemon registry configuration. They only influence the PentAGI application image or worker image selection after Docker is already able to pull the required images. Note that the main Compose stack already includes a service from `quay.io` (`postgres-exporter`), the optional observability stack includes an image from `gcr.io`, and the optional Langfuse stack includes one from `cgr.dev` (`minio`). A Docker Hub mirror alone is therefore not sufficient for a full deployment — those registries also need to be reachable or individually mirrored.

See Docker's official documentation for [registry mirrors](https://docs.docker.com/docker-hub/image-library/mirror/) and [daemon proxy configuration](https://docs.docker.com/engine/daemon/proxy/).

### 5. Update Availability Checks

`GatherUpdatesInfo` (`checker/updates.go`) asks the PentAGI Cloud API what is available for this installation. It never fails the surrounding gather: a check that cannot be made is a fact *about* the installation, recorded in the result, not a reason to throw away everything else the checker learned.

The call goes to `https://update.pentagi.com` (`UPDATE_SERVER_HOST` overrides the host) through the installer's own `PROXY_URL`, and carries `LICENSE_KEY` when it is present and holds up. A key that does not hold up is deliberately **not** fatal — the check runs without it and the reason is written to the log, because otherwise the only symptom is a tier the user did not expect. `UPDATE_STRATEGY` selects the release channel (`preview` by default, also `stable` and `nightly`); an unrecognised value falls back to `preview` with a warning rather than failing the check, since a typo should not leave an installation unable to learn about security updates. The default is `preview` because `stable` measures the answer against curated releases, and a stack with no published release is then offered nothing at all — which reads as "no updates exist" when it means "no releases are published for this yet".

#### Facts are gathered per component, not per stack

`checker/components.go` turns the running installation into the component list the request carries. Two lists are sent, and they are separate all the way through: an **image** is identified by a registry reference and a digest, a **file** by a version and a hash, and no artefact is ever both.

The list has to be complete. The answer only covers what was asked about, so a component left out comes back as "nothing to report" — which reads exactly like "up to date", and that is the more dangerous of the two mistakes.

For every container image the request carries:

- **Component** — the artefact's own name, taken from an explicit component → container table. Nothing here is derived: for four of the sixteen images the container name is neither the compose service name nor the component name — Langfuse's `postgres`, `clickhouse`, `redis` and `minio` services run in containers called `langfuse-postgres`, `langfuse-clickhouse`, `langfuse-redis` and `langfuse-minio`, while the component sent stays the bare `postgres`, `clickhouse`, `redis`, `minio` — and a wrong container name is indistinguishable from an absent one, so a guess would silently drop the component instead of failing. The container name is what the table holds and what `ContainerInspect` is given; the compose `hostname:` field (`otelcol` for the `otel` service) is never read.
- **Status** — `running` when the container is up, `installed` when it exists but is stopped.
- **OS and Arch** — the **image's**, read from `ImageInspect` (`.Os`, `.Architecture`), never the host's. A container image is `linux` even on macOS and Windows, where Docker runs in a VM, and on Apple silicon an `amd64` image may well be running under emulation. Substituting `runtime.GOOS`/`runtime.GOARCH` is not caught by validation — `darwin` is a valid value, it just matches no published artefact — so the installation would silently be told "no updates" forever. An image on a platform the contract has no name for (say `arm/v7`) is dropped with a log line rather than rounded to the nearest value: rounding asks about one image and gets an answer about another.
- **Repository and Tag** — parsed from the reference the container was configured with, with a Docker Hub host and the implicit `library/` removed and the host of any other registry kept, and the tag `docker pull` would apply filled in when the reference omits one. Both halves are required: an update is the newest image *under a reference*, so a component that cannot name its reference cannot be resolved at all.
- **Image hash** — the image's **config digest**, i.e. what `docker inspect` returns as `Id`.

Note the order: `ContainerInspect` first, then `ImageInspect` **by the config digest that inspection returned**, not by the tag. A tag can have been moved onto a different image since the container started, and inspecting by tag would describe something other than what is actually running.

#### The three digests, and which one is sent

Every image has three sha256 identities: the **index digest** (the whole multi-platform publication, and what a pull by tag records in `RepoDigests`), the per-platform **manifest digest**, and the **config digest** (`docker inspect` → `Id`).

The installer sends the config digest, for three reasons: it is always present, it is per-platform — so it confirms the architecture as well as the build — and it needs no round-trip to the registry. The answer may carry all three (`image_hash`, `config_hash`, `index_hash`), and a match against **any one of them** means the installed artefact is what is being offered.

Digests are sent as **bare lowercase hex, 64 characters, without the `sha256:` prefix** (`normalizeDigest`). The daemon reports the prefixed form, so the conversion is mandatory, and it is not cosmetic: the contract validates against `^[0-9a-f]{64}$`, so the prefixed form and the same 64 characters in uppercase are both rejected — and a rejected request costs the answer for every component in it, not just the offending one. A value that does not normalise is **omitted** rather than sent: a missing digest means "nothing to compare", a malformed one takes the whole request down. Never truncate a digest to the 12 characters the CLI displays.

#### The three components that are not compose containers

- **Worker image.** It has no container and no compose service — it is pulled into the worker Docker environment on demand — so it is reported only when that environment is reachable and the image is actually present. What is reported is what is there, taken from `DOCKER_DEFAULT_IMAGE_FOR_PENTEST` (default `vxcontrol/kali-linux`); substituting the built-in default would claim an image the machine may not have. Status is always `installed`, because the image being present is the whole story. The image is authoritative about its own platform; the host architecture is used only as a fallback for a daemon that will not describe it, and if the result is still not a contract value the component is dropped.
- **Installer binary.** The one component that is a file on *this* host, and therefore the only one whose `os`/`arch` are legitimately the host's own. It is reported with its normalised version and, when the executable path can be read, the sha256 of the running binary. The hash is a nice-to-have — installer builds are identified by version — so failing to read our own path does not drop the component. The request's top-level `installer_version` / `installer_os` / `installer_arch` describe the host for the same reason.
- **Jaeger storage plugin** (`jaeger-clickhouse`). Reported as a **file** component, **once per architecture**, from `observability/jaeger/bin/jaeger-clickhouse-linux-amd64` and `…-arm64` relative to the directory the environment file lives in. Both are reported whatever the host is, because which one runs is decided *inside* the Jaeger container by the architecture of the Docker VM. Each is identified by the **sha256 of the file** and nothing else — the plugin ships without a version — with `os: linux`, because the binary runs in the container. A file that cannot be read is skipped; the two are emitted in a fixed order so two runs on the same machine produce the same request. The directory and file names are exported (`JaegerPluginDir`, `JaegerPluginBinaries`) so that the path the processor downloads into and the path reported here stay one table: if they drift, an update writes a file nothing reads.

#### What is deliberately left out

- A **container that does not exist** — the component is not deployed here, and is not worth a slot in the request.
- Every image of a stack pointing at an **external address**. Such a stack runs somewhere that cannot be inspected, so there is no reference to resolve, and reporting it would claim somebody else's deployment as ours. The Jaeger plugin is skipped for an external observability stack for the same reason.
- Every component of a stack that is **not configured at all**.
- Any component whose platform or image reference cannot be described completely — see above. One dropped component costs one answer; one invalid field costs all of them.

The request may report at most **30 artefacts, counting images and files together**. Today's inventory is 16 stack images + worker + installer + two plugin files = 20, but the list grows with every stack, so the ceiling is enforced during gathering. Components are ordered so the stacks whose updates matter most come first, files are trimmed before images, and whatever is dropped is **named in the log** — a silently truncated list would report the dropped components as complete.

#### Stack statuses

Alongside the components the request says how each product stack is deployed: `installed`, `connected`, `external` or `unused`. Nothing in the answer depends on this — it is reported for its own sake, because a stack hosted elsewhere contributes no components at all and is otherwise indistinguishable from a stack nobody uses. Every stack is reported, including the unused ones: silence cannot be told from a client too old to speak about stacks. `external` takes precedence over `installed`, since it is the more specific statement — it says who *operates* the stack — and an installation pointing at somebody else's Langfuse may well also have the compose file on disk.

#### What the answer contains

`CheckResult.StackUpdates` holds the answer in full; the per-stack `*IsUpToDate` booleans the menus read are the same verdict reduced to one bit. A stack **missing from the answer stays up to date**: the answer covers what was asked about, and silence is not evidence — offering an update nobody has grounds for is the worse failure.

`StackUpdate`:

| Field | Meaning |
|---|---|
| `HasUpdate` | The stack-level verdict, and the only thing the menu needs. |
| `CurrentVersion` / `LatestVersion` | What this installation is attributed to, and what is offered. |
| `CurrentVersionMixed` | The stack's components were attributed to **different** releases, so `CurrentVersion` names the oldest of them rather than a version this installation as a whole ever was. Legitimate — a component nobody rebuilt stays on its old release — but "you are on 2.1.0" and "the oldest thing you have is from 2.1.0" are different sentences, and a screen must not print the first when it means the second. |
| `Releases` | Every curated release this update crosses, **oldest first, the last entry being the target**. Empty for two different reasons: there is genuinely nothing to cross, or the answer predates the field — so an empty list is not by itself evidence of a single-step update. |
| `ReleasesTruncated` | The list was cut, and it is the **oldest** entries that went. |
| `Changelog` / `ReleaseNotes` | The **target** release's text only. When `Releases` is present its last entry carries the same text. |
| `Resolution` | How the answer was arrived at: `release`, `channel`, `ahead_of_release`, `no_artifact_for_tag`, `not_tracked`. **Diagnostic only** — it is logged and nothing more. It exists so that `has_update: false` can be *read* instead of guessed at, most of all in the case that used to be silent: an installation running something newer than anything curated. Branching on it would couple the installer to wording that is free to grow. |
| `Components` | The per-artefact plan, below. |

`ComponentUpdate`. Being listed here does **not** mean the artefact is outdated: under `stable` the answer names every artefact of the release that matches something reported, whether or not it differs, so the installation can be attributed to a version with release notes.

| Field | Meaning |
|---|---|
| `Action` | What to do about this artefact: `current`, `install`, `upgrade`, `downgrade`, `unknown`. Exactly one comes back per reported component, including the ones nothing could be resolved for. |
| `Reason` | Set only for `unknown`: `tag_not_published` (the component is published, but nothing under the reference this installation follows), `repository_not_tracked` (an image nobody publishes — usually a private rebuild), `component_not_tracked` (nothing published for this component on this platform at all), `no_release_artifact_for_tag` (the component exists, but the release the stack is measured against links no artefact of it). This is the difference between "nothing to do" and "we publish nothing for what you are running" — opposite situations that used to arrive identically. |
| `PullReference` | What to write into the compose variable before pulling, ready to use. It is **not** always `repository:tag`: under `stable` it pins the most specific immutable tag so the pull is reproducible, under `preview` and `nightly` it names the moving tag. Write it as given — composing a reference locally undoes that. |
| `Outdated` | Whether this artefact would actually change. |
| `Verifiable` | Whether **this machine** had something to compare — a digest for an image, a version or hash for a file. It describes the installed side only; it says nothing about what the answer carried. |
| `CurrentVersion` | What is installed, in whatever identity the comparison used: the config digest for an image, the version — or, for the plugin, the file hash — for a file. |
| `TargetVersion` | For an image, the offered **config digest**: the same identity the installed side is measured in, and therefore the value post-update verification compares a pulled image against. For a file, the offered version. Empty when it is not known, and verification then degrades honestly to "cannot verify". |

#### How `Outdated` and `Verifiable` are decided

Images are compared **by digest, not by tag** — a moving tag such as `latest` points at different bytes over time, so the tag alone says nothing — and a match against any of the three offered identities counts. Files are compared by version, and by hash when no version was reported, which is the Jaeger plugin's case.

Where the comparison has nothing to work with, **the server's `Action` is the authority**:

- `install`, `upgrade`, `downgrade` set `Outdated` unconditionally. `install` is the case that matters: nothing has been pulled, so there is no digest to compare and the comparison alone leaves `Outdated` false — which reads as "nothing to do" for exactly the component that most needs doing.
- `unknown` clears both `Outdated` and `Verifiable`: there is no artefact at all, so whatever a local comparison produced means nothing.
- An artefact with no installed counterpart is reported **unverifiable, not mismatched** — "this will change" and "we cannot tell" are different statements, and conflating them offers an update on no evidence. The guard is on the **installed** side only, though: `Verifiable` is set whenever this machine had a digest to compare, whatever the answer carried. An offer that names the image only by `image_hash` — the per-platform manifest digest, which no Docker API reports and which therefore can never equal the config digest the installer sends — comes back verifiable *and* outdated, not "cannot tell". Where that case does degrade honestly is one layer later: `TargetVersion` stays empty, so the post-update verification reports "cannot verify" instead of a mismatch.

`CheckResult.InstalledComponents` keeps what this check observed on the machine, recorded **before** the call is made, since it is a local fact whether or not the server answers. Post-update verification reads it instead of interrogating Docker for a second time.

#### When there is no answer

A failed check sets `UpdateServerAccessible = false`, clears `StackUpdates`, and resets **all six** per-stack flags. Resetting all of them matters: a flag left behind keeps whatever verdict an earlier successful check put there, and the interface then presents a stale answer as a current one — the worker flag used to be the one left behind.

`UpdateFailure` says why, in terms the interface can act on, because a spent quota and a broken proxy call for opposite reactions. `Retryable` separates "come back later" from "this needs a different license", and `RetryAfter` carries the advertised cooldown when there is one.

| Reason | What it means | Retryable |
|---|---|---|
| `unreachable` | Nothing reached a verdict: no network, no DNS, a proxy that refused, a connection that died mid-flight. | yes |
| `timeout` | A deadline — the caller's, or the proof-of-work challenge exceeding its budget on slow hardware. | yes |
| `rate_limited` | Too many requests in a rolling window; `RetryAfter` says when it reopens. | yes |
| `quota_exceeded` | The allowance for the period is spent; `RetryAfter` says when it resets. | yes |
| `server_error` | The server failed on its side. | yes |
| `forbidden` | This client is not allowed to make the call at all — a license tier without it, or a blocked client. Waiting does not help; a different license does. | no |
| `not_found` | No such package or version. That is an answer, not an outage. | no |
| `rejected` | The request was refused as malformed, including replay and signature refusals. That is our bug, not the user's, and resending an identical request cannot succeed. | no |

## Public API

### Main Entry Points
```go
// Gather performs all system checks using provided application state
func Gather(ctx context.Context, appState state.State) (CheckResult, error)

// GatherWithHandler allows custom CheckHandler implementation
func GatherWithHandler(ctx context.Context, handler CheckHandler) (CheckResult, error)
```

### Availability Helper Methods
The CheckResult provides helper methods to determine available operations:
```go
func (c *CheckResult) IsReadyToContinue() bool      // Pre-installation checks passed
func (c *CheckResult) CanInstallAll() bool          // Can perform installation
func (c *CheckResult) CanStartAll() bool            // Can start services
func (c *CheckResult) CanStopAll() bool             // Can stop services
func (c *CheckResult) CanUpdateAll() bool           // Updates available
func (c *CheckResult) CanFactoryReset() bool        // Can reset installation
```

## Implementation Details

### OS-Specific Implementations
- **Memory Checks**:
  - Linux: Reads /proc/meminfo for MemAvailable
  - macOS: Parses vm_stat output for free + inactive + purgeable pages
- **Disk Space Checks**:
  - Uses `df` command with appropriate flags per OS

### Docker Integration
- Supports both local and remote Docker environments
- Handles TLS configuration for secure remote connections
- Compatible with Docker contexts and environment variables

### Error Handling Philosophy
- Network failures are treated as "assume OK" to avoid blocking on transient issues
- Missing system information defaults to "sufficient resources"
- Only critical failures (missing env file, Docker API inaccessible) prevent continuation

### Version Parsing
- Flexible regex-based extraction from various version output formats
- Semantic version comparison for compatibility checks
- Handles both docker-compose and docker compose command variants

### Image Information Extraction
- Parses complex Docker image references (registry/namespace/name:tag@hash)
- Handles various edge cases in image naming conventions
- Extracts version information for update comparison

### Helper Functions for Code Reusability
To avoid code duplication, the package provides several shared helper functions:

- **calculateRequiredMemoryGB**: Calculates total memory requirements based on components that need to be started
- **calculateRequiredDiskGB**: Computes disk space requirements considering worker images and local components
- **countLocalComponentsToInstall**: Counts how many components need local installation
- **determineComponentNeeds**: Determines which components need to be started based on their current state
- **getAvailableMemoryGB**: Platform-specific memory availability detection
- **getAvailableDiskGB**: Platform-specific disk space availability detection
- **getNetworkFailures**: Collects detailed network connectivity failure messages
- **getProxyURL**: Centralized proxy URL retrieval from application state
- **getDockerErrorType**: Identifies specific Docker error types (not installed, not running, permission issues)
- **checkDirIsWritable**: Tests write permissions by creating a temporary file

These functions ensure consistent calculations across different parts of the codebase and make maintenance easier.

## Constants and Thresholds

Key configuration values are defined as constants for easy adjustment:
- Container names for each service
- Minimum resource requirements
- Default endpoints for services
- Update server configuration
- Version compatibility thresholds

## Thread Safety

The default implementation uses mutex protection for Docker client management, ensuring safe concurrent access during information gathering operations.

---

### Backend Docs Installer Installer Architecture Design (`backend/docs/installer/installer-architecture-design.md`)

# PentAGI Installer Architecture & Design Patterns

> Architecture patterns, design decisions, and implementation strategies specific to the PentAGI installer.

## 🏗️ **Unified App Architecture**

### **Central Orchestrator Pattern**
The installer implements a centralized app controller that manages all global concerns:

```go
// File: wizard/app.go
type App struct {
    // Navigation state
    navigator *Navigator
    currentModel tea.Model

    // Shared resources (injected into all models)
    controller *controllers.StateController
    styles     *styles.Styles
    window     *window.Window

    // Global state
    eulaAccepted bool
    systemReady  bool
}

func (a *App) View() string {
    header := a.renderHeader()    // Screen-specific header
    footer := a.renderFooter()    // Dynamic footer with actions
    content := a.currentModel.View()  // Content only from model

    // App.go enforces layout constraints
    contentWidth, contentHeight := a.window.GetContentSize()
    contentArea := a.styles.Content.
        Width(contentWidth).
        Height(contentHeight).
        Render(content)

    return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```

### **Responsibilities Separation**
- **App Layer**: Navigation, layout, global state, resource management
- **Model Layer**: Screen-specific logic, user interaction, content rendering
- **Controller Layer**: Business logic, environment variables, configuration
- **Styles Layer**: Presentation, theming, responsive calculations
- **Window Layer**: Terminal size management, dimension coordination
- **Cloud Layer**: The only route out to the PentAGI Cloud API — request construction, envelope handling, error classification

## 🏗️ **Navigation Architecture**

### **Composite ScreenID System**
**Innovation**: Parameters embedded in screen identifiers for type-safe navigation

```go
// Screen ID structure: "screen§arg1§arg2§..."
type ScreenID string

// Helper methods for parsing composite IDs
func (s ScreenID) GetScreen() string {
    parts := strings.Split(string(s), "§")
    return parts[0]
}

func (s ScreenID) GetArgs() []string {
    parts := strings.Split(string(s), "§")
    if len(parts) <= 1 {
        return []string{}
    }
    return parts[1:]
}

// Type-safe creation
func CreateScreenID(screen string, args ...string) ScreenID {
    if len(args) == 0 {
        return ScreenID(screen)
    }
    return ScreenID(screen + "§" + strings.Join(args, "§"))
}
```

### **Navigator Implementation**
```go
type Navigator struct {
    stack        []ScreenID
    stateManager StateManager // Persists stack across sessions
}

func (n *Navigator) Push(screenID ScreenID) {
    n.stack = append(n.stack, screenID)
    n.persistState()
}

func (n *Navigator) Pop() ScreenID {
    if len(n.stack) <= 1 {
        return n.stack[0] // Can't pop welcome screen
    }
    popped := n.stack[len(n.stack)-1]
    n.stack = n.stack[:len(n.stack)-1]
    n.persistState()
    return popped
}

// Universal ESC behavior
func (a *App) handleGlobalNavigation(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
    switch msg.String() {
    case "esc":
        if a.navigator.Current().GetScreen() != string(WelcomeScreen) {
            a.navigator.stack = []ScreenID{WelcomeScreen}
            a.navigator.persistState()
            a.currentModel = a.createModelForScreen(WelcomeScreen, nil)
            return a, a.currentModel.Init()
        }
    }
    return a, nil
}
```

### **Args-Based Model Construction**
```go
func (a *App) createModelForScreen(screenID ScreenID, data any) tea.Model {
    baseScreen := screenID.GetScreen()
    args := screenID.GetArgs()

    switch ScreenID(baseScreen) {
    case LLMProviderFormScreen:
        providerID := "openai" // default
        if len(args) > 0 {
            providerID = args[0]
        }
        return NewLLMProviderFormModel(a.controller, a.styles, a.window, []string{providerID})

    case SummarizerFormScreen:
        summarizerType := "general" // default
        if len(args) > 0 {
            summarizerType = args[0]
        }
        return NewSummarizerFormModel(a.controller, a.styles, a.window, []string{summarizerType})
    }
}
```

## 🏗️ **Screen Registration Architecture**

### **Where the Update Screens Sit**
Two screens were inserted into the maintenance branch of the navigation graph, and both took over a destination that used to belong to the generic processor operation form:

```
MaintenanceScreen (list)
  ├── UpdateOverviewScreen ──enter──> UpdatePentagiScreen   ("processor_operation_form§compose§update")
  └── InstallerUpdateScreen                                  (replaced "processor_operation_form§installer§update")
```

- **`UpdateOverviewScreen`** stands *between* the menu entry and the operation form.
The entry used to lead straight into "are you sure?", asking for consent to something the interface had never described.
The overview describes it, then continues to the same form — the confirmation there is untouched.
The screen makes no network call: everything it renders comes from the last check result held by the controller, so it cannot fail and has no error state.
`Enter` is deliberately *not* bound to page-down as it is on the EULA screen: this screen asks for no consent, so its one unambiguous key belongs to its one action.

- **`InstallerUpdateScreen`** is a screen rather than an operation form because what it does is describe a build and download one verified file, and the generic form's help text claimed the running binary would be replaced and the app would exit — something no version of the installer has ever done.
It names version, platform, size and target path *first*, then offers a single action, and confirms only the one thing worth confirming: a file already occupying the target name.

### **The Four Registration Steps**
A new screen is not a single file. It is four edits, and only the first of them is enforced by the compiler:

| # | Where | What |
|---|---|---|
| 1 | `wizard/models/types.go` | the `ScreenID` constant |
| 2 | `wizard/models/types.go` — `RestoreModel` | a `case` for the concrete model type |
| 3 | `wizard/registry/registry.go` — `initScreens` | construct the screen and store it under its `ScreenID` |
| 4 | `wizard/locale/locale.go` + `app.initHotkeysLocale` | the screen's strings, and a caption for every hotkey it returns |

```go
// Step 2 — the branch without which the screen is inert
func RestoreModel(model tea.Model) BaseScreenModel {
    switch m := model.(type) {
    // ...
    case *UpdateOverviewModel:
        return m
    case *InstallerUpdateModel:
        return m
    default:
        return nil
    }
}
```

### **Why Step 2 Is the Dangerous One**
`RestoreModel` is the *only* path by which a model returned from `Update` gets back into the
app:

```go
func (app *App) forwardMsgToCurrentModel(msg tea.Msg) tea.Cmd {
    model, cmd := app.currentModel.Update(msg)
    if newModel := models.RestoreModel(model); newModel != nil {
        app.currentModel = newModel
    }
    return cmd
}
```

A type the switch does not know yields `nil` and the app keeps the model it already had.

What that costs is worth stating precisely, because the obvious answer is wrong: every screen here has a pointer receiver on `Update` and returns itself, so the mutation has already happened in place and the pointer the app is holding *is* the updated model — nothing freezes.

The branch is load-bearing for the case the signature allows but no screen has used yet: a model whose `Update` returns a *different* model. That one is dropped silently, with no panic, no log line and no error anywhere.

The compile-time assertion each screen carries (`var _ BaseScreenModel = (*InstallerUpdateModel)(nil)`) does **not** catch it either, because `RestoreModel` is a runtime type switch and a missing `case` is not a type error.

`TestTypes_RestoreModel_RestoresTheUpdateScreens` (`models/types_test.go`) therefore enumerates both new screens, and that test is the only guard there is. The contract it encodes is "every screen is listed" rather than "list the ones that need it", because the day a screen starts returning something else is not the day anybody remembers this rule.

Steps 3 and 4 fail just as quietly, which is why they are listed rather than assumed:

- A screen missing from `initScreens` is not a crash — `GetScreen` falls back to `initMockScreen` and the user gets a placeholder form where the feature should be.
- The footer renders a hotkey only if its key is present in `app.hotkeys` (`if localeHotKey, ok := app.hotkeys[hotkey]; ok`). A screen returning a key with no caption simply shows nothing in the footer.

## 🏗️ **Adaptive Layout Strategy**

### **Responsive Design Pattern**
The installer implements a sophisticated responsive design that adapts to terminal capabilities:

```go
// Layout constants define breakpoints
const (
    MinTerminalWidth = 80        // Minimum for horizontal layout
    MinMenuWidth     = 38        // Minimum left panel width
    MaxMenuWidth     = 66        // Maximum left panel width (prevents too wide forms)
    MinInfoWidth     = 34        // Minimum right panel width
    PaddingWidth     = 8         // Total horizontal padding
)

// Layout decision logic
func (m *Model) isVerticalLayout() bool {
    contentWidth := m.window.GetContentWidth()
    return contentWidth < (MinMenuWidth + MinInfoWidth + PaddingWidth)
}

// Dynamic width allocation
func (m *Model) renderHorizontalLayout(leftPanel, rightPanel string, width, height int) string {
    leftWidth, rightWidth := MinMenuWidth, MinInfoWidth
    extraWidth := width - leftWidth - rightWidth - PaddingWidth

    // Distribute extra space intelligently, but cap left panel
    if extraWidth > 0 {
        leftWidth = min(leftWidth+extraWidth/2, MaxMenuWidth)
        rightWidth = width - leftWidth - PaddingWidth/2
    }

    leftStyled := lipgloss.NewStyle().Width(leftWidth).Padding(0, 2, 0, 2).Render(leftPanel)
    rightStyled := lipgloss.NewStyle().Width(rightWidth).PaddingLeft(2).Render(rightPanel)

    return lipgloss.JoinHorizontal(lipgloss.Top, leftStyled, rightStyled)
}
```

### **Content Hiding Strategy**
```go
func (m *Model) renderVerticalLayout(leftPanel, rightPanel string, width, height int) string {
    verticalStyle := lipgloss.NewStyle().Width(width).Padding(0, 4, 0, 2)

    leftStyled := verticalStyle.Render(leftPanel)
    rightStyled := verticalStyle.Render(rightPanel)

    // Show both panels if they fit
    if lipgloss.Height(leftStyled)+lipgloss.Height(rightStyled)+2 < height {
        return lipgloss.JoinVertical(lipgloss.Left,
            leftStyled,
            verticalStyle.Height(1).Render(""),
            rightStyled,
        )
    }

    // Hide right panel if insufficient space - show only essential content
    return leftStyled
}
```

## 🏗️ **Form Architecture Patterns**

### **Production Form Model Structure**
```go
type FormModel struct {
    // Standard dependencies (injected)
    controller *controllers.StateController
    styles     *styles.Styles
    window     *window.Window

    // Form state
    fields       []FormField
    focusedIndex int
    showValues   bool
    hasChanges   bool

    // Environment integration
    configType         string
    typeName          string
    initiallySetFields map[string]bool // Track for cleanup

    // Navigation state
    args []string // From composite ScreenID

    // Viewport as permanent property (preserves scroll state)
    viewport     viewport.Model
    formContent  string
    fieldHeights []int
}
```

### **Dynamic Field Generation Pattern**
```
/* Detailed source-code truncated for AI context efficiency. */
```

### **Environment Variable Integration**
```
/* Detailed source-code truncated for AI context efficiency. */
```

## 🏗️ **Advanced Form Field Patterns**

### **Boolean Field with Auto-completion**
```go
func (m *FormModel) addBooleanField(key, title, description string, envVar loader.EnvVar) {
    input := textinput.New()
    input.Prompt = ""
    input.PlaceholderStyle = m.styles.FormPlaceholder
    input.ShowSuggestions = true
    input.SetSuggestions([]string{"true", "false"})

    // Show default in placeholder
    if envVar.Default == "true" {
        input.Placeholder = "true (default)"
    } else {
        input.Placeholder = "false (default)"
    }

    // Set value only if actually present in environment
    if envVar.Value != "" && envVar.IsPresent() {
        input.SetValue(envVar.Value)
    }

    field := FormField{
        Key:         key,
        Title:       title,
        Description: description,
        Input:       input,
        Type:        "boolean",
    }

    m.fields = append(m.fields, field)
}
```

### **Integer Field with Validation**
```go
func (m *FormModel) addIntegerField(key, title, description string, envVar loader.EnvVar, min, max int) {
    input := textinput.New()
    input.Prompt = ""
    input.PlaceholderStyle = m.styles.FormPlaceholder

    // Parse and format default value
    defaultValue := 0
    if envVar.Default != "" {
        if val, err := strconv.Atoi(envVar.Default); err == nil {
            defaultValue = val
        }
    }

    // Human-readable placeholder with default
    input.Placeholder = fmt.Sprintf("%s (%s default)",
        m.formatNumber(defaultValue), m.formatBytes(defaultValue))

    // Set value only if present
    if envVar.Value != "" && envVar.IsPresent() {
        input.SetValue(envVar.Value)
    }

    // Add validation range to description
    fullDescription := fmt.Sprintf("%s (Range: %s - %s)",
        description, m.formatBytes(min), m.formatBytes(max))

    field := FormField{
        Key:         key,
        Title:       title,
        Description: fullDescription,
        Input:       input,
        Type:        "integer",
        Min:         min,
        Max:         max,
    }

    m.fields = append(m.fields, field)
}
```

## 🏗️ **Controller Integration Pattern**

### **StateController Bridge**
```go
type StateController struct {
    state *state.State
}

func NewStateController(state *state.State) *StateController {
    return &StateController{state: state}
}

// Environment variable management
func (c *StateController) GetVar(name string) (loader.EnvVar, error) {
    return c.state.GetVar(name)
}

func (c *StateController) SetVar(name, value string) error {
    return c.state.SetVar(name, value)
}

// Higher-level configuration management
func (c *StateController) GetLLMProviders() map[string]ProviderConfig {
    // Aggregate multiple environment variables into structured config
    providers := make(map[string]ProviderConfig)

    for _, providerID := range []string{"openai", "anthropic", "gemini", "bedrock", "deepseek", "glm", "kimi", "qwen", "ollama", "custom"} {
        config := c.loadProviderConfig(providerID)
        providers[providerID] = config
    }

    return providers
}

func (c *StateController) loadProviderConfig(providerID string) ProviderConfig {
    prefix := strings.ToUpper(providerID) + "_"

    apiKey, _ := c.GetVar(prefix + "API_KEY")
    baseURL, _ := c.GetVar(prefix + "BASE_URL")

    return ProviderConfig{
        ID:         providerID,
        Configured: apiKey.IsPresent() && baseURL.IsPresent(),
        APIKey:     apiKey.Value,
        BaseURL:    baseURL.Value,
    }
}
```

## 🏗️ **Cloud Client Architecture**

### **Single Egress Pattern**
Everything the installer asks of the PentAGI Cloud API goes through one package: `cmd/installer/cloud`.

Both `checker` and `processor` build a `cloud.Config` from the installer state and call methods on the client. Neither constructs requests, parses responses, or opens connections to `update.pentagi.com` directly.

The `cloud` package is independent from the rest of the installer — it does not depend on `checker`, `processor`, or `wizard`. The dependency arrow points only one way. In tests, the client can be exercised by substituting the call functions, with no actual server involved.

```go
// cloud/client.go — three routes bound in one build
const (
    routeUpdatesCheck     = "updates_check"      // POST /api/v1/proxy/updates/check
    routePackagesInfo     = "packages_info"      // GET  /api/v1/proxy/packages/info
    routePackagesDownload = "packages_download"  // GET  /api/v1/proxy/packages/download
)

type Client struct {
    checkUpdates    sdk.CallReqBytesRespBytes
    packageInfo     sdk.CallReqQueryRespBytes
    downloadPackage sdk.CallReqQueryRespWriter
    // ...
}
```

Route names are part of the contract, not internal labels: the server dispatches on the name, and one it does not recognise is refused outright rather than reported as a missing page.

### **One Proxy Setting Governs Every Call**
The SDK's default transport reads the proxy from the process environment. The installer has its own `PROXY_URL` setting — often carrying credentials the environment does not — and that setting has to win, because it is the one the user actually configured.

```go
func newTransport(proxyURL string) (*http.Transport, error) {
    transport := sdk.DefaultTransport()
    value := strings.TrimSpace(proxyURL)
    if value == "" {
        return transport, nil // nothing configured: leave the environment behaviour alone
    }
    parsed, err := url.Parse(value)
    // ... reject a URL with no host ...
    transport.Proxy = http.ProxyURL(parsed)
    return transport, nil
}
```

The transport is built once in `New` and shared by all three calls, so there is no route that can quietly bypass it. Both construction sites — the checker's update check and the processor's package download — read the same `PROXY_URL`, which is what makes "one setting" true in practice rather than only by intent.

Two neighbouring normalisations belong to the same idea of a single, predictable egress:

- **`NormalizeHost`** reduces the configured address to `host[:port]`. A scheme and trailing path are tolerated because the setting historically held a full URL, but an explicit `http://` is an **error** rather than a silent upgrade: the connection is always encrypted, and quietly doing the opposite of what the user wrote would hide a misconfiguration.

- **`NormalizeVersion`** turns the build version into one the contract accepts, falling back to `0.0.0` for anything that is not a version number — a branch name, a bare commit hash, an empty string. The original is not lost: it travels in the `User-Agent`, where it identifies the exact build without having to satisfy the version grammar.

### **The License Key Is Validated Before It Is Handed Over**
```go
if key := strings.TrimSpace(cfg.LicenseKey); key != "" {
    if _, err := sdk.IntrospectLicenseKey(key); err != nil {
        client.licenseWarning = fmt.Errorf("license key is not usable, continuing without it: %w", err)
    } else {
        options = append(options, sdk.WithLicenseKey(key))
    }
}
```

The SDK drops a key it cannot decode without saying so, and the call then runs anonymously — which from the outside looks exactly like the license not being honoured. Validating first turns that into a stated reason, available from `LicenseWarning()`.

An unusable key is deliberately **not** fatal. The update check is the one thing that still works without a license, so refusing to build the client would leave the user with no way to discover that their key is the problem. `New` fails only on configuration that cannot produce working calls at all: an unusable host or an unusable proxy.

### **Answers Arrive Inside an Envelope**
Every non-streaming answer is wrapped as `{"status": …, "data": …}`, and unwrapping it is not optional:

```go
answer, err := c.checkUpdates(ctx, body)
if err != nil {
    return nil, Classify(err)
}
// Decoding straight into the response type SUCCEEDS against the wrapper — every field
// simply stays at its zero value. Skipping this reports "no updates" for every answer.
response, err := models.ParseEnvelope[models.CheckUpdatesResponse](answer)
```

This is the sharpest edge in the whole client. `json.Unmarshal` of an envelope into the payload type produces no error and no populated fields, so the installer concludes there is nothing to install and never learns it was wrong.

`ParseEnvelope` also turns a `2xx` body whose status is not `success` into an `*APIError` describing what the server reported. Binary downloads are the one exception — they carry the file itself, with no envelope around it.

### **Failure Classification**
`Classify` maps SDK errors onto reasons the interface can act on, wrapping the original so `errors.Is`/`errors.As` keep working through it:

| Reason | What the user can do about it |
|---|---|
| `unreachable` | check the network — nothing ever reached a verdict |
| `timeout` | a deadline, including the proof-of-work budget on slow hardware |
| `rate_limited` / `quota_exceeded` | wait; `RetryAfter` says how long |
| `forbidden` | retrying never helps; the license does |
| `not_found` | an answer, not an outage — no such package or version |
| `rejected` | the request was malformed: our bug, and resending it cannot succeed |
| `server_error` | worth retrying later |

`Retryable()` collapses the table back down for callers that only need the one bit. Ordering inside `Classify` matters: the typed errors are matched before the sentinels they wrap, because that is where the server-advertised cooldown lives. Collapsing all of this into a single "update server unavailable" flag — as the installer used to — tells the user nothing they can act on.

### **Verification Belongs to the Client, Not the Caller**

`PackageInfo` validates the answer before returning it, because everything `DownloadPackage` checks is taken from that answer: an unverified description of a file is not a basis for trusting the file.

The download then feeds one pass over the stream into three checks at once:

```go
digest := sha256.New()
counter := &countingWriter{}
signed := info.Signature.ValidateWrapWriter(io.MultiWriter(dst, digest, counter))
```

The expected length has to come from `info.Size` because the encrypted response carries no `Content-Length` — there is no header to read it from. It catches nothing the sha256 and the signature would miss; what it buys is a legible error ("package is N bytes, expected M") ahead of a bare hash mismatch.

Verification can only complete once the last byte is written, so on any error `dst` holds a partial or unverified file. Discarding it is the caller's job, and the two callers discharge that obligation differently on purpose.

`installPluginFile` streams into `<target>.new` and renames only once the write has returned clean, because the file it replaces is the one a running Jaeger has mounted — a download that dies halfway would otherwise leave a truncated plugin and the container would come back up unable to start.

`fetchInstaller` does the opposite: it writes the binary under its final name in the environment directory and removes it on error. Staging an executable through the system temp directory and moving it into place is a pattern antivirus and EDR products score heavily, and a false positive there costs the user their update.

## 🏗️ **Resource Estimation Architecture**

### **Token Calculation Pattern**
```
/* Detailed source-code truncated for AI context efficiency. */
```

## 🏗️ **Auto-Scrolling Form Architecture**

### **Viewport-Based Scrolling**
```go
func (m *FormModel) ensureFocusVisible() {
    if m.focusedIndex >= len(m.fieldHeights) {
        return
    }

    // Calculate Y position of focused field
    focusY := 0
    for i := 0; i < m.focusedIndex; i++ {
        focusY += m.fieldHeights[i]
    }

    visibleRows := m.viewport.Height
    offset := m.viewport.YOffset

    // Scroll up if field is above visible area
    if focusY < offset {
        m.viewport.YOffset = focusY
    }

    // Scroll down if field is below visible area
    if focusY+m.fieldHeights[m.focusedIndex] >= offset+visibleRows {
        m.viewport.YOffset = focusY + m.fieldHeights[m.focusedIndex] - visibleRows + 1
    }
}

// Enhanced field navigation with auto-scroll
func (m *FormModel) focusNext() {
    if len(m.fields) == 0 {
        return
    }
    m.fields[m.focusedIndex].Input.Blur()
    m.focusedIndex = (m.focusedIndex + 1) % len(m.fields)
    m.fields[m.focusedIndex].Input.Focus()
    m.updateFormContent()
    m.ensureFocusVisible() // Key addition for auto-scroll
}
```

## 🏗️ **Layout Integration Architecture**

### **Content Area Management**
```go
// Models handle ONLY content area
func (m *Model) View() string {
    leftPanel := m.renderForm()
    rightPanel := m.renderHelp()

    // Adaptive layout decision
    if m.isVerticalLayout() {
        return m.renderVerticalLayout(leftPanel, rightPanel, width, height)
    }
    return m.renderHorizontalLayout(leftPanel, rightPanel, width, height)
}

// App.go handles complete layout structure
func (a *App) View() string {
    header := a.renderHeader()    // Screen-specific header (logo or title)
    footer := a.renderFooter()    // Dynamic actions based on screen
    content := a.currentModel.View()  // Content from model

    // Calculate content area size
    contentWidth, contentHeight := a.window.GetContentSize()
    contentArea := a.styles.Content.
        Width(contentWidth).
        Height(contentHeight).
        Render(content)

    return lipgloss.JoinVertical(lipgloss.Left, header, contentArea, footer)
}
```

This architecture provides:
- **Clean Separation**: Each layer has clear responsibilities
- **Type Safety**: Compile-time navigation validation
- **State Persistence**: Complete session restoration
- **Responsive Design**: Adaptive to terminal capabilities
- **Resource Awareness**: Real-time estimation and optimization
- **User Experience**: Professional interaction patterns
- **Single Egress**: One package, one proxy setting, one place where answers are unwrapped and failures are classified

---

### Backend Docs Installer Installer Base Screen (`backend/docs/installer/installer-base-screen.md`)

# BaseScreen Architecture Guide

> Practical guide for implementing new installer screens and migrating existing ones using the BaseScreen architecture.

## 🏗️ **Architecture Overview**

The `BaseScreen` provides a unified foundation for all installer form screens, encapsulating:

- **State Management**: `initialized`, `hasChanges`, `focusedIndex`, `showValues`
- **Form Handling**: `fields []FormField`, viewport management, auto-scrolling
- **Navigation**: Composite ScreenID support, GoBack patterns
- **Layout**: Responsive horizontal/vertical layouts
- **Lists**: Optional dropdown lists with delegates

### **Core Components**

```go
type BaseScreen struct {
    // Dependencies (injected)
    controller *controllers.StateController
    styles     *styles.Styles
    window     *window.Window

    // State
    initialized  bool
    hasChanges   bool
    focusedIndex int
    showValues   bool

    // Form data
    fields       []FormField
    fieldHeights []int

    // UI components
    viewport    viewport.Model
    formContent string

    // Handlers (must be implemented)
    handler     BaseScreenHandler
    listHandler BaseListHandler // optional
}
```

### **Required Interfaces**

```go
type BaseScreenHandler interface {
    BuildForm() tea.Cmd
    GetFormSummary() string
    GetFormTitle() string
    GetHelpContent() string
    HandleSave() error
    HandleReset()
    OnFieldChanged(fieldIndex int, oldValue, newValue string)
    GetFormFields() []FormField
    SetFormFields(fields []FormField)
}

type BaseListHandler interface { // Optional
    GetList() *list.Model
    OnListSelectionChanged(oldSelection, newSelection string)
    GetListHeight() int
}
```

## 🚀 **Creating New Screens**

### **1. Basic Form Screen**

```
/* Detailed source-code truncated for AI context efficiency. */
```

### **2. Screen with List Selection**

```
/* Detailed source-code truncated for AI context efficiency. */
```

> The two examples above predate two changes to the base: `NewBaseScreen` no longer takes an `args []string` parameter, and `BuildForm()` now returns a `tea.Cmd`.
> The two screens below are written against the current shape.

### **3. Document Screen (viewport, no BaseScreen)**

`wizard/models/update_overview.go` is the shape to copy when a screen shows *text* rather than fields. It does not embed `BaseScreen` at all — it implements `BaseScreenModel` directly, the way `eula.go` does.

Everything `BaseScreen` contributes (fields, focus, save/reset, the form/help split) is dead weight for a document, and the one thing a document needs — a full-width scrolling viewport over rendered markdown — `BaseScreen` does not provide.

```go
type UpdateOverviewModel struct {
    styles     styles.Styles
    window     window.Window
    controller controller.Controller

    viewport viewport.Model

    source     string // the markdown
    content    string // that markdown, rendered at renderedAt columns
    renderedAt int
    ready      bool
    scrolled   bool
}

// only the model interface: there is no form, so there is no handler
var _ BaseScreenModel = (*UpdateOverviewModel)(nil)
```

**Keep the markdown and its rendered form as two separate fields.** A resize re-renders *from the source*. Re-wrapping the already-rendered string would wrap the ANSI escape sequences glamour wrote into it, and the screen fills with fragments of colour codes.

**Re-render only when the wrap width actually changed:**

```go
func (m *UpdateOverviewModel) updateViewport() {
    contentWidth, contentHeight := m.window.GetContentSize()
    if contentWidth <= 0 || contentHeight <= 0 || m.source == "" {
        return
    }

    if m.content == "" || m.renderedAt != contentWidth {
        rendered, err := m.styles.GetRendererForWidth(contentWidth).Render(m.source)
        // ... on error, fall back to the raw markdown plus the error
        m.content = rendered
        m.renderedAt = contentWidth
    }

    if !m.ready {
        m.viewport = viewport.New(contentWidth, contentHeight)
        m.ready = true
    } else {
        m.viewport.Width, m.viewport.Height = contentWidth, contentHeight
    }

    m.viewport.SetContent(m.content)
    m.updateScrollStatus()
}
```

Dragging a window border delivers one `tea.WindowSizeMsg` per column and glamour is not cheap, so `renderedAt` is what stops the whole changelog being re-rendered for a resize that cannot change how it wraps — a height-only drag, or one of the resizes `registry.HandleMsg` forwards to *every* registered screen whether or not the user is looking at it. The comparison is against the raw content width, so widths outside the clamp band below still re-render (into identical output); the renderer itself is memoised in either case.

**Create the viewport on the first real size, never in the constructor.**  
`window.GetContentSize()` is 0×0 until the first `WindowSizeMsg` arrives, and a viewport built at that size shows nothing afterwards. `ready` is the flag that says the viewport exists, and each of its readers does something different: `View()` falls back to the loading text, `GetScrollInfo()` returns zeroes rather than measure an unsized viewport, and every scrolling key sits behind an `if !m.ready { break }` guard.  
`enter` is deliberately *in front of* that guard — continuing needs no rendered document, so the one action the screen offers is never blocked by a size message that has not arrived yet.

**Deliver the content as a message even when nothing is fetched.**  
`loadOverview` only reads the last check result out of the controller and cannot fail, but it is still a `tea.Cmd` returning `UpdateOverviewLoadedMsg` — the same shape as the EULA screen, and it keeps `Init()` the single place that decides what a fresh visit shows.

**`GetFormTitle()` returns the empty string.**  
The document carries its own H1, and the header would otherwise print the heading twice.

**`GetScrollInfo() (scrolled, atEnd bool, percent int)`** is the accessor a footer uses to show scroll position; it returns zeroes while `!ready` rather than reading an unsized viewport.  
`app.renderFooter` consults it only on the EULA screen today, which needs "you have reached the end" for consent.  
This screen instead announces what the keys do, and only once there is something to scroll:

```go
func (m *UpdateOverviewModel) GetFormHotKeys() []string {
    hotkeys := []string{"enter"}
    if m.ready && m.content != "" {
        hotkeys = append(hotkeys, "up|down", "pgup|pgdown", "home|end")
    }
    return hotkeys
}
```

**Build the document in a pure function.**
`overviewDocument(updates []checker.StackUpdate, checkSucceeded bool) string` lives
outside the model and `update_overview_test.go` tests that function. There is no
harness for a bubbletea screen in the wizard, and what is worth testing is what the
document *says*, not how it scrolls.

### **4. Action Screen with a Terminal Panel**

`wizard/models/installer_update.go` is the other new shape: no form fields at all, but
the `BaseScreen` layout and help panel are worth keeping. It embeds `BaseScreen`,
implements `BaseScreenHandler` with empty form plumbing, and puts a
`terminal.Terminal` where the form viewport would be.

```go
type InstallerUpdateModel struct {
    *BaseScreen

    processor processor.ProcessorModel
    terminal  terminal.Terminal

    offered  *processor.InstallerPackage // what the server offers
    offerErr error                       // ...or why there is none

    running    bool
    confirming bool
}

var _ BaseScreenModel = (*InstallerUpdateModel)(nil)
var _ BaseScreenHandler = (*InstallerUpdateModel)(nil)
```

**`BuildForm()` builds the panel, not fields — and sets `initialized` itself:**

```go
func (m *InstallerUpdateModel) BuildForm() tea.Cmd {
    m.SetFormFields([]FormField{}) // action screen: nothing to focus, nothing to save

    contentWidth, contentHeight := m.getViewportFormSize()
    if m.terminal == nil {
        if !m.isVerticalLayout() {
            contentWidth -= 2
        }
        m.terminal = terminal.NewTerminal(contentWidth-2, contentHeight-1,
            terminal.WithAutoScroll(), terminal.WithAutoPoll(), terminal.WithCurrentEnv())
    } else {
        m.terminal.Clear()
    }

    m.terminal.Append(locale.InstallerUpdateAsking)

    if !m.initialized {
        m.initialized = true
        return m.terminal.Init()
    }

    return nil
}
```

The overridden `View()` also calls `BuildForm()` while `!initialized` and discards the command it returns. Without setting the flag here, every frame would clear the panel and re-append the first line, wiping out the output of a running download on each redraw.

**Re-size the terminal by hand.** The terminal is not part of the `BaseScreen` viewports, so `updateViewports()` does not manage it. In the `tea.WindowSizeMsg` branch, the same arithmetic from `BuildForm` is repeated (`getViewportFormSize()`, subtracting two columns in horizontal layout and accounting for the border) before calling `m.terminal.SetSize(...)`. It is important to keep these two pieces of arithmetic in sync.

**Never let key messages fall through to `BaseScreen`.** In the base screen logic, `enter` means save-and-return and `ctrl+r` means reset. On an action screen, this behavior is undesirable, as pressing `enter` could exit mid-download. Therefore, the `tea.KeyMsg` branch for this model handles `enter`, `y`, and `n` itself and passes all other keys to the terminal for scrolling. While `m.terminal.IsRunning()` is true, every key message goes directly to the terminal unfiltered, making the screen's own keys unreachable by construction rather than by explicit checks.

**Load asynchronously, and carry the failure in the same message.**

```go
func (m *InstallerUpdateModel) Init() tea.Cmd {
    m.running, m.confirming = false, false
    m.offered, m.offerErr = nil, nil

    return tea.Batch(m.BaseScreen.Init(), m.loadPackage)
}

type InstallerPackageLoadedMsg struct {
    Package *processor.InstallerPackage
    Err     error
}
```

Registry screens are singletons that are built once at start-up, but `Init()` runs on every navigation to the screen. This means per-visit state has to be cleared here. Otherwise, on a second visit, the screen would display the previous answer, and if `confirming` were still set, pressing `y` could act on an outdated offer. Both the package and error outcomes travel in a single message and are stored, because the logic for allowing a download is based on `canDownload()`—defined as `!running && offered != nil && offerErr == nil`. If the screen fails to load the package, it becomes inactive: pressing `enter` has no effect, and `GetFormHotKeys()` returns no keys to show available actions.

**A confirmation only where there is a question.** Pressing `enter` starts the download immediately. The confirmation prompt appears only if the file about to be written already exists—only in this case do `y` and `n` have meaning. The hotkeys are managed to match these three states exactly, ensuring that the footer never suggests a key that the screen would ignore.

```go
func (m *InstallerUpdateModel) GetFormHotKeys() []string {
    if m.terminal != nil && m.terminal.IsRunning() {
        return nil
    }
    if m.confirming {
        return []string{"y|n"}
    }
    if !m.canDownload() {
        return nil
    }

    return []string{"enter"}
}
```

Note that this shadows `BaseScreen.GetFormHotKeys()`. The base gates `down|up`, `ctrl+s`, and `ctrl+r` behind having fields or a list, so on a screen with neither it offers none of them — but it appends `enter` unconditionally. `enter` is exactly the key this screen has to be able to withhold: while the download is running, and on a screen whose offer never arrived, pressing it does nothing and the footer must not say otherwise.

The declining branch matters as much as the confirming one: `n` clears `confirming` and rewrites the panel with what the user now needs, rather than returning to a screen that still reads as if nothing happened.

**Text in a pure function, again.** `installerUpdateLines(running, offered, err)` returns the panel contents as `[]string`, so `installer_update_test.go` can assert what the screen *claims* without a terminal.

## 🔄 **Migrating Existing Screens**

### **Step 1: Analyze Current Screen**

From existing screen, identify:
- Form fields and their types
- List components (if any)
- Special keyboard handling
- Save/reset logic

### **Step 2: Refactor Structure**

```go
// Before:
type OldFormModel struct {
    controller   *controllers.StateController
    styles       *styles.Styles
    window       *window.Window
    args         []string
    initialized  bool
    hasChanges   bool
    focusedIndex int
    showValues   bool
    fields       []FormField
    viewport     viewport.Model
    formContent  string
    fieldHeights []int
    // ... config-specific fields
}

// After:
type NewFormModel struct {
    *BaseScreen                    // Embedded base screen
    // Only config-specific fields
    config *controllers.Config
    list   list.Model             // If needed
}
```

### **Step 3: Update Constructor**

```go
// Before:
func NewOldFormModel(...) *OldFormModel {
    return &OldFormModel{
        controller: controller,
        styles:     styles,
        // ... lots of boilerplate
    }
}

// After:
func NewNewFormModel(...) *NewFormModel {
    m := &NewFormModel{
        config: controller.GetConfig(),
    }

    // Initialize list if needed
    m.initializeList()

    m.BaseScreen = NewBaseScreen(controller, styles, window, m, m)
    return m
}
```

### **Step 4: Implement Required Interfaces**

Move existing methods to interface implementations:

```go
// Move: buildForm() → BuildForm()
// Move: save logic → HandleSave()
// Move: reset logic → HandleReset()
// Move: help content → GetHelpContent()
```

### **Step 5: Remove Redundant Methods**

Delete these methods from migrated screens:
- `getInputWidth()`, `getViewportSize()`, `updateViewport()`
- `focusNext()`, `focusPrev()`, `toggleShowValues()`
- `renderVerticalLayout()`, `renderHorizontalLayout()`
- `ensureFocusVisible()`

## 📋 **Environment Variable Integration**

Follow existing patterns for environment variable handling:

```go
func (m *FormModel) BuildForm() tea.Cmd {
    // Track initially set fields for cleanup
    m.initiallySetFields = make(map[string]bool)

    for _, fieldConfig := range m.fieldConfigs {
        envVar, _ := m.GetController().GetVar(fieldConfig.EnvVarName)
        m.initiallySetFields[fieldConfig.Key] = envVar.IsPresent()

        field := m.createFieldFromEnvVar(fieldConfig, envVar)
        fields = append(fields, field)
    }

    m.SetFormFields(fields)

    return nil // a form screen has nothing to start; only a panel screen returns a cmd
}

func (m *FormModel) HandleSave() error {
    // First pass: Remove cleared fields
    for _, field := range m.GetFormFields() {
        value := strings.TrimSpace(field.Input.Value())
        if value == "" && m.initiallySetFields[field.Key] {
            m.GetController().SetVar(field.EnvVarName, "")
        }
    }

    // Second pass: Save non-empty values
    for _, field := range m.GetFormFields() {
        value := strings.TrimSpace(field.Input.Value())
        if value != "" {
            m.GetController().SetVar(field.EnvVarName, value)
        }
    }

    return nil
}
```

## 🎯 **Navigation Integration**

### **Screen Registration — four mandatory steps**

All four are required. Skipping any of them fails *quietly*: nothing panics and nothing is logged. Three of them show a symptom somewhere other than the code you changed; the fourth — step 2 — shows no symptom at all with the screens written so far, which is exactly why it is the one that gets skipped.

**1. The `ScreenID` constant** in `wizard/models/types.go`:

```go
const (
    ExampleFormScreen ScreenID = "example_form"
    // parameterised screens join their arguments with "§"
    UpdatePentagiScreen ScreenID = "processor_operation_form§compose§update"
)
```

The ID is the only key the navigator, the registry and every menu share. Without it there is nothing to push onto the navigation stack and nothing to key the registry by.

**2. A case in `models.RestoreModel`** (`types.go`):

```go
case *ExampleFormModel:
    return m
```

`app.forwardMsgToCurrentModel` calls `Update` on the current model and adopts the result only when `RestoreModel` recognises it — when it returns `nil` there is no fallback and no log: the app keeps the pointer it already holds, and returns the command either way. What a missing branch actually costs is worth stating precisely, because the obvious answer is wrong. Every screen in `wizard/models` has a pointer receiver on `Update` and returns *itself*, so the object the app renders is the object `Update` has just mutated — nothing freezes, and a missing branch shows no symptom at all. The branch is load-bearing for the case the type allows but nobody has written yet: a model that returns a *different* model, dropped silently with nothing anywhere to say why. That is why the contract is "every screen is listed" rather than "list the ones that need it" — the day a screen starts returning something else is not the day anybody remembers this rule. The `nil` is load-bearing in the other direction too: a `MaintenanceModel` delegates `Update` to its embedded `*ListScreen`, which returns *itself*, and `RestoreModel` not knowing `*ListScreen` is exactly what keeps the app pointing at the whole screen instead of one of its parts — so list the concrete screen type and nothing else.

`TestTypes_RestoreModel_RestoresTheUpdateScreens` in `types_test.go` pins membership:

```go
for _, model := range []tea.Model{(*InstallerUpdateModel)(nil), (*UpdateOverviewModel)(nil)} {
    if RestoreModel(model) == nil {
        t.Errorf("%T is missing its branch in RestoreModel", model)
    }
}
```

**3. Construction in `registry.initScreens`** (`wizard/registry/registry.go`):

```go
r.screens[models.ExampleFormScreen] = models.NewExampleFormModel(r.controller, r.styles, r.window)
```

Screens are built once here and reused; `registry.GetScreen` falls back to a `MockFormModel` for an unknown ID **and caches it under that ID**, so the screen you wrote is never constructed at all. The first symptom is not the screen but the menu: `ListScreen` labels each entry by asking the registry for the model behind its ID (`GetFormName`, `GetFormOverview`, `GetCurrentConfiguration`, `IsConfigured`), so the entry reads "Development Screen" and opens a placeholder. An unregistered screen also never receives `tea.WindowSizeMsg`, which `registry.HandleMsg` forwards to registered screens only.

**4. Strings in `wizard/locale/locale.go`, and hotkey captions in `app.initHotkeysLocale`** (`wizard/app.go`). All user-visible text lives in `locale`; the footer resolves each name from `GetFormHotKeys()` through the caption map:

```go
if localeHotKey, ok := app.hotkeys[hotkey]; ok {
    actions = append(actions, localeHotKey)
}
```

An unknown name is skipped without complaint, so the key keeps working and is simply never announced — the quietest of the four failures, and the only one whose symptom is a hint that is missing rather than a screen that is wrong. Neither new screen needed a new caption: `enter`, `up|down`, `pgup|pgdown`, `home|end` and `y|n` are already in the map.

### **Width-aware markdown rendering**

`styles.GetRenderer()` returns a single renderer built once with `glamour.WithWordWrap(DefaultRenderWidth)` — 80 columns, whatever the terminal is doing. That was enough while the only markdown screen was the EULA, which renders once at load and is prose written to be read at a fixed measure. It is not enough for a changelog: at 80 columns in a 200-column window the document uses a third of the screen, and in a terminal narrower than 80 every line breaks two or three times, which is what a list of component names does worst.

`styles.GetRendererForWidth(width)` is the replacement for screens that know how much room they have:

```go
func (s *Styles) GetRendererForWidth(width int) *glamour.TermRenderer {
    width = min(max(width, MinRenderWidth), MaxRenderWidth) // [40, 120]
    // ... memoised per width; on any failure returns the fixed 80-column renderer
}
```

Four properties are worth knowing before you use it:

- **It clamps to `[MinRenderWidth, MaxRenderWidth]` = `[40, 120]`.** The floor keeps a narrow terminal from splitting every line; the ceiling is a readability limit — a line spanning a maximised terminal is measurably harder to read.
- **It never returns `nil`.** A width that cannot be built falls back to the default renderer, so no render path needs a nil check — and a nil check that someone forgets is how a changelog ends up not being shown at all.
- **The cache is behind a pointer** (`renderers *rendererCache`), because `styles.Styles` is copied by value into every screen and a value cache would be cloned per screen, defeating the point.
- **The cache is mutex-guarded**, because renderers are built inside `tea.Cmd` goroutines — off the update loop, where two screens can ask for the same width at once.

`GetRenderer()` still exists and still wraps at 80; prefer `GetRendererForWidth` on anything that renders inside a sized panel.

### **Navigation Usage**

```go
// Navigate to screen with parameters
return m, func() tea.Msg {
    return NavigationMsg{
        Target: CreateScreenID("example_form", "config_type"),
    }
}

// Return to previous screen
return m, func() tea.Msg {
    return NavigationMsg{GoBack: true}
}
```

## 🔧 **Interface Validation**

Add compile-time interface checks:

```go
// Ensure interfaces are implemented
var _ BaseScreenHandler = (*ExampleFormModel)(nil)
var _ BaseListHandler = (*ListFormModel)(nil)

// A screen that embeds BaseScreen asserts both: the registry stores it as a model,
// and BaseScreen calls back into it as a handler
var _ BaseScreenModel = (*InstallerUpdateModel)(nil)
var _ BaseScreenHandler = (*InstallerUpdateModel)(nil)

// A document screen has no form, so it asserts only the model interface
var _ BaseScreenModel = (*UpdateOverviewModel)(nil)
```

## 📊 **Benefits Summary**

- **Code Reduction**: 50-60% less boilerplate per screen
- **Consistency**: Unified behavior across all forms
- **Maintainability**: Centralized bug fixes and improvements
- **Development Speed**: Faster new screen implementation

## 🎯 **Quick Reference**

### **Screen Types**

1. **Simple Form**: Inherit BaseScreen, implement BaseScreenHandler
2. **Form with List**: Inherit BaseScreen, implement both handlers
3. **Menu Screen**: Use existing patterns without BaseScreen
4. **Document**: No BaseScreen — implement BaseScreenModel over a viewport (`eula.go`, `update_overview.go`)
5. **Action with terminal panel**: Inherit BaseScreen with zero fields, own the terminal and override `View()` (`installer_update.go`, `apply_changes.go`)

### **Required Methods**

- `BuildForm()` - Create form fields
- `HandleSave()` - Save configuration with validation
- `HandleReset()` - Reset to defaults
- `GetFormTitle()` - Screen title
- `GetHelpContent()` - Right panel content

### **Optional Methods**

- `OnFieldChanged()` - Real-time validation
- List handler methods (if using lists)

This architecture enables rapid development of new installer screens while maintaining consistency and reducing code duplication.

---

### Backend Docs Installer Installer Troubleshooting (`backend/docs/installer/installer-troubleshooting.md`)

# PentAGI Installer Troubleshooting Guide

> Comprehensive troubleshooting guide including recent fixes, performance optimization, and common issues.

## 🚨 **Development-Specific Issues**

### **TUI Application Constraints**
**Problem**: Running installer breaks terminal session during development
**Solution**: Build-only development workflow

```bash
# ✅ CORRECT: Build and test separately
cd backend/
go build -o ../build/installer ./cmd/installer/main.go

# Test in separate terminal session
cd ../build/
./installer

# ❌ WRONG: Running during development
cd backend/
go run ./cmd/installer/main.go  # Breaks active terminal!
```

**Debug Monitoring**:
```bash
# Monitor debug output during development
tail -f log.json | jq '.'

# Filter by component
tail -f log.json | jq 'select(.component == "FormModel")'

# Pretty print timestamps
tail -f log.json | jq -r '"\(.timestamp) [\(.level)] \(.message)"'
```

## 🔧 **Recent Fixes & Improvements**

### ✅ **Composite ScreenID Navigation System**
**Problem**: Need to preserve selected menu items and provider selections across navigation
**Solution**: Implemented composite ScreenIDs with `§` separator for parameter passing

**Before** (❌ Problematic):
```go
// Lost selection on navigation
func (m *MenuModel) handleSelection() (tea.Model, tea.Cmd) {
    return NavigationMsg{Target: LLMProvidersScreen} // No context preserved
}
```

**After** (✅ Fixed):
```go
// Preserves selection context
func (m *MenuModel) handleSelection() (tea.Model, tea.Cmd) {
    selectedItem := m.getSelectedItem()
    return NavigationMsg{
        Target: CreateScreenID("llm_providers", selectedItem.ID),
    }
}

// Results in: "llm_providers§openai" - selection preserved
```

**Benefits**:
- Type-safe parameter passing via `GetScreen()`, `GetArgs()`, `CreateScreenID()`
- Automatic state restoration - user returns to exact selection after ESC
- Clean navigation stack with full context preservation

### ✅ **Complete Localization Architecture**
**Problem**: Hardcoded strings scattered throughout UI components
**Solution**: Centralized all user-visible text in `locale.go` with structured constants

**Implementation**:
```go
// Multi-line text stored as single constants
const MainMenuLLMProvidersInfo = `Configure AI language model providers for PentAGI.

Supported providers:
• OpenAI (GPT-4, GPT-3.5-turbo)
• Anthropic (Claude-3, Claude-2)
...`

// Usage in components
sections = append(sections, m.styles.Paragraph.Render(locale.MainMenuLLMProvidersInfo))
```

**Coverage**: 100% of user-facing text moved to locale constants
- Menu descriptions and help text
- Form labels and error messages
- Provider-specific documentation
- Keyboard shortcuts and hints

### ✅ **Viewport-Based Form Scrolling**
**Problem**: Forms with many fields don't fit on smaller terminals
**Solution**: Implemented auto-scrolling viewport with focus tracking

**Key Features**:
- **Auto-scroll**: Focused field automatically stays visible
- **Smart positioning**: Calculates field heights for precise scroll positioning
- **Seamless navigation**: Tab/Shift+Tab scroll form as needed
- **No extra hotkeys**: Uses existing navigation keys

**Technical Implementation**:
```go
// Auto-scroll on field focus change
func (m *FormModel) ensureFocusVisible() {
    focusY := m.calculateFieldPosition(m.focusedIndex)
    if focusY < m.viewport.YOffset {
        m.viewport.YOffset = focusY  // Scroll up
    }
    if focusY >= m.viewport.YOffset + m.viewport.Height {
        m.viewport.YOffset = focusY - m.viewport.Height + 1  // Scroll down
    }
}
```

### ✅ **Enhanced Provider Configuration**
**Problem**: Missing configuration fields for several LLM providers
**Solution**: Added complete field sets for all supported providers

**Provider-Specific Field Sets:**

- **OpenAI/Anthropic/Gemini**: Base URL + API Key
- **AWS Bedrock**: Region + Default Auth OR Bearer Token OR (Access Key + Secret Key + Session Token) + Base URL + Config Path
- **DeepSeek**: Base URL + API Key + Provider Name (for LiteLLM prefix, e.g., 'deepseek')
- **GLM**: Base URL + API Key + Provider Name (for LiteLLM prefix, e.g., 'zai')
- **Kimi**: Base URL + API Key + Provider Name (for LiteLLM prefix, e.g., 'moonshot')
- **Qwen**: Base URL + API Key + Provider Name (for LiteLLM prefix, e.g., 'dashscope')
- **Ollama**: Base URL + API Key (cloud only) + Model + Config Path + Pull/Load settings
  - Local scenario: No API key needed
  - Cloud scenario: API key required from https://ollama.com/settings/keys
- **Custom**: Base URL + API Key + Model + Config Path + Provider Name + Reasoning options

**Dynamic Form Generation**: Forms adapt based on provider type with appropriate validation and help text.

## 🔧 **Common Issues & Solutions**

### **macOS Reports the Installer as Malware**

**Symptoms**: macOS blocks a downloaded PentAGI installer and reports that it may contain malware.

**What is currently known**:
- Use only the official PentAGI installer downloads linked from the project README.
- Select the archive that matches your Mac CPU architecture: `darwin/amd64` for Intel Macs or `darwin/arm64` for Apple silicon.
- Maintainer notes in [issue #249](https://github.com/vxcontrol/pentagi/issues/249) report that the installer files are signed with the project certificate and that the checked `arm64` and `amd64` builds had zero VirusTotal detections.
- The maintainers have contacted Apple about the warning and are awaiting a response.

**Recommended handling**:
- Treat any unexpected malware warning as a reason to stop and verify the download source before continuing.
- Re-download the installer from the official PentAGI link if the archive came from a mirror, chat attachment, or any unofficial source.
- If you trust the official release after verification, try to open the installer once. Then open **System Settings** -> **Privacy & Security**, find the security message that says the installer was blocked, click **Open Anyway**, and confirm the follow-up prompt to allow that specific app to run.
- If your version of macOS shows different wording, follow Apple's guidance for opening a Mac app from an unidentified developer: <https://support.apple.com/guide/mac-help/open-a-mac-app-from-an-unidentified-developer-mh40616/mac>.
- Do not reuse an installer archive that was downloaded from an unknown source, even if its filename matches the official archive name.

### **Navigation Issues**

#### **Navigation Stack Corruption**
**Symptoms**: User gets stuck on screens, ESC doesn't work, back navigation fails
**Cause**: Circular navigation patterns or corrupted navigation stack

**Debug**:
```go
func (n *Navigator) debugStack() {
    stackInfo := make([]string, len(n.stack))
    for i, screenID := range n.stack {
        stackInfo[i] = string(screenID)
    }

    logger.LogWithData("Navigation Stack", map[string]interface{}{
        "stack":   stackInfo,
        "current": string(n.Current()),
        "depth":   len(n.stack),
    })
}
```

**Solution**:
```go
// ✅ CORRECT: Use GoBack to prevent loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    if err := m.saveConfiguration(); err != nil {
        return m, nil  // Stay on form if save fails
    }
    return m, func() tea.Msg {
        return NavigationMsg{GoBack: true}  // Return to previous screen
    }
}

// ❌ WRONG: Direct navigation creates loops
func (m *FormModel) saveAndReturn() (tea.Model, tea.Cmd) {
    m.saveConfiguration()
    return m, func() tea.Msg {
        return NavigationMsg{Target: ProvidersScreen} // Creates navigation loop!
    }
}
```

#### **Lost Selection State**
**Symptoms**: Menu selections reset, provider choices forgotten, configuration lost
**Cause**: Models not constructed with proper args

**Solution**:
```go
// ✅ CORRECT: Args-based construction
func NewModel(controller *StateController, styles *Styles,
              window *Window, args []string) *Model {
    selectedIndex := 0
    if len(args) > 0 && args[0] != "" {
        // Restore selection from navigation args
        for i, item := range items {
            if item.ID == args[0] {
                selectedIndex = i
                break
            }
        }
    }
    return &Model{selectedIndex: selectedIndex, args: args}
}
```

### **Form Issues**

#### **Form Field Width Problems**
**Symptoms**: Input fields too narrow/wide, don't adapt to terminal size
**Cause**: Fixed width assignments during field creation

**Debug**:
```go
func (m *FormModel) debugFormDimensions() {
    width, height := m.styles.GetSize()
    viewportWidth, viewportHeight := m.getViewportSize()
    inputWidth := m.getInputWidth()

    logger.LogWithData("Form Dimensions", map[string]interface{}{
        "terminal_size":  fmt.Sprintf("%dx%d", width, height),
        "viewport_size":  fmt.Sprintf("%dx%d", viewportWidth, viewportHeight),
        "input_width":    inputWidth,
        "is_vertical":    m.isVerticalLayout(),
        "field_count":    len(m.fields),
    })
}
```

**Solution**:
```go
// ✅ CORRECT: Dynamic width calculation
func (m *FormModel) updateFormContent() {
    inputWidth := m.getInputWidth()

    for i, field := range m.fields {
        // Apply width during rendering, not initialization
        field.Input.Width = inputWidth - 3
        field.Input.SetValue(field.Input.Value()) // Trigger width update
    }
}

// ❌ WRONG: Fixed width at creation
func (m *FormModel) addField() {
    input := textinput.New()
    input.Width = 50 // Breaks responsive design!
}
```

#### **Form Scrolling Issues**
**Symptoms**: Can't reach all fields, focused field goes off-screen
**Cause**: Missing auto-scroll implementation or incorrect field height calculation

**Debug**:
```go
func (m *FormModel) debugScrollState() {
    logger.LogWithData("Scroll State", map[string]interface{}{
        "focused_index":    m.focusedIndex,
        "viewport_offset":  m.viewport.YOffset,
        "viewport_height":  m.viewport.Height,
        "content_height":   lipgloss.Height(m.formContent),
        "field_heights":    m.fieldHeights,
        "total_fields":     len(m.fields),
    })
}
```

**Solution**:
```go
// ✅ CORRECT: Auto-scroll implementation
func (m *FormModel) focusNext() {
    m.fields[m.focusedIndex].Input.Blur()
    m.focusedIndex = (m.focusedIndex + 1) % len(m.fields)
    m.fields[m.focusedIndex].Input.Focus()
    m.updateFormContent()
    m.ensureFocusVisible() // Critical for auto-scroll
}
```

### **Environment Variable Issues**

#### **Configuration Not Persisting**
**Symptoms**: Settings lost between sessions, environment variables not saved
**Cause**: Not calling controller save methods or incorrect cleanup logic

**Debug**:
```go
func (m *FormModel) debugEnvVarState() {
    for _, field := range m.fields {
        envVar, _ := m.controller.GetVar(m.getEnvVarName(getEnvSuffixFromKey(field.Key)))

        logger.LogWithData("Field State", map[string]interface{}{
            "field_key":       field.Key,
            "input_value":     field.Input.Value(),
            "env_var_name":    m.getEnvVarName(getEnvSuffixFromKey(field.Key)),
            "env_var_value":   envVar.Value,
            "env_var_default": envVar.Default,
            "is_present":      envVar.IsPresent(),
            "initially_set":   m.initiallySetFields[field.Key],
        })
    }
}
```

**Solution**:
```go
// ✅ CORRECT: Proper save implementation
func (m *FormModel) saveConfiguration() error {
    // First pass: Remove cleared fields
    for _, field := range m.fields {
        value := strings.TrimSpace(field.Input.Value())

        if value == "" && m.initiallySetFields[field.Key] {
            // Field was set but now empty - remove from environment
            if err := m.controller.SetVar(field.EnvVarName, ""); err != nil {
                return fmt.Errorf("failed to clear %s: %w", field.EnvVarName, err)
            }
            logger.Log("[FormModel] SAVE: cleared %s", field.EnvVarName)
        }
    }

    // Second pass: Save non-empty values
    for _, field := range m.fields {
        value := strings.TrimSpace(field.Input.Value())
        if value != "" {
            if err := m.controller.SetVar(field.EnvVarName, value); err != nil {
                return fmt.Errorf("failed to set %s: %w", field.EnvVarName, err)
            }
            logger.Log("[FormModel] SAVE: set %s=%s", field.EnvVarName, value)
        }
    }

    return nil
}
```

### **Layout Issues**

#### **Content Not Adapting to Terminal Size**
**Symptoms**: Content cut off, panels don't resize, horizontal scrolling
**Cause**: Missing responsive layout logic or incorrect dimension handling

**Debug**:
```go
func (m *Model) debugLayoutState() {
    width, height := m.styles.GetSize()
    contentWidth, contentHeight := m.window.GetContentSize()

    logger.LogWithData("Layout State", map[string]interface{}{
        "terminal_size":    fmt.Sprintf("%dx%d", width, height),
        "content_size":     fmt.Sprintf("%dx%d", contentWidth, contentHeight),
        "is_vertical":      m.isVerticalLayout(),
        "min_terminal":     MinTerminalWidth,
        "min_menu_width":   MinMenuWidth,
        "min_info_width":   MinInfoWidth,
    })
}
```

**Solution**:
```go
// ✅ CORRECT: Responsive layout implementation
func (m *Model) View() string {
    width, height := m.styles.GetSize()

    leftPanel := m.renderContent()
    rightPanel := m.renderInfo()

    if m.isVerticalLayout() {
        return m.renderVerticalLayout(leftPanel, rightPanel, width, height)
    }
    return m.renderHorizontalLayout(leftPanel, rightPanel, width, height)
}

func (m *Model) isVerticalLayout() bool {
    contentWidth := m.window.GetContentWidth()
    return contentWidth < (MinMenuWidth + MinInfoWidth + PaddingWidth)
}
```

#### **Footer Height Inconsistency**
**Symptoms**: Footer takes more/less space than expected, layout calculations wrong
**Cause**: Using border-based footer approach instead of background approach

**Solution**:
```go
// ✅ CORRECT: Background approach (always 1 line)
func (a *App) renderFooter() string {
    actions := a.buildFooterActions()
    footerText := strings.Join(actions, " • ")

    return a.styles.Footer.Render(footerText)
}

// In styles.go
func (s *Styles) updateStyles() {
    s.Footer = lipgloss.NewStyle().
        Width(s.width).
        Background(lipgloss.Color("240")).
        Foreground(lipgloss.Color("255")).
        Padding(0, 1, 0, 1)
}

// ❌ WRONG: Border approach (height varies)
footer := lipgloss.NewStyle().
    Height(1).
    Border(lipgloss.Border{Top: true}).
    Render(text)
```

## 🔧 **Performance Issues**

### **Slow Rendering**
**Symptoms**: Laggy UI, delayed responses to keystrokes
**Cause**: Multiple glamour renderers, excessive content updates

**Debug**:
```go
func (m *Model) debugRenderPerformance() {
    start := time.Now()
    content := m.buildContent()
    buildDuration := time.Since(start)

    start = time.Now()
    m.viewport.SetContent(content)
    setContentDuration := time.Since(start)

    start = time.Now()
    view := m.viewport.View()
    viewDuration := time.Since(start)

    logger.LogWithData("Render Performance", map[string]interface{}{
        "content_size":       len(content),
        "rendered_size":      len(view),
        "build_ms":           buildDuration.Milliseconds(),
        "set_content_ms":     setContentDuration.Milliseconds(),
        "view_render_ms":     viewDuration.Milliseconds(),
    })
}
```

**Solution**:
```go
// ✅ CORRECT: Single shared renderer
// In styles.go
func New() *Styles {
    renderer, _ := glamour.NewTermRenderer(
        glamour.WithAutoStyle(),
        glamour.WithWordWrap(80),
    )
    return &Styles{renderer: renderer}
}

// Usage
rendered, err := m.styles.GetRenderer().Render(content)

// ❌ WRONG: Multiple renderers
func (m *Model) renderMarkdown(content string) string {
    renderer, _ := glamour.NewTermRenderer(...) // Performance killer!
    return renderer.Render(content)
}
```

### **Memory Leaks**
**Symptoms**: Increasing memory usage, application becomes sluggish over time
**Cause**: Not properly cleaning up resources, creating multiple renderer instances

**Solution**:
```go
// ✅ CORRECT: Complete state reset
func (m *Model) Init() tea.Cmd {
    // Reset ALL state completely
    m.content = ""
    m.ready = false
    m.error = nil
    m.initialized = false

    // Reset component state
    m.viewport.GotoTop()
    m.viewport.SetContent("")

    // Reset form state
    m.focusedIndex = 0
    m.hasChanges = false
    for i := range m.fields {
        m.fields[i].Input.Blur()
    }

    return m.loadContent
}
```

## 🔧 **Error Recovery Patterns**

### **Graceful State Recovery**
```go
func (m *Model) recoverFromError(err error) tea.Cmd {
    logger.Errorf("[%s] ERROR: %v", m.componentName, err)

    // Try to recover state
    m.error = err
    m.ready = true

    // Attempt graceful recovery
    return func() tea.Msg {
        logger.Log("[%s] RECOVERY: attempting state recovery", m.componentName)

        // Try to reload content
        if content, loadErr := m.loadFallbackContent(); loadErr == nil {
            logger.Log("[%s] RECOVERY: fallback content loaded", m.componentName)
            return ContentLoadedMsg{content}
        }

        logger.Log("[%s] RECOVERY: using minimal content", m.componentName)
        return ContentLoadedMsg{"# Error\n\nContent temporarily unavailable."}
    }
}
```

### **Safe Async Operations**
```go
func (m *Model) loadContent() tea.Cmd {
    return func() tea.Msg {
        defer func() {
            if r := recover(); r != nil {
                logger.Errorf("[%s] PANIC: recovered from panic: %v", m.componentName, r)
                return ErrorMsg{fmt.Errorf("panic in loadContent: %v", r)}
            }
        }()

        content, err := m.loadFromSource()
        if err != nil {
            return ErrorMsg{err}
        }

        return ContentLoadedMsg{content}
    }
}
```

## 🔧 **Testing Strategies**

### **Manual Testing Checklist**
```go
// Test dimensions
// 1. Resize terminal to various sizes
// 2. Test minimum dimensions (80x24)
// 3. Test very narrow terminals (< 80 cols)
// 4. Test very short terminals (< 24 rows)

func testDimensions() {
    testSizes := []struct{ width, height int }{
        {80, 24},   // Standard
        {40, 12},   // Small
        {120, 40},  // Large
        {20, 10},   // Tiny
    }

    for _, size := range testSizes {
        logger.LogWithData("Dimension Test", map[string]interface{}{
            "test_size":   fmt.Sprintf("%dx%d", size.width, size.height),
            "layout_mode": getLayoutMode(size.width, size.height),
        })
    }
}
```

### **Navigation Flow Testing**
```go
func testNavigationFlow() {
    testSteps := []struct {
        action   string
        expected string
    }{
        {"start", "welcome"},
        {"continue", "main_menu"},
        {"select_providers", "llm_providers"},
        {"select_openai", "llm_provider_form§openai"},
        {"go_back", "llm_providers§openai"},
        {"esc", "welcome"},
    }

    for _, step := range testSteps {
        logger.LogWithData("Navigation Test", map[string]interface{}{
            "action":   step.action,
            "expected": step.expected,
            "actual":   string(navigator.Current()),
        })
    }
}
```

This troubleshooting guide provides comprehensive solutions for:
- **Development Workflow**: TUI-safe development patterns
- **Navigation Issues**: Stack management and state preservation
- **Form Problems**: Responsive design and scrolling
- **Configuration**: Environment variable management
- **Performance**: Optimization and resource management
- **Recovery**: Graceful error handling and state restoration

---

### Backend Docs Installer Processor Implementation (`backend/docs/installer/processor-implementation.md`)

# Processor Implementation Summary

## Overview
Processor package implements the operational engine for PentAGI installer operations per [processor.md](processor.md). Core lifecycle flows, file integrity logic, Docker/Compose orchestration, and Bubble Tea integration are implemented. Cloud-facing work — the update check, the installer download and the Jaeger storage plugin — is implemented too and talks to the PentAGI Cloud API through `cmd/installer/cloud`. Post-update verification is implemented as well, but it makes no request of its own: it compares the components observed after the update against targets captured locally from the answer that was already in hand.

## Implementation Notes

### Architecture Decisions
- **Interface-based design**: Internal interfaces per operation type (`fileSystemOperations`, `dockerOperations`, `composeOperations`, `updateOperations`) enable separation of concerns and testability
- **Two-track execution**: Docker API SDK for worker environment; Compose stacks via console commands with live output streaming
- **OperationOption pattern**: Functional options applied to an internal `operationState` support force mode, embedded terminal integration and the reset-password value via `WithForce`, `WithTerminal` and `WithPasswordValue`
- **State machine logic**: `ApplyChanges` implements four-phase stack management (Observability → Langfuse → Graphiti → PentAGI) with integrity validation; the wizard performs a pre-phase interactive integrity check with Y/N decision for force mode
- **Single-responsibility operations**: business logic delegates to compose layer; strict purge of images is implemented as `purgeImagesStack` alongside other compose operations
- **One lock for everything**: every entry point on `Processor` takes the processor mutex, including `InstallerPackage`, which changes nothing. It takes it because it *remembers* something the download that follows will read — see [Installer package description](#installer-package-description)

### Key Features Implemented
1. **File System Operations** (`fs.go`):
   - Ensure/verify stack file integrity with force mode support
   - Handle embedded directory trees (observability) and compose files
   - YAML validation and automatic file recovery
   - Support deployment modes (embedded/external/disabled) for applicable stacks
   - Excluded files policy for integrity verification (`filesToExcludeFromVerification`): presence ensured, content changes tolerated — service configs (`observability/otel/config.yml`, `observability/grafana/config/grafana.ini`), user-editable presets (`example.custom.provider.yml`, `example.ollama.provider.yml`, `example.bedrock.provider.yml`, `neo4j/conf/neo4j.conf`, `neo4j/conf/apoc.conf`, `graphiti/{custom,gemini,litellm,openai}.yaml`) and both Jaeger storage plugin binaries

2. **Docker Operations** (`docker.go`):
   - Worker and default image management with progress reporting
   - Worker container lifecycle management (removal, purging)
   - Support for custom Docker configuration via environment variables

3. **Compose Operations** (`compose.go`):
   - Stack lifecycle management with dependency ordering
   - Rolling updates with health checks
   - Live output streaming to TUI callbacks
   - Environment variable injection for compose commands
    - `purgeStack` (down -v) and `purgeImagesStack` (down --rmi all -v) placed together for clarity

4. **Update Operations** (`update.go`, `jaeger_plugin.go`, `verify.go`):
   - `checkUpdates` refreshes the shared `checker.CheckResult` through `GatherUpdatesInfo` and returns nothing: the menus, the update screens and the operations all read the answer from there, and a second copy would be a second thing to keep current. Nothing calls it today — the answer is refreshed by `GatherAllInfo` at startup and by the re-gather every operation does when it finishes — so it is an entry point on the interface rather than a step on any current code path
   - `downloadInstaller` / `updateInstaller` fetch the offered build over `/api/v1/proxy/packages/info` → `/api/v1/proxy/packages/download` and leave it beside the environment file as `installer_<version>[.exe]`; both are the same operation reported under two names, because what would separate them — swapping the running binary — is out of scope
   - `installerPackage` describes the offered build without downloading it, for the screen that asks the user to agree to the download
   - `updateJaegerPlugin` fetches the Jaeger storage plugin, which is a mounted file rather than an image and so is invisible to `docker compose pull`
   - `removeInstaller` is the one member of the interface still returning "not implemented". It is reached by `Remove`/`Purge` on the installer stack, and the `all` fan-out visits that stack last, so both maintenance screens report a failure from their final step. Under "Remove" that is the whole cost: everything ahead of it has already run and nothing follows the loop. Under "Purge" it costs more — the error aborts the fan-out, and what sits after the loop never happens: the three custom Docker networks are not removed, and the deferred refresh of the update answer is skipped because it returns early on a failed operation

5. **Remove/Purge Operations**:
   - Soft removal (preserve data) vs purge (complete cleanup); strict image purge via compose in `purgeImagesStack`
   - Proper cleanup ordering and external/existing deployment handling

### Critical Implementation Details
- **Four-phase execution**: Observability → Langfuse → Graphiti → PentAGI with state validation after each phase
- **Force mode behavior**: Aggressive file overwriting and state correction when explicitly requested
- **File integrity logic**: `ensureStackIntegrity` for missing files, `verifyStackIntegrity` for existing files; modified files are explicitly skipped and logged when `force=false`; excluded files are ensured to exist but not overwritten when modified
- **State consistency**: `Gather*Info` calls after each phase validate operation success
- **Error isolation**: Phase failures don't affect other stacks, partial state preserved
- **Compose environment tweaks**: `COMPOSE_IGNORE_ORPHANS=1`, `PYTHONUNBUFFERED=1`; ANSI disabled on narrow terminals via `COMPOSE_ANSI=never`
- **Ordering inside a stack update**: the Jaeger plugin first — for observability only — then `downloadStack`, then `updateStack`, then a `restart` of the jaeger service when the plugin was actually replaced, and last the refresh of the answer. Writing the pull references is not a step in this sequence: `applyPullReferences` runs at the top of `downloadStack` and `updateStack` themselves, the only two compose operations that fetch anything, rather than at each of the five call sites that reach them — a call site that forgot would not fail to compile, because `${<COMPONENT>_IMAGE:-<default>}` quietly resolves to the default the compose file ships with and the operation would report success having fetched something the update server never offered. The plugin goes first because it is a mounted file no pull can bring up to date, and one that cannot be fetched must abort the update with nothing else disturbed, rather than leave images pulled and the stack restarted around a plugin that never arrived. The restart goes last because `up -d` recreates a container only when its spec changed, and a different file inside a bind mount is not a spec change — without it Jaeger goes on serving through the plugin it exec'd when it started, and the update reports success anyway
- **The Jaeger plugin is exempt from integrity verification**: both plugin paths are in `filesToExcludeFromVerification`, because the embedded copy and the update server are two sources for one path. Without the exemption a downloaded plugin reads as "modified", and a stack update — which runs with force — copies the embedded version back over it, undoing itself

### Operation Interfaces
The four internal interfaces in `processor.go` are how the business logic in `logic.go` reaches anything external. A method takes the `*operationState` when it produces output, because that is what carries the terminal panel and the message chain; the three that produce a value instead of output — `checkStackIntegrity`, `determineComposeFile` and `installerPackage` — do not:

```go
type updateOperations interface {
    checkUpdates(ctx context.Context, state *operationState) error
    downloadInstaller(ctx context.Context, state *operationState) error
    updateInstaller(ctx context.Context, state *operationState) error
    removeInstaller(ctx context.Context, state *operationState) error
    installerPackage(ctx context.Context) (*InstallerPackage, error)
    updateJaegerPlugin(ctx context.Context, state *operationState) (bool, error)
}
```

`installerPackage` is the odd one out here: it produces a value for a screen to render rather than output for a panel to show. `updateJaegerPlugin` is both at once: it takes a `state` because it downloads tens of megabytes and has to say so while it does, and it returns a `bool` beside the error — whether it actually replaced a file — because that is what decides the jaeger restart afterwards.

`updateOperationsImpl` is the only implementation and holds two mutable fields — `described` and `describedKey`, the remembered package description. They are plain fields rather than guarded ones because every path that touches them runs under the processor mutex: `installerPackage` takes it in `processor.InstallerPackage`, and the operations that reach `fetchInstaller` run under it too.

### Streaming Output Into the Terminal Panel
There are two producers of panel content, and they differ in whether the output is a whole view or a single line.

- `runCommand` hands the `*exec.Cmd` to `state.terminal.Execute`, waits for both the process and the terminal, and then sends `terminal.View()` as a **full** replacement (`isPartial=false`). Compose redraws its own progress in place, so the panel has to be the terminal's rendering rather than an append-only log.
- `appendLog` writes one line through `terminal.Append` and again sends the whole `View()`. This is what every cloud-facing operation uses: download progress, the plugin lines, the verification verdicts.

Both end at `operationState.sendOutput`, which appends a `ProcessorOutputMsg` to `state.msgs` under `state.mx`, stamped with its own 1-based position. Nothing is pushed to Bubble Tea — `HandleMsg` polls: handed a message numbered *n*, it returns `msgs[n]` (the next one) when it has arrived, and otherwise a `ProcessorWaitMsg` carrying the same *n* after 100 ms, which brings it back. The operation itself runs in a goroutine started by `wrapCommand`, which returns the first `ProcessorWaitMsg` as soon as the operation finishes or after 500 ms, whichever comes first. `ProcessorCompletionMsg` ends the chain by returning no command, and a `ProcessorWaitMsg` carrying an error stops the polling too. Walking an accumulated slice rather than draining a channel is what keeps the chain ordered and replayable when the screen is slower than the operation.

When no terminal is supplied, `newOperationState` builds a headless one (80×24, `WithNoPty`), so an operation invoked outside the wizard still produces the same message chain.

### Installer Package Description
`InstallerPackage` is the plain form of what the update server offers for this host — version, OS, arch, size — plus three fields the screen cannot compute for itself:

- `Path`: where the download will land, `installer_<version>[.exe]` in the directory holding `.env`. It is computed here rather than by the caller so that the name the screen promises and the name the download produces cannot drift apart.
- `Downloaded`: whether a file already sits at `Path`. It is the only thing the screen asks the user to confirm — downloading over nothing needs no permission, downloading over a file somebody may have put there on purpose does.
- `CurrentPath`: the running binary, or `""` when the OS would not say. `MoveCommand()` turns the pair into the exact `mv` (`move /Y` on Windows) the user runs afterwards, with both paths quoted so an installation under a directory with a space does not become two arguments. The installer never runs it: a process cannot reliably replace the binary it is executing — Linux answers `ETXTBSY`, Windows holds the file locked — so the operation honestly ends at "downloaded and verified".

**Why the description is reused by the download.** The screen has to state the size *before* the user agrees, which means `packages/info` is called on entering the screen and not only inside the download. Asking twice would cost a second proof-of-work challenge and a second slice of the account's daily allowance for one keypress, so `packageInfo` caches the answer in `described` and `fetchInstaller` reuses it.

The cache key is `version/GOOS/GOARCH`, not the version alone: the same version is published per platform with its own size, sha256 and signature, so a description of another platform's build would fail all three checks with a message about a corrupt download. On a failed download the description is dropped along with the partial file — it was the yardstick every check used, so a build republished under the same version would otherwise make every retry fail identically forever.

### Where Downloaded Files Are Written
Neither download goes through the system temp directory. Writing an executable to `%TEMP%` and then moving it is a pattern antivirus and EDR products score heavily, and a false positive there costs the user the update.

- **The installer** is written straight under its final name in the directory the installation lives in. Before opening it, `fetchInstaller` refuses to write to a path that is the running binary, comparing by inode rather than by string — a symlink or a different spelling of the same directory compares unequal as text while naming one file. The versioned name normally makes the collision impossible, but an installer renamed to `installer_<its own version>` lands exactly there, and `O_TRUNC` would empty the binary the user is running: Linux answers `ETXTBSY`, macOS does not.
- **The Jaeger plugin** is staged as `<target>.new` in the *same* directory as its target and moved with `os.Rename`, which is atomic only within one filesystem. Staging matters because the file being replaced is the one a running Jaeger has loaded; an in-place write cut short would leave a truncated plugin and the container would come back up with an opaque gRPC error. Mode `0755` is set at creation and not afterwards — `os.Rename` does not preserve the destination's mode, so a plugin created `0644` and renamed over an executable one ends up not executable, and Jaeger again fails with a gRPC error rather than a permission one.

On failure both paths delete what they wrote: verification only completes on the last byte, so anything left behind is unverified. Leftover `.new` files from an interrupted run are removed at the start of `updateJaegerPlugin` — always, even when there is nothing to fetch, because integrity verification only walks the files the installer ships and would never see a ~28 MB stranger. They are deleted by exact name rather than by globbing `*.new`: the directory is mounted into a container and an operator may keep their own files in it.

Both architectures of the plugin are kept in step regardless of the host, because which one runs is decided inside the container by the Docker VM's architecture — the compose entrypoint picks the file by `uname -m` at start. Only components the check marked outdated are fetched: membership in the answer is not the test, since a curated release names every artefact it knows about, and fetching everything listed would mean ~28 MB per architecture on every observability update. A component whose OS is not linux is skipped — the file name says `linux`, and content for another platform would be written under a linux name.

### Download Progress
`progressWriter` sits in an `io.MultiWriter` beside the destination file, so the bytes are counted on their way past without a second pass.

Its percentage is computed against `info.Size` from the package description, **not** a `Content-Length`: the response body is encrypted, which removes that header, so the description is the only size the client ever learns. This is also why `installerPackage` and `fetchInstaller` both need a description in hand before anything can start.

Two details are deliberate. A line is printed once per `progressStepPercent` (10%) rather than per read — a line every few kilobytes would bury the file path and the verification verdict in a panel that scrolls. And the percentage is clamped to 100: a stream longer than advertised is caught by the length check in `cloud.DownloadPackage`, and it is not the indicator's job to report it. With `total == 0` the writer stays silent instead of dividing by zero.

`FormatSize` is exported so the screen and the progress lines it prints into use one renderer; they end up in the same panel, and two formatters would eventually disagree about the same number in front of the user.

`cloud.DownloadPackage` checks three things in the same single pass the progress writer counts in — the byte count against `info.Size`, the sha256 against `info.Hash`, and an Ed25519 signature over the file's SHA-512 digest. The verdict therefore already exists when the call returns, which is why the line printed after it is past tense: `Checksum and signature verified`.

### Post-Update Verification
`verify.go` answers one question after a stack update: did the artefacts now installed become the ones that were offered? Targets are captured **before** the update (`captureStackTargets`), because the deferred `GatherUpdatesInfo` replaces the answer — comparing against the refreshed one would compare the result with itself and always agree.

The comparison runs inside the same `defer`, after the refresh, because the fresh answer is what decides whether anything is still pending. That distinction is made on evidence rather than heuristics: when a component differs from its captured target *and* the fresh answer no longer offers an update for the stack, the registry moved ahead between the answer and the pull, and the line says so; when the offer is still there, it is a mismatch. A failed refresh has no appeal — `stackHasNoUpdate` returns false — and a difference counts as a mismatch.

Verification never fails the operation. The update either worked or already reported an error; turning "the registry moved ahead" into a failed update would be worse than saying nothing. The installer stack is excluded from `verifiableUpdateStacks` on purpose: its operation downloads a build without applying it, so comparing against the running binary would report the intended outcome as a mismatch.

### Testing Strategy
Comprehensive tests include:
- Mock implementations for external dependencies (state, checker, files)
- Unit tests for file system integrity operations (ensure/verify/cleanup, excluded files policy, YAML validation)
- Validation tests for operation applicability
- Factory reset, lifecycle, and ordering behavior at logic level
- `update_test.go` — progress stepping and clamping, silence without a size, the description key, cleanup of file and description after a failed download, the versioned file name, the inode comparison guarding the running binary, `MoveCommand` quoting, `FormatSize`
- `jaeger_plugin_test.go` — only outdated linux components are fetched, a failed download leaves the running plugin alone, the installed plugin is executable, staging leftovers are removed by exact name
- `verify_test.go` — targets captured before the answer is replaced, an already current component's target is what it runs, an unverifiable component carries no target, "newer" is distinguished from "wrong", the installer is not verified against the running binary
- `pull_reference_test.go` — the compose variable is derived from the component name, only resolved components get a reference, and every `image:` in the shipped compose files is parameterised

### Integration Points
- **State management**: Integrates with `state.State` for configuration and environment variables
- **System assessment**: Uses `checker.CheckResult` for current system state analysis
- **File handling**: Integrates with `files.Files` for embedded content extraction
- **TUI integration**: Bubble Tea integration via `ProcessorModel` with message polling; wizard performs pre-phase integrity scan (Enter → scan; Y/N → overwrite decision; Ctrl+C → cancel integrity stage)

## Files Created/Modified

### Core Implementation
- `processor.go` - Processor interface, options, operation interfaces, and synchronous operations entry points
- `model.go` - Bubble Tea `ProcessorModel` with `HandleMsg` polling
- `logic.go` - Business logic (ApplyChanges, lifecycle operations, factory reset)
- `state.go` - `operationState`, message types and the `sendOutput`/`sendCompletion` chain
- `fs.go` - File system operations and integrity verification
- `docker.go` - Docker API/CLI operations and worker image/volumes management
- `compose.go` - Docker Compose stack lifecycle management (including `purgeImagesStack`)
- `update.go` - Update check, installer package description and download, progress reporting
- `jaeger_plugin.go` - Jaeger storage plugin download with staged install
- `verify.go` - Post-update comparison of installed artefacts against what was offered
- `pull_reference.go` - Writing the server-chosen image reference into `.env` before a pull
- `reference_drift.go` - Reporting compose variables that differ from what this build ships
- `locale.go` - Operation message strings

### Testing
- `fixtures_test.go` - Mocks for interfaces with call tracking
- `logic_test.go` - Business logic tests (state machine and sequencing)
- `fs_test.go` - File system operations tests (including excluded files policy, the Jaeger plugin among them)
- `update_test.go`, `jaeger_plugin_test.go`, `verify_test.go`, `pull_reference_test.go`, `reference_drift_test.go`, `compose_test.go`, `pg_test.go`, `state_test.go`, `model_test.go`

## Status
✅ **COMPLETE** - Processor functionality implemented and tested, with one known gap
- Lifecycle, file integrity, Docker/Compose orchestration are production-ready
- Bubble Tea integration via `ProcessorModel` is complete
- Installer download/update, Jaeger plugin download and post-update verification are implemented and covered by tests
- `removeInstaller` remains unimplemented; because `Remove`/`Purge` on `all` end on the installer stack, both maintenance screens finish with a "not implemented" error. Under "Remove" the real work is already done by then; under "Purge" the failure also skips everything after the fan-out — the three custom Docker networks stay in place and the update answer is not refreshed

## Current Architecture

### ProcessorModel Integration
The processor integrates with Bubble Tea through `ProcessorModel` that wraps operations as `tea.Cmd` and provides a polling handler:

```go
// ProcessorModel provides tea.Cmd wrappers for all operations and a polling handler
type ProcessorModel interface {
    ApplyChanges(ctx context.Context, opts ...OperationOption) tea.Cmd
    CheckFiles(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    FactoryReset(ctx context.Context, opts ...OperationOption) tea.Cmd
    Install(ctx context.Context, opts ...OperationOption) tea.Cmd
    Update(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    Download(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    Remove(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    Purge(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    Start(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    Stop(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    Restart(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    ResetPassword(ctx context.Context, stack ProductStack, opts ...OperationOption) tea.Cmd
    HandleMsg(msg tea.Msg) tea.Cmd

    // InstallerPackage describes the installer build on offer without downloading it.
    InstallerPackage(ctx context.Context) (*InstallerPackage, error)
}
```

`InstallerPackage` is blocking rather than a `tea.Cmd`, unlike everything above it: it is one short request with one answer, not an operation that streams output into a terminal panel, and the screen that needs it already wraps it in a command of its own.

### Message Types
Processor operations communicate via messages:
- `ProcessorStartedMsg` - operation started
- `ProcessorOutputMsg` - command output (partial or full)
- `ProcessorFilesCheckMsg` - file statuses computed during `CheckFiles`
- `ProcessorCompletionMsg` - operation completed
- `ProcessorWaitMsg` - polling tick

### Terminal Integration
Operations support real-time terminal output through `WithTerminal(term terminal.Terminal)`; ANSI is auto-disabled for narrow terminals (< 56 columns) and on Windows. Compose commands inherit current env plus `COMPOSE_IGNORE_ORPHANS=1` and `PYTHONUNBUFFERED=1`. Without the option an operation still runs, against a headless 80×24 terminal built by `newOperationState`. See [Streaming Output Into the Terminal Panel](#streaming-output-into-the-terminal-panel) for how a line becomes a message.

---

### Backend Docs Installer Processor Logic Implementation (`backend/docs/installer/processor-logic-implementation.md`)

# Processor Business Logic Implementation

## Core Concepts

The processor package manages PentAGI stack lifecycle through state-oriented approach. Main idea: maintain consistency between desired user state (state.State) and actual system state (checker.CheckResult).

## Key Principles

1. **State-Driven Operations**: All operations based on comparing current and target state
2. **Stack Independence**: Each stack (observability, langfuse, pentagi) managed independently
3. **Force Mode**: Aggressive state correction ignoring warnings
4. **Idempotency**: Repeated operation calls do not cause side effects
5. **User-Facing Automation**: Installer automates manual Docker/file operations with real-time feedback

## Stack Architecture

### ProductStack Hierarchy
```
ProductStackAll
├── ProductStackObservability (optional, embedded/external/disabled)
├── ProductStackLangfuse (optional, embedded/external/disabled)
└── ProductStackPentagi (mandatory, always embedded)
```

### Deployment Modes
- **Embedded**: Full local stack with Docker Compose files
- **External**: Using external service (configuration only)
- **Disabled**: Functionality turned off

## ApplyChanges Algorithm

### Purpose
Bring system to state matching user configuration. Main installation/update/configuration function.

### Prerequisites
- docker accessibility is validated via `checker.CheckResult` flags
- required compose networks are ensured via `ensureMainDockerNetworks` before stack operations

### Implementation Strategy

#### Pre-phase: interactive file integrity check (wizard)
- before starting ApplyChanges, the wizard runs an integrity scan using processor file checking helpers
- if outdated or missing files are detected, the user is prompted to choose:
  - proceed with updates (force=true) – modified files will be overwritten from embedded content;
  - proceed without updates (force=false) – installer will try to apply changes without touching modified files.
- hotkeys on the screen: Enter (start integrity scan), then Y/N to choose scenario; Ctrl+C cancels the integrity stage and returns to the initial prompt.

#### Phase 1: Observability Stack Management
```go
if p.isEmbeddedDeployment(ProductStackObservability) {
    // user wants embedded observability
    if !p.checker.ObservabilityExtracted {
        // extract files (docker-compose + observability directory)
         p.fsOps.ensureStackIntegrity(ctx, ProductStackObservability, state)
    } else {
        // verify file integrity, update if force=true
         p.fsOps.verifyStackIntegrity(ctx, ProductStackObservability, state)
    }
    // update/start containers
     p.composeOps.updateStack(ctx, ProductStackObservability, state)
} else {
    // user wants external/disabled observability
    if p.checker.ObservabilityInstalled {
        // remove containers but keep files (user might re-enable)
         p.composeOps.removeStack(ctx, ProductStackObservability, state)
    }
}
// refresh state to verify operation success
p.checker.GatherObservabilityInfo(ctx)
```

**Rationale**: Observability processed first as most complex stack (directory + compose file). Force mode used for file conflict resolution. For external/disabled modes containers are removed but files are preserved.

#### Phase 2: Langfuse Stack Management
```go
if p.isEmbeddedDeployment(ProductStackLangfuse) {
    // only docker-compose-langfuse.yml file
     p.fsOps.ensureStackIntegrity(ctx, ProductStackLangfuse, state)
     p.composeOps.updateStack(ctx, ProductStackLangfuse, state)
} else {
    if p.checker.LangfuseInstalled {
         p.composeOps.removeStack(ctx, ProductStackLangfuse, state)
    }
}
p.checker.GatherLangfuseInfo(ctx)
```

**Rationale**: Langfuse simpler than observability (single file only), but follows same logic. As a precondition for local start, configuration must be connected (see checker `LangfuseConnected`).

#### Phase 3: PentAGI Stack Management
```go
// PentAGI always embedded, always required
p.fsOps.ensureStackIntegrity(ctx, ProductStackPentagi, state)
p.composeOps.updateStack(ctx, ProductStackPentagi, state)
p.checker.GatherPentagiInfo(ctx)
```

**Rationale**: PentAGI - main stack, always installed, only file integrity check.

### Critical Implementation Details

#### File System Integrity
- **ensureStackIntegrity**: creates missing files from embed, overwrites with force=true
- **verifyStackIntegrity**: checks existence, updates with force=true
- **Embedded Provider**: uses `files.Files` for embedded content access
- correctness of directory checks: when modified files are detected and force=false, we log skip explicitly and keep files intact; the final directory log reflects whether modified files were present
- excluded files policy: `observability/otel/config.yml`, `observability/grafana/config/grafana.ini`, `example.custom.provider.yml`, `example.ollama.provider.yml`, `example.bedrock.provider.yml` are ensured to exist but not overwritten if modified

#### Container State Management
- `updateStack`: executes `docker compose up -d` for rolling update
- `removeStack`: executes `docker compose down` without removing volumes
- `purgeStack`: executes `docker compose down -v`
- `purgeImagesStack`: executes `docker compose down --rmi all -v`
- dependency ordering: observability → langfuse → pentagi
- environment: `COMPOSE_IGNORE_ORPHANS=1`, `PYTHONUNBUFFERED=1`; ANSI disabled on narrow terminals via `COMPOSE_ANSI=never`

#### State Consistency
- After each phase corresponding `Gather*Info()` method called
- CheckResult updated for next decisions
- On errors state remains partially updated (no rollback)
 - optimization with `state.IsDirty()`: optional early exit can be used by the UI to avoid unnecessary work; installer remains consistent without it.

## Force Mode Behavior

### Normal Mode (force=false)
- Does not overwrite existing files
- Stops on filesystem conflicts
- Conservative approach, minimal changes

### Force Mode (force=true)
- Overwrites any files without warnings
- Ignores validation errors
- Maximum effort to reach target state
- Used on explicit user request

### Disabled branches and YAML validation
- when a stack is configured as disabled, compose operations are skipped and file system changes are not required (except prior installation remains preserved);
- YAML validation is performed on compose files during integrity ensuring to fail fast in case of syntax errors.

## Error Handling Strategy

### Fail-Fast Principle
- Each phase can interrupt execution
- Partial state preserved (no rollback)
- Errors bubbled up with context

### Recovery Scenarios
- User can repeat operation with force=true
- Partial installation can be completed
- Remove/Purge operations for complete cleanup

## Operation Algorithms

### Update Operation
- checks `checker.*IsUpToDate` flags for compose stacks
- compose stacks: download then `docker compose up -d`, gather info
- worker: pull images only, gather info
- installer: stubbed (download/update/remove return not implemented), checksum and replace helpers exist
- refreshes updates info at the end of successful flows

### FactoryReset Operation
Correct cleanup sequence ensuring no dangling resources:
1. `purgeStack(all)` - removes all compose containers, networks, volumes
2. Remove worker containers/volumes (Docker API managed)
3. Remove residual networks (fallback cleanup)
4. Restore default .env from embedded
5. Restore all stack files with force=true
6. Refresh checker state

### Install Operation
- ensures docker networks exist before any stack operations
- checks `*Installed` flags to skip already installed components
- follows the same three-phase approach as ApplyChanges
- designed for fresh system setup

### Remove vs Purge
- **Remove**: Soft operation preserving user data (`docker compose down`)
- **Purge**: Complete cleanup including volumes (`docker compose down -v`)
- Files preserved in both cases for potential re-enablement

## Code Organization

### Clean Architecture
- Each operation delegates to specialized handlers (compose/docker/fs/update)
- No duplicate logic - single responsibility principle
- Force mode propagated through operationState
- Global mutex prevents concurrent modifications

## Integration Points

### State Management
- `state.State.IsDirty()` determines need for operations
- `state.State.Commit()` commits changes
- Environment variables control deployment modes

### Checker Integration
- `checker.CheckResult` contains current state
- Gather methods update state after operations
- Boolean flags optimize decision logic

### Files Integration
- `files.Files` provides embedded content access
- Fallback to filesystem when embedded missing
- Copy operations with rewrite flag for force mode

## Testing Strategy

### Unit Tests Focus
- State transition logic (current → target)
- Force mode behavior verification
- Error handling and partial state recovery
- Integration between components

### Mock Requirements
- files.Files for embedded content control
- checker.CheckResult with mockCheckHandler for state simulation
- baseMockFileSystemOperations, baseMockDockerOperations, baseMockComposeOperations with call tracking
- mockCheckHandler configured via mockCheckConfig for various scenarios

### Test Implementation
- Base mocks provide call logging and positive scenarios
- Error injection through setError() method on base mocks
- CheckResult states controlled via mockCheckHandler configuration
- Helper functions: testState(), testOperationState(), assertNoError(), assertError()

## Performance Characteristics

### Time Complexity
- O(1) for each stack (processed in parallel)
- File operations: O(n) where n = number of files in stack
- Container operations: depend on Docker/network latency

### Memory Usage
- Minimal: only state metadata
- Files read streaming without full memory loading
- CheckResult caches results until explicit refresh

### Network Impact
- Docker image pulls only when necessary
- Container updates use efficient rolling strategy
- External service connections minimal (checks only)

---

### Backend Docs Installer Processor Wizard Integration (`backend/docs/installer/processor-wizard-integration.md`)

# Processor-Wizard Integration Guide

> Technical documentation for embedded terminal integration in PentAGI installer wizard using Bubble Tea and pseudoterminals.

## Architecture Decision: Pseudoterminal vs Built-in Bubble Tea

After extensive research of `tea.ExecProcess` and alternative approaches, **pseudoterminal solution was chosen** for the following reasons:

**Why Not `tea.ExecProcess`:**
- ❌ Fullscreen takeover - cannot be embedded in viewport regions
- ❌ Blocking execution - pauses entire Bubble Tea program
- ❌ No real-time output streaming to specific UI components
- ❌ Cannot handle interactive Docker commands within constrained areas

**Built-in Approach Limitations:**
- Limited to `os/exec` with pipes (no true terminal semantics)
- Manual ANSI escape sequence handling required
- Reduced interactivity (no Ctrl+C, terminal properties)
- Complex input/output coordination

**Pseudoterminal Advantages:**
- ✅ Embedded in specific viewport regions
- ✅ Real-time command output with ANSI colors/formatting
- ✅ Full interactivity (stdin/stdout/stderr, Ctrl+C)
- ✅ Professional terminal experience within TUI
- ✅ Docker compatibility (`docker exec -it`, progress bars)

## Core Architecture

### Key Components

**`Processor`** interface (defined in [`processor.go`](../../cmd/installer/processor/processor.go)):
```go
type Processor interface {
    ApplyChanges(ctx context.Context, opts ...OperationOption) error
    CheckFiles(ctx context.Context, stack ProductStack, opts ...OperationOption) (FilesCheckResult, error)
    FactoryReset(ctx context.Context, opts ...OperationOption) error
    Install(ctx context.Context, opts ...OperationOption) error
    Update(ctx context.Context, stack ProductStack, opts ...OperationOption) error
    Download(ctx context.Context, stack ProductStack, opts ...OperationOption) error
    Remove(ctx context.Context, stack ProductStack, opts ...OperationOption) error
    Purge(ctx context.Context, stack ProductStack, opts ...OperationOption) error
    Start(ctx context.Context, stack ProductStack, opts ...OperationOption) error
    Stop(ctx context.Context, stack ProductStack, opts ...OperationOption) error
    Restart(ctx context.Context, stack ProductStack, opts ...OperationOption) error
    ResetPassword(ctx context.Context, stack ProductStack, opts ...OperationOption) error

    // InstallerPackage describes the installer build on offer without downloading it.
    InstallerPackage(ctx context.Context) (*InstallerPackage, error)
}
```

**Bubble Tea integration** in [`model.go`](../../cmd/installer/processor/model.go), [`state.go`](../../cmd/installer/processor/state.go), and [`logic.go`](../../cmd/installer/processor/logic.go):
- Wraps processor operations as `tea.Cmd` values through `ProcessorModel`
- Streams operation updates through `ProcessorStartedMsg`, `ProcessorOutputMsg`, `ProcessorFilesCheckMsg`, and `ProcessorCompletionMsg`
- Pumps that stream with `ProcessorWaitMsg` — the screen's reply to each message is what asks for the next one (see [The message chain](#the-message-chain))
- Executes commands through the shared `terminal.Terminal` abstraction when available
- Falls back to non-ANSI compose output on small terminals and Windows by setting `COMPOSE_ANSI=never`

`ProcessorModel.InstallerPackage(ctx)` is the one exception to the pattern: it returns `(*InstallerPackage, error)` rather than a `tea.Cmd`. It is a single short request with a single answer, not an operation that streams into a terminal panel, and the one screen that needs it wraps it in a command of its own.

### Integration Pattern

**Wizard Screen Integration** (see [`apply_changes.go`](../../cmd/installer/wizard/models/apply_changes.go)):
```go
type ApplyChangesFormModel struct {
    processor processor.ProcessorModel
    running   bool
    terminal  terminal.Terminal
}

// Create terminal model for the screen
m.terminal = terminal.NewTerminal(width, height, terminal.WithAutoScroll(), terminal.WithAutoPoll())

// Start operation through the processor model
return m.processor.ApplyChanges(context.Background(), processor.WithTerminal(m.terminal))
```

## Message Flow Architecture

The integration is **message-driven in both directions**: the operation appends messages to its own state and the screen asks for the next one, while the terminal panel is fed directly by the running command and signals the UI when its content changes (see [The message chain](#the-message-chain)):

```mermaid
graph TD
    A[User Action - Enter] --> B[ProcessorModel Command]
    B --> C[Operation State]
    C --> D[Terminal Abstraction]
    D --> E[Execute Command - docker compose]
    E --> F[Terminal Update Loop]
    F --> G[Processor Messages]
    G --> H[HandleMsg Polling]
    H --> I[ProcessorOutputMsg]
    I --> J[appendOutput + Viewport Update]
    J --> K[UI Refresh - Real-time Display]

    L[User Input] --> M[handleTerminalInput]
    M --> N{Key Type?}
    N -->|Scroll Keys| O[Viewport Handling]
    N -->|Other Keys| P[keyToTerminalSequence]
    P --> Q[Direct PTY Write]

    subgraph "Real-time Flow"
        F
        G
        H
        I
        J
    end

    subgraph "Input Handling"
        M
        N
        O
        P
        Q
    end

    subgraph "Terminal Display"
        C
        R[Blue Border Only]
        S[Maximized Content]
        T[Auto-scroll Bottom]
    end
```

## Apply Changes integrity pre-check (Wizard)

Before invoking `processor.ApplyChanges()`, the Apply Changes screen performs an embedded files integrity scan:

- Enter: start async scan using `GetStackFilesStatus(files, ProductStackAll, workingDir)`
- If outdated/missing files found: prompt user to update (Y) or proceed without updates (N)
- Ctrl+C: cancel the integrity stage and return to initial instruction screen

Hotkeys on this screen:
- Initial: Enter
- During scan: Ctrl+C
- When prompt is shown: Y/N, Ctrl+C

Depending on choice, `processor.ApplyChanges()` is called with/without `WithForce()`. This keeps user in control of overwriting modified files while still allowing a smooth path when no updates are required.

Note: the integrity prompt lists only modified files; missing files are considered normal on a fresh installation and are not shown to the user.

**Key Improvements:**
- **Message replay, not polling** — the operation appends to `operationState.msgs` and `HandleMsg` hands back the next entry; it falls back to a 100 ms wait only when there is nothing new
- **No message cap and no shared channel** — `msgs` is a slice that grows with the operation, so nothing about a slow screen can block the operation
- **Direct key mapping** - no intermediate input buffers
- **Viewport delegation** for scrolling (PageUp/PageDown, mouse wheel)

## Maintenance entries and the screens behind them

The maintenance list ([`maintenance.go`](../../cmd/installer/wizard/models/maintenance.go)) carries no behaviour of its own: `LoadItems` appends a `ListItem` holding a `ScreenID`, and `HandleSelection` turns the selected one into a `NavigationMsg{Target: item.ID}`. Changing what an entry does is therefore changing which screen its `ScreenID` names. Two entries no longer name the generic operation form.

**Update PentAGI** points at `UpdateOverviewScreen` (`"update_overview"`), not at `UpdatePentagiScreen` (`"processor_operation_form§compose§update"`). [`update_overview.go`](../../cmd/installer/wizard/models/update_overview.go) renders what the update would change — the version move per stack, the per-component plan, and the release notes crossed on the way — and on `enter` emits `NavigationMsg{Target: UpdatePentagiScreen}`. The overview is inserted **between** the entry and the form; the form, its availability check and its `y/n` confirmation are untouched, so the sequence is entry → overview → form, not a replacement.

The overview screen holds no `ProcessorModel` and starts no operation — `NewUpdateOverviewModel` takes only the controller, styles and window. Everything it shows already sits in the last check result read through `controller.GetChecker()`, so it makes no network call and cannot fail. What separates "nothing to update" from "we never found out" is `CheckResult.UpdateFailure`, not the emptiness of `StackUpdates`: the screen is reachable from a menu entry drawn off an earlier check, and printing "up to date" after a check that never completed would be a claim in exactly the wrong direction.

`enter` here means *continue*, not *page down* as it does on the EULA screen. EULA spends `enter` on paging because consent there is given by separate `y`/`n` keys; the overview asks for no consent, so its one unambiguous key belongs to the one action it offers, and scrolling is left to the arrows, `pgup`/`pgdown` and `home`/`end`.

**Update Installer** points at `InstallerUpdateScreen` (`"installer_update"`), a screen of its own in [`installer_update.go`](../../cmd/installer/wizard/models/installer_update.go). The `ScreenID` it replaced was `"processor_operation_form§installer§update"` — a generic operation form — and the `ProductStackInstaller` branches went out with it: `getOperationInfo`, `isActionAvailable` and `renderEffectsText` in [`processor_operation_form.go`](../../cmd/installer/wizard/models/processor_operation_form.go) no longer mention that stack, and the help and effects strings those branches used — `ProcessorHelpUpdateInstaller`, `EffectsUpdateInstaller` — are gone from the locale. The two the menu entry itself is drawn from stayed: `MaintenanceUpdateInstaller` and `MaintenanceUpdateInstallerDesc` are what the new screen returns from `GetFormName`, `GetFormTitle` and `GetFormDescription`, deliberately, so the entry reads the way it always did. This is a replacement rather than a rename: the form's help promised to replace the running binary and exit, which no version of the installer has ever done, and a confirmation guarding an operation that does not exist is worse than none. The screen names the running version, the offered version, the platform, the download size and the full target path first, then offers one action.

`processor.ProductStackInstaller` itself still exists and is still what the screen passes to `processor.Update` — only the wizard's generic branch for it is gone.

**Confirmation is not shared between the two screens.** The generic form keeps `requiresConfirmation` for the stack operations that have it — update of `all`/`compose`, factory reset, purge — and on `enter` it clears the terminal, prints the confirmation and waits for `y`/`n`. The installer screen asks nothing about the download itself; its only question is whether to write over a file already sitting at the target path (`InstallerPackage.Downloaded`), because downloading over nothing needs no permission. `n` there is not a dead end: the file the user chose to keep is the build they came for, so the screen prints the same move command a finished download would have printed.

**Adding a screen takes four registrations, and the second one has no symptom when it is missing.** A `ScreenID` constant in [`types.go`](../../cmd/installer/wizard/models/types.go); a branch in `models.RestoreModel` in the same file; registration in `registry.initScreens` ([`registry.go`](../../cmd/installer/wizard/registry/registry.go)); and the locale strings, plus a hotkey caption in `app.initHotkeysLocale` if the screen uses a combination the map does not already have.

What a missing `RestoreModel` branch costs is worth stating precisely, because the obvious answer is wrong. `app.forwardMsgToCurrentModel` calls `app.currentModel.Update(msg)` and reassigns `app.currentModel` only when `RestoreModel` returns non-nil — but every screen here has a pointer receiver and returns itself, so the mutation has already happened in place and the returned `tea.Cmd` is propagated either way. Nothing freezes today; the skipped step is a reassignment of a pointer the app already holds. The branch is load-bearing for the case the type allows but nobody has written yet: a screen whose `Update` returns a *different* model, which would then be dropped with no error anywhere. `TestTypes_RestoreModel_RestoresTheUpdateScreens` (`models/types_test.go`) asserts the two update screens' presence in `RestoreModel` for that reason — the contract is "every screen is listed" rather than "list the ones that need it", because the day a screen starts returning something else is not the day anybody remembers this rule.

## The message chain

A screen does not poll the processor; it answers it. Understanding this is the difference between a screen that finishes an operation and one that appears to hang halfway through.

`wrapCommand` in [`model.go`](../../cmd/installer/processor/model.go) starts the operation on a goroutine under the processor mutex and then waits before returning a `tea.Cmd` producing a `ProcessorWaitMsg` — until the operation returns, until the context is cancelled, or 500 ms, whichever comes first. That wait happens inside the caller, so the screen's `Update` or key handler can be held for up to half a second before the first frame is drawn. Meanwhile the operation appends messages to `operationState.msgs` ([`state.go`](../../cmd/installer/processor/state.go)): `ProcessorStartedMsg`, one `ProcessorOutputMsg` per output update, `ProcessorFilesCheckMsg` where files are scanned, and a final `ProcessorCompletionMsg`. Each message carries the state it came from and its own index.

`ProcessorModel.HandleMsg` is what advances it: given a message, it replays the next entry of `state.msgs` if one exists, and otherwise waits 100 ms and emits another `ProcessorWaitMsg` for the same index. So the rule for a screen is mechanical — **return `m.processor.HandleMsg(msg)` for every processor message received**, including the ones it has nothing to do with:

```go
case processor.ProcessorStartedMsg:
    return m, m.processor.HandleMsg(msg)

case processor.ProcessorOutputMsg:
    // ignore (handled by terminal)
    return m, m.processor.HandleMsg(msg)

case processor.ProcessorWaitMsg:
    return m, m.processor.HandleMsg(msg)

case processor.ProcessorCompletionMsg:
    m.handleCompletion(msg)
    return m, m.processor.HandleMsg(msg)
```

Swallow any one of these types and the chain stops there: no further output arrives and the completion never does, so the screen sits on its "in progress" line forever with the operation actually finished behind it.

Two ends are deliberate. `HandleMsg` returns `nil` for `ProcessorCompletionMsg` — it is the last message, and asking for another would only spin. It also returns `nil` for a `ProcessorWaitMsg` carrying an error, which is how a cancelled context stops the polling rather than leaving a command re-arming itself every 100 ms.

`ProcessorOutputMsg` is forwarded but its content is ignored by screens that pass `WithTerminal(m.terminal)`: the operation writes into that same terminal directly, so the panel is already current, and the message exists here only to keep the chain moving. What a screen actually reacts to is `ProcessorCompletionMsg` — that is where it clears its `running` flag and appends whatever it has left to say. In [`installer_update.go`](../../cmd/installer/wizard/models/installer_update.go) that is `handleCompletion`: on `msg.Error != nil` it appends the failure line and stops, and otherwise it appends the move instructions — where the build is, and the exact command that puts it in place, replaced by a plain "move it over the installer you launched" when `os.Executable` gave no path to build a command from. Windows gets one line more, because it will not replace a program that is running. What the screen does not repeat is the verification: the operation has already reported that the checksum and signature matched, and saying it twice would read as a second event.

Only the current screen sees these messages. `app.Update` broadcasts `tea.WindowSizeMsg` to every registered screen through `registry.HandleMsg`, but everything else goes to `forwardMsgToCurrentModel` — which is why each screen that runs operations spells out the processor cases itself rather than inheriting them.

## Implementation Guide

### 1. Screen Model Setup

Add terminal integration to any wizard screen following this pattern:

```go
type YourFormModel struct {
    *BaseScreen
    processor processor.ProcessorModel
    terminal  terminal.Terminal
}

func (m *YourFormModel) BuildForm() tea.Cmd {
    contentWidth, contentHeight := m.getViewportFormSize()

    if m.terminal == nil {
        m.terminal = terminal.NewTerminal(
            contentWidth-2,
            contentHeight-1,
            terminal.WithAutoScroll(),
            terminal.WithAutoPoll(),
            terminal.WithCurrentEnv(),
        )
    }

    return m.terminal.Init()
}
```

### 2. Update Method Integration

Route each message to exactly one owner. Terminal updates are restored through `terminal.RestoreModel`, processor messages are answered with `HandleMsg`, and anything unrecognised falls through to the terminal so scrolling keeps working:

```
/* Detailed source-code truncated for AI context efficiency. */
```

See [The message chain](#the-message-chain) for why the three processor cases that appear to do nothing still have to be there.

### 3. Operation Triggers

Start processor operations through terminal model:

```go
// In your action handler
func (m *YourFormModel) startProcess() tea.Cmd {
    m.running = true

    return m.processor.Install(
        context.Background(),
        processor.WithTerminal(m.terminal),
    )
}
```

Operations taking a `ProductStack` take it as the second argument — the installer screen, for instance, runs `m.processor.Update(ctx, processor.ProductStackInstaller, processor.WithTerminal(m.terminal))`. The `running` flag is the screen's own; it is set here and cleared on `ProcessorCompletionMsg`, and it is what keeps `enter` from starting a second operation while the first is still going.

### 4. Display Integration

Render terminal within your screen layout - simplified approach:

```go
func (m *YourFormModel) renderMainPanel() string {
    if m.terminal != nil {
        // Terminal model returns complete styled content with blue border
        return m.terminal.View()
    }

    return m.GetStyles().Error.Render(locale.ApplyChangesTerminalIsNotInitialized)
}
```

**Terminal View Structure:**
- **No header/footer** - maximized content space
- **Blue border only** - clean visual boundaries
- **Auto-scrolling viewport** - always shows latest output
- **Responsive sizing** - adapts to window changes via `SetSize()`

## Features & Capabilities

### Interactive Terminal
- **Real-time output**: the running command's stdout/stderr are read by scanner goroutines straight into the terminal's buffer, and a single-waiter notifier wakes the UI with a `TerminalUpdateMsg` when the content changes — no polling interval to tune
- **Native input handling**: Direct key-to-terminal-sequence conversion
- **ANSI support**: Colors, formatting, progress bars rendered correctly
- **Smart scrolling**: PageUp/PageDown and mouse wheel for viewport, all other keys to PTY

### Docker Integration
- **Interactive commands**: `docker exec -it container bash`
- **Progress visualization**: Docker pull progress bars with colors
- **Color output**: Docker's colored status messages preserved
- **Signal handling**: Ctrl+C, Ctrl+D, Ctrl+Z properly handled

### UI Features
- **Maximized space**: No headers, input lines, or inner borders
- **Blue border styling**: Clean visual boundaries with `lipgloss.Color("62")`
- **Dynamic resizing**: `SetSize()` method for window changes
- **Toggle mode**: Ctrl+T switches between embedded terminal and message logs
- **Auto-scroll**: Always shows latest output, bottom-aligned

## Command Configuration

Processor operations support flexible configuration via [`processor.go`](../../cmd/installer/processor/processor.go):

```go
// Available options
processor.WithForce()                   // Skip validation checks
processor.WithTerminal(terminal)        // Terminal-backed integration
processor.WithPasswordValue(password)   // Reset password operation input
```

Choose integration method based on screen requirements:
- **Embedded terminal**: Interactive operations, real-time output, full PTY support
- **Processor model**: Bubble Tea command wrappers and message polling for wizard screens (see [`model.go`](../../cmd/installer/processor/model.go))

**Alternative Integration:** For simpler use cases or Windows compatibility, the current processor model uses `ProcessorOutputMsg` events from [`state.go`](../../cmd/installer/processor/state.go) and disables Docker Compose ANSI output when needed in [`logic.go`](../../cmd/installer/processor/logic.go).

## Limitations & Considerations

### Performance
- **Memory usage**: `operationState.msgs` grows for the life of the operation and is not capped — a very chatty operation keeps every message it produced until the state is dropped
- **Goroutine management**: one goroutine per operation, started by `wrapCommand` and finished when the operation returns; the terminal's reader goroutines end with the command's pipes
- **Resource overhead**: minimal — nothing spins while an operation is idle
- **Update frequency**: `HandleMsg` returns the next message immediately when one exists, and waits 100 ms only when none has arrived yet

### Platform Compatibility
- **Unix/Linux/macOS**: Full pseudoterminal support via `github.com/creack/pty v1.1.21`
- **Windows**: Compose ANSI output is disabled in the command runner when the terminal is small or the host is Windows

### UI Constraints
- **Minimum size**: `width-2, height-2` for border space
- **Input delegation**: Terminal captures all input except scroll keys when running
- **Layout integration**: Single terminal per screen, full area utilization

## Testing & Debugging

### Processor Model Testing
Processor model behavior can be tested independently of wizard integration (see [`logic_test.go`](../../cmd/installer/processor/logic_test.go) and [`fixtures_test.go`](../../cmd/installer/processor/fixtures_test.go) for current mock patterns).

### Debug Features
- **Ctrl+T toggle**: Switch to message-based mode for debugging
- **Output logging**: All terminal output available for inspection
- **Error reporting**: Terminal errors bubble up to UI state

## Future Enhancements

### Potential Improvements
- **Session recording**: Capture terminal sessions for replay/debugging
- **Multiple terminals**: Support for concurrent operations with tabs
- **Terminal themes**: Customizable color schemes via lipgloss
- **Buffer persistence**: Save terminal output between screen switches
- **Copy/paste**: Terminal text selection and clipboard integration

### Integration Opportunities
- **Service management**: Real-time `docker ps` monitoring in terminal
- **Log streaming**: Live log viewing with `docker logs -f`
- **Interactive debugging**: Terminal-based troubleshooting tools
- **Configuration editing**: Embedded editors for compose files

### Architecture Extensions
- **Message broadcasting**: Share terminal output across multiple UI components
- **Operation queuing**: Sequential command execution with progress tracking
- **Error recovery**: Automatic retry mechanisms with user confirmation

This clean, message-driven architecture provides a maintainable foundation for embedding professional terminal functionality within Bubble Tea applications while maximizing screen real estate and user experience.

---

### Backend Docs Installer Processor (`backend/docs/installer/processor.md`)

# Processor Package Architecture

## Overview

The `processor` package serves as the core orchestrator for PentAGI installer operations, managing system interactions, Docker environments, and file system operations across different product stacks. It acts as the operational engine that executes user configuration changes determined through the TUI wizard interface.

## Installer Integration Architecture

```mermaid
graph TB
    subgraph "Installer Application"
        Main[main.go<br/>Entry Point]
        State[state package<br/>Configuration Management]
        Checker[checker package<br/>System Assessment]
        Files[files package<br/>Embedded Content]
        Wizard[wizard package<br/>TUI Interface]
        Processor[processor package<br/>Operation Engine]
    end

    subgraph "External Systems"
        Docker[Docker Engine<br/>Container Management]
        FS[File System<br/>Host OS]
        UpdateServer[PentAGI Cloud API<br/>update.pentagi.com]
        Compose[Docker Compose<br/>Stack Orchestration]
    end

    Main --> State
    Main --> Checker
    Main --> Wizard
    State --> Wizard
    Checker --> Wizard
    Wizard --> Processor
    Processor --> State
    Processor --> Checker
    Processor --> Files
    Processor --> Docker
    Processor --> FS
    Processor --> UpdateServer
    Processor --> Compose

    classDef core fill:#f9f,stroke:#333,stroke-width:2px
    classDef external fill:#bbf,stroke:#333,stroke-width:2px

    class Main,State,Checker,Files,Wizard,Processor core
    class Docker,FS,UpdateServer,Compose external
```

## Terms and Definitions

- **ProductStack**: Logical grouping of services that can be managed as a unit (pentagi, graphiti, langfuse, observability, compose, worker, installer, all)
- **Deployment Modes**: embedded (full local stack), external (existing service), disabled (no functionality)
- **State**: Persistent configuration storage including .env variables and wizard navigation stack
- **Checker**: System environment assessment providing current installation status and capabilities
- **TUI Wizard**: Terminal User Interface providing guided configuration flow
- **Embedded Content**: Docker compose files and configurations bundled within installer binary via go:embed
- **Worker Images**: Container images for AI agent tasks (default: debian:latest, pentest: vxcontrol/kali-linux)
- **ApplyChanges**: Deterministic state machine that transitions system from current to target configuration
- **Stack "all"**: Context-dependent operation affecting applicable stacks (excludes worker/installer for lifecycle operations)
- **Terminal Model**: Embedded pseudoterminal (`github.com/creack/pty`) providing real-time interactive command execution within TUI
- **Message Integration**: Channel-based processor integration via `ProcessorMessage` events for simple operations

## Architecture

### Main Components

- **processor.go**: Processor interface, options (`WithForce`, `WithTerminal`), and synchronous operation entry points
- **model.go**: Bubble Tea `ProcessorModel` with command wrappers and `HandleMsg` polling
- **compose.go**: Docker Compose operations and YAML file management, including strict purge `purgeImagesStack`
- **docker.go**: Docker API/CLI interactions for images, containers, networks, and volumes (worker + main)
- **fs.go**: File system operations and embedded content extraction with excluded files policy
- **logic.go**: Business logic (ApplyChanges state machine, lifecycle operations, factory reset)
- **update.go**: Cloud-facing operations — update check, installer package description, installer download with progress reporting
- **pull_reference.go**: Writes the image references the cloud chose into `.env` before a compose pull
- **jaeger_plugin.go**: Downloads the Jaeger storage plugin (a file, not an image) and puts it in place
- **verify.go**: Post-update comparison of what is installed against what was offered
- **reference_drift.go**: Reports compose image variables pointing at a repository this build does not ship
- **state.go**: Operation state, messages, and terminal integration helpers

### ProductStack Types

```go
type ProductStack string

const (
    ProductStackPentagi       ProductStack = "pentagi"       // Main stack (docker-compose.yml)
    ProductStackGraphiti      ProductStack = "graphiti"      // Knowledge graph (docker-compose-graphiti.yml)
    ProductStackLangfuse      ProductStack = "langfuse"      // LLM observability (docker-compose-langfuse.yml)
    ProductStackObservability ProductStack = "observability" // System monitoring (docker-compose-observability.yml)
    ProductStackCompose       ProductStack = "compose"       // The four compose stacks together
    ProductStackWorker        ProductStack = "worker"        // Docker images for AI agent tasks
    ProductStackInstaller     ProductStack = "installer"     // Installer binary itself
    ProductStackAll           ProductStack = "all"           // Context-dependent multi-stack operation
)
```

## Detailed Operation Scenarios

### Lifecycle Management
- **Start(stack)**:
  - pentagi/langfuse/observability: `docker compose ... up -d` (honors embedded mode for non-destructive ops)
  - all: sequential start in order observability → langfuse → pentagi
  - worker/installer: not applicable

- **Stop(stack)**:
  - pentagi/langfuse/observability: `docker compose ... stop`
  - all: sequential stop in reverse order (pentagi → langfuse → observability)
  - worker/installer: not applicable

- **Restart(stack)**:
  - implemented as stop + small delay + start to avoid dependency race
  - worker/installer: not applicable

### Installation & Content Management
- **Download(stack)**:
  - pentagi: `docker compose pull`; graphiti/langfuse/observability: the same, but only in embedded mode (external/disabled stacks are skipped)
  - worker: `docker pull ${DOCKER_DEFAULT_IMAGE_FOR_PENTEST}` (default 6GB+), then refresh worker and update facts
  - installer: download and verify the offered installer build (no-op when already current, error when the update server is unreachable) — see *Installer self-update* below
  - compose: the four compose stacks in order
  - all: compose stacks, then worker, then installer

- **Install(stack)**:
  - pentagi: extract compose file and example provider config
  - langfuse: extract compose file (embedded mode only)
  - observability: extract compose file and directory tree (embedded mode only)
  - graphiti: extract compose file plus the `graphiti/` and `neo4j/` directory trees (embedded mode only)
  - worker: download images
  - installer: not applicable
  - all: install all configured stacks (only the ones not yet installed)

- **Update(stack)**:
  - pentagi/graphiti/langfuse/observability: no-op when the checker already says the stack is up to date; otherwise (observability only) update the Jaeger storage plugin → `docker compose pull` → `docker compose up -d` → restart the `jaeger` service if that plugin was actually replaced → re-gather that stack's facts. Writing the image references is not a separate first step: the pull and the `up -d` each write them into `.env` themselves (see *Image references are written before the pull*)
  - worker: pull worker images only (no forced restart)
  - installer: same download-and-verify as `Download`; the running binary is never replaced
  - compose: the four compose stacks in order observability → langfuse → graphiti → pentagi
  - all: the compose stacks in that order, then worker, then installer
  - every update ends with a fresh update check followed by a verification report (see *Post-update verification*)

#### Image references are written before the pull

Every `image:` in the shipped compose files is `${<COMPONENT>_IMAGE:-<default>}`, so the variable — not the file — decides what `docker compose pull` fetches. `applyPullReferences(stack)` reads `PullReference` from the components of that stack in the last check answer and writes them into `.env` via `state.WriteVars`, logging every `NAME=reference` it wrote.

`WriteVars` and **not** `SetVars`. Everything else in the state package stages: `SetVar`/`SetVars` record an intention that reaches `.env` only when the user applies their changes. The pull that follows is `docker compose --env-file <.env>`, so a staged variable is one compose has never heard of — `${PENTAGI_IMAGE:-vxcontrol/pentagi:latest}` falls back to the default, the pull fetches something the server never offered, and the update reports success. `WriteVars` re-reads the file from disk, sets these values on it and saves, so the user's pending edits are neither applied nor lost, and the installation is not left dirty by a value the installer wrote for itself.

- The variable name is derived from the component name (upper case, `-` → `_`, suffix `_IMAGE`) rather than kept in a table, so a new component needs no second registration; a mismatch degrades to "the compose default was used", which `docker compose config` shows.
- It runs **before** the pull. Writing afterwards would pull the old reference and then record the new intention — an update that reports success and changes nothing.
- The installer does not choose a tag. `pull_reference` is ready to use as it arrives and is not always `repository:tag` — it may name a more specific, immutable tag than the one the compose file carries, and a client that rebuilt `repository:tag` locally would silently pull something it was never offered.
- Components the answer did not resolve carry no reference and are skipped. Writing an empty value would work (`${VAR:-default}` treats empty as absent) but would erase a reference an earlier answer pinned.
- A write failure is returned rather than logged and swallowed, for the same reason.
- Skipping is safe by construction: an installation that never got an answer falls back to the reference the compose file shipped with and behaves exactly as before.
- Every operation that fetches writes them, because the write lives in `updateStack` and `downloadStack` rather than at the call sites. There are more call sites than is comfortable — `Update` (which calls both), `Download`, `Install` and the four `ApplyChanges` phases — and the one that forgot would not be a compile error: `${COMPONENT_IMAGE:-default}` would quietly resolve to the shipped default and the operation would report success having fetched something the server never offered. The tear-down operations deliberately do not write: `down` needs no reference, and writing one while removing a stack would be a side effect nobody asked for.
- `ReferenceDrift()` is the separate query for the opposite problem: variables pointing at a different *repository* than this build ships. A different tag of the same repository is the mechanism working, not drift, so it is not reported; realignment is a deliberate user action, never a side effect of an update.

#### Jaeger storage plugin (observability only)

The plugin is a **file** mounted into the Jaeger container, which loads it over gRPC — no amount of image pulling updates it. Both architectures (`observability/jaeger/bin/jaeger-clickhouse-linux-amd64` and `-arm64`) are kept in step, because which one runs is decided inside the container by the architecture of the Docker VM and the entrypoint picks the file by `uname -m` at start.

- Fetched **before** the images. A plugin that cannot be downloaded aborts the stack update with nothing else disturbed; doing it afterwards would leave images pulled and the stack restarted around a plugin that never arrived.
- `updateJaegerPlugin` reports **whether it replaced anything**, and when it did, the update restarts the `jaeger` service after `up -d`. The `up -d` is not enough on its own: compose recreates a container only when its *spec* changes, and a different file inside a bind mount is not a spec change. The plugin is exec'd by the entrypoint at container start, so when the plugin was replaced but the Jaeger image was not, the container keeps running the plugin it loaded when it came up — the update looks applied while nothing about it is. The restart is skipped when nothing was replaced, so an observability update that only moves images does not bounce Jaeger for no reason.
- Only components the checker marked outdated are fetched. Membership in the answer is not the test — it also lists artefacts that are already current — so "download everything listed" would mean ~28MB per architecture on every observability update. Components whose OS is not linux are skipped as well: the file name says linux, and content for another platform would land under a linux name.
- Written to `<target>.new` **in the target's own directory** and renamed into place. The file being replaced is the one a running Jaeger loads, so an interrupted in-place write would leave a truncated plugin and the container would come back up with an opaque gRPC error. The staging file is a sibling because `os.Rename` is atomic only within one filesystem, and because writing an executable to the system temp directory and moving it is the pattern antivirus and EDR products score heavily.
- Created with mode `0755` **at creation**, not afterwards: `os.Rename` does not carry the destination's mode, the result keeps the staging file's, and a plugin created `0644` ends up non-executable — with Jaeger failing on a gRPC plugin error rather than a permission one.
- Leftover `.new` files are removed at the start of every run, whether or not anything is to be fetched: an interrupted download leaves ~28MB that nothing else would ever clean up, because integrity verification only walks files the installer ships. Removal is by exact name and never by glob — the directory is mounted into a container and an operator may keep their own files there.
- Both plugin paths are in `filesToExcludeFromVerification` with the "create if missing, never overwrite" policy. Without that the update would undo itself: the downloaded plugin differs from the embedded copy, so integrity verification calls it modified and under force — which is the path a stack update takes — copies the shipped version back over it.

#### Post-update verification

Every successful update answers one question in two steps: did the artefacts that are now installed become the ones that were offered?

- The targets are captured **before** the update runs, because the update replaces the answer; comparing against the new answer would compare the result with itself and always agree.
- **Step 1 — local re-check.** The components observed after the update are compared with the captured targets, and three outcomes are stated in different words: *matches the published build*, *no published digest to compare against* (the answer carried no digest, or nothing is installed — not a failure), and *expected X, got Y*.
- **Step 2 — a fresh check against the cloud.** It runs in the same deferred call, and its answer is what repaints the maintenance menu: only the server knows whether something newer appeared in the meantime.
- The comparison runs **after** the fresh check, because the fresh check settles the one legitimate disagreement. A moving tag can move between the answer and the pull, so what is installed may match no target and still be right. If the server no longer offers an update for the stack, the difference is reported as *newer than the offered build*; if the offer stands, it is a mismatch. A re-check that itself failed is no authority, so a difference stays a mismatch.
- Verification never fails the operation. The update either worked or reported an error; this is a report about the result, and turning a registry that moved ahead into a failed update would be worse than saying nothing.
- The installer stack is excluded from verification deliberately: its operation downloads a build without applying it, so comparing against the running binary would report the intended outcome as a mismatch.

#### Installer self-update

`Download(installer)` and `Update(installer)` are two menu entries over the same operation, and that operation ends at "downloaded and verified".

- The build is described first (`packages/info`), then streamed to `installer_<version>` (`.exe` on Windows) in the directory holding `.env`, with length, sha256 and Ed25519 signature checked as the bytes arrive. Progress is printed every 10% — one line per read would bury the surrounding output.
- **The running binary is never replaced.** A process cannot reliably replace the file it is executing: Linux refuses the write (`ETXTBSY`), Windows holds it locked, and getting it wrong leaves the user with no installer at all. Instead the screen prints the exact command — `mv "<downloaded>" "<running installer>"`, or `move /Y` on Windows, both paths quoted so an installation under a path with a space does not become two arguments. If `os.Executable` says nothing, no command is printed at all: a command with a guessed destination is worse than none.
- The version is part of the file name so a download cannot overwrite the installer in use, and an inode-based guard refuses to write to a file that *is* the running binary (Linux would answer `ETXTBSY`, macOS would silently truncate it).
- The file is written under its final name in the installation directory rather than through the system temp directory, for the same antivirus/EDR reason as the Jaeger plugin.
- An unfinished download is deleted — verification only completes on the last byte, so a file named like a release but unverified must not be left next to the installation — and the held package description is dropped with it, so a build republished under the same version does not make every retry fail identically.
- `InstallerPackage(ctx)` describes the offer without downloading: version, os/arch, size, the exact path the download will use, whether a file is already there, and the running binary's path. The screen needs the size *before* asking for consent; the description is remembered and reused by the download, so one press of the action does not pay for the same request twice (every call carries a proof-of-work challenge and counts against the account's daily allowance). The only thing the screen asks the user to confirm is overwriting a file that is already there.
- `removeInstaller` is the one operation still unimplemented: it returns an error.

### Removal Operations
- **Remove(stack)**:
  - pentagi/langfuse/observability: `docker compose down` (keep volumes/images)
  - worker: remove images via Docker API and related containers
  - installer: remove flow stubbed
  - all: remove all stacks

- **Purge(stack)**:
  - pentagi/langfuse/observability: `down --rmi all -v` for strict purge; standard purge `down -v` is also available
  - worker: remove all containers, images, and volumes in worker environment
  - installer: complete removal flow stubbed
  - all: purge all stacks and remove custom networks

### State Management
- **ApplyChanges()**:
  - pre-phase (wizard): integrity scan, user selects overwrite (force) or keep (no force)
  - phase 1: observability (ensure/verify files → update stack or remove if external/disabled)
  - phase 2: langfuse (same logic; local start requires `LangfuseConnected`)
  - phase 3: graphiti (same logic; local start requires `GraphitiConnected`)
  - phase 4: pentagi (always embedded; ensure/verify → update)
  - refresh checker state after each phase
  - `Install` reuses the same four phases, each guarded by "not installed yet"

- **ResetChanges()**:
  - Call `state.State.Reset()` to discard pending configuration changes
  - Preserve committed state from previous successful operations
  - Reset wizard navigation stack to last stable point

## Implementation Strategy

### High-Level Method Organization
Each specialized file should contain business-logic level methods that directly support processor interface operations, not low-level utilities.

### Stack-Specific Operations
- **compose.go**:
  - `startStack(stack)`, `stopStack(stack)`, `restartStack(stack)` - orchestrate docker compose commands
  - `updateStack(stack)` / `downloadStack(stack)` - the only two operations that fetch, and the two that write the chosen image references first
  - `removeStack(stack)` / `purgeStack(stack)` / `purgeImagesStack(stack)` - `down`, `down -v`, `down --rmi all -v`
- **docker.go**:
  - `pullWorkerImage()` - download pentest image (DOCKER_DEFAULT_IMAGE_FOR_PENTEST: vxcontrol/kali-linux)
  - `pullDefaultImage()` - download general image (DOCKER_DEFAULT_IMAGE: debian:latest)
  - `removeWorkerContainers()` - cleanup running worker containers (ports 28000-32000 range)
  - `purgeWorkerImages()` - complete image removal including fallback images
- **fs.go**:
  - `ensureStackIntegrity(stack, force)` - create missing files, force update existing ones
  - `verifyStackIntegrity(stack, force)` - validate existing files, update if force=true
  - `cleanupStackFiles(stack)` - remove extracted files and directories
  - File integrity validation with YAML syntax checking
  - Embedded directory tree handling for observability stack
- **update.go**:
  - `checkUpdates()` - refresh what the PentAGI Cloud API says about this installation; the answer lands on the shared `checker.CheckResult` instead of being returned, so there is only one copy of it
  - `installerPackage()` - describe the offered build (version, platform, size, target path) without downloading it
  - `downloadInstaller()` / `updateInstaller()` - the same operation under two names: fetch `installer_<version>` and verify it; the running binary is left alone
  - `updateJaegerPlugin()` - bring the Jaeger storage plugin files up to the offered version (in `jaeger_plugin.go`)
- **pull_reference.go**:
  - `applyPullReferences(stack)` - write the chosen image references into `.env` before a pull
- **verify.go**:
  - `captureStackTargets(result, stack)` / `verifyStackUpdate(stack, targets, state)` - the two halves of the post-update comparison
- **logic.go**:
  - `remove(stack)` / `purge(stack)` - the removal orchestration; the compose helpers above do the work

### Critical Implementation Details
- **Two-Track Command Execution**:
  - Worker stack: Docker API SDK (uses DOCKER_HOST, PENTAGI_DOCKER_CERT_PATH, DOCKER_TLS_VERIFY from config)
  - Compose stacks: Console commands with live output streaming to TUI
- **TUI Integration Modes**:
  - **Embedded Terminal**: Real-time pseudoterminal integration (`ProcessorTerminalModel`) via `github.com/creack/pty`
  - **Message Channel**: Simple progress tracking via `ProcessorMessage` events through channels
  - **Toggle Support**: Ctrl+T switches between modes for debugging/compatibility
- **Deployment Mode Handling**:
  - Langfuse: embedded (full stack), external (existing server), disabled (no analytics); local start guarded by `LangfuseConnected`
  - Observability: embedded (full stack), external (OTEL collector), disabled (no monitoring)
- **Environment Variable Handling**: Compose files use --env-file parameter for environment variables, only special cases require file patching. Image references are part of this: every `image:` is `${<COMPONENT>_IMAGE:-<default>}`, and every operation that fetches writes those variables from the cloud answer into `.env` before pulling — bypassing the staging area, since compose reads the file
- **Progress Tracking**: Worker downloads (vxcontrol/kali-linux 6GB+ → 13GB disk) with real-time progress via terminal; file downloads (installer build, Jaeger plugin) report every 10% against the size the package description gave
- **Docker Configuration**: Support NET_ADMIN capability for network scanning, Docker socket access for container management
- **Dependency Ordering**: PentAGI must start before Langfuse/Observability for network creation
- **State Persistence**: All operations update checker.CheckResult and state.State for consistency
- **Atomicity**: Only file downloads are atomic (staged sibling + rename for the Jaeger plugin, delete-on-failure for the installer build). Compose operations are not reversible — a failed update leaves whatever compose managed to do

### Integration Patterns
- **Wizard → Processor**: Called via `wizard.controllers.StateController` on user action
- **State Coordination**: Processor updates both `state.State` (configuration) and internal state tracking
- **File System Layout**: Working directory contains .env + extracted compose files + .state/ subdirectory
- **Container Naming**: Follows patterns in checker constants (PentagiContainerName, etc.)

### Key Environment Variables
- **LLM providers**: OPEN_AI_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, BEDROCK_*, DEEPSEEK_*, GLM_*, KIMI_*, QWEN_*, OLLAMA_SERVER_URL
- **Provider configs**: PENTAGI_LLM_SERVER_CONFIG_PATH (host path), PENTAGI_OLLAMA_SERVER_CONFIG_PATH (host path), PENTAGI_BEDROCK_CONFIG_PATH (host path)
- **Monitoring**: LANGFUSE_BASE_URL, LANGFUSE_PROJECT_ID, OTEL_HOST
- **Docker config**: DOCKER_HOST, PENTAGI_DOCKER_CERT_PATH (host path), DOCKER_TLS_VERIFY, DOCKER_CERT_PATH (container path, managed)
- **Deployment modes**: envs determine embedded vs external vs disabled
- **Worker images**: DOCKER_DEFAULT_IMAGE (debian:latest), DOCKER_DEFAULT_IMAGE_FOR_PENTEST (vxcontrol/kali-linux)
- **Path migration**: DoMigrateSettings() migrates old DOCKER_CERT_PATH/LLM_SERVER_CONFIG_PATH/OLLAMA_SERVER_CONFIG_PATH/BEDROCK_CONFIG_PATH to PENTAGI_* variants on startup

### Error Handling Strategy
- **Validation errors**: validate stack applicability before operation
- **System errors**: Docker API failures, file permissions, network issues
- **State conflicts**: partial installations, conflicting configs, version mismatches
- **Rollback logic**: none for compose operations — they are fail-fast. Only file downloads are protected: the Jaeger plugin is staged next to its target and renamed only after a clean write, and an unverified installer download is deleted rather than left behind

## Integration Points

### Dependencies
- **checker package**: System state assessment via `checker.CheckResult`
- **state package**: Configuration management via `state.State`
- **files package**: Embedded content access via `files.Files`
- **loader package**: .env file operations

### External Systems
- Docker Compose CLI for stack orchestration
- Docker API for container/image management
- HTTP client (`cloud` package) for update checks and file downloads from the PentAGI Cloud API
- File system for content extraction and cleanup

## ApplyChanges State Machine

Deterministic operation sequence based on current vs target configuration:

```mermaid
graph TD
    Start([ApplyChanges Called]) --> Assess[Assess Current State]
    Assess --> Compare[Compare with Target State]
    Compare --> Plan[Generate Operation Plan]

    Plan --> NeedInstall{Need Install?}
    NeedInstall -->|Yes| Install[Install Stack Files]
    NeedInstall -->|No| NeedDownload{Need Download?}
    Install --> NeedDownload

    NeedDownload -->|Yes| Download[Download Images/Binaries]
    NeedDownload -->|No| NeedUpdate{Need Update?}
    Download --> NeedUpdate

    NeedUpdate -->|Yes| Update[Update Running Services]
    NeedUpdate -->|No| NeedStart{Need Start?}
    Update --> NeedStart

    NeedStart -->|Yes| Start[Start Services]
    NeedStart -->|No| Success[Operation Complete]
    Start --> Success

    Install --> Rollback{Error?}
    Download --> Rollback
    Update --> Rollback
    Start --> Rollback
    Rollback -->|Yes| Cleanup[Rollback Changes]
    Rollback -->|No| Success
    Cleanup --> Failure[Operation Failed]
```

### Decision Matrix
- **Fresh Install**: Install → Download → Start
- **Update Available**: Download → Update
- **Configuration Change**: Install → Update → Restart
- **Multi-Stack Setup**: Sequential operations with dependency handling

## User Scenarios & Integration

### Primary Use Cases
1. **First-time Installation**: User runs installer, configures via TUI, calls ApplyChanges to deploy complete stack
2. **Configuration Updates**: User modifies .env settings via TUI, ApplyChanges determines minimal required operations
3. **Stack Management**: User enables/disables Langfuse or Observability, system installs/removes appropriate components
4. **System Updates**: Periodic update checks trigger Download/Update operations for newer versions
5. **Troubleshooting**: Remove/Install cycles for component reset, Purge for complete cleanup

### Wizard Integration Flow
1. `main.go` initializes state, checker, launches wizard
2. `wizard.App` provides TUI for configuration changes
3. `wizard.controllers.StateController` manages state modifications
4. User triggers "Apply Changes" → wizard runs integrity scan (Enter), prompts user for update decision (Y/N), then executes `processor.ApplyChanges()` with/without force
5. Operations execute with real-time feedback to TUI; Ctrl+C cancels integrity stage only
6. `state.Commit()` persists successful changes, `state.Reset()` on failure

### Processor Usage in Wizard

**Controller Integration**: StateController creates processor instance and delegates operations
```go
// wizard/controllers/state_controller.go
type StateController struct {
    state     *state.State
    checker   *checker.CheckResult
    processor processor.Processor
}

func (c *StateController) ApplyUserChanges() error {
    return c.processor.ApplyChanges(context.Background(),
        processor.WithForce(), // User explicitly requested changes
    )
}
```

**Screen-Level Operations**: Modern approach with embedded terminal model
```go
// wizard/models/apply_changes.go (current implementation)
type ApplyChangesFormModel struct {
    processor           processor.Processor
    terminalModel       processor.ProcessorTerminalModel
    useEmbeddedTerminal bool
}

func (m *ApplyChangesFormModel) startApplyProcess() tea.Cmd {
    if m.useEmbeddedTerminal && m.terminalModel != nil {
        return m.terminalModel.StartOperation("ApplyChanges", processor.ProductStackAll)
    }

    // Fallback for message-based integration
    messageChan := make(chan processor.ProcessorMessage, 100)
    return processor.CreateApplyChangesCommand(m.processor,
        processor.WithForce(),
        processor.WithTea(messageChan),
    )
}

// Terminal model integration in Update method
func (m *ApplyChangesFormModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    if m.useEmbeddedTerminal && m.terminalModel != nil {
        updatedModel, terminalCmd := m.terminalModel.Update(msg)
        if terminalModel, ok := updatedModel.(processor.ProcessorTerminalModel); ok {
            m.terminalModel = terminalModel
        }
        return m, terminalCmd
    }
    // Handle other cases...
}
```

**Real-Time Integration**: Terminal model provides native real-time feedback
```go
// No manual polling required - terminal model handles real-time updates automatically
func (m *ApplyChangesFormModel) renderTerminalPanel() string {
    if m.useEmbeddedTerminal && m.terminalModel != nil {
        return m.terminalModel.View() // Complete terminal interface
    }
    return m.renderFallbackPanel() // Message-based fallback
}

// Dynamic resizing support
case tea.WindowSizeMsg:
    if m.useEmbeddedTerminal && m.terminalModel != nil {
        contentWidth, contentHeight := m.getViewportFormSize()
        m.terminalModel.SetSize(contentWidth-2, contentHeight-2)
    }
```

## Implementation Requirements

### CommandOption Design

```go
type commandConfig struct {
    // Execution control
    Force        bool                     // Skip validation checks and attempt maximum operations

    // TUI Integration - two integration modes available
    Tea          chan ProcessorMessage    // Message-based integration via channel
    TerminalModel ProcessorTerminalModel  // Embedded terminal model integration
}

// Simplified option pattern (only essential options)
type CommandOption func(*commandConfig)

func WithForce() CommandOption {
    return func(c *commandConfig) { c.Force = true }
}

func WithTea(messageChan chan ProcessorMessage) CommandOption {
    return func(c *commandConfig) { c.Tea = messageChan }
}

func WithTerminalModel(terminal ProcessorTerminalModel) CommandOption {
    return func(c *commandConfig) { c.TerminalModel = terminal }
}
```

### Update Server Protocol
- Every call goes through the `cloud` package to the PentAGI Cloud API at `UPDATE_SERVER_HOST` (default `update.pentagi.com`), always honoring `PROXY_URL` — the installer's own proxy setting, which the process environment may not carry. `SUPPORT_SERVER_HOST` (default `support.pentagi.com`) is written by the same settings form and belongs to the Support API; no call here targets it yet
- `/api/v1/proxy/updates/check` reports the installed components and answers per stack: whether an update exists, the version transition, and per component the action, the pull reference and the target digest
- `/api/v1/proxy/packages/info` then `/api/v1/proxy/packages/download` fetch a file (installer build, Jaeger storage plugin). The description carries the size, sha256 and signature the stream is checked against; that size is also the only length the client ever learns, since the encrypted body carries no `Content-Length`, so download progress is computed from it
- No atomic self-replacement and no post-update exit: the downloaded installer is left on disk with the command to move it into place

### Critical Safety Measures
- **Pre-flight Checks**: Validate system resources before major operations
- **Backup Strategy**: Create backups before destructive operations (Purge)
- **Network Isolation**: Respect proxy settings from environment configuration
- **Permission Handling**: Graceful handling of Docker socket access requirements

### Current Architecture

**Core Components**:
- **processor.go**: Interface implementation and delegation
- **model.go**: ProcessorModel for tea.Cmd wrapping
- **logic.go**: Business logic (ApplyChanges, lifecycle operations)
- **state.go**: Operation state management
- **compose.go, docker.go, fs.go, update.go**: Specialized operations
- **pull_reference.go, jaeger_plugin.go, verify.go, reference_drift.go**: The update path — image references, plugin file, post-update comparison, drift reporting

**Testing Infrastructure**:
- **fixtures_test.go**: Comprehensive mocks with call tracking
- **logic_test.go**: Business logic tests
- **fs_test.go**: File system operation tests
- **update_test.go, pull_reference_test.go, jaeger_plugin_test.go, verify_test.go**: Download progress and package description reuse, image reference selection, plugin staging and cleanup, the four verification outcomes
- **Mock CheckHandler**: Flexible system state simulation

**Integration Points**:
- **Wizard**: Uses ProcessorModel with terminal integration
- **Checker**: Uses CheckHandler interface for state assessment
- **State**: Configuration management and persistence

---

### Backend Docs Installer Reference Config Pattern (`backend/docs/installer/reference-config-pattern.md`)

# Reference Configuration Pattern

> Reference implementation of configuration using ScraperConfig as an example for all future configurations in StateController.

## 🎯 **Core Principles**

### 1. **Use loader.EnvVar for direct form mapping**
```go
type ReferenceConfig struct {
    // ✅ Direct form mapping - use loader.EnvVar
    DirectField1 loader.EnvVar // SOME_ENV_VAR
    DirectField2 loader.EnvVar // ANOTHER_ENV_VAR

    // ✅ Computed fields - simple types
    ComputedMode string // computed from DirectField1

    // ✅ Temporary data for processing
    TempData string // not saved, used for logic
}
```

### 2. **Reference configuration structure**
```go
// ScraperConfig represents scraper configuration settings
type ScraperConfig struct {
    // direct form field mappings using loader.EnvVar
    // these fields directly correspond to environment variables and form inputs (not computed)
    PublicURL             loader.EnvVar // SCRAPER_PUBLIC_URL
    PrivateURL            loader.EnvVar // SCRAPER_PRIVATE_URL
    LocalUsername         loader.EnvVar // LOCAL_SCRAPER_USERNAME
    LocalPassword         loader.EnvVar // LOCAL_SCRAPER_PASSWORD
    MaxConcurrentSessions loader.EnvVar // LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS

    // computed fields (not directly mapped to env vars)
    // these are derived from the above EnvVar fields
    Mode string // "embedded", "external", "disabled" - computed from PrivateURL

    // parsed credentials for external mode (extracted from URLs)
    PublicUsername  string
    PublicPassword  string
    PrivateUsername string
    PrivatePassword string
}
```

### 3. **Constants for default values**
```go
const (
    DefaultScraperBaseURL = "https://scraper/"
    DefaultScraperDomain  = "scraper"
    DefaultScraperSchema  = "https"
)
```

## 🔧 **Configuration Methods**

### 1. **GetConfig() - Retrieve configuration**
```go
func (c *StateController) GetScraperConfig() *ScraperConfig {
    // get all environment variables using the state controller
    publicURL, _ := c.GetVar("SCRAPER_PUBLIC_URL")
    privateURL, _ := c.GetVar("SCRAPER_PRIVATE_URL")
    localUsername, _ := c.GetVar("LOCAL_SCRAPER_USERNAME")
    localPassword, _ := c.GetVar("LOCAL_SCRAPER_PASSWORD")
    maxSessions, _ := c.GetVar("LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS")

    config := &ScraperConfig{
        PublicURL:             publicURL,
        PrivateURL:            privateURL,
        LocalUsername:         localUsername,
        LocalPassword:         localPassword,
        MaxConcurrentSessions: maxSessions,
    }

    // compute derived fields using multiple inputs
    config.Mode = c.determineScraperMode(privateURL.Value, publicURL.Value)

    // for external mode, extract credentials from URLs
    if config.Mode == "external" {
        config.PublicUsername, config.PublicPassword = c.extractCredentialsFromURL(publicURL.Value)
        config.PrivateUsername, config.PrivatePassword = c.extractCredentialsFromURL(privateURL.Value)
    }

    return config
}
```

### 2. **UpdateConfig() - Update configuration**
```
/* Detailed source-code truncated for AI context efficiency. */
```

### 3. **ResetConfig() - Reset to defaults**
```go
func (c *StateController) ResetScraperConfig() *ScraperConfig {
    // reset all scraper-related environment variables to their defaults
    vars := []string{
        "SCRAPER_PUBLIC_URL",
        "SCRAPER_PRIVATE_URL",
        "LOCAL_SCRAPER_USERNAME",
        "LOCAL_SCRAPER_PASSWORD",
        "LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS",
    }

    if err := c.ResetVars(vars); err != nil {
        return nil
    }

    return c.GetScraperConfig()
}
```

## 🧩 **Helper Methods**

### 1. **Mode determination with multiple inputs**
```go
func (c *StateController) determineScraperMode(privateURL, publicURL string) string {
    if privateURL == "" && publicURL == "" {
        return "disabled"
    }

    parsedURL, err := url.Parse(privateURL)
    if err != nil {
        return "external"
    }

    if parsedURL.Scheme == DefaultScraperSchema && parsedURL.Hostname() == DefaultScraperDomain {
        return "embedded"
    }

    return "external"
}
```

### 2. **URL credential handling with defaults**
```go
func (c *StateController) addCredentialsToURL(urlStr, username, password string) string {
    if username == "" || password == "" {
        return urlStr
    }

    if urlStr == "" {
        urlStr = DefaultScraperBaseURL
    }

    parsedURL, err := url.Parse(urlStr)
    if err != nil {
        return urlStr
    }

    // set user info
    parsedURL.User = url.UserPassword(username, password)

    return parsedURL.String()
}
```

### 3. **Public method for safe display**
```go
// RemoveCredentialsFromURL removes credentials from URL - public method for form display
func (c *StateController) RemoveCredentialsFromURL(urlStr string) string {
    if urlStr == "" {
        return urlStr
    }

    parsedURL, err := url.Parse(urlStr)
    if err != nil {
        return urlStr
    }

    parsedURL.User = nil
    return parsedURL.String()
}
```

## 📋 **Form Integration**

### 1. **Field creation**
```go
func (m *FormModel) createURLField(key, title, description, placeholder string) FormField {
    input := textinput.New()
    input.Placeholder = placeholder

    var value string
    switch key {
    case "public_url":
        value = m.config.PublicURL.Value
    case "private_url":
        value = m.config.PrivateURL.Value
    }

    if value != "" {
        input.SetValue(value)
    }

    return FormField{
        Key:         key,
        Title:       title,
        Description: description,
        Input:       input,
        Value:       input.Value(),
    }
}
```

### 2. **Save handling with validation**
```
/* Detailed source-code truncated for AI context efficiency. */
```

### 3. **Safe display in overview**
```go
func (m *ScraperFormModel) GetFormOverview() string {
    config := m.GetController().GetScraperConfig()

    var sections []string
    sections = append(sections, "Current Configuration:")

    switch config.Mode {
    case "embedded":
        sections = append(sections, "• Mode: Embedded")
        if config.PublicURL.Value != "" {
            sections = append(sections, "• Public URL: " + config.PublicURL.Value)
        }
        if config.LocalUsername.Value != "" {
            sections = append(sections, "• Local Username: " + config.LocalUsername.Value)
        }

    case "external":
        sections = append(sections, "• Mode: External")
        if config.PublicURL.Value != "" {
            // show clean URL without credentials for security
            cleanURL := m.GetController().RemoveCredentialsFromURL(config.PublicURL.Value)
            sections = append(sections, "• Public URL: " + cleanURL)
        }
        if config.PrivateURL.Value != "" {
            cleanURL := m.GetController().RemoveCredentialsFromURL(config.PrivateURL.Value)
            sections = append(sections, "• Private URL: " + cleanURL)
        }

    case "disabled":
        sections = append(sections, "• Mode: Disabled")
    }

    return strings.Join(sections, "\n")
}
```

## ✅ **Benefits of Reference Approach**

1. **State tracking**: `loader.EnvVar` tracks changes, file presence, default values
2. **Metadata preservation**: Information about changes, presence, defaults
3. **Security**: Public methods for safe display of sensitive data
4. **Consistency**: Uniform behavior across all configurations
5. **Reliability**: Minimizes errors in different usage scenarios
6. **URL handling**: Uses `net/url` package for robust URL parsing
7. **Default management**: Constants for maintainable default values

## 🔄 **Key Patterns**

### 1. **Data Types**
| Field Type | Data Type | Usage |
|------------|-----------|-------|
| **Direct mapping** | `loader.EnvVar` | Form fields, env variables |
| **Computed** | `string`/`bool`/`int` | Modes, status, flags |
| **Temporary** | `string` | Parsing, processing |

### 2. **Method signatures**
- `GetConfig() *Config` - retrieves with metadata
- `UpdateConfig(config *Config) error` - saves with validation
- `ResetConfig() *Config` - resets to defaults
- `PublicMethod()` - exported for form usage

### 3. **Error handling**
- Always validate input parameters
- Use `fmt.Errorf` with context
- Handle URL parsing errors gracefully
- Provide meaningful error messages

## 🚀 **Creating New Configurations**

```go
// 1. Define structure
type NewServiceConfig struct {
    // direct fields
    APIKey    loader.EnvVar // NEW_SERVICE_API_KEY
    BaseURL   loader.EnvVar // NEW_SERVICE_BASE_URL
    Enabled   loader.EnvVar // NEW_SERVICE_ENABLED

    // computed fields
    IsConfigured bool
}

// 2. Add constants
const (
    DefaultNewServiceURL = "https://api.newservice.com"
)

// 3. Implement methods
func (c *StateController) GetNewServiceConfig() *NewServiceConfig { /* ... */ }
func (c *StateController) UpdateNewServiceConfig(config *NewServiceConfig) error { /* ... */ }
func (c *StateController) ResetNewServiceConfig() *NewServiceConfig { /* ... */ }

// 4. Create form following ScraperFormModel pattern
```

This reference approach ensures reliable and consistent operation of all configurations in the system.

---

### Backend Docs Installer Terminal Wizard Integration (`backend/docs/installer/terminal-wizard-integration.md`)

# Terminal Integration Guide for Wizard Screens

This guide covers integration of the `terminal` package into wizard configuration screens, providing command execution capabilities with real-time UI updates.

## Core Architecture

The terminal system consists of three layers:
- **Virtual Terminal (VT)**: Low-level ANSI parsing and screen management (`terminal/vt/`)
- **Terminal Interface**: High-level command execution with PTY/pipe support (`terminal/`)
- **Wizard Integration**: Screen-specific integration patterns (`wizard/models/`)

## Terminal Modes

### PTY Mode (Default on Unix)
- Full pseudoterminal emulation via `creack/pty`
- ANSI escape sequence processing through VT layer
- Interactive command support (vim, less, etc.)
- Proper terminal environment variables

### Pipe Mode (Windows/NoPty)
- Standard stdin/stdout/stderr pipes
- Line-by-line output processing
- Simpler but limited interactivity
- Plain text output handling

## Configuration Options

Terminal behavior is controlled via functional options:

```go
// Essential options for wizard integration
terminal.NewTerminal(width, height,
    terminal.WithAutoScroll(),     // Auto-scroll to bottom on updates
    terminal.WithAutoPoll(),       // Continuous update polling
    terminal.WithCurrentEnv(),     // Inherit process environment
)

// Advanced options
terminal.WithNoStyled()            // Disable ANSI styling (PTY only)
terminal.WithNoPty()              // Force pipe mode
terminal.WithStyle(lipgloss.Style) // Custom viewport styling
```

## Integration Patterns

### Complete Integration Template

```go
type YourFormModel struct {
    *BaseScreen
    terminal terminal.Terminal
    // other screen-specific fields
}

func NewYourFormModel(controller *controllers.StateController, styles *styles.Styles, window *window.Window, args []string) *YourFormModel {
    m := &YourFormModel{}
    m.BaseScreen = NewBaseScreen(controller, styles, window, args, m, nil)
    return m
}

// Required BaseScreenHandler implementation
func (m *YourFormModel) BuildForm() tea.Cmd {
    contentWidth, contentHeight := m.getViewportFormSize()

    // Initialize or reset terminal
    if m.terminal == nil {
        m.terminal = terminal.NewTerminal(
            contentWidth-4,  // Account for border + padding
            contentHeight-1, // Account for border
            terminal.WithAutoScroll(),
            terminal.WithAutoPoll(),
            terminal.WithCurrentEnv(),
        )
    } else {
        m.terminal.Clear()
    }

    // Set initial content
    m.terminal.Append("Terminal initialized...")

    // CRITICAL: Return terminal init for update subscription (idempotent)
    // repeated calls to Init() are safe: only a single waiter will receive
    // the next TerminalUpdateMsg; others will return nil quietly
    return m.terminal.Init()
}
```

### Sizing Calculations

The sizing adjustments account for UI elements:
- **Width -4**: Left border (1) + left padding (1) + right padding (1) + right border (1)
- **Height -1**: Top/bottom borders, content area needs space for text
- Use `m.getViewportFormSize()` from BaseScreen for consistent calculations
- Handle dynamic resizing in Update() method

### Event Flow Architecture

The system uses a single-waiter update notifier for real-time updates:

1. **Update Notifier**: Manages single-waiter update notifications (`teacmd.go`)
2. **Update Messages**: `TerminalUpdateMsg` carries terminal ID
3. **Subscription Model**: Commands wait for `release()` signalling the next update
4. **Auto-polling**: Continuous listening when `WithAutoPoll()` enabled
5. **Single-waiter guarantee**: For a given `Terminal`, at most one pending waiter is active at any time. Multiple `Init()` calls are safe; only one will receive the next `TerminalUpdateMsg` after `release()`, others return nil.

### Complete Update Method Implementation

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Event Processing Order

Process events in this strict order to ensure proper functionality:

1. **Window Resize**: Update terminal dimensions before any other processing
2. **Terminal Updates**: Handle `TerminalUpdateMsg` immediately for real-time updates
3. **Key Routing**: Route based on `IsRunning()` state
   - **Running commands**: All keys forwarded to terminal for interaction
   - **Idle terminal**: Screen hotkeys first, then terminal scrolling
4. **Other Messages**: Delegate to terminal for potential internal handling

## Command Execution

### Single Command with Error Handling
```go
func (m *YourFormModel) executeCommand() {
    cmd := exec.Command("echo", "hello")

    err := m.terminal.Execute(cmd)
    if err != nil {
        // Display error to user through terminal
        m.terminal.Append(fmt.Sprintf("❌ Command failed: %v", err))
        m.terminal.Append("Please check the command and try again.")
        return
    }

    // Wait for completion if needed
    go func() {
        for m.terminal.IsRunning() {
            time.Sleep(100 * time.Millisecond)
        }
        m.terminal.Append("✅ Command completed successfully")
    }()
}
```

### Sequential Commands with Robust Error Handling
```
/* Detailed source-code truncated for AI context efficiency. */
```go
// Terminal lifecycle follows screen lifecycle
func (m *YourFormModel) BuildForm() tea.Cmd {
    // Create terminal once per screen instance
    if m.terminal == nil {
        m.terminal = terminal.NewTerminal(...)
    } else {
        // Reset content when re-entering screen
        m.terminal.Clear()
    }
    return m.terminal.Init()
}

// No manual cleanup needed - handled by finalizers
// Terminal will be cleaned up when screen model is garbage collected
```

### Screen Navigation Considerations
- **Terminal Persistence**: Terminal remains active during screen navigation
- **Content Reset**: Use `Clear()` when re-entering screens to avoid content buildup
- **Resource Cleanup**: Automatic via finalizers when screen model is destroyed
- **State Preservation**: Terminal state (size, options) persists across `BuildForm()` calls

### State Checking and Debugging
```go
// Essential state checks
if m.terminal == nil {
    // Terminal not initialized - call BuildForm()
}

if m.terminal.IsRunning() {
    // Command is executing - avoid new commands
    // Show spinner or disable UI elements
}

// Debugging helpers
terminalID := m.terminal.ID()           // Unique identifier for logging
width, height := m.terminal.GetSize()  // Current dimensions
view := m.terminal.View()               // Current rendered content

// For debugging terminal content
if DEBUG {
    log.Printf("Terminal %s: %dx%d, running=%t",
        terminalID, width, height, m.terminal.IsRunning())
}
```

### Resource Management Details
Resources managed via Go finalizers (`terminal.go:131-142`):
- **PTY file descriptors**: Automatically closed when terminal is garbage collected
- **Process termination**: Running processes killed during cleanup
- **Notifier shutdown**: Wait channel closed and state reset
- **Mutex-protected cleanup**: Thread-safe resource cleanup
- **No manual Close()**: Resources cleaned automatically, no explicit cleanup needed

## Virtual Terminal Capabilities

The VT layer provides advanced features:
- **Screen Buffer**: Main and alternate screen support
- **Scrollback**: Configurable history buffer
- **ANSI Processing**: Full VT100/xterm compatibility
- **Color Support**: 256-color palette + true color
- **Cursor Modes**: Various cursor styles and visibility
- **Character Sets**: GL/GR charset switching

## Testing Strategies

Key test patterns from `terminal_test.go`:
- **Command Output**: Verify content appears in `View()`
- **Interactive Input**: Simulate key sequences via `Update()`
- **Resource Cleanup**: Manual finalizer calls for verification
- **Concurrent Access**: `Execute` refuses a second command while one runs
- **Error Handling**: Invalid commands and process failures

## Concurrency and Threading

### Thread Safety
```go
// Terminal methods are thread-safe for these operations:
m.terminal.Append("message")       // Safe from any goroutine
m.terminal.IsRunning()            // Safe to check from any goroutine
m.terminal.ID()                   // Safe to call from any goroutine

// UI operations must be on main thread:
m.terminal.Update(msg)            // Only from main BubbleTea thread
m.terminal.View()                 // Only from main rendering thread
m.terminal.SetSize(w, h)          // Only from main thread
```

### Command Execution Patterns
```go
// CORRECT: Run commands in separate goroutine
go func() {
    m.terminal.Append("Starting long operation...")
    cmd := exec.Command("long-running-command")
    err := m.terminal.Execute(cmd)
    // Error handling...
}()

// INCORRECT: Blocking main thread
cmd := exec.Command("long-running-command")
m.terminal.Execute(cmd) // Will block UI updates
```

### AutoPoll vs Manual Updates
- **WithAutoPoll()**: Continuous listening, higher CPU but immediate updates; still single-waiter per terminal ensures no message storm. Updates are triggered internally via `release()` when content changes.
- **Manual polling**: Call `terminal.Init()` only when needed, lower resource usage
- **Use AutoPoll**: For active terminal screens with frequent updates
- **Skip AutoPoll**: For background or rarely updated terminals

## Troubleshooting Guide

### Terminal Not Updating
**Problem**: Terminal content doesn't appear or update

**Solutions**:
1. Ensure `terminal.Init()` is returned from `BuildForm()`
2. Check `TerminalUpdateMsg` handling in `Update()` method — it should return next wait command to continue listening
3. Verify `handleTerminal()` function calls `RestoreModel()`
4. Add debug logging to track message flow

```go
case terminal.TerminalUpdateMsg:
    log.Printf("Received terminal update: %s", msg.ID)
    return handleTerminal(msg)
```

### Commands Not Executing
**Problem**: `Execute()` returns nil but nothing happens

**Solutions**:
1. Check if previous command is still running: `m.terminal.IsRunning()`
2. Verify command path and arguments
3. Check terminal initialization
4. Add error logging and terminal output

```go
if m.terminal.IsRunning() {
    m.terminal.Append("⚠️ Previous command still running")
    return
}
```
/* Detailed source-code truncated for AI context efficiency. */
```go
// Creation and lifecycle
terminal.NewTerminal(width, height, options...)
m.terminal.Init()                    // Subscribe to updates
m.terminal.Clear()                   // Reset content

// Command execution
m.terminal.Execute(cmd)              // Run command
m.terminal.IsRunning()               // Check execution status
m.terminal.Append(text)              // Add content

// UI integration
m.terminal.Update(msg)               // Handle messages
m.terminal.View()                    // Render content
m.terminal.SetSize(width, height)    // Update dimensions
```

### Common Error Patterns to Avoid
- ❌ Creating multiple terminals per screen
- ❌ Running `Execute()` on main thread for long commands
- ❌ Forgetting to return `terminal.Init()` from `BuildForm()`
- ❌ Not handling `TerminalUpdateMsg` in `Update()`
- ❌ Calling UI methods from background goroutines
- ❌ Manual resource cleanup (use finalizers instead)

### Integration Checklist for New Screens
1. ✅ Add `terminal terminal.Terminal` to model struct
2. ✅ Initialize in `BuildForm()` with proper sizing
3. ✅ Return `terminal.Init()` from `BuildForm()`
4. ✅ Handle `TerminalUpdateMsg` first in `Update()`
5. ✅ Implement proper key routing based on `IsRunning()`
6. ✅ Handle window resize events
7. ✅ Run commands in goroutines with error handling
8. ✅ Add progress feedback via `Append()`

This comprehensive architecture provides robust terminal integration with wizard screens while maintaining proper resource management, real-time UI updates, and cross-platform compatibility.

---

--- METRICS ---
- Files Extracted: 40
- Estimated Token Budget: ~336120 tokens
- Canonical Reference: https://codewiki.google/github.com/vxcontrol/pentagi