# Technical Documentation: tailcallhq/forgecode > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/tailcallhq/forgecode) ### Overview (`README.md`)

⚒️ Forge: AI-Enhanced Terminal Development Environment

A comprehensive coding agent that integrates AI capabilities with your development environment

curl -fsSL https://forgecode.dev/cli | sh

[](https://github.com/tailcallhq/forgecode/actions) [](https://github.com/tailcallhq/forgecode/releases) [](https://discord.gg/kRZBPpkgwq) [](https://cla-assistant.io/tailcallhq/forgecode) ---
Table of Contents - [Quickstart](#quickstart) - [Usage Examples](#usage-examples) - [Why Forge?](#why-forge) - [How Forge Works: Three Modes](#how-forge-works-three-modes) - [Interactive Mode (TUI)](#interactive-mode-tui) - [One-Shot CLI Mode](#one-shot-cli-mode) - [ZSH Plugin Mode (`:` prefix)](#zsh-plugin-mode--prefix) - [ZSH Plugin: The `:` Prefix System](#zsh-plugin-the--prefix-system) - [Agents](#agents) - [Sending Prompts](#sending-prompts) - [Attaching Files](#attaching-files) - [Conversation Management](#conversation-management) - [Git Integration](#git-integration) - [Shell Command Tools](#shell-command-tools) - [Session & Configuration](#session--configuration) - [Skills](#skills) - [Customizing Agent Behavior](#customizing-agent-behavior) - [Semantic Search (Workspace)](#semantic-search-workspace) - [Quick Reference: All `:` Commands](#quick-reference-all--commands) - [Command-Line Options](#command-line-options) - [Advanced Configuration](#advanced-configuration) - [Provider Configuration](#provider-configuration) - [Managing Provider Credentials](#managing-provider-credentials) - [Deprecated: Environment Variables](#deprecated-environment-variables) - [forge.yaml Configuration Options](#forgeyaml-configuration-options) - [Environment Variables](#environment-variables) - [MCP Configuration](#mcp-configuration) - [Example Use Cases](#example-use-cases) - [Usage in Multi-Agent Workflows](#usage-in-multi-agent-workflows) - [Documentation](#documentation) - [Community](#community) - [Support Us](#support-us)
--- ## Quickstart To get started with Forge, run the command below: ```bash curl -fsSL https://forgecode.dev/cli | sh ``` On first run, Forge will guide you through setting up your AI provider credentials using the interactive login flow. Alternatively, you can configure providers beforehand: ```bash # Configure your provider credentials interactively forge provider login # Then start Forge forge ``` That's it! Forge is now ready to assist you with your development tasks. ## Usage Examples Forge can be used in different ways depending on your needs. Here are some common usage patterns:
Code Understanding ``` > Can you explain how the authentication system works in this codebase? ``` Forge will analyze your project's structure, identify authentication-related files, and provide a detailed explanation of the authentication flow, including the relationships between different components.
Implementing New Features ``` > I need to add a dark mode toggle to our React application. How should I approach this? ``` Forge will suggest the best approach based on your current codebase, explain the steps needed, and even scaffold the necessary components and styles for you.
Debugging Assistance ``` > I'm getting this error: "TypeError: Cannot read property 'map' of undefined". What might be causing it? ``` Forge will analyze the error, suggest potential causes based on your code, and propose different solutions to fix the issue.
Code Reviews ``` > Please review the code in src/components/UserProfile.js and suggest improvements ``` Forge will analyze the code, identify potential issues, and suggest improvements for readability, performance, security, and maintainability.
Learning New Technologies ``` > I want to integrate GraphQL into this Express application. Can you explain how to get started? ``` Forge will provide a tailored tutorial on integrating GraphQL with Express, using your specific project structure as context.
Database Schema Design ``` > I need to design a database schema for a blog with users, posts, comments, and categories ``` Forge will suggest an appropriate schema design, including tables/collections, relationships, indexes, and constraints based on your project's existing database technology.
Refactoring Legacy Code ``` > Help me refactor this class-based component to use React Hooks ``` Forge can help modernize your codebase by walking you through refactoring steps and implementing them with your approval.
Git Operations ``` > I need to merge branch 'feature/user-profile' into main but there are conflicts ``` Forge can guide you through resolving git conflicts, explaining the differences and suggesting the best way to reconcile them.
## Why Forge? Forge is designed for developers who want to enhance their workflow with AI assistance while maintaining full control over their development environment. - **Zero configuration** - Just add your API key and you're ready to go - **Seamless integration** - Works right in your terminal, where you already work - **Multi-provider support** - Use OpenAI, Anthropic, or other LLM providers - **Secure by design** - Restricted shell mode limits file system access and prevents unintended changes - **Open-source** - Transparent, extensible, and community-driven Forge helps you code faster, solve complex problems, and learn new technologies without leaving your terminal. --- ## How Forge Works: Three Modes Forge has three distinct ways to use it. Understanding this distinction upfront will save you confusion. ### Interactive Mode (TUI) Running `forge` with no arguments starts the interactive terminal UI, a persistent session where you type prompts and the AI responds in a conversational loop. This is the primary way to do multi-step work. ```bash forge # Start a new interactive session forge conversation resume # Resume a specific saved conversation in interactive mode forge --conversation-id # Same: resume conversation by ID forge --agent # Start interactive session with a specific agent forge -C /path/to/project # Start in a specific directory forge --sandbox experiment-name # Create an isolated git worktree + branch, then start there ``` Once inside interactive mode, type your prompt and press Enter. Forge reads files, writes patches, runs commands, and maintains context across the whole session. ### One-Shot CLI Mode Pass `-p` (or `--prompt`) to run a single prompt and exit. Forge does the work and returns to your shell. Useful for scripts, piping output, or quick tasks. ```bash forge -p "Explain the purpose of src/main.rs" forge -p "Add error handling to the parse() function in lib.rs" echo "What does this do?" | forge # Pipe input as the prompt forge commit # Generate an AI commit message and commit (exits when done) forge commit --preview # Generate commit message, print it, then exit forge suggest "find large log files" # Translate natural language to a shell command, then exit ``` > **Note:** `forge conversation resume ` opens the interactive TUI. It does **not** just print a message and exit. If you run it and see the cursor waiting, you are inside the interactive session. Type your prompt or press `Ctrl+C` to exit. ### ZSH Plugin Mode (`:` prefix) Install the ZSH plugin once with `forge setup`, then use `:` commands directly at your shell prompt without ever typing `forge`. This is the fastest mode for day-to-day development: send prompts, switch conversations, commit, and suggest commands without leaving your shell. ```zsh : refactor the auth module # Send a prompt to the active agent :commit # AI-powered git commit :suggest "find large log files" # Translate description → shell command in your buffer :conversation # Browse saved conversations with interactive picker ``` See the full [ZSH Plugin reference below](#zsh-plugin-the--prefix-system) for all commands and aliases. --- ## ZSH Plugin: The `:` Prefix System When you install the ZSH plugin (`forge setup`), you get a `:` prefix command system at your shell prompt. This is the fastest way to use Forge during normal development; you never leave your shell. **How it works:** Lines starting with `:` are intercepted before the shell sees them and routed to Forge. Everything else runs normally. ```zsh : # Send a prompt to the active agent :sage # Send a prompt to a specific agent by name (sage, muse, forge, or any custom agent) :agent # Switch the active agent; opens interactive picker if no name given ``` ### Agents Forge ships with three built-in agents, each with a different role: | Agent | Alias | Purpose | Modifies files? | |---|---|---|---| | `forge` | (default) | Implementation: builds features, fixes bugs, and runs tests | Yes | | `sage` | `:ask` | Research: maps architecture, traces data flow, and reads code | No | | `muse` | `:plan` | Planning: analyzes structure and writes implementation plans to `plans/` | No | ### Sending Prompts ```zsh : refactor the auth module to use the new middleware :sage how does the caching layer work? # sage = read-only research agent :muse design a deployment strategy # muse = planning agent (writes to plans/) :ask how does X work? # alias for :sage :plan create a migration plan # alias for :muse ``` The agent context persists. Typing `:sage` alone (no prompt text) switches the active agent to sage for all subsequent `: ` commands. ### Attaching Files Type `@` in a prompt, then press Tab to fuzzy-search and select files. The path is inserted as `@[filename]` and attached as context to the AI. ```zsh : review this code @[src/auth.rs] @[tests/auth_test.rs] ``` ### Conversation Management Forge saves every conversation. You can switch between them like switching directories. ```zsh :new # Start a fresh conversation (saves current for :conversation -) :new # Start a new conversation and immediately send a prompt :conversation # Open interactive picker: browse and switch conversations with preview :conversation # Switch directly to a conversation by ID :conversation - # Toggle between current and previous conversation (like cd -) :clone # Branch the current conversation (try a different direction) :clone # Clone a specific conversation by ID :rename # Rename the current conversation :conversation-rename # Rename a conversation via interactive picker :retry # Retry the last prompt (useful if the AI misunderstood) :copy # Copy the last AI response to clipboard as markdown :dump # Export conversation as JSON :dump html # Export conversation as formatted HTML :compact # Manually compact context to free up token budget ``` ### Git Integration ```zsh :commit # AI reads your diff, writes a commit message, and commits immediately :commit # Same, but pass extra context: :commit fix typo in readme :commit-preview # AI generates the message and puts "git commit -m '...'" in your buffer # so you can review/edit the message before pressing Enter ``` ### Shell Command Tools ```zsh :suggest # Translate natural language to a shell command and put it in your buffer :edit # Open $EDITOR to compose a complex multi-line prompt, then send it ``` ### Session & Configuration Some commands change settings for the current session only. Others persist to your config file (`~/forge/.forge.toml`). The distinction matters: ```zsh # Session-only (reset when you close the terminal; not saved to config) :model # Change model for this session only :reasoning-effort # Set reasoning effort: none/minimal/low/medium/high/xhigh/max :agent # Switch active agent for this session # Persistent (saved to config file) :config-model # Set default model globally (alias: :cm) :config-provider # Switch provider globally (alias: :provider, :p) :config-reasoning-effort # Set default reasoning effort globally (alias: :cre) :config-commit-model # Set model used for :commit (alias: :ccm) :config-suggest-model # Set model used for :suggest (alias: :csm) :config-reload # Reset session overrides back to global config (alias: :cr) # View & edit config :info # Show current session info (model, agent, conversation ID) :config # Display effective resolved configuration in TOML format :config-edit # Open config file in $EDITOR (alias: :ce) :tools # List available tools for the current agent :skill # List available skills ``` ### Skills Skills are reusable workflows the AI can invoke as tools. Forge ships three built-in skills: - **`create-skill`**: scaffold a new custom skill - **`execute-plan`**: execute a plan file from `plans/` - **`github-pr-description`**: generate a PR description from your diff Use `:skill` to list available skills. The AI invokes them automatically when relevant, or you can ask explicitly: `: generate a PR description using the github-pr-description skill`. **Custom skills** live in `SKILL.md` files with YAML front-matter. Precedence (highest first): | Location | Path | Scope | |---|---|---| | Project-local | `.forge/skills//SKILL.md` | This project only | | Global | `~/forge/skills//SKILL.md` | All projects | | Built-in | Embedded in binary | Always available | Project-local skills override global ones, which override built-in ones. To scaffold a new skill, ask: `: create a new skill`. ### Customizing Agent Behavior **`AGENTS.md`:** Create this file in your project root (or `~/forge/AGENTS.md` globally) to give all agents persistent instructions such as coding conventions, commit message style, and things to avoid. Forge reads it automatically at the start of every conversation. **Custom agents:** Place a `.md` file with YAML front-matter in `.forge/agents/` (project) or `~/forge/agents/` (global) to define additional agents with their own models, tools, and system prompts. Project-local agents override global ones. The built-in agent files in `crates/forge_repo/src/agents/` are good examples of the format. **Custom commands:** Place YAML files in `.forge/commands/` (project) or `~/forge/commands/` (global) to define shortcut commands available via `:commandname`. Commands can also be defined inline in `forge.yaml` under the `commands:` key. ### Semantic Search (Workspace) ```zsh :sync # Index your codebase for semantic search :workspace-init # Initialize workspace for indexing :workspace-status # Show indexing status :workspace-info # Show workspace details ``` After running `:sync`, the AI can search your codebase by meaning rather than exact text matches. Indexing sends file content to the workspace server, which defaults to `https://api.forgecode.dev`. Set `FORGE_WORKSPACE_SERVER_URL` to override this if self-hosting. ### Quick Reference: All `:` Commands | Command | Alias | What it does | |---|---|---| | `: ` | | Send prompt to active agent | | `:new` | `:n` | Start new conversation | | `:conversation` | `:c` | Browse/switch conversations (interactive picker) | | `:conversation -` | | Toggle to previous conversation | | `:clone` | | Branch current conversation | | `:rename ` | `:rn` | Rename current conversation | | `:conversation-rename` | | Rename conversation (interactive picker) | | `:retry` | `:r` | Retry last prompt | | `:copy` | | Copy last response to clipboard | | `:dump` | `:d` | Export conversation as JSON | | `:compact` | | Compact context | | `:commit` | | AI commit (immediate) | | `:commit-preview` | | AI commit (review first) | | `:suggest ` | `:s` | Translate natural language to command | | `:edit` | `:ed` | Compose prompt in $EDITOR | | `:sage ` | `:ask` | Q&A / code understanding agent | | `:muse ` | `:plan` | Planning agent | | `:agent ` | `:a` | Switch active agent (interactive picker if no name given) | | `:model ` | `:m` | Set model for this session only | | `:config-model ` | `:cm` | Set default model (persistent) | | `:reasoning-effort ` | `:re` | Set reasoning effort for session | | `:config-reload` | `:cr` | Reset session overrides to global config | | `:info` | `:i` | Show session info | | `:sync` | `:workspace-sync` | Index codebase for semantic search | | `:tools` | `:t` | List available tools | | `:skill` | | List available skills | | `:login` | `:provider-login` | Login to a provider | | `:logout` | | Logout from a provider | | `:keyboard-shortcuts` | `:kb` | Show keyboard shortcuts | | `:doctor` | | Run shell environment diagnostics | --- ## Command-Line Options Here's a quick reference of Forge's command-line options: | Option | Description | | ----------------------------------- | ------------------------------------------------------------------------ | | `-p, --prompt ` | Direct prompt to process without entering interactive mode | | `-e, --event ` | Dispatch an event to the workflow in JSON format | | `--conversation ` | Path to a JSON file containing the conversation to execute | | `--conversation-id ` | Resume or continue an existing conversation by ID | | `--agent ` | Agent ID to use for this session | | `-C, --directory ` | Change to this directory before starting | | `--sandbox ` | Create an isolated git worktree + branch for safe experimentation | | `--verbose` | Enable verbose logging output | | `-h, --help` | Print help information | | `-V, --version` | Print version | ### Subcommands ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Advanced Configuration ### Provider Configuration Forge supports multiple AI providers. The recommended way to configure providers is using the interactive login command: ```bash forge provider login ``` This will: 1. Show you a list of available providers 2. Guide you through entering the required credentials #### Managing Provider Credentials ```bash # Login to a provider (add or update credentials) forge provider login # Remove provider credentials forge provider logout # List supported providers forge provider list ``` #### Deprecated: Environment Variables > **⚠️ DEPRECATED**: Using `.env` files for provider configuration is deprecated and will be removed in a future version. Please use `forge provider login` instead. For backward compatibility, Forge still supports environment variables. On first run, any credentials found in environment variables will be automatically migrated to file-based storage.
Legacy Environment Variable Setup (Deprecated)
OpenRouter ```bash # .env OPENROUTER_API_KEY= ```
Requesty ```bash # .env REQUESTY_API_KEY= ```
x-ai ```bash # .env XAI_API_KEY= ```
z.ai ```bash # .env ZAI_API_KEY= # Or for coding plan subscription ZAI_CODING_API_KEY= ```
Cerebras ```bash # .env CEREBRAS_API_KEY= ```
Neuralwatt ```bash # .env NEURALWATT_API_KEY= ```
OrcaRouter ```bash # .env ORCAROUTER_API_KEY= ```
Meta ```bash # .env META_API_KEY= ``` ```yaml # forge.yaml model: muse-spark-1.1 ```
IO Intelligence ```bash # .env IO_INTELLIGENCE_API_KEY= ``` ```yaml # forge.yaml model: meta-llama/Llama-3.3-70B-Instruct ```
OpenAI ```bash # .env OPENAI_API_KEY= ``` ```yaml # forge.yaml model: o3-mini-high ```
Anthropic ```bash # .env ANTHROPIC_API_KEY= ``` ```yaml # forge.yaml model: claude-3.7-sonnet ```
Google Vertex AI **Setup Instructions:** 1. **Install Google Cloud CLI** and authenticate: ```bash gcloud auth login gcloud config set project YOUR_PROJECT_ID ``` 2. **Get your authentication token**: ```bash gcloud auth print-access-token ``` 3. **Use the token when logging in via Forge**: ```bash forge provider login # Select Google Vertex AI and enter your credentials ``` **Legacy `.env` setup:** ```bash # .env PROJECT_ID= LOCATION= VERTEX_AI_AUTH_TOKEN= ``` ```yaml # forge.yaml model: google/gemini-2.5-pro ``` **Available Models:** - Claude models: `claude-sonnet-4@20250514` - Gemini models: `gemini-2.5-pro`, `gemini-2.0-flash` Use the `/model` command in Forge CLI to see all available models.
OpenAI-Compatible Providers ```bash # .env OPENAI_API_KEY= OPENAI_URL= ``` ```yaml # forge.yaml model: ```
Groq ```bash # .env OPENAI_API_KEY= OPENAI_URL=https://api.groq.com/openai/v1 ``` ```yaml # forge.yaml model: deepseek-r1-distill-llama-70b ```
Amazon Bedrock To use Amazon Bedrock models with Forge, you'll need to first set up the [Bedrock Access Gateway](https://github.com/aws-samples/bedrock-access-gateway): 1. **Set up Bedrock Access Gateway**: - Follow the deployment steps in the [Bedrock Access Gateway repo](https://github.com/aws-samples/bedrock-access-gateway) - Create your own API key in Secrets Manager - Deploy the CloudFormation stack - Note your API Base URL from the CloudFormation outputs 2. **Configure in Forge**: ```bash forge provider login # Select OpenAI-compatible provider and enter your Bedrock Gateway details ``` **Legacy `.env` setup:** ```bash # .env OPENAI_API_KEY= OPENAI_URL= ``` ```yaml # forge.yaml model: anthropic.claude-3-opus ```
ForgeCode Services ```bash # .env FORGE_API_KEY= ``` ```yaml # forge.yaml model: claude-3.7-sonnet ```
--- ### forge.yaml Configuration Options ### Environment Variables Forge supports several environment variables for advanced configuration and fine-tuning. These can be set in your `.env` file or system environment.
Retry Configuration Control how Forge handles retry logic for failed requests: ```bash # .env FORGE_RETRY_INITIAL_BACKOFF_MS=1000 # Initial backoff time in milliseconds (default: 1000) FORGE_RETRY_BACKOFF_FACTOR=2 # Multiplier for backoff time (default: 2) FORGE_RETRY_MAX_ATTEMPTS=3 # Maximum retry attempts (default: 3) FORGE_SUPPRESS_RETRY_ERRORS=false # Suppress retry error messages (default: false) FORGE_RETRY_STATUS_CODES=429,500,502 # HTTP status codes to retry (default: 429,500,502,503,504) ```
HTTP Configuration Fine-tune HTTP client behavior for API requests: ```bash # .env FORGE_HTTP_CONNECT_TIMEOUT=30 # Connection timeout in seconds (default: 30) FORGE_HTTP_READ_TIMEOUT=900 # Read timeout in seconds (default: 900) FORGE_HTTP_POOL_IDLE_TIMEOUT=90 # Pool idle timeout in seconds (default: 90) FORGE_HTTP_POOL_MAX_IDLE_PER_HOST=5 # Max idle connections per host (default: 5) FORGE_HTTP_MAX_REDIRECTS=10 # Maximum redirects to follow (default: 10) FORGE_HTTP_USE_HICKORY=false # Use Hickory DNS resolver (default: false) FORGE_HTTP_TLS_BACKEND=default # TLS backend: "default" or "rustls" (default: "default") FORGE_HTTP_MIN_TLS_VERSION=1.2 # Minimum TLS version: "1.0", "1.1", "1.2", "1.3" FORGE_HTTP_MAX_TLS_VERSION=1.3 # Maximum TLS version: "1.0", "1.1", "1.2", "1.3" FORGE_HTTP_ADAPTIVE_WINDOW=true # Enable HTTP/2 adaptive window (default: true) FORGE_HTTP_KEEP_ALIVE_INTERVAL=60 # Keep-alive interval in seconds (default: 60, use "none"/"disabled" to disable) FORGE_HTTP_KEEP_ALIVE_TIMEOUT=10 # Keep-alive timeout in seconds (default: 10) FORGE_HTTP_KEEP_ALIVE_WHILE_IDLE=true # Keep-alive while idle (default: true) FORGE_HTTP_ACCEPT_INVALID_CERTS=false # Accept invalid certificates (default: false) - USE WITH CAUTION FORGE_HTTP_ROOT_CERT_PATHS=/path/to/cert1.pem,/path/to/cert2.crt # Paths to root certificate files (PEM, CRT, CER format), multiple paths separated by commas ``` > **⚠️ Security Warning**: Setting `FORGE_HTTP_ACCEPT_INVALID_CERTS=true` disables SSL/TLS certificate verification, which can expose you to man-in-the-middle attacks. Only use this in development environments or when you fully trust the network and endpoints.
API Configuration Override default API endpoints and provider/model settings: ```bash # .env FORGE_API_URL=https://api.forgecode.dev # Custom Forge API URL (default: https://api.forgecode.dev) FORGE_WORKSPACE_SERVER_URL=http://localhost:8080 # URL for the indexing server (default: https://api.forgecode.dev/) ```
Tool Configuration Configuring the tool calls settings: ```bash # .env FORGE_TOOL_TIMEOUT=300 # Maximum execution time in seconds for a tool before it is terminated to prevent hanging the session. (default: 300) FORGE_MAX_IMAGE_SIZE=10485760 # Maximum image file size in bytes for read_image operations (default: 10485760 - 10 MB) FORGE_DUMP_AUTO_OPEN=false # Automatically open dump files in browser (default: false) FORGE_DEBUG_REQUESTS=/path/to/debug/requests.json # Write debug HTTP request files to specified path (supports absolute and relative paths) ```
ZSH Plugin Configuration Configure the ZSH plugin behavior: ```bash # .env FORGE_BIN=forge # Command to use for forge operations (default: "forge") ``` The `FORGE_BIN` environment variable allows you to customize the command used by the ZSH plugin when transforming `:` prefixed commands. If not set, it defaults to `"forge"`.
Display Configuration Configure display options for the Forge UI and ZSH theme: ```bash # .env FORGE_CURRENCY_SYMBOL="$" # Currency symbol for cost display in ZSH theme (default: "$") FORGE_CURRENCY_CONVERSION_RATE=1.0 # Conversion rate for currency display (default: 1.0) NERD_FONT=1 # Enable Nerd Font icons in ZSH theme (default: auto-detected, set to "1" or "true" to enable, "0" or "false" to disable) USE_NERD_FONT=1 # Alternative variable for enabling Nerd Font icons (same behavior as NERD_FONT) ``` The `FORGE_CURRENCY_SYMBOL` and `FORGE_CURRENCY_CONVERSION_RATE` variables control how costs are displayed in the ZSH theme right prompt. Use these to customize the currency display for your region or preferred currency.
System Configuration System-level environment variables (usually set automatically): ```bash # .env FORGE_CONFIG=/custom/config/dir # Base directory for all Forge config files (default: ~/.forge) FORGE_MAX_SEARCH_RESULT_BYTES=10240 # Maximum bytes for search results (default: 10240 - 10 KB) FORGE_HISTORY_FILE=/path/to/history # Custom path for Forge history file (default: uses system default location) FORGE_BANNER="Your custom banner text" # Custom banner text to display on startup (default: Forge ASCII art) FORGE_MAX_CONVERSATIONS=100 # Maximum number of conversations to show in list (default: 100) FORGE_MAX_LINE_LENGTH=2000 # Maximum characters per line for file read operations (default: 2000) FORGE_STDOUT_MAX_LINE_LENGTH=2000 # Maximum characters per line for shell output (default: 2000) SHELL=/bin/zsh # Shell to use for command execution (Unix/Linux/macOS) COMSPEC=cmd.exe # Command processor to use (Windows) ```
Semantic Search Configuration Configure semantic search behavior for code understanding: ```bash # .env FORGE_SEM_SEARCH_LIMIT=200 # Maximum number of results to return from initial vector search (default: 200) FORGE_SEM_SEARCH_TOP_K=20 # Top-k parameter for relevance filtering during semantic search (default: 20) ```
Logging Configuration Configure logging verbosity and output: ```bash # .env FORGE_LOG=forge=info # Log filter level (default: forge=debug when tracking disabled, forge=info when tracking enabled) ``` The `FORGE_LOG` variable controls the logging level for Forge's internal operations using the standard tracing filter syntax. Common values: - `forge=error` - Only errors - `forge=warn` - Warnings and errors - `forge=info` - Informational messages (default when tracking enabled) - `forge=debug` - Debug information (default when tracking disabled) - `forge=trace` - Detailed tracing
Tracking Configuration Control tracking of user-identifying metadata in telemetry events: ```bash # .env FORGE_TRACKER=false # Disable tracking enrichment metadata (default: true) ``` The `FORGE_TRACKER` variable controls whether tracking enrichment metadata is included in telemetry events.
The `forge.yaml` file supports several advanced configuration options that let you customize Forge's behavior.
Custom Rules Add your own guidelines that all agents should follow when generating responses. ```yaml # forge.yaml custom_rules: | 1. Always add comprehensive error handling to any code you write. 2. Include unit tests for all new functions. 3. Follow our team's naming convention: camelCase for variables, PascalCase for classes. ```
Commands Define custom commands as shortcuts for repetitive prompts: ```yaml # forge.yaml commands: - name: "refactor" description: "Refactor selected code" prompt: "Please refactor this code to improve readability and performance" ```
Model Specify the default AI model to use for all agents in the workflow. ```yaml # forge.yaml model: "claude-3.7-sonnet" ```
Max Walker Depth Control how deeply Forge traverses your project directory structure when gathering context. ```yaml # forge.yaml max_walker_depth: 3 # Limit directory traversal to 3 levels deep ```
Temperature Adjust the creativity and randomness in AI responses. Lower values (0.0-0.3) produce more focused, deterministic outputs, while higher values (0.7-2.0) generate more diverse and creative results. ```yaml # forge.yaml temperature: 0.7 # Balanced creativity and focus ```
Tool Max Failure Limit Control how many times a tool can fail before Forge forces completion to prevent infinite retry loops. This helps avoid situations where an agent gets stuck repeatedly trying the same failing operation. ```yaml # forge.yaml max_tool_failure_per_turn: 3 # Allow up to 3 failures per tool before forcing completion ``` Set to a higher value if you want more retry attempts, or lower if you want faster failure detection.
Max Requests Per Turn Limit the maximum number of requests an agent can make in a single conversation turn. This prevents runaway conversations and helps control API usage and costs. ```yaml # forge.yaml max_requests_per_turn: 50 # Allow up to 50 requests per turn ``` When this limit is reached, Forge will: - Ask you if you wish to continue - If you respond with 'Yes', it will continue the conversation - If you respond with 'No', it will end the conversation
---
Model Context Protocol (MCP) The MCP feature allows AI agents to communicate with external tools and services. This implementation follows Anthropic's [Model Context Protocol](https://docs.anthropic.com/en/docs/claude-code/tutorials#set-up-model-context-protocol-mcp) design. ### MCP Configuration Configure MCP servers using the CLI: ```bash # List all MCP servers forge mcp list # Import a server from JSON forge mcp import # Show server configuration details forge mcp show # Remove a server forge mcp remove # Reload servers and rebuild caches forge mcp reload ``` Or manually create a `.mcp.json` file with the following structure: ```json { "mcpServers": { "server_name": { "command": "command_to_execute", "args": ["arg1", "arg2"], "env": { "ENV_VAR": "value" } }, "another_server": { "url": "http://localhost:3000/events" } } } ``` MCP configurations are read from two locations (project-local takes precedence): 1. **Project-local:** `.mcp.json` in your project directory 2. **Global:** `~/forge/.mcp.json` ### Example Use Cases MCP can be used for various integrations: - Web browser automation - External API interactions - Tool integration - Custom service connections ### Usage in Multi-Agent Workflows MCP tools can be used as part of multi-agent workflows, allowing specialized agents to interact with external systems as part of a collaborative problem-solving approach.
--- ## Documentation For comprehensive documentation on all features and capabilities, please visit the [documentation site](https://github.com/tailcallhq/forgecode/tree/main/docs). --- ## Installation ```bash # YOLO curl -fsSL https://forgecode.dev/cli | sh # Package managers nix run github:tailcallhq/forgecode # for latest dev branch ``` --- ## Community Join our vibrant Discord community to connect with other Forge users and contributors, get help with your projects, share ideas, and provide feedback! [](https://discord.gg/kRZBPpkgwq) --- ## Support Us Your support drives Forge's continued evolution! By starring our GitHub repository, you: - Help others discover this powerful tool 🔍 - Motivate our development team 💪 - Enable us to prioritize new features 🛠️ - Strengthen our open-source community 🌱 --- ### Overview (`benchmarks/README.md`) # Forge Code Evaluations A flexible evaluation framework for running automated tests and benchmarks against Forge Code commands. ## Quick Start ### Setup Before running evaluations, create a `forgee` symlink to the debug binary: ```bash # Create symlink in your PATH (e.g., ~/bin or /usr/local/bin) ln -sf /path/to/code-forge/target/debug/forge ~/forgee # Or if ~/bin is in your PATH ln -sf $(pwd)/target/debug/forge ~/bin/forgee ``` **Why is this needed?** Tasks execute in temporary directories, so relative paths like `../../target/debug/forge` won't work. The `forgee` symlink provides a stable absolute path that works from any directory. ### Running Evaluations ```bash # Run an evaluation npm run eval ./evals/create_skill/task.yml # Set custom log level LOG_LEVEL=debug npm run eval ./evals/create_skill/task.yml ``` ## How It Works The evaluation system executes commands based on task definitions and validates their output. It supports: - **Parallel execution** with configurable concurrency - **Timeout handling** for long-running tasks - **Data-driven testing** using CSV files - **Output validation** using regex patterns - **Debug artifacts** stored with timestamps ## Project Structure ``` benchmarks/ ├── cli.ts # Main CLI entry point ├── command-generator.ts # Command template rendering ├── task-executor.ts # Task execution with timeout support ├── model.ts # TypeScript types for tasks ├── parse.ts # CLI argument parsing └── evals/ # Evaluation definitions └── create_skill/ ├── task.yml # Task definition ├── create_skill_tasks.csv # Test data └── debug/ # Debug outputs (timestamped) ``` ## Creating an Evaluation ### 1. Create an Evaluation Directory ```bash mkdir -p evals/my_eval ``` ### 2. Create a `task.yml` File The task file defines how to run your evaluation: ```yaml # Optional: Commands to run before evaluation starts before_run: - cargo build - npm install # Required: Command(s) to execute for each test case # Single command run: forgee -p '{{prompt}}' # Or multiple commands (executed sequentially) run: - echo "Step 1: {{task}}" - forgee -p '{{prompt}}' - echo "Step 2: Complete" # Execution configuration parallelism: 10 # Number of tasks to run in parallel (default: 1) timeout: 60 # Timeout in seconds (optional) early_exit: true # Stop execution when validations pass (optional) # Optional: Validations to run on output validations: - name: "Check success message" type: regex regex: \[[0-9:]*\] Skill create-skill # Required: Data sources for test cases sources: - csv: my_tasks.csv ``` #### Task File Schema **`before_run`** (optional): Array of shell commands to execute before running tasks - Runs sequentially before the main command execution - Executes in a temporary directory created for the evaluation run - Useful for building binaries or setting up dependencies **`run`** (required): Command(s) to execute for each test case - Can be a single string or an array of strings - Commands support template placeholders (e.g., `{{variable}}`) - Multiple commands are executed sequentially - If any command fails, subsequent commands are skipped **`parallelism`** (optional): Number of tasks to run concurrently (default: 1) **`timeout`** (optional): Maximum execution time in seconds per task **`early_exit`** (optional): Stop command execution when all validations pass **`validations`** (optional): Array of validation rules - `name`: Human-readable description - `type`: Validation type. Supported values: - `regex`: Match output against a regular expression pattern - `shell`: Execute a shell command with output as stdin - For `regex` type: - `regex`: Regular expression pattern to match in output - For `shell` type: - `command`: Shell command to execute (receives task output via stdin) - `exit_code`: Expected exit code (default: 0) **`sources`** (required): Array of data sources - Currently supports CSV files: `- csv: filename.csv` - Future: Command output: `- cmd: command` ### 3. Create Test Data (CSV) Create a CSV file with columns matching your template variables: ```csv prompt,expected_output "Create a backup script","backup.sh" "Generate API client","api_client.py" ``` The column names (e.g., `prompt`, `expected_output`) become template variables you can use in your command with `{{column_name}}`. ### 4. Run the Evaluation ```bash npm run eval ./evals/my_eval/task.yml ``` ## Template Variables Commands support Handlebars template syntax. Variables are populated from CSV columns: ```yaml run: command: ./tool --input '{{input_file}}' --format {{format}} --verbose ``` With CSV: ```csv input_file,format data1.txt,json data2.txt,yaml ``` ## Output and Debugging ### Debug Artifacts Each run creates a timestamped debug directory: ``` evals/my_eval/debug/2025-11-23T10-30-45-123Z/ ├── task-1.log ├── task-2.log └── task-3.log ``` Each log file contains the full output (stdout + stderr) from the command execution. ### Task Status Tasks can have four statuses: - **`passed`**: Task completed successfully and passed all validations - **`validation_failed`**: Task completed but failed one or more validations - **`timeout`**: Task exceeded the timeout limit - **`failed`**: Task execution failed (non-zero exit code) ### Logging **Human-readable output (default):** ```bash npm run eval ./evals/my_eval/task.yml ``` **Machine-readable JSON output:** ```bash LOG_JSON=1 npm run eval ./evals/my_eval/task.yml | jq . ``` **Debug logging:** ```bash LOG_LEVEL=debug npm run eval ./evals/my_eval/task.yml ``` ## Examples ### Example 1: Simple Sequential Execution ```yaml run: echo "Processing {{name}}" sources: - csv: names.csv ``` ```csv name Alice Bob Charlie ``` ### Example 2: Parallel Execution with Timeout ```yaml run: ./slow_task --id {{task_id}} parallelism: 5 timeout: 30 sources: - csv: tasks.csv ``` ### Example 3: Multiple Commands ```yaml run: - echo "Starting task {{id}}" - ./process --input {{file}} - echo "Task {{id}} complete" parallelism: 3 timeout: 120 sources: - csv: tasks.csv ``` ### Example 4: Shell Command Validation ```yaml run: echo "{{message}}" parallelism: 3 validations: # Using grep to check if output contains specific text - name: "Contains 'test' word" type: shell command: grep -q "test" exit_code: 0 # Count words and ensure it's greater than 2 - name: "More than 2 words" type: shell command: test $(wc -w | awk '{print $1}') -gt 2 exit_code: 0 # Traditional regex validation (for comparison) - name: "Contains test or validation" type: regex regex: "test|validation" sources: - csv: messages.csv ``` ### Example 5: Regex Validation ```yaml run: cargo test {{test_name}} validations: - name: "All tests passed" type: regex regex: test result:\s+ok sources: - csv: tests.csv ``` ## Tips 1. **Use quotes in commands**: When passing CSV values with spaces, wrap them in quotes: ```yaml command: forge -p '{{prompt}}' ``` 2. **Build before running**: Use `before_run` to ensure binaries are up-to-date: ```yaml before_run: - cargo build ``` 3. **Start with low parallelism**: Test with `parallelism: 1` first, then increase: ```yaml parallelism: 1 # Start here ``` 4. **Set appropriate timeouts**: Add timeouts to prevent hanging: ```yaml timeout: 60 # seconds ``` 5. **Check debug logs**: When tasks fail, check the debug directory for full output: ```bash cat evals/my_eval/debug/*/task-1.log ``` ## Exit Codes The CLI exits with: - **0**: All tasks passed - **1**: One or more tasks failed (excluding timeouts and validation failures) --- ### Overview (`shell-plugin/README.md`) # Forge ZSH Plugin A powerful ZSH plugin that provides intelligent command transformation, file tagging, and conversation management for the Forge AI assistant. ## Features - **Smart Command Transformation**: Convert `:command` syntax into forge executions - **Agent Selection**: Tab completion for available agents using `:agent_name` - **File Tagging**: Interactive file selection with `@[filename]` syntax - **Syntax Highlighting**: Visual feedback for commands and tagged files - **Conversation Continuity**: Automatic session management across commands - **Interactive Completion**: Fuzzy finding for files and agents via built-in picker ## Prerequisites Before using this plugin, ensure you have the following tools installed: - **fd** - Fast file finder (alternative to find) - **forge** - The Forge CLI tool ### Installation of Prerequisites ```bash # macOS (using Homebrew) brew install fd # Ubuntu/Debian sudo apt install fd-find # Arch Linux sudo pacman -S fd ``` ## Usage ### Starting a Conversation Begin any command with `:` followed by your prompt: ```bash : Get the current time ``` This automatically starts a new conversation with the default Forge agent. ### Using Specific Agents Specify an agent by name after the colon: ```bash :sage How does caching work in this system? :muse Create a deployment strategy for my app ``` **Tab Completion**: Type `:` followed by partial agent name and press `TAB` for interactive selection. ### File Tagging Tag files in your commands using the `@[filename]` syntax: ```bash : Review this code @[src/main.rs] : Explain the configuration in @[config.yaml] ``` **Interactive Selection**: Type `@` and press `TAB` to search and select files interactively using fuzzy finder. ### Conversation Continuity Commands within the same session maintain context: ```bash # First command : My project uses React and TypeScript # Second command (remembers previous context) : How can I optimize the build process? ``` The plugin automatically manages conversation IDs to maintain context across related commands. ### Command Naming Shell commands should follow the **Object-Action** format. Examples: - `:provider-login` - `:sync-status` For backward compatibility, `:login` remains available as an alias for `:provider-login`. ### Session Management #### Starting New Sessions Clear the current conversation context and start fresh: ```bash :new # or use the alias :n ``` This will: - Clear the current conversation ID - Show the banner with helpful information - Reset the session state - Display a confirmation message with timestamp #### System Information View system and project information: ```bash :info # or use the alias :i ``` This displays: - System information - Project details - Current configuration - Current configuration #### Switching Conversations Browse and switch between conversations interactively: ```bash :conversation # or use the alias :c ``` This will display an interactive list of all conversations with preview, allowing you to select and switch. Switch to a specific conversation by ID: ```bash :conversation ``` Toggle between current and previous conversation (like `cd -`): ```bash :conversation - # or :c - ``` The plugin remembers your previous conversation, allowing you to quickly toggle back and forth. This works just like `cd -` in your shell, and **also works with `:new`** - when you start a new conversation, you can toggle back to your previous one. If there's no previous conversation tracked (e.g., first time using the plugin), `:c -` will show the conversation list popup, allowing you to select a conversation. This is useful when: - You need to temporarily check another conversation and come back - You're comparing or referencing information between two conversations - You want to quickly switch context between related tasks - You started a new conversation but want to reference the previous one #### Cloning Conversations Create a copy of an existing conversation with interactive selection: ```bash :clone ``` This will: - Display an interactive list of all conversations with preview - Allow you to select a conversation to clone - Create a new conversation with the same content - Automatically switch to the cloned conversation - Show the cloned conversation content and details You can also clone a specific conversation by providing its ID: ```bash :clone ``` This is useful when you want to: - Create a backup before making significant changes - Start a new conversation branch from an existing context - Experiment with different approaches while preserving the original #### Session Status The plugin automatically displays session information including: - Conversation ID when starting new sessions - Active agent information - New session confirmations with timestamps ## Syntax Highlighting The plugin provides visual feedback through syntax highlighting: - **Tagged Files** (`@[filename]`): Displayed in **green bold** - **Agent Commands** (`:agent`): Agent names in **yellow bold** - **Command Text**: Remaining text in **white bold** ## Configuration Customize the plugin behavior by setting these variables before loading the plugin: ```bash # Custom forge binary location export FORGE_BIN="/path/to/custom/forge" ``` ### Available Configuration Variables - `FORGE_BIN`: Path to the forge executable (default: `forge`) - `FORGE_EDITOR`: Editor command to use for `:edit` command (default: `$EDITOR` or `nano`) - `FORGE_SYNC_ENABLED`: Enable/disable automatic workspace sync (default: `true`) - `FORGE_MAX_COMMIT_DIFF`: Maximum diff size for commit message generation in bytes (default: `100000`) - `FORGE_SKIP_INTERACTIVE`: Skip interactive prompts (internal use) - `FORGE_CURRENCY_SYMBOL`: Currency symbol for cost display in ZSH theme (default: `"$"`) - `FORGE_CURRENCY_CONVERSION_RATE`: Conversion rate for currency display (default: `1.0`) - `NERD_FONT`: Enable Nerd Font icons in ZSH theme (default: auto-detected, set to `"1"` or `"true"` to enable, `"0"` or `"false"` to disable) - `USE_NERD_FONT`: Alternative variable for enabling Nerd Font icons (same behavior as `NERD_FONT`) - Internal pattern matching for conversation syntax (`:`) - New session command keyword: `:new` or `:n` ### Codebase Indexing Sync your codebase for semantic search: ```bash :sync ``` This will index the current directory for semantic code search. ### Environment Diagnostics Run comprehensive environment diagnostics to check your Forge setup: ```bash :doctor ``` This will check: - ZSH version and terminal information - Forge installation and version - Plugin and theme loading status - Completions availability - Dependencies (fd, bat) - ZSH plugins (autosuggestions, syntax-highlighting) - Editor configuration and PATH setup - Nerd Font support for icons ### .forge Directory The plugin creates a `.forge` directory in your current working directory (similar to `.git`) for temporary files: - `FORGE_EDITMSG.md`: Temporary file used when opening an external editor with `:edit` ## Advanced Features ### Command History All transformed commands are properly saved to ZSH history, allowing you to: - Navigate command history with arrow keys - Search previous forge commands with `Ctrl+R` - Reuse complex commands with file tags ### Keyboard Shortcuts - **Tab**: Interactive completion for files (`@`) and agents (`:`) - **Enter**: Transform and execute `:commands` - **Ctrl+C**: Interrupt running forge commands ## Examples ### Basic Usage ```bash : What's the weather like? :sage Explain the MVC pattern :planner Help me structure this project ``` ### With File Tagging ```bash : Review this implementation @[src/auth.rs] : Debug the issue in @[logs/error.log] @[config/app.yml] ``` ### Session Flow ```bash : I'm working on a Rust web API : What are the best practices for error handling? : Show me an example with @[src/errors.rs] :info :new : New conversation starts here ``` ### Codebase Indexing ```bash # Sync current directory for semantic search :sync ``` --- ### Overview (`crates/forge_select/README.md`) # forge_select A centralized crate for user interaction prompts using dialoguer. ## Purpose This crate provides a unified interface for terminal user interactions across the forge codebase. It encapsulates all direct dependencies on `dialoguer`, ensuring no other crates need to depend on it directly. ## Features - **Select prompts**: Choose from a list of options - **Confirm prompts**: Yes/no questions - **Input prompts**: Text input from user - **Multi-select prompts**: Choose multiple options from a list - **Consistent theming**: All prompts use a unified color scheme - **Error handling**: Graceful handling of user interruptions ## Usage ### Select from options ```rust use forge_select::ForgeSelect; let options = vec!["Option 1", "Option 2", "Option 3"]; let selected = ForgeSelect::select("Choose an option:", options) .with_starting_cursor(1) .prompt()?; ``` ### Confirm (yes/no) ```rust use forge_select::ForgeSelect; let confirmed = ForgeSelect::confirm("Are you sure?") .with_default(true) .prompt()?; ``` ### Text input ```rust use forge_select::ForgeSelect; let name = ForgeSelect::input("Enter your name:") .allow_empty(false) .with_default("John") .prompt()?; ``` ### Multi-select ```rust use forge_select::ForgeSelect; let options = vec!["Red", "Green", "Blue"]; let selected = ForgeSelect::multi_select("Choose colors:", options) .prompt()?; ``` ## Design ### Builder Pattern All prompt types use a builder pattern for configuration: - Create the builder with `ForgeSelect::select()`, `ForgeSelect::confirm()`, etc. - Configure options with `.with_*()` methods - Execute with `.prompt()` ### Ownership vs Clone Two variants for select operations: - `select()`: Requires `Clone` for options, useful when you need the list after selection - `select_owned()`: Takes ownership, no `Clone` required, more efficient ### Theme All prompts use a consistent `ColorfulTheme` from dialoguer, providing a unified look across the application. ## Integration This crate is used by: - `forge_main`: For CLI user interactions - `forge_infra`: For implementing the `UserInfra` trait No other crates should depend on `dialoguer` directly - use this crate instead. --- ### Overview (`.forge/skills/create-plan/README.md`) # Create Plan Skill Tools and scripts for creating and validating implementation plans. ## Files - `SKILL.md` - Main skill instructions for AI agents - `validate-plan.sh` - Validates a single plan file - `validate-all-plans.sh` - Validates all plans in a directory ## Validation Scripts ### validate-plan.sh Validates the structure and content of a single plan file. **Usage:** ```bash ./.forge/skills/create-plan/validate-plan.sh plans/2025-11-27-example-v1.md ``` **Checks:** - ✓ Filename follows convention: `YYYY-MM-DD-task-name-vN.md` - Year is reasonable (2020 to current year + 1) - Month is valid (01-12) - Day is valid (01-31) - Task name is meaningful (not generic like "test", "task", "temp") - Task name length is reasonable (5-60 characters) - Version number is reasonable (not > 50) - No uppercase letters or underscores (use lowercase and hyphens only) - ✓ File is in `plans/` directory - ✓ All required sections present: - Main heading (`# Title`) - `## Objective` - `## Implementation Plan` - `## Verification Criteria` - `## Potential Risks and Mitigations` - `## Alternative Approaches` - ✓ Implementation Plan uses checkbox format (`- [ ]`) - ✓ No numbered lists or plain bullets in Implementation Plan - ✓ No code blocks (` ``` `) in the plan - ✓ No code snippets (detects suspicious patterns) - ✓ No placeholder tasks (TODO, TBD, etc.) - ✓ **Task quality and density:** - Task descriptions are descriptive (≥ 20 characters recommended) - Average task length is substantial (30-200 chars recommended) - No generic/vague descriptions ("implement feature", "fix bug", etc.) - No duplicate or highly similar tasks - Consistent and sequential numbering if tasks are numbered - ✓ Verification criteria have content - ✓ Risks include mitigations - ✓ Reasonable number of tasks (3-20) **Exit Codes:** - `0` - Validation passed - `1` - Validation failed (errors found) ### validate-all-plans.sh Validates all plan files in a directory. **Usage:** ```bash # Validate all plans in default directory (plans/) ./.forge/skills/create-plan/validate-all-plans.sh # Validate plans in custom directory ./.forge/skills/create-plan/validate-all-plans.sh path/to/plans ``` **Exit Codes:** - `0` - All plans passed validation - `1` - One or more plans failed validation ## Integration ### Pre-commit Hook Add to `.git/hooks/pre-commit`: ```bash #!/bin/bash # Validate plans before committing if git diff --cached --name-only | grep -q "^plans/.*\.md$"; then echo "Validating modified plans..." ./.forge/skills/create-plan/validate-all-plans.sh plans/ exit $? fi ``` ### CI/CD Add to your CI pipeline: ```yaml - name: Validate Plans run: ./.forge/skills/create-plan/validate-all-plans.sh plans/ ``` ## Example Valid Plan See `SKILL.md` for the complete plan template structure. ## Common Validation Errors 1. **Invalid filename format**: Must follow `YYYY-MM-DD-task-name-vN.md` pattern - Use lowercase letters only - Use hyphens (not underscores) to separate words - Use valid date (month 01-12, day 01-31, year 2020+) - Avoid generic names like "test", "task", "temp" 2. **Missing checkboxes**: Use `- [ ]` not `1.` or `-` 3. **Code blocks**: Plans should use natural language, not code 4. **Missing sections**: All required sections must be present 5. **Empty sections**: Sections should have meaningful content 6. **Poor task quality**: Tasks should be descriptive and specific - Avoid short descriptions like "Do this", "Fix that", "Update code" - Avoid generic descriptions like "implement feature", "add functionality" - Include rationale and context in task descriptions - Aim for 30-150 characters per task description --- ### Overview (`.github/scripts/bounty/README.md`) # Bounty Automation Automates the full lifecycle of issue bounties — from label propagation when a PR is opened, through claiming when work begins, to rewarding when a PR is merged. ## Flow ``` Issue created └── maintainer adds bounty: $N label │ ├── parse-sync-generic-bounty.ts → adds generic bounty label │ ▼ Issue assigned to contributor └── parse-sync-claimed.ts → adds bounty: claimed │ ▼ Contributor opens PR with "Closes #N" / "Fixes #N" / "Resolves #N" └── parse-propagate-label.ts → copies bounty: $N to PR → posts comment on issue: "PR #X opened by @author" │ ▼ PR is merged └── parse-mark-rewarded.ts → adds bounty: rewarded to PR → adds bounty: rewarded to linked issue(s) → removes bounty: claimed from linked issue(s) │ ▼ Bounty lifecycle complete Removal path: maintainer removes last bounty: $N label └── parse-sync-generic-bounty.ts → removes generic bounty label ``` ## Labels | Label | Applied to | Set by | | -------------------------------- | ---------- | ----------------------------------------------------------- | | `bounty: $100` … `bounty: $5500` | Issue | Maintainer (manually) | | `bounty` | Issue | `parse-sync-generic-bounty.ts` on value label add/remove | | `bounty: claimed` | Issue | `parse-sync-claimed.ts` on assignment | | `bounty: rewarded` | Issue + PR | `parse-mark-rewarded.ts` on merge | Bounty values follow the Fibonacci sequence: **$100, $200, $300, $500, $800, $1300, $2100, $3400, $5500**. ## Pipeline Each workflow job runs a single Node process (`run.ts`) that executes all three stages in memory — no temp files, no shell piping: ``` parse → ParsedIntent (in memory) → plan → BatchPlan (in memory) → execute ``` **Stage 1 — parse** (`parse-*.ts`): Reads the GitHub Actions event payload from `GITHUB_EVENT_PATH` and returns a `ParsedIntent` object. Pure — makes no API calls. **Stage 2 — plan** (`plan.ts`): Receives the `ParsedIntent`, fetches current labels only for targets not already known from the event payload, diffs desired vs actual state, and returns a `BatchPlan`. Minimises API calls by reusing label data already present in the event. **Stage 3 — execute** (`execute.ts`): Receives the `BatchPlan` and applies all mutations. Label additions per target are sent as a single batched `POST /labels` call. Each removal is a separate `DELETE` (GitHub has no bulk-remove endpoint). Comments are posted last. This design means: - The parse stage is trivially unit-testable (pure function, no mocks needed). - The plan stage is testable with a minimal mock that only needs `getLabels`. - The execute stage is testable with a mock that tracks calls — no HTTP. - API calls are minimised: additions are batched per target; label state already in the event payload is never re-fetched. ## Scripts The workflow invokes a single orchestrator script (`run.ts`) per job. The parse, plan, and execute modules are called directly in the same Node process, passing objects in memory. ### `run.ts` The entrypoint used by `bounty.yml`. Accepts a `--script` flag to select which parse module to run, then calls `plan` and `execute` in sequence — all in-process. ```sh npx tsx .github/scripts/bounty/run.ts \ --script \ --repo \ --token \ [--pr | --issue ] ``` ### `parse-propagate-label.ts` Triggered by: `pull_request` — opened, edited, reopened. 1. Parses the PR body for closing keywords (`closes`, `fixes`, `resolves`, case-insensitive). 2. Returns a `ParsedIntent` with a `labelCopies` field — the plan stage fetches each linked issue's labels and copies any `bounty: $N` ones onto the PR. 3. Includes a comment mutation per linked issue (the plan stage drops it if the issue has no bounty labels). ### `parse-sync-claimed.ts` Triggered by: `issues` — assigned, unassigned. - **assigned**: returns `add: ["bounty: claimed"]` if the issue has a `bounty: $N` label. - **unassigned**: returns `remove: ["bounty: claimed"]` only when no assignees remain. - Issue labels are already in the event payload and supplied as `knownLabels` — no extra fetch. ### `parse-sync-generic-bounty.ts` Triggered by: `issues` — labeled, unlabeled. Keeps the generic `bounty` label in sync with value labels. Inspects `event.label` (the label that just changed) and only acts when it matches `bounty: $`. - **labeled**: returns `add: ["bounty"]`. - **unlabeled**: returns `remove: ["bounty"]` only when no value labels remain (guards against mid-tier-swap removal when a maintainer swaps one value label for another). - Issue labels from the event are supplied as `knownLabels` — no extra fetch. ### `parse-mark-rewarded.ts` Triggered by: `pull_request_target` — closed (merged only). 1. Returns empty intent if the PR was not merged or has no `bounty: $N` label. 2. Returns `add: ["bounty: rewarded"]` for the PR (labels known from event, no fetch). 3. Parses the PR body for linked issues; returns `add: ["bounty: rewarded"], remove: ["bounty: claimed"]` for each (plan stage fetches their labels). Uses `pull_request_target` so the job has write access to issues and PRs from forks. ### `plan.ts` Receives a `ParsedIntent` from the parse stage. Resolves `labelCopies` by fetching source issue labels. Fetches current labels for any target not already in `knownLabels`. Filters out no-op adds and removes. Returns a `BatchPlan`. ### `execute.ts` Receives a `BatchPlan` from the plan stage. For each mutation: one batched `POST` for all additions, one `DELETE` per removal, one `POST` per comment. ## Shared Module `github-api.ts` defines: - Event payload types (`PullRequestEvent`, `IssuesEvent`) - Pipeline types (`ParsedIntent`, `BatchPlan`, `TargetMutation`) - The `GitHubApi` interface (injectable for testing) - `GitHubRestApi` — the production implementation using `node:https` All scripts import types from `github-api.ts` and accept a `GitHubApi` instance in their `run()` / `plan()` / `execute()` function signature, making every step independently mockable. ## Tests Unit tests live alongside each script (`*.test.ts`) and use Node's built-in `node:test` runner. ```sh npm run test:bounty ``` - Parse tests: pure — no mock needed, just call `parse()` with a synthetic event. - Plan and execute tests: use a mock `GitHubApi` that tracks calls and returns preset label lists. - The CLI entrypoint in each script (yargs parsing + `GITHUB_EVENT_PATH` read) is guarded behind an `import.meta.url` check so it does not execute on import. ## Workflow Source `bounty.yml` is auto-generated from Rust source in `crates/forge_ci`. Do not edit it by hand — modify `crates/forge_ci/src/workflows/bounty.rs` and regenerate with: ```sh cargo test -p forge_ci ``` --- ### Overview (`benchmarks/evals/sem_search/README.md`) # Semantic Search Tool Selection Evaluation This evaluation validates that the Forge agent correctly identifies and uses the `sem_search` (Codebase Search) tool when presented with conceptual, functionality-based code location queries. ## What We're Testing The agent's ability to recognize when a user query requires semantic understanding rather than exact pattern matching. Specifically, we test whether the agent: 1. **Correctly invokes `sem_search`** for conceptual queries about code functionality 2. **Understands the distinction** between semantic search (conceptual) vs regex search (exact patterns) 3. **Recognizes appropriate use cases** such as: - Finding code by its purpose/behavior (e.g., "retry logic with exponential backoff") - Locating implementation patterns (e.g., "authentication token validation") - Discovering architectural components (e.g., "message transformation between AI providers") - Identifying system behaviors (e.g., "rate limiting for API requests") ## Test Scenarios The evaluation uses real-world queries that describe: - **Message transformation** between different AI provider formats - **Retry mechanisms** with exponential backoff patterns - **Authentication flows** including token validation and refresh - **Error handling** for network failures - **Caching strategies** for API responses - **Streaming response** handling from LLM APIs - **File validation** including upload and size checks - **Rate limiting** implementations for API requests - **Tool registration** and availability management - **Context management** including conversation history truncation ## Expected Behavior For all test queries, the agent should invoke the `sem_search` tool (displayed as "Codebase Search" in logs) rather than falling back to regex-based `search` tool, demonstrating understanding that these are conceptual queries requiring semantic code comprehension. --- ### Overview (`benchmarks/evals/semantic_search_quality/README.md`) # Semantic Search Quality Evaluation This evaluation measures the quality of semantic search queries generated by Forge using an LLM-as-judge approach. ## Quick Start ### Test Your Queries Manually Before running the full eval, test your queries to get immediate feedback: ```bash # Interactive mode npx tsx benchmarks/evals/semantic_search_quality/test_queries.ts # Non-interactive mode npx tsx benchmarks/evals/semantic_search_quality/test_queries.ts \ "retry mechanism with exponential backoff" \ "I need to find the implementation code for retry logic" ``` The query tester provides: - Detailed scores for embedding and reranking queries - Specific strengths and weaknesses - Actionable suggestions for improvement - Differentiation analysis ### Run the Full Evaluation ```bash # Run the entire semantic search quality eval npm run eval semantic_search_quality # Run with specific parallelism npm run eval semantic_search_quality -- --parallelism 10 # Run with specific model npm run eval semantic_search_quality -- --model "anthropic/claude-sonnet-4.5" ``` ## Purpose The semantic search feature uses two types of queries: 1. **Embedding Query (`query`)**: Converted to vector embeddings for semantic similarity search 2. **Reranking Query (`use_case`)**: Used by the reranker to filter results based on intent This eval verifies that: - Queries are well-formed and contain sufficient context - The two queries are different and serve distinct purposes - Results match the user's intent (implementation vs docs vs tests, etc.) - Correct file types are returned based on intent - Unwanted file types are avoided ## How It Works ### 1. Test Execution The eval runs various semantic search tasks covering different intents: - **Implementation**: Find actual code, not documentation - **Flow Understanding**: Understand how systems work - **Architecture**: Understand structural design - **Tests**: Find test examples and fixtures - **Documentation**: Find docs and guides - **Debugging**: Investigate issues - **Modification**: Find code to modify - **Configuration**: Find config files and schemas ### 2. LLM Judge Evaluation After each task, an LLM judge (Gemini 3 Pro) evaluates: #### Query Quality (40 points) - **Embedding Query (15pts)**: Domain terms, technical context, behavior description - **Reranking Query (15pts)**: Intent clarity, context about WHY, specificity - **Query Differentiation (10pts)**: Queries serve different purposes #### Result Relevance (60 points) - **Intent Matching (25pts)**: Results match the stated intent - **File Type Accuracy (20pts)**: Correct file types returned - **Avoidance Compliance (15pts)**: Unwanted file types avoided #### Pass Criteria - Total score >= 70/100 - No critical issues (e.g., identical queries, severe intent mismatch) ### 3. Structured Output The judge uses structured output (Zod schema) to provide: - Detailed scores for each dimension - Specific feedback on what's good/bad - List of critical issues - Overall pass/fail determination ## Environment Variables Required: - `GOOGLE_APPLICATION_CREDENTIALS`: Path to Google Cloud service account JSON credentials file Or authenticate using `gcloud auth application-default login`. The LLM judge uses Vertex AI to run Gemini 3 Pro. It supports Google's Application Default Credentials through the `google-auth-library`. The most common authentication method is to set the path to a JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. ## Test Cases The eval includes diverse test cases: ### Implementation Queries ```yaml - task: "Find the retry mechanism with exponential backoff implementation" intent: "implementation" expected_file_types: "rust,typescript,javascript" should_avoid: "markdown,txt,documentation" ``` ### Flow Understanding ```yaml - task: "How does the authentication flow work from request to response?" intent: "flow_understanding" expected_file_types: "rust,typescript" should_avoid: "test,markdown" ``` ### Documentation ```yaml - task: "Find documentation on how to configure semantic search" intent: "documentation" expected_file_types: "markdown,txt" should_avoid: "rust,typescript,javascript" ``` ## Interpreting Results ### High Scores (>80) Queries are excellent - domain-specific, well-differentiated, and would return highly relevant results. ### Medium Scores (60-80) Queries are functional but have room for improvement. Check feedback for specific suggestions. ### Low Scores (<60) Queries have significant issues: - Too generic or vague - Embedding and reranking queries too similar - Intent mismatch - Would return wrong file types ## Common Issues 1. **Identical Queries**: Embedding and reranking queries are the same or too similar - **Fix**: Make reranking query focus on intent, not just rephrasing 2. **Generic Queries**: Lack domain-specific context - **Fix**: Add technical terms, describe behavior, include framework names 3. **Intent Mismatch**: Queries would return docs when implementation is needed - **Fix**: Be explicit in reranking query about what type of code is needed 4. **Poor Differentiation**: Both queries try to do the same thing - **Fix**: Embedding = WHAT code does, Reranking = WHY you need it ## Architecture ``` task.yml ├── Runs Forge with semantic search task ├── Captures debug output (context.json) └── Validates with llm_judge.ts ├── Extracts sem_search calls ├── Calls Gemini 3 Pro with structured output └── Returns evaluation with scores ``` ## Extending the Eval To add new test cases: 1. Add to `task.yml` under `sources`: ```yaml - task: "Your task description" intent: "implementation|flow_understanding|tests|documentation|debugging|modification|architecture|configuration" expected_file_types: "comma,separated,extensions" should_avoid: "comma,separated,extensions" ``` 2. Choose appropriate intent: - `implementation`: Actual code implementation - `flow_understanding`: Understanding how things work - `architecture`: Structural/design understanding - `tests`: Test files and fixtures - `documentation`: Docs, README, guides - `debugging`: Finding bugs/issues - `modification`: Code to modify/extend - `configuration`: Config files, schemas ## Future Improvements - [ ] Add actual result verification (not just query quality) - [ ] Measure embedding quality directly - [ ] Test reranking effectiveness with real results - [ ] Add precision/recall metrics - [ ] Compare against human-labeled ground truth - [ ] Test edge cases (very short queries, multi-lingual code, etc.) --- ### Overview (`.forge/skills/debug-cli/scripts/README.md`) # Scripts This directory contains helper scripts for CLI debugging and testing. ## test_cli.sh Basic smoke test script that: - Builds the forge CLI - Tests main help command - Tests version command - Tests help for various subcommands Usage: ```bash ./scripts/test_cli.sh ``` --- ### Tool Guidelines (`docs/tool-guidelines.md`) # Claude Tool Description Best Practices ## Key Principles When defining tools for Claude to use, the quality of your tool descriptions dramatically affects performance. Here are the most important best practices extracted from Anthropic's documentation: ### 1. Provide Extremely Detailed Descriptions This is by far the most important factor in tool performance. Your descriptions should explain every detail about the tool, including: - **What the tool does** - Be explicit about the tool's primary function and purpose - **When it should be used** - Specify the scenarios where this tool is appropriate - **When it should NOT be used** - Clarify scenarios where the tool is not suitable - **What each parameter means** - Explain how each parameter affects the tool's behavior - **Limitations and caveats** - Note what information the tool does not return or handle - **Additional context** - If the tool name is unclear, provide extra clarification ### 2. Use Comprehensive But Concise Descriptions The more context you can give Claude about your tools, the better it will be at deciding when and how to use them. Aim for 3-4 sentences per tool description, or more if the tool is complex. > **IMPORTANT**: Tool descriptions must never exceed 1024 characters. This is enforced by tests to ensure compatibility with LLM API constraints. ### 3. Prioritize Descriptions Over Examples While you can include examples of how to use a tool in its description or in the accompanying prompt, this is less important than having a clear and comprehensive explanation of the tool's purpose and parameters. Only add examples after you've fully fleshed out the description. ### 4. Register All Tools in the Registry Every tool must be registered in the `crates/forge_services/src/tools/registry.rs` file to be available for use. The `ToolRegistry::tools()` method returns all available tools configured with the given infrastructure. ## Example Comparison ### Good Tool Description Example: ```json { "name": "get_stock_price", "description": "Retrieves the current stock price for a given ticker symbol. The ticker symbol must be a valid symbol for a publicly traded company on a major US stock exchange like NYSE or NASDAQ. The tool will return the latest trade price in USD. It should be used when the user asks about the current or most recent price of a specific stock. It will not provide any other information about the stock or company.", "input_schema": { "type": "object", "properties": { "ticker": { "type": "string", "description": "The stock ticker symbol, e.g. AAPL for Apple Inc." } }, "required": ["ticker"] } } ``` ### Poor Tool Description Example: ```json { "name": "get_stock_price", "description": "Gets the stock price for a ticker.", "input_schema": { "type": "object", "properties": { "ticker": { "type": "string" } }, "required": ["ticker"] } } ``` ## Why the Good Description Works Better The good description clearly explains: - What the tool does (retrieves the current stock price) - What data format it returns (latest trade price in USD) - When to use it (when the user asks about current/recent price) - What limitations it has (won't provide other company info) - What the parameter means (ticker symbol for publicly traded companies) The poor description is too brief and leaves Claude with many open questions about the tool's behavior and usage. It doesn't explain what kind of data is returned, what format it's in, when to use it, or what parameters are expected. ## Practical Tips 1. **Be specific about data formats** - Explain what format the data will be returned in 2. **Clarify parameter constraints** - Note any restrictions on parameter values 3. **Explain tool selection criteria** - Help Claude understand when to pick this tool over others 4. **Describe error handling** - Note how the tool behaves with invalid inputs 5. **Include domain-specific details** - Add contextual information related to the tool's domain 6. **Keep descriptions under 1024 characters** - Ensure compatibility with LLM API constraints 7. **Register your tool in the registry** - Add your tool to the registry.rs file for availability Thorough yet concise tool descriptions lead to more accurate tool selection, fewer clarification questions, and better overall performance when using Claude with tools. Source: [Anthropic Documentation on Tool Use](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview#example-poor-tool-description) --- --- METRICS --- - Files Extracted: 10 - Estimated Token Budget: ~19987 tokens - Canonical Reference: https://codewiki.google/github.com/tailcallhq/forgecode