# Technical Documentation: CodebuffAI/codebuff

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/CodebuffAI/codebuff)

### Overview (`README.md`)

# Freebuff

English | [简体中文](./README.zh-CN.md)

**Five free AI products for coding, building, and research.** No subscription, credits, or API key required.

[Freebuff](https://freebuff.com) brings specialized agents and a choice of leading models to your terminal, desktop, browser, and GitHub repositories. Text ads support access to the included models.

## Choose your Freebuff

| Product              | What it does                        | Get started                                                           |
| -------------------- | ----------------------------------- | --------------------------------------------------------------------- |
| **Freebuff Desktop** | Run parallel agents locally         | [Download for macOS, Windows, or Linux](https://freebuff.com/desktop) |
| **Freebuff CLI**     | Code from your terminal             | [Install the CLI](https://freebuff.com/cli)                           |
| **Freebuff Web**     | Build and ship full-stack apps      | [Build an app](https://freebuff.com/web)                              |
| **Freebuff Cloud**   | Run agents on any GitHub repository | [Connect a repository](https://freebuff.com/cloud)                    |
| **Freebuff Chat**    | Research and think with AI          | [Start a chat](https://freebuff.com/chat)                             |

## Quick start

Run Freebuff in any project from your terminal:

```bash
npm install freebuff
cd ~/my-project
freebuff
```

Then describe what you want. Freebuff finds the relevant files, makes changes, and runs the checks that matter for your project.

## Models

Freebuff includes a curated model catalog. The regular picker currently offers:

| Model                       | Access                  | Best for                                                          |
| --------------------------- | ----------------------- | ----------------------------------------------------------------- |
| **MiMo v2.6 Flash Fast**    | Full and limited access | The default everywhere; balanced performance with image support   |
| **GLM 5.3 Flash**           | Full and limited access | Deepest reasoning, unmetered                                      |
| **DeepSeek V4.1 Flash** | Full and limited access | Fast coding and tool use, unmetered                               |
| **DeepSeek V4.1 Flash Fast** | Full access; paid plans at limited access | Fast mode: the same model on DeepSeek's own API, splitting work across parallel subagents and verifying the result; priced on DeepSeek's peak and off-peak hours |
| **GPT-6 Luna**              | Full access             | Strong all-around with native images; runs on OpenAI flex capacity |
| **MiMo 2.6 Pro**            | Full access             | Xiaomi's stronger reasoning model, with image support |
| **Solar Mini 4**            | Full and limited access | Upstage's fast, compact model; 524K context, text only |
| **Solar Pro 4**             | Full and limited access | Upstage's larger, stronger model; 524K context, text only |
| **Gemini 3.8 Flash**        | Paid plans              | 1M context, and the only model that accepts audio, video and PDF |
| **Muse Spark 1.3**          | Paid plans              | Meta's agentic coding model; 1M context. When it is busy or unavailable, it answers on DeepSeek V4.1 Flash rather than making you wait |
| **Gemini 4 Argon**          | US, or paid plans elsewhere | Google's flagship reasoning model at a temporary promotional price, one session a day for every account; 1M context, images |
| **GPT-6.1 Sol**             | US, or paid plans elsewhere | OpenAI's flagship, one session a day for every account; 1M context, images |

Most models draw on your normal daily sessions rather than a separate limit. GLM 5.3 Flash, DeepSeek V4.1 Flash, MiMo v2.6 Flash Fast, Solar Mini 4 and Solar Pro 4 are unmetered at full access and cost no session at all. Models may still serve from a quantized (Q8_0) build.

DeepSeek V4 Pro was retired from the catalog; GLM 5.3 Flash replaces it as the deep-reasoning pick.

Beyond the regular picker:

- **Referrals and bounties** earn extra sessions on top of the free daily allowance.
- **Gemini 3.1 Flash Lite** powers specialist tasks such as file finding and research rather than appearing in the main picker.

Availability and limits depend on your access tier, product, and current capacity. Freebuff Desktop can also run locally installed Claude Code and Codex agents using your existing provider account; those connected models are separate from Freebuff's included catalog.

## How Freebuff works

Freebuff uses specialized agents instead of sending every task through one model and one prompt. Depending on the task, agents gather context, plan, edit or research, run tools, and review the result.

- **Codebase context** — File-finding agents map the relevant parts of a project before editing.
- **Implementation and review** — Agents can divide work, make changes, run commands, and inspect the result.
- **Research and browser use** — Agents can investigate documentation and test applications in a real browser.
- **Parallel local work** — Desktop isolates concurrent agents in separate workspaces.
- **Hosted environments** — Web and Cloud provide sandboxes, previews, terminals, and deployment workflows.

## Free access

Freebuff is available in every country. Supported regions receive full access; other regions and VPN users receive limited access to GLM 5.3 Flash, DeepSeek V4.1 Flash, MiMo v2.6 Flash Fast, Solar Mini 4, and Solar Pro 4. Every model is paid for in hours at the multiplier shown in the picker: an hour of MiMo v2.6 Flash Slow uses one hour of balance, a 3x model uses three. The free limited-access allowance is 5 hours a day, or 4 on a VPN or proxy, spent first on every model, GLM included; earned hours go into your wallet and are used after it. (Balances read in hours since 2026-10-15; 1 hour = 5 Freebucks.)

Text ads support the included models. Freebuff shows the applicable session limits and any model-specific data-use notice before you start.



**Is my data used to train AI?** Only when a model or feature says data may be used for AI training. Freebuff or the provider may then keep submissions to develop, train, test, evaluate, fine-tune, and improve AI models or products, and Freebuff may share or sell them to third parties. Separately, Freebuff may use your data to train the models that choose and measure ads.

**How is my data used and stored?** Freebuff uses your messages, agent traces, and files to run the service and respond to your requests. We may analyze this data, and how you use the app, to personalize ads and train ad models. We do not give separately uploaded files, project files, or connected repositories to advertising providers. Restricted partners may evaluate repositories you connect to Freebuff Cloud, but cannot otherwise use, broadly share, or train on them. See the Privacy Policy for details.

See the [Privacy Policy](https://freebuff.com/privacy-policy) for complete details.



## Contributing

Freebuff is a TypeScript monorepo built with Bun. Contributions to the products, agents, tools, documentation, and underlying runtime are welcome.

Local development requires Docker and a configured `.env.local`; see the
[Contributing Guide](./CONTRIBUTING.md) before starting the services.

```bash
git clone https://github.com/CodebuffAI/freebuff.git
cd freebuff
bun install
bun up
```

Start the CLI separately with:

```bash
bun start-cli
```

See the [Contributing Guide](./CONTRIBUTING.md), [development guide](./docs/development.md), and [testing guide](./docs/testing.md) for environment setup and the checks to run before opening a pull request.

## Built on Codebuff

Freebuff is built on [Codebuff](https://codebuff.com), the open multi-agent framework that powers its orchestration, tools, and SDK. To create custom agents or embed them in another application, see the [Codebuff documentation](https://codebuff.com/docs) and [`@codebuff/sdk`](https://www.npmjs.com/package/@codebuff/sdk).

## Links

- [Website](https://freebuff.com)
- [GitHub](https://github.com/CodebuffAI/freebuff)
- [Discord](https://discord.gg/yXG3w7wxfs)
- [Privacy Policy](https://freebuff.com/privacy-policy)
- [License](./LICENSE)

---

### Overview (`cli/README.md`)

# @codebuff/cli

A Terminal User Interface (TUI) package built with OpenTUI and React.

## Installation

```bash
bun install
```

## Development

Run the TUI in development mode:

```bash
bun run dev
```

## Testing

Run the test suite:

```bash
bun test
```

### Interactive E2E Testing

For testing interactive CLI features, install tmux:

```bash
# macOS
brew install tmux

# Ubuntu/Debian
sudo apt-get install tmux

# Windows (via WSL)
wsl --install
sudo apt-get install tmux
```

Then run the proof-of-concept:

```bash
bun run test:tmux-poc
```

**Note:** When sending input to the CLI via tmux, you must use bracketed paste mode. Standard `send-keys` drops characters.

```bash
# ❌ Broken: tmux send-keys -t session "hello"
# ✅ Works:  tmux send-keys -t session $'\e[200~hello\e[201~'
```

See [tmux.knowledge.md](tmux.knowledge.md) for comprehensive tmux documentation and [src/__tests__/README.md](src/__tests__/README.md) for testing documentation.

## Build

Build the package:

```bash
bun run build
```

## Run

Run the built TUI:

```bash
bun run start
```

Or use the binary directly:

```bash
codebuff-tui
```

## Features

- Built with OpenTUI for modern terminal interfaces
- Uses React for declarative component-based UI
- TypeScript support out of the box

---

### Overview (`freebuff/README.md`)

# Freebuff

**The free coding agent.** No subscription. No configuration. Start in seconds.

An AI coding agent that runs in your terminal — describe what you want, and Freebuff edits your code.

## Install

```bash
npm install freebuff
```

## Usage

```bash
cd ~/my-project
freebuff
```

## Project Structure

```
freebuff/
├── cli/       # CLI build & npm release files
└── web/       # Freebuff website
```

## Building from Source

```bash
# From the repo root
bun freebuff/cli/build.ts 1.0.0
```

---

For everything else — what Freebuff does, how it works, FAQ, and how it relates to Codebuff — see the [repo root README](../README.md). We keep that one up to date as the single source of truth.

## License

MIT

---

### Overview (`sdk/README.md`)

# @codebuff/sdk

Official SDK for Codebuff - AI coding agent and framework

## Installation

```bash
npm install @codebuff/sdk
```

## Prerequisites

- Create a Codebuff account and get your [Codebuff API key here](https://www.codebuff.com/api-keys).

## Usage

### Basic Example

```typescript
import { CodebuffClient } from '@codebuff/sdk'

async function main() {
  const client = new CodebuffClient({
    // You need to pass in your own API key here.
    // Get one here: https://www.codebuff.com/api-keys
    apiKey: process.env.CODEBUFF_API_KEY,
    cwd: process.cwd(),
  })

  // First run
  const runState1 = await client.run({
    // The agent id. Any agent on the store (https://codebuff.com/store)
    agent: 'codebuff/base@0.0.16',
    prompt: 'Create a simple calculator class',
    handleEvent: (event) => {
      // All events that happen during the run: agent start/finish, tool calls/results, text responses, errors.
      console.log('Codebuff Event', JSON.stringify(event))
    },
  })

  // Continue the same session with a follow-up
  const runOrError2 = await client.run({
    agent: 'codebuff/base@0.0.16',
    prompt: 'Add unit tests for the calculator',
    previousRun: runState1, // <-- this is where your next run differs from the previous run
    handleEvent: (event) => {
      console.log('Codebuff Event', JSON.stringify(event))
    },
  })
}

main()
```

### Example 2: Custom Agents and Tools

Here, we create a full agent and custom tools that can be reused between runs.

```
/* Detailed source-code truncated for AI context efficiency. */
```

## API Reference

### Knowledge Files

Knowledge files provide project context to the agent. The SDK auto-discovers:

- **Project files**: `knowledge.md`, `AGENTS.md`, or `CLAUDE.md` in each directory (priority order)
- **User files**: `~/.knowledge.md`, `~/.AGENTS.md`, or `~/.CLAUDE.md` (case-insensitive)

Override with `knowledgeFiles` (replaces project files) or `userKnowledgeFiles` (merges with home directory files):

```typescript
await client.run({
  agent: 'codebuff/base@0.0.16',
  prompt: 'Help me refactor',
  knowledgeFiles: { 'knowledge.md': '# Guidelines\n- Use TypeScript' },
  userKnowledgeFiles: { '~/.knowledge.md': '# Preferences\n- Be concise' },
})
```

### File Filtering

The SDK always blocks `.env` and `.env.*` files case-insensitively, except for
the `.env.example`, `.env.sample`, and `.env.template` templates. Use
`fileFilter` to add project-specific rules:

```typescript
const client = new CodebuffClient({
  apiKey: process.env.CODEBUFF_API_KEY,
  fileFilter: (filePath) => {
    if (filePath.endsWith('.pem')) return { status: 'blocked' }
    return { status: 'allow' }
  },
})
```

**Statuses:** `'blocked'` (returns `[BLOCKED]`), `'allow-example'` (prefixes content with `[TEMPLATE]`), `'allow'` (normal read).

**Default behavior:** The built-in `.env` policy always applies. When no
`fileFilter` is provided, gitignore checking also applies automatically,
including for the allowed env templates. When a `fileFilter` is provided, the
caller owns any additional filtering, but env templates remain subject to
gitignore. This built-in policy is scoped to
SDK-mediated agent file reads (including `read_files`); it does not add
terminal-command or internal-edit restrictions.

Custom `overrideTools.read_files` implementations must preserve project
gitignore behavior for env templates. The built-in CLI, Desktop, Web, and Cloud
bridges already do this.

### `loadLocalAgents(options)`

Loads agent definitions from `.agents` directories on disk.

```typescript
import { loadLocalAgents, CodebuffClient } from '@codebuff/sdk'

// Load from default locations (.agents in cwd, parent, or home)
const agents = await loadLocalAgents({ verbose: true })

// Or load from a specific directory
// const agents = await loadLocalAgents({ agentsPath: './my-agents' })

// Or load and validate agents (invalid agents are filtered out)
// const agents = await loadLocalAgents({ validate: true, verbose: true })

// Access source file path for debugging
for (const agent of Object.values(agents)) {
  console.log(`${agent.id} loaded from ${agent._sourceFilePath}`)
}

// Use the loaded agents with client.run()
const client = new CodebuffClient({ apiKey: process.env.CODEBUFF_API_KEY })
const result = await client.run({
  agent: 'my-custom-agent',
  agentDefinitions: Object.values(agents),
  prompt: 'Hello',
})
```

#### Parameters

- **`agentsPath`** (string, optional): Path to a specific agents directory. If omitted, searches in `{cwd}/.agents`, `{cwd}/../.agents`, and `{homedir}/.agents`.
- **`verbose`** (boolean, optional): Whether to log errors during loading. Defaults to `false`.
- **`validate`** (boolean, optional): Whether to validate agents after loading. Invalid agents are filtered out. Defaults to `false`.

#### Returns

Returns a `Promise<LoadedAgents>` - a `Record<string, LoadedAgentDefinition>` of agent definitions keyed by their ID.

Each `LoadedAgentDefinition` extends `AgentDefinition` with:

- **`_sourceFilePath`** (string): The file path the agent was loaded from

#### Supported File Types

- `.ts`, `.tsx` - TypeScript files (automatically transpiled)
- `.js`, `.mjs`, `.cjs` - JavaScript files

Files ending in `.d.ts` or `.test.ts` are excluded.

### `client.run(options)`

Runs a Codebuff agent with the specified options.

#### Parameters

- **`agent`** (string, required): The agent to run. Use `'base'` for the default agent, or specify a custom agent ID if you made your own agent definition (passed with the `agentDefinitions` param).

- **`prompt`** (string, required): The user prompt describing what you want the agent to do.

- **`params`** (object, optional): Additional parameters for the agent. Most agents don't use this, but some custom agents can take a JSON object as input in addition to the user prompt string.

- **`handleEvent`** (function, optional): Callback function that receives every event during execution (assistant messages, tool calls, etc.). This allows you to stream the agent's progress in real-time. We will likely add a token-by-token streaming callback in the future.

- **`previousRun`** (object, optional): JSON state returned from a previous `run()` call. Use this to continue a conversation or session with the agent, maintaining context from previous interactions.

- **`projectFiles`** (object, optional): All the files in your project as a plain JavaScript object. Keys should be the full path from your current directory to each file, and values should be the string contents of the file. Example: `{ "src/index.ts": "console.log('hi')" }`. This helps Codebuff pick good source files for context. Note: This parameter was previously named `allFiles` but has been renamed for clarity.

- **`knowledgeFiles`** (object, optional): Knowledge files to inject into every `run()` call. Uses the same schema as `projectFiles` - keys are file paths and values are file contents. These files are added directly to the agent's context.

- **`agentDefinitions`** (array, optional): Array of custom agent definitions. Each object should satisfy the AgentDefinition type.

- **`customToolDefinitions`** (array, optional): Array of custom tool definitions that extend the agent's capabilities. Each tool definition includes a name, Zod schema for input validation, and a handler function. These tools can be called by the agent during execution.

- **`maxAgentSteps`** (number, optional): Maximum number of steps the agent can take before stopping. Use this as a safety measure in case your agent starts going off the rails. A reasonable number is around 20.

#### Returns

Returns a Promise that resolves to either a "success" or a "failure" object.

- The "success" object contains a `RunState` object which can be passed into subsequent runs via the `previousRun` parameter to resume the conversation.
- The "failure" object contains an `Error` object with a `name`, `message`, and `stack` properties.

The `RunState` object contains:

- `sessionState`: Internal state to be passed to the next run
- `output`: The agent's output (text, error, or other types)

## License

MIT

---

### Overview (`cli/release-core/README.md`)

# Release launcher core

This directory is the canonical implementation of the npm launchers used by
Codebuff, Codecane, and Freebuff. Each product keeps a small `index.js` in its
release package that supplies product-specific configuration to
`createLauncher()`.

The npm packages must remain standalone, so their `prepack` scripts copy
`launcher.js` and `http.js` into the package directory. `postpack` removes those
generated, gitignored files again. No lifecycle scripts run when users install
or uninstall the packages.

## Download integrity

The launcher downloads a `.tar.gz` from
`https://codebuff.com/api/releases/download/<version>/<file>` (a 302 to the
GitHub release asset) and verifies its sha256 **before** extracting it. The
expected hashes never come from the download origin:

- For the wrapper's own version they are read from the package's own
  `package.json` field `binaryChecksums` (`{ "<target>": "<sha256 hex>" }`).
- For a self-update to `latest` (or any other version) they are read from the
  npm registry's version document, which carries the same field.

That makes the npm registry (TLS + the package manifest) the trust root and
leaves the download host, and every redirect hop, unable to substitute a binary.
Verification fails **closed**: a missing or malformed checksum aborts the
install, so `write-binary-checksums.js` runs in each release workflow's npm
publish job — after the binaries are built — to stamp one hash per target into
the package manifest, and its `verify` mode fails the release if any target is
uncovered. Wrappers published before this field existed ignore it, so old
installs keep working.

Transport rules, in `http.js` / `launcher.js`: the `NEXT_PUBLIC_CODEBUFF_APP_URL`
override is honoured only over `https:` (or `http:` on localhost), redirects
never downgrade `https:` to `http:`, and a redirect may only land on the
original host, `codebuff.com`, `freebuff.com`, `github.com` (with `www.`
variants) or `*.githubusercontent.com`. Extraction admits only the binary and
`tree-sitter.wasm` as root-level plain files.

When changing launcher behavior, edit this directory and test all package
assemblies with:

```bash
npm pack ./cli/release --dry-run
npm pack ./cli/release-staging --dry-run
npm pack ./freebuff/cli/release --dry-run
```

---

### Overview (`cli/release-staging/README.md`)

# 🚀 Codecane - The most powerful coding agent (STAGING)

**⚠️ This is a staging/beta release for testing purposes.**

Codecane is a CLI tool that writes code for you.

1. Run `codecane` from your project directory
2. Tell it what to do
3. It will read and write to files and run commands to produce the code you want

Note: Codecane will run commands in your terminal as it deems necessary to fulfill your request.

## Installation

To install Codecane (staging), run:

```bash
npm install codecane@beta
```

(Use `sudo` if you get a permission error.)

## Usage

After installation, you can start Codecane by running:

```bash
codecane [project-directory]
```

If no project directory is specified, Codecane will use the current directory.

Once running, simply chat with Codecane to say what coding task you want done.

## Features

- Understands your whole codebase
- Creates and edits multiple files based on your request
- Can run your tests or type checker or linter; can install packages
- It's powerful: ask Codecane to keep working until it reaches a condition and it will.

Our users regularly use Codecane to implement new features, write unit tests, refactor code, write scripts, or give advice.

## Knowledge Files

To unlock the full benefits of modern LLMs, we recommend storing knowledge alongside your code. Add a `knowledge.md` file anywhere in your project to provide helpful context, guidance, and tips for the LLM as it performs tasks for you.

Codecane can fluently read and write files, so it will add knowledge as it goes. You don't need to write knowledge manually!

Some have said every change should be paired with a unit test. In 2024, every change should come with a knowledge update!

## Tips

1. Type '/help' or just '/' to see available commands.
2. Create a `knowledge.md` file and collect specific points of advice. The assistant will use this knowledge to improve its responses.
3. Type `undo` or `redo` to revert or reapply file changes from the conversation.
4. Press `Esc` or `Ctrl+C` while Codecane is generating a response to stop it.

## Troubleshooting

If you are getting permission errors during installation, try using sudo:

```
sudo npm install codecane@beta
```

If you still have errors, it's a good idea to [reinstall Node](https://nodejs.org/en/download).

## Feedback

We value your input! Please email your feedback to `founders@freebuff.com`. Thank you for using Codecane!

---

### Overview (`cli/release/README.md`)

# The most powerful coding agent

Codebuff is a CLI tool that writes code for you.

1. Run `codebuff` from your project directory
2. Tell it what to do
3. It will read and write to files and run commands to produce the code you want

Note: Codebuff will run commands in your terminal as it deems necessary to fulfill your request.

## Installation

To install Codebuff, run:

```bash
npm install codebuff
```

(Use `sudo` if you get a permission error.)

## Usage

After installation, you can start Codebuff by running:

```bash
codebuff [project-directory]
```

If no project directory is specified, Codebuff will use the current directory.

Once running, simply chat with Codebuff to say what coding task you want done.

## Features

- Understands your whole codebase
- Creates and edits multiple files based on your request
- Can run your tests or type checker or linter; can install packages
- It's powerful: ask Codebuff to keep working until it reaches a condition and it will.

Our users regularly use Codebuff to implement new features, write unit tests, refactor code,write scripts, or give advice.

## Knowledge Files

To unlock the full benefits of modern LLMs, we recommend storing knowledge alongside your code. Add a `knowledge.md` file anywhere in your project to provide helpful context, guidance, and tips for the LLM as it performs tasks for you.

Codebuff can fluently read and write files, so it will add knowledge as it goes. You don't need to write knowledge manually!

Some have said every change should be paired with a unit test. In 2024, every change should come with a knowledge update!

## Tips

1. Type '/help' or just '/' to see available commands.
2. Create a `knowledge.md` file and collect specific points of advice. The assistant will use this knowledge to improve its responses.
3. Type `undo` or `redo` to revert or reapply file changes from the conversation.
4. Press `Esc` or `Ctrl+C` while Codebuff is generating a response to stop it.

## Troubleshooting

### Permission Errors

If you are getting permission errors during installation, try using sudo:

```
sudo npm install codebuff
```

If you still have errors, it's a good idea to [reinstall Node](https://nodejs.org/en/download).

### Corporate Proxy / Firewall

If you see `Failed to download codebuff: Request timeout` or `Failed to determine latest version`, you may be behind a corporate proxy or firewall.

Codebuff respects standard proxy environment variables. Set `HTTPS_PROXY` to route traffic through your proxy:

**Linux / macOS (bash/zsh):**
```bash
export HTTPS_PROXY=http://your-proxy-server:port
codebuff
```

**Windows (PowerShell):**
```powershell
$env:HTTPS_PROXY = "http://your-proxy-server:port"
codebuff
```

**Windows (CMD):**
```cmd
set HTTPS_PROXY=http://your-proxy-server:port
codebuff
```

To make it permanent, add the `export` or `set` line to your shell profile (e.g. `~/.bashrc`, `~/.zshrc`, or Windows System Environment Variables).

**Supported environment variables:**

| Variable | Purpose |
|---|---|
| `HTTPS_PROXY` / `https_proxy` | Proxy for HTTPS requests (recommended) |
| `HTTP_PROXY` / `http_proxy` | Fallback proxy for HTTP requests |
| `NO_PROXY` / `no_proxy` | Comma-separated list of hostnames to bypass the proxy (port suffixes are ignored) |

Both `http://` and `https://` proxy URLs are supported. Proxy authentication is supported via URL credentials (e.g. `http://user:password@proxy:port`).

## Feedback

We value your input! Please email your feedback to `founders@freebuff.com`. Thank you for using Codebuff!

---

### Overview (`evals/buffbench/README.md`)

# BuffBench

BuffBench is Codebuff's evaluation framework for measuring AI coding agent performance through real-world git commit reconstruction tasks.

## Overview

BuffBench evaluates coding agents by having them reconstruct actual git commits from open source repositories. The system uses AI-powered judging to assess both the quality of implementation and the agent's problem-solving process.

### Key Features

- **Parallel Agent Comparison**: Run multiple agents simultaneously on the same tasks for direct performance comparison
- **Trace Analysis**: Deep analysis of agent execution traces to understand decision-making patterns and workflows
- **Meta-Analysis**: Aggregate analysis across all tasks to identify consistent strengths, weaknesses, and patterns
- **Structured Output**: Judges use structured output for more reliable and consistent scoring
- **Final Check Commands**: Run custom validation commands (tests, lints, etc.) after agent implementations
- **Binary Installation**: Support for installing required binaries in isolated environments
- **Enhanced Metrics**: Track completion score, code quality score, and overall score separately

## Architecture

### System Components

1. **Evaluation Orchestration** (`run-buffbench.ts`)
   - Manages the complete evaluation pipeline
   - Handles parallel execution of multiple agents on multiple tasks
   - Coordinates between agents, judges, and analyzers
   - Saves comprehensive logs and traces

2. **Agent Runner** (`agent-runner.ts`)
   - Executes coding agents in isolated test repositories
   - Captures execution traces and metrics
   - Extracts git diffs and context files
   - Runs optional validation commands

3. **Judging System** (`judge.ts`)
   - Uses AI (GPT-5) to score implementations
   - Runs 2 parallel judges (GPT and Sonnet), averaging their scores and keeping the median judge's analysis; warns loudly if one fails, since a silently-dropped judge turns the panel into a single opinion
   - Compares agent output against ground truth git diffs
   - Provides detailed analysis with strengths and weaknesses
   - Scores on three dimensions: completion, code quality, and overall

4. **Trace Analyzer** (`trace-analyzer.ts`)
   - Analyzes agent execution traces after each task
   - Compares how different agents approached the same problem
   - Identifies process strengths, weaknesses, and recommendations
   - Focuses on agent workflows and decision-making patterns

5. **Meta Analyzer** (`meta-analyzer.ts`)
   - Performs aggregate analysis across all tasks
   - Identifies consistent patterns in agent behavior
   - Provides high-level insights and development priorities
   - Compares agent configurations and their impact on performance

6. **Evaluation Generation** (`gen-evals.ts`, `gen-repo-eval.ts`)
   - Generates evaluation tasks from git commits
   - Creates natural language prompts and specifications
   - Extracts file diffs and supplemental context
   - Supports batch processing of multiple commits

### Evaluation Workflow

```mermaid
sequenceDiagram
    participant Orchestrator as BuffBench Orchestrator
    participant AgentRunner as Agent Runner
    participant Agent as Coding Agent
    participant Judge as AI Judge (3x)
    participant TraceAnalyzer as Trace Analyzer
    participant MetaAnalyzer as Meta Analyzer

    loop For each task
        Orchestrator->>AgentRunner: Run all agents in parallel
        
        par Agent 1
            AgentRunner->>Agent: Execute with prompt
            Agent->>AgentRunner: Return trace + changes
        and Agent 2
            AgentRunner->>Agent: Execute with prompt
            Agent->>AgentRunner: Return trace + changes
        end
        
        AgentRunner->>Judge: Judge each agent's output (3 judges)
        Judge->>AgentRunner: Return median scores + analysis
        
        AgentRunner->>TraceAnalyzer: Analyze all agent traces
        TraceAnalyzer->>AgentRunner: Return process insights
        
        Orchestrator->>Orchestrator: Save task results + analysis
    end
    
    Orchestrator->>MetaAnalyzer: Analyze all tasks
    MetaAnalyzer->>Orchestrator: Return aggregate insights
```

## Key Features

### Multi-Agent Comparison

- **Parallel Execution**: Run multiple agents simultaneously on the same tasks
- **Fair Comparison**: All agents work from identical repository states
- **Comprehensive Metrics**: Compare scores, costs, duration, and success rates
- **Process Analysis**: Understand how agents differ in their problem-solving approaches

### Advanced Analysis

#### Trace Analysis (Per-Task)
After each task, the trace analyzer examines:
- How agents approached the problem
- Tool usage patterns and sequences
- Context gathering strategies
- Decision-making workflows
- Relative performance comparison

#### Meta-Analysis (Across All Tasks)
After all tasks complete, the meta analyzer identifies:
- Consistent strengths and weaknesses per agent
- Performance trends and patterns
- Cost vs quality trade-offs
- Reliability and consistency metrics
- Actionable development priorities

### Comprehensive Scoring

The AI judge evaluates three dimensions:

- **Completion Score (0-10)**: How completely was the prompt addressed?
- **Code Quality Score (0-10)**: How well-structured and maintainable is the code?
- **Overall Score (0-10)**: Combined assessment of implementation quality

### Validation Support

- **Final Check Commands**: Run tests, lints, or other validation after agent execution
- **Binary Installation**: Install required tools (e.g., linters, test runners) in isolated environments
- **Custom Environment**: Set environment variables for evaluation runs

### External CLI Agents

BuffBench supports running external CLI coding agents for comparison:

- **Claude Code**: Use `external:claude` - requires `claude` CLI installed
- **Codex**: Use `external:codex` - requires `codex` CLI installed
- **OpenCode**: Use `external:opencode` - requires `opencode` CLI installed

Example comparing Codebuff vs Claude Code:

```typescript
await runBuffBench({
  evalDataPath: 'evals/buffbench/eval-codebuff.json',
  agents: ['base2', 'external:claude'],
  taskConcurrency: 3,
})
```

### Prerequisites for External Agents

**Claude Code CLI:**
```bash
npm install @anthropic-ai/claude-code
# Set ANTHROPIC_API_KEY or CLAUDE_CODE_KEY environment variable
```

**Codex CLI:**
```bash
npm install @openai/codex
# Set OPENAI_API_KEY environment variable
```

**OpenCode CLI:**
```bash
# Install from https://opencode.ai/docs/install
# Set OPENCODE_API_KEY environment variable
# BuffBench uses opencode/kimi-k2.6 by default; override with OPENCODE_MODEL if needed.
```

## Directory Structure

```
evals/buffbench/
├── run-buffbench.ts          # Main orchestrator
├── main.ts                   # Example usage
├── main-nightly.ts           # Nightly evaluation runner with email reporting
├── main-single-eval.ts       # Single evaluation runner
│
├── agent-runner.ts           # Executes agents in test repos
├── freestyle-sandbox.ts      # --freestyle: one Freestyle VM per agent run
├── freestyle-task.ts         # The in-VM entry point of a Freestyle run
├── judge.ts                  # AI judging system
├── trace-analyzer.ts         # Per-task trace analysis
├── meta-analyzer.ts          # Cross-task meta analysis
│
├── gen-evals.ts              # Generate eval tasks from commits
├── gen-repo-eval.ts          # End-to-end eval creation
├── eval-task-generator.ts    # Task prompt generation
│
├── format-output.ts          # Result formatting utilities
├── trace-utils.ts            # Trace processing utilities
├── lessons-extractor.ts      # Extract lessons from failures
│
├── types.ts                  # Type definitions
│
├── eval-codebuff.json        # Codebuff project evaluations
├── eval-manifold.json        # Manifold evaluations
├── eval-plane.json           # Plane project evaluations
└── eval-saleor.json          # Saleor e-commerce evaluations
```

## Usage

### Running Evaluations

#### Basic Example

```typescript
import { runBuffBench } from './run-buffbench'

await runBuffBench({
  evalDataPath: 'eval-codebuff.json',
  agents: ['base2', 'base2-fast'],
  taskConcurrency: 3,
})
```

#### Command Line

```bash
# Run evaluations
bun run evals/buffbench/main.ts

# Run single evaluation
bun run evals/buffbench/main-single-eval.ts

# Run nightly evaluation with email reporting
bun run evals/buffbench/main-nightly.ts
```

#### Running in Freestyle VMs (recommended)

Pass `--freestyle` to any `main-*.ts` entry point (or `sandbox: 'freestyle'` to `runBuffBench`) and every agent run gets its own [Freestyle](https://freestyle.sh) VM, so a laptop that sleeps delays the run without tainting it:

```bash
cd evals
export NEXT_PUBLIC_CB_ENVIRONMENT=prod NEXT_PUBLIC_CODEBUFF_APP_URL=https://www.codebuff.com
PORT=4242 bun --env-file=../.env.local run buffbench/main-fast-harness.ts 5 all --freestyle
```

The VM receives this checkout's bench code (uncommitted edits included, never the `eval-*.json` files), runs the clone, agent and final checks as a local run does (`freestyle-task.ts`), and powers off; this machine polls and judges (`freestyle-sandbox.ts`). It needs `FREESTYLE_API_KEY` and a backend the VM can reach (not `localhost`). Running VMs = `taskConcurrency` × agents, which must fit the Freestyle plan. A run that ends without a result is an error carrying the VM's log.

### Creating New Evaluations

#### 1. Generate from Specific Commits

```bash
bun run evals/buffbench/gen-evals.ts \
  https://github.com/user/repo \
  abc123 \
  def456 \
  ghi789
```

#### 2. Generate from Repository (End-to-End)

```bash
bun run evals/buffbench/gen-repo-eval.ts \
  https://github.com/user/repo
```

This will:
1. Clone the repository
2. Select high-quality commits using AI
3. Generate evaluation tasks for each commit
4. Save the complete evaluation file

## Evaluation Data Format

### Evaluation File Structure

```typescript
interface EvalDataV2 {
  repoUrl: string              // Source repository URL
  testRepoName?: string        // Optional repo name override
  generationDate: string       // ISO timestamp of creation
  initCommand?: string         // Optional setup command
  binInstalls?: BinInstall[]   // Binaries to install
  env?: Record<string, string> // Environment variables
  finalCheckCommands?: string[] // Validation commands
  evalCommits: EvalCommitV2[]  // List of evaluation tasks
}

interface EvalCommitV2 {
  id: string                   // Unique task identifier
  sha: string                  // Target commit SHA
  parentSha: string            // Parent commit SHA
  spec: string                 // Technical specification
  prompt: string               // Natural language prompt
  supplementalFiles: string[]  // Context files
  fileDiffs: FileDiff[]        // Ground truth changes
}

interface FileDiff {
  path: string                 // File path
  status: 'modified' | 'added' | 'deleted' | 'renamed'
  oldPath?: string             // For renamed files
  diff: string                 // Unified diff
}
```

### Results Format

```typescript
interface AgentEvalResults {
  agentId: string
  runs: EvalRun[]
  averageScore: number               // Overall score across valid runs
  averageScoreExcludingFailures: number // Score excluding failures (≤1.0)
  averageCost: number
  averageDuration: number
}

interface EvalRun {
  commitSha: string
  prompt: string
  diff: string                       // Agent's changes
  judging: JudgingResult
  cost: number
  durationMs: number
  error?: string
  finalCheckOutputs?: FinalCheckOutput[]
}

interface JudgingResult {
  analysis: string                   // Detailed analysis
  strengths: string[]
  weaknesses: string[]
  completionScore: number            // 0-10
  codeQualityScore: number           // 0-10
  overallScore: number               // 0-10
}
```

## Advanced Features

### Binary Installation

Install required binaries in isolated environments:

```json
{
  "binInstalls": [
    {
      "name": "ruff",
      "installScript": "curl -LsSf https://astral.sh/ruff/install.sh | sh -s -- --to $INSTALL_DIR",
      "binPath": "ruff"
    }
  ]
}
```

### Final Check Commands

Run validation commands after agent execution:

```json
{
  "finalCheckCommands": [
    "npm run typecheck",
    "npm test",
    "npm run lint"
  ]
}
```

### Nightly Evaluations

Run automated nightly evaluations with email reporting:

The nightly evaluation runner automatically sends email reports with results.

```bash
bun run evals/buffbench/main-nightly.ts
```

## Output Files

BuffBench creates a timestamped logs directory for each run:

```
logs/YYYY-MM-DDTHH-MM_agent1_vs_agent2/
├── 1-task-id-agent1-abc123.json      # Agent 1 trace for task 1
├── 1-task-id-agent2-abc123.json      # Agent 2 trace for task 1
├── 1-task-id-ANALYSIS-abc123.json    # Trace analysis for task 1
├── 2-task-id-agent1-def456.json      # Agent 1 trace for task 2
├── 2-task-id-agent2-def456.json      # Agent 2 trace for task 2
├── 2-task-id-ANALYSIS-def456.json    # Trace analysis for task 2
└── FINAL_RESULTS.json                # Complete results + meta-analysis
```

### FINAL_RESULTS.json Structure

```json
{
  "metadata": {
    "timestamp": "2024-01-15T10:30:00.000Z",
    "evalDataPath": "eval-codebuff.json",
    "agentsTested": ["base2", "base2-fast"],
    "commitsEvaluated": 10,
    "logsDirectory": "logs/..."
  },
  "metaAnalysis": {
    "overallComparison": "...",
    "agentInsights": [...],
    "keyFindings": [...]
  },
  "base2": {
    "agentId": "base2",
    "runs": [...],
    "averageScore": 7.5,
    "averageCost": 0.0234,
    "averageDuration": 45000
  },
  "base2-fast": {...}
}
```

## Configuration

### Environment Variables

```bash
# Required
CODEBUFF_API_KEY=your-api-key

# Optional
EVAL_RESULTS_EMAIL=team@freebuff.com  # For nightly email reports
```

### Task Concurrency

Control parallel execution:

```typescript
taskConcurrency: 1   // Sequential (safest, slowest)
taskConcurrency: 3   // Moderate parallelism
taskConcurrency: 10  // High parallelism (faster, more resources)
```

## Examples

The `evals/buffbench/` directory contains several example evaluation files:

- **eval-codebuff.json** - Codebuff project evaluations
- **eval-manifold.json** - Manifold prediction market evaluations
- **eval-plane.json** - Plane project management evaluations
- **eval-saleor.json** - Saleor e-commerce platform evaluations

These demonstrate the evaluation format and provide ready-to-use test cases.

---

### Overview (`freebuff/e2e/README.md`)

# Freebuff E2E Tests

End-to-end tests for the Freebuff CLI binary. Tests verify that the compiled binary works correctly by interacting with it via tmux.

## Architecture

Two testing approaches are supported:

### 1. Direct tmux tests (fast, deterministic)

Use the `FreebuffSession` class to start the binary in tmux, send commands, capture output, and assert directly.

```typescript
import { describe, test, expect, afterEach } from 'bun:test'
import { FreebuffSession, requireFreebuffBinary } from '../utils'

describe('My Feature', () => {
  let session: FreebuffSession | null = null

  afterEach(async () => {
    if (session) await session.stop()
    session = null
  })

  test('works correctly', async () => {
    const binary = requireFreebuffBinary()
    session = await FreebuffSession.start(binary)

    await session.send('/help')
    const output = await session.capture(2)

    expect(output).toContain('Shortcuts')
  }, 60_000)
})
```

### 2. SDK agent-driven tests (AI-powered verification)

Use the Codebuff SDK to run a testing agent that interacts with Freebuff via custom tmux tools. The agent reasons about the CLI output and verifies complex behaviors.

```typescript
import { describe, test, expect, afterEach } from 'bun:test'
import { CodebuffClient } from '@codebuff/sdk'
import { freebuffTesterAgent } from '../agent/freebuff-tester'
import { createFreebuffTmuxTools, requireFreebuffBinary } from '../utils'

describe('Agent Test', () => {
  let cleanup: (() => Promise<void>) | null = null

  afterEach(async () => {
    if (cleanup) await cleanup()
    cleanup = null
  })

  test('verifies startup', async () => {
    const apiKey = process.env.CODEBUFF_API_KEY
    if (!apiKey) return // Skip if no API key

    const binary = requireFreebuffBinary()
    const tmuxTools = createFreebuffTmuxTools(binary)
    cleanup = tmuxTools.cleanup

    const client = new CodebuffClient({ apiKey })
    const result = await client.run({
      agent: freebuffTesterAgent.id,
      prompt: 'Start Freebuff and verify the branding is correct.',
      agentDefinitions: [freebuffTesterAgent],
      customToolDefinitions: tmuxTools.tools,
      handleEvent: () => {},
    })

    expect(result.output.type).not.toBe('error')
  }, 180_000)
})
```

### 3. Live smoke against production

`tests/live-turn.e2e.test.ts` is the one file here that talks to the real backend:
it navigates the landing picker to DeepSeek V4.1 Flash, starts a session, sends a
prompt, waits for the answer, then runs `/end-session` and checks the backend shows
no open session. It skips unless `FREEBUFF_SMOKE_API_KEY` (or
`CODEBUFF_API_KEY`) is set, and needs a binary built with the production public
env. It is not in `freebuff-e2e.yml` — `.github/workflows/prod-smoke.yml`
runs it on a schedule and before each release.

```bash
NEXT_PUBLIC_CB_ENVIRONMENT=prod NEXT_PUBLIC_CODEBUFF_APP_URL=https://www.codebuff.com \
NEXT_PUBLIC_FREEBUFF_APP_URL=https://freebuff.com NEXT_PUBLIC_SUPPORT_EMAIL=support@freebuff.com \
NEXT_PUBLIC_POSTHOG_API_KEY=test NEXT_PUBLIC_POSTHOG_HOST_URL=http://127.0.0.1:9 \
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=test NEXT_PUBLIC_STRIPE_CUSTOMER_PORTAL=http://127.0.0.1:9 \
NEXT_PUBLIC_WEB_PORT=3000 bun freebuff/cli/build.ts 0.0.0-smoke
CODEBUFF_API_KEY=<your token> bun test freebuff/e2e/tests/live-turn.e2e.test.ts --timeout=300000
```

The CLI runs with its own `FREEBUFF_CONFIG_DIR`, so your real profile is untouched.

## Prerequisites

- **tmux** must be installed: `brew install tmux` (macOS) or `sudo apt-get install tmux` (Ubuntu)
- **Freebuff binary** must be built: `bun freebuff/cli/build.ts 0.0.0-dev`
- **SDK built** (for agent tests): `cd sdk && bun run build`
- **CODEBUFF_API_KEY** (for agent tests only): Set this environment variable

## Running Tests

### Build the binary first

```bash
bun freebuff/cli/build.ts 0.0.0-dev
```

### Run all tests

```bash
bun test freebuff/e2e/tests/
```

### Run a specific test

```bash
bun test freebuff/e2e/tests/version.e2e.test.ts
bun test freebuff/e2e/tests/startup.e2e.test.ts
bun test freebuff/e2e/tests/help-command.e2e.test.ts
bun test freebuff/e2e/tests/agent-startup.e2e.test.ts
```

### Use a custom binary path

```bash
FREEBUFF_BINARY=/path/to/freebuff bun test freebuff/e2e/tests/
```

## Adding New Tests

1. Create a new file in `freebuff/e2e/tests/` with the naming convention `<feature>.e2e.test.ts`
2. Add an `e2e - <name>` step to the `build-and-e2e-freebuff` job in
   `.github/workflows/freebuff-e2e.yml`, with the same `if:` as its siblings so it
   still runs when another suite fails. Keep it off the network: that job has no
   credentials, so anything that needs a real backend belongs in `prod-smoke.yml`.

## CI Workflow

The `.github/workflows/freebuff-e2e.yml` workflow:

1. **Builds** the Freebuff binary once (linux-x64) and smoke-boots it
2. **Runs each suite as its own step** in that same job, against that binary
3. **Uploads tmux session logs** on failure for debugging
4. **Builds and smoke-boots on Windows** in a separate job (no tmux there)

Triggers: pull requests to `main` or `great-db-merge-v2`, pushes to `main`, manual
`workflow_dispatch`, and `freebuff-release.yml` before each release.

## Utilities Reference

### `FreebuffSession`

| Method | Description |
|--------|-------------|
| `FreebuffSession.start(binaryPath)` | Start binary in tmux, returns session |
| `session.send(text)` | Send text input (presses Enter) |
| `session.sendKey(key)` | Send special key (e.g. `'C-c'`, `'Escape'`) |
| `session.capture(waitSec?)` | Capture terminal output |
| `session.captureLabeled(label, waitSec?)` | Capture and save to session logs |
| `session.waitForText(pattern, timeoutMs?)` | Poll until text appears |
| `session.stop()` | Stop session and clean up |

### `createFreebuffTmuxTools(binaryPath)`

Creates SDK custom tools for agent-driven testing:
- `start_freebuff` - Launch the CLI
- `send_to_freebuff` - Send text input
- `capture_freebuff_output` - Capture terminal output
- `stop_freebuff` - Stop and clean up

### Helper functions

| Function | Description |
|----------|-------------|
| `requireFreebuffBinary()` | Get binary path, throws if not found |
| `getFreebuffBinaryPath()` | Get binary path (may not exist) |

---

### Overview (`scripts/tmux/README.md`)

# tmux TUI Testing Scripts

Helper scripts for testing TUI applications (Terminal User Interface apps) using tmux. These scripts abstract away the complexity of tmux communication, particularly the **bracketed paste mode** requirement.

## Features

- **Automatic bracketed paste mode** - Input is wrapped in escape sequences so characters don't get dropped
- **Automatic session logs** - Every capture is saved to `debug/tmux-sessions/{session}/` for debugging
- **Works with any TUI app** - Codebuff, Claude Code, Codex, or any custom TUI application
- **Paper trail** - Both the subagent and parent agent can review what the app displayed

## Why These Scripts?

Many TUI apps (using frameworks like OpenTUI, Ink, Textual, etc.) process keyboard input character-by-character. When tmux sends characters rapidly via `send-keys`, they get dropped or garbled. These scripts automatically wrap input in bracketed paste escape sequences (`\e[200~...\e[201~`), which tells the terminal to process the input atomically.

**Without these scripts:**
```bash
# ❌ Characters get dropped!
tmux send-keys -t session "hello world"
# Result: Only "d" appears in the input!
```

**With these scripts:**
```bash
# ✅ Works correctly
./scripts/tmux/tmux-send.sh session "hello world"
# Result: "hello world" appears correctly
```

## Quick Start

### Testing Any TUI App

```bash
# Start a test session with any command
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "claude")
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "codex chat")
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "python my_tui.py")
echo "Started session: $SESSION"

# Send a command
./scripts/tmux/tmux-cli.sh send "$SESSION" "/help"

# Wait and capture output
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 2

# Clean up
./scripts/tmux/tmux-cli.sh stop "$SESSION"
```

### Testing Codebuff (Default)

```bash
# Start a test session (defaults to Codebuff dev server)
SESSION=$(./scripts/tmux/tmux-cli.sh start)
echo "Started session: $SESSION"

# Send a command
./scripts/tmux/tmux-cli.sh send "$SESSION" "/help"

# Wait and capture output
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 2

# Clean up
./scripts/tmux/tmux-cli.sh stop "$SESSION"
```

### Testing Compiled Binaries

```bash
# Build the binary first (Codebuff example)
cd cli && bun run build:binary

# Test with binary at default location (./cli/bin/codebuff)
SESSION=$(./scripts/tmux/tmux-cli.sh start --binary)

# Or specify a custom binary path
SESSION=$(./scripts/tmux/tmux-cli.sh start --binary /path/to/binary)

# Or use environment variable
CODEBUFF_BINARY=./cli/bin/codebuff ./scripts/tmux/tmux-cli.sh start
```

The session-info.yaml will record which mode was used (`cli_mode: custom`, `cli_mode: binary`, or `cli_mode: dynamic`).

## Scripts

### `tmux-cli.sh` (Unified Interface)

The main entry point with subcommands:

```bash
./scripts/tmux/tmux-cli.sh start      # Start a session
./scripts/tmux/tmux-cli.sh send       # Send input
./scripts/tmux/tmux-cli.sh capture    # Capture output
./scripts/tmux/tmux-cli.sh stop       # Stop a session
./scripts/tmux/tmux-cli.sh list       # List sessions
./scripts/tmux/tmux-cli.sh help       # Show help
```

### `tmux-start.sh`

Start a new tmux session with a TUI app.

```bash
# Start with a custom command (any TUI app)
./scripts/tmux/tmux-start.sh --command "claude"
./scripts/tmux/tmux-start.sh --command "codex chat"
./scripts/tmux/tmux-start.sh --command "python my_app.py"

# Default settings (Codebuff dev server)
./scripts/tmux/tmux-start.sh
# Output: tui-test-1234567890

# Custom session name
./scripts/tmux/tmux-start.sh --name my-test

# Custom dimensions
./scripts/tmux/tmux-start.sh -w 160 -h 40

# Custom wait time for app initialization
./scripts/tmux/tmux-start.sh --wait 6

# Test a compiled binary (default location: ./cli/bin/codebuff)
./scripts/tmux/tmux-start.sh --binary

# Test a compiled binary at custom path
./scripts/tmux/tmux-start.sh --binary /path/to/binary
```

### `tmux-send.sh`

Send input to a running session.

```bash
# Send text (auto-presses Enter)
./scripts/tmux/tmux-send.sh SESSION "your prompt"

# Send without pressing Enter
./scripts/tmux/tmux-send.sh SESSION "partial" --no-enter

# Send special keys
./scripts/tmux/tmux-send.sh SESSION --key Escape
./scripts/tmux/tmux-send.sh SESSION --key C-c
./scripts/tmux/tmux-send.sh SESSION --key Enter

# Send and wait for CLI to finish streaming (for rapid message tests)
./scripts/tmux/tmux-send.sh SESSION "hello" --wait-idle 2

# Paste clipboard content and submit immediately
./scripts/tmux/tmux-send.sh SESSION --paste

# Paste clipboard but don't submit (useful for testing attachment UI)
# This lets you capture a screenshot of the attachment card before sending
./scripts/tmux/tmux-send.sh SESSION --paste --no-enter
```

### `tmux-capture.sh`

Capture output from a session. **Automatically saves captures** to `debug/tmux-sessions/{session}/`.

```bash
# Basic capture (auto-saves to session logs)
./scripts/tmux/tmux-capture.sh SESSION

# Capture with a descriptive label (recommended)
./scripts/tmux/tmux-capture.sh SESSION --label "after-help-command"
# Saves to: debug/tmux-sessions/SESSION/capture-{sequence}-after-help-command.txt

# Wait before capturing
./scripts/tmux/tmux-capture.sh SESSION --wait 3

# Capture without auto-saving to session logs
./scripts/tmux/tmux-capture.sh SESSION --no-save

# Preserve colors
./scripts/tmux/tmux-capture.sh SESSION --colors

# Save to specific file (disables auto-save)
./scripts/tmux/tmux-capture.sh SESSION -o output.txt
```

### `tmux-stop.sh`

Stop/kill sessions.

```bash
# Stop specific session
./scripts/tmux/tmux-stop.sh SESSION

# Stop all test sessions (tui-test-* and cli-test-*)
./scripts/tmux/tmux-stop.sh --all

# List sessions first, then stop
./scripts/tmux/tmux-stop.sh --list SESSION
```

## Example: Full Test Workflow

```bash
#!/bin/bash

# Start session with a TUI app
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "my-tui-app" --name my-test)
echo "Testing in: $SESSION"

# Verify app started (capture auto-saved with label)
./scripts/tmux/tmux-cli.sh capture "$SESSION" --label "initial-state"

# Send /status command
./scripts/tmux/tmux-cli.sh send "$SESSION" "/status"
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 2 --label "after-status"

# Test a prompt
./scripts/tmux/tmux-cli.sh send "$SESSION" "hello world"
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 5 --label "after-prompt"

# Clean up
./scripts/tmux/tmux-cli.sh stop "$SESSION"
echo "Test complete!"

# View all session logs
ls -la debug/tmux-sessions/$SESSION/
```

## Session Logs Directory Structure

```
debug/tmux-sessions/
└── tui-test-1234567890/
    ├── session-info.yaml             # Session metadata (YAML format)
    ├── commands.yaml                 # Log of all commands sent (YAML array)
    ├── capture-001-initial-state.txt # Captures with YAML front-matter
    ├── capture-002-after-status.txt
    └── capture-003-after-prompt.txt
```

### Session Info (session-info.yaml)

```yaml
session: tui-test-1234567890
started: 2025-01-01T12:00:00Z
started_local: Wed Jan  1 12:00:00 PST 2025
dimensions:
  width: 120
  height: 30
cli_mode: custom  # or "binary" or "dynamic"
cli_command: claude  # or path to binary, or "bun --cwd=cli run dev"
status: active
```

### Commands Log (commands.yaml)

The `commands.yaml` file records every input sent to the app as a YAML array:

```yaml
- timestamp: 2025-01-01T12:00:05Z
  type: text
  input: "/help"
  auto_enter: true

- timestamp: 2025-01-01T12:00:10Z
  type: text
  input: "hello world"
  auto_enter: true

- timestamp: 2025-01-01T12:00:15Z
  type: key
  input: "Escape"

- timestamp: 2025-01-01T12:00:20Z
  type: text
  input: "partial input"
  auto_enter: false
```

### Capture Files with YAML Front-Matter

Each capture file includes YAML front-matter with metadata:

```yaml
---
sequence: 1
label: initial-state
timestamp: 2025-01-01T12:00:30Z
after_command: null
dimensions:
  width: 120
  height: 30
---
[actual terminal output below]
```

The front-matter provides:
- **sequence**: Order of captures (1, 2, 3, ...)
- **label**: Descriptive label if provided via `--label`
- **timestamp**: ISO 8601 timestamp
- **after_command**: The last command sent before this capture (or `null`)
- **dimensions**: Terminal dimensions at capture time

This structured format enables both human reading and programmatic parsing (AI agents, viewers, etc.).

## Debugging

### Attach to a Session Interactively

```bash
tmux attach -t SESSION_NAME
# Detach with Ctrl+B, D
```

### List All Sessions

```bash
./scripts/tmux/tmux-cli.sh list
# or
tmux list-sessions
```

### Check If Session Exists

```bash
tmux has-session -t SESSION_NAME && echo "exists" || echo "not found"
```

## Prerequisites

- **tmux** must be installed:
  - macOS: `brew install tmux`
  - Ubuntu: `sudo apt-get install tmux`
  - Arch: `sudo pacman -S tmux`

## Used By

These scripts are used by TUI testing agents:
- `@codebuff-tester` - Tests the Codebuff CLI
- `@claude-code-tester` - Tests Claude Code CLI
- `@codex-tester` - Tests OpenAI Codex CLI
- `@gemini-tester` - Tests Google Gemini CLI
- Custom agents for other TUI apps can easily be created following the same pattern

---

### Overview (`sdk/e2e/README.md`)

# SDK Tests & Examples

This folder contains end-to-end tests, integration tests, unit tests, and runnable examples for the Codebuff SDK.

## Directory Structure

```
sdk/e2e/
├── streaming/          # E2E tests for streaming behavior
├── workflows/          # E2E tests for multi-step workflows  
├── custom-agents/      # E2E tests for custom agent & tool integration
├── features/           # E2E tests for SDK feature coverage
├── integration/        # Integration tests (SDK + API mechanics)
├── examples/           # Runnable example scripts (not tests)
└── utils/              # Shared utilities
    ├── __tests__/      # Unit tests for utilities
    ├── event-collector.ts
    ├── get-api-key.ts
    └── test-fixtures.ts
```

## Test Categories

### E2E Tests (`test:e2e`)
Full end-to-end tests that exercise complete user workflows with real API calls.

```bash
bun run test:e2e
```

| Category | Description |
|----------|-------------|
| `streaming/` | Subagent streaming, concurrent streams |
| `workflows/` | Multi-turn conversations, error recovery |
| `custom-agents/` | Custom agents with custom tools |
| `features/` | projectFiles, knowledgeFiles, maxAgentSteps |

### Integration Tests (`test:integration`)
Tests that verify SDK + API integration mechanics.

```bash
bun run test:integration
```

| Test | Description |
|------|-------------|
| `event-types` | Validates all PrintModeEvent types are emitted |
| `event-ordering` | Validates event sequence (start → content → finish) |
| `stream-chunks` | Tests handleStreamChunk callback |
| `connection-check` | Tests checkConnection() method |

### Unit Tests (`test:unit:e2e`)
Pure unit tests with no external dependencies.

```bash
bun run test:unit:e2e
```

| Test | Description |
|------|-------------|
| `event-collector.test.ts` | Tests EventCollector utility class |

### Examples
Runnable scripts demonstrating SDK usage patterns. Not tests - just examples!

```bash
# Run an example
bun run sdk/e2e/examples/code-reviewer.example.ts
```

| Example | Description |
|---------|-------------|
| `code-reviewer.example.ts` | AI code review |
| `code-explainer.example.ts` | Explain code in plain English |
| `commit-message-generator.example.ts` | Generate commit messages from diffs |
| `sdk-lint.example.ts` | AI-powered linter |
| `sdk-refactor.example.ts` | Code refactoring |
| `sdk-test-gen.example.ts` | Unit test generation |

## Running All Tests

```bash
# All E2E tests
bun run test:e2e

# All integration tests
bun run test:integration

# All unit tests
bun run test:unit:e2e

# Everything
bun run test:e2e && bun run test:integration && bun run test:unit:e2e
```

## Prerequisites

- **API Key**: Set `CODEBUFF_API_KEY` for E2E and integration tests
- **Opt-in**: Set `RUN_CODEBUFF_E2E=true` for local live API runs (CI runs automatically)
- Tests skip gracefully if API key is not set

## Writing Tests

### E2E Test Pattern
```typescript
import { describe, test, expect, beforeAll } from 'bun:test'
import { CodebuffClient } from '../../src/client'
import { EventCollector, getApiKey, skipIfNoApiKey, isAuthError, DEFAULT_AGENT, DEFAULT_TIMEOUT } from '../utils'

describe('E2E: My Test', () => {
  let client: CodebuffClient

  beforeAll(() => {
    if (skipIfNoApiKey()) return
    client = new CodebuffClient({ apiKey: getApiKey() })
  })

  test('does something', async () => {
    if (skipIfNoApiKey()) return
    const collector = new EventCollector()
    
    const result = await client.run({
      agent: DEFAULT_AGENT,
      prompt: 'Test prompt',
      handleEvent: collector.handleEvent,
    })

    if (isAuthError(result.output)) return
    
    expect(result.output.type).not.toBe('error')
  }, DEFAULT_TIMEOUT)
})
```

### Unit Test Pattern
```typescript
import { describe, test, expect, beforeEach } from 'bun:test'
import { EventCollector } from '../event-collector'

describe('Unit: EventCollector', () => {
  let collector: EventCollector

  beforeEach(() => {
    collector = new EventCollector()
  })

  test('collects events', () => {
    collector.handleEvent({ type: 'start', messageHistoryLength: 0 })
    expect(collector.events).toHaveLength(1)
  })
})
```

---

### Overview (`cli/src/__tests__/README.md`)

# CLI Testing

Comprehensive testing suite for the Codebuff CLI using tmux for interactive terminal emulation.

## Test Naming Convention

**IMPORTANT:** Follow these patterns for automatic tmux detection:

- **Unit tests:** `*.test.ts` (e.g., `cli-args.test.ts`)
- **E2E tests:** `e2e-*.test.ts` (e.g., `e2e-cli.test.ts`)
- **Integration tests:** `integration-*.test.ts` (e.g., `integration-tmux.test.ts`)

Files matching `*integration*.test.ts` or `*e2e*.test.ts` trigger automatic tmux availability checking in `.bin/bun`.

## Quick Start

```bash
cd cli
bun test
```

## Prerequisites

### For Integration Tests

Install tmux for interactive CLI testing:

```bash
# macOS
brew install tmux

# Ubuntu/Debian
sudo apt-get install tmux

# Windows (via WSL)
wsl --install
sudo apt-get install tmux
```

### For E2E Tests

Build the SDK first:

```bash
cd sdk
bun run build
cd ../cli
```

## Running Tests

### All Tests

```bash
bun test
```

### Specific Test Suites

```bash
# Unit tests
bun test cli-args.test.ts

# E2E tests (requires SDK)
bun test e2e-cli.test.ts

# Integration tests (requires tmux)
bun test integration-tmux.test.ts
```

### Manual tmux POC

```bash
bun run test:tmux-poc
```

## Automatic tmux Detection

The `.bin/bun` wrapper automatically checks for tmux when running integration/E2E tests:

- **Detects** test files matching `*integration*.test.ts` or `*e2e*.test.ts`
- **Checks** if tmux is installed
- **Shows** installation instructions if missing
- **Skips** tests gracefully if tmux unavailable

**Benefits:**

- ✅ Project-wide (works in any package)
- ✅ No hardcoded paths
- ✅ Clear test categorization
- ✅ Automatic dependency validation

## Test Structure

### Unit Tests

Test individual functions in isolation:

```typescript
import { describe, test, expect } from 'bun:test'

describe('CLI Arguments', () => {
  test('parses --agent flag', () => {
    // Test implementation
  })
})
```

### Integration Tests (tmux)

Test interactive CLI with full terminal emulation:

```typescript
import { describe, test, expect } from 'bun:test'
import { isTmuxAvailable } from './test-utils'

const tmuxAvailable = isTmuxAvailable()

describe.skipIf(!tmuxAvailable)('CLI Integration Tests', () => {
  test('handles user input', async () => {
    // Create tmux session
    // Send commands
    // Verify output
  })
})
```

### E2E Tests

Test complete CLI workflows:

```typescript
import { describe, test, expect } from 'bun:test'
import { isSDKBuilt } from './test-utils'

const sdkBuilt = isSDKBuilt()

describe.skipIf(!sdkBuilt)('CLI E2E Tests', () => {
  test('runs --help command', async () => {
    // Test CLI behavior
  })
})
```

## Test Utilities

Shared utilities in `test-utils.ts`:

```typescript
import { isTmuxAvailable, isSDKBuilt, sleep } from './test-utils'

// Check for tmux
if (isTmuxAvailable()) {
  // Run tmux tests
}

// Check for SDK
if (isSDKBuilt()) {
  // Run E2E tests
}

// Async delay
await sleep(1000)
```

## tmux Testing

**See [`../../tmux.knowledge.md`](../../tmux.knowledge.md) for comprehensive tmux documentation**, including:

- Why standard `send-keys` doesn't work (must use bracketed paste mode)
- Helper functions for Bash and TypeScript
- Complete example scripts
- Debugging and troubleshooting tips

**Quick reference:**

```typescript
// ❌ Broken:
await tmux(['send-keys', '-t', session, 'hello'])

// ✅ Works:
await tmux(['send-keys', '-t', session, '-l', '\x1b[200~hello\x1b[201~'])
```

## Debugging Tests

### View Test Output

```bash
# Verbose test output
bun test --verbose

# Watch mode
bun test --watch
```

## Contributing

When adding new tests:

1. **Follow naming convention** (`*integration*.test.ts` or `*e2e*.test.ts`)
2. **Use test-utils.ts** for shared functionality
3. **Add graceful skipping** for missing dependencies
4. **Clean up resources** (tmux sessions, temp files)
5. **Document test purpose** clearly in test descriptions

## Troubleshooting

### tmux Not Found

```
⚠️  tmux not found but required for integration/E2E tests
```

**Solution:** Install tmux (see Prerequisites above)

### SDK Not Built

```
✓ Build SDK for E2E tests: cd sdk && bun run build [skip]
```

**Solution:** Build the SDK first (see Prerequisites above)

### Tests Hanging

- Check tmux session isn't waiting for input
- Ensure proper cleanup in `finally` blocks
- Use timeouts for tmux operations

### Session Already Exists

- Use unique session names (e.g., timestamp suffix)
- Clean up sessions in `beforeEach`/`afterEach`

## Performance

- **Unit tests:** ~100ms total
- **Integration tests:** ~2-5s per test (tmux overhead)
- **E2E tests:** ~3-10s per test (full CLI startup)

## CI/CD

For CI environments:

```yaml
# Install tmux in CI
- name: Install tmux
  run: |
    sudo apt-get update
    sudo apt-get install -y tmux

# Run tests
- name: Run tests
  run: bun test
```

---

### Overview (`freebuff/cli/release/README.md`)

# Freebuff

**The free coding agent.** No subscription. No configuration. Start in seconds.

An AI coding agent that runs in your terminal — describe what you want, and Freebuff edits your code.

## Install

```bash
npm install freebuff
```

## Usage

```bash
cd ~/my-project
freebuff
```

### Older Intel Macs (without AVX2)

The npm launcher automatically switches to the `darwin-x64-baseline` build if
the standard Intel binary exits with an illegal-instruction error (`SIGILL`).
To select that build before the first launch:

```bash
FREEBUFF_BINARY_TARGET=darwin-x64-baseline freebuff
```

For manual installation, download `freebuff-darwin-x64-baseline.tar.gz` from the
[Freebuff CLI releases](https://github.com/CodebuffAI/codebuff-community/releases?q=freebuff-v)
and keep the extracted `freebuff` binary and `tree-sitter.wasm` together.

## Why Freebuff?

**Choice of leading models** — The regular CLI picker includes MiMo v2.6 Flash Fast, GLM 5.3 Flash, DeepSeek, DeepSeek V4.1 Flash Fast, GPT-6 Luna, MiMo 2.6 Pro, Solar Mini 4, Solar Pro 4, Space Bunny Alpha, Gemini 3.8 Flash, Muse Spark 1.3, Gemini 4 Argon, and GPT-6.1 Sol, with MiMo v2.6 Flash Fast selected by default.

**Fast** — 5–10× speed up. 3–5× tokens per second compared to Claude, plus context gathering in seconds.

**Loaded** — Built-in web research, browser use, and more.

## FAQ

**How can it be free?** Freebuff is supported by text ads.

**What models do you use?** In full mode, the regular picker includes MiMo v2.6 Flash Fast, GLM 5.3 Flash, DeepSeek V4.1 Flash, DeepSeek V4.1 Flash Fast, GPT-6 Luna, MiMo 2.6 Pro, Solar Mini 4, Solar Pro 4, Space Bunny Alpha, Gemini 3.8 Flash, Muse Spark 1.3, Gemini 4 Argon, and GPT-6.1 Sol. Space Bunny Alpha is a beta stealth model from an anonymous provider that retains prompts. Gemini 3.8 Flash and Muse Spark 1.3 require a paid plan: without one they are listed but locked, and selecting one opens the plans page. Gemini 4 Argon and GPT-6.1 Sol are free in the US and need a paid plan elsewhere, and every account, paid or not, gets one session a day on each; Gemini 4 Argon's price is a temporary promotion. GPT-6 Luna and MiMo 2.6 Pro are open to every full-access account. DeepSeek V4.1 Flash Fast is fast mode: the same model on DeepSeek's own API, splitting work across parallel subagents. Muse Spark 1.3 has a 1M-token context, and when it is busy or unavailable it answers on DeepSeek V4.1 Flash instead of leaving you waiting. MiMo v2.6 Flash Fast is the CLI default, and GLM 5.3 Flash is the cheapest model to run. Limited mode uses MiMo v2.6 Flash Fast, GLM 5.3 Flash, DeepSeek V4.1 Flash, Solar Mini 4, Solar Pro 4, and Space Bunny Alpha; the models a paid plan unlocks there are listed too, locked until you have one. Every model is paid for in hours, one multiplier per one-hour session, shown in the model picker: an hour of MiMo v2.6 Flash Slow uses one hour of balance, a 3x model uses three. Models may still serve from a quantized (Q8_0) build.

**Which countries is Freebuff available in?** All countries. Freebuff runs in "full" mode in the US, Canada, UK, EU, and other select countries, and in "limited" mode everywhere else (or while using a VPN). See [freebuff.com](https://freebuff.com) for the full list.

**What is limited mode?** Limited mode lets you use Freebuff outside the full-access countries, or while using a VPN. It includes GLM 5.3 Flash, DeepSeek V4.1 Flash, MiMo v2.6 Flash Fast, Solar Mini 4, Solar Pro 4, and Space Bunny Alpha. The free limited-access allowance is 5 hours a day, or 4 on a VPN or proxy, and each model's multiplier per one-hour session is shown in the picker. Today's allowance is spent first, whichever model you pick, including GLM 5.3 Flash; Hours you earn on the Earn page go into your wallet and are used once today's allowance runs out.



**Is my data used to train AI?** Only when a model or feature says data may be used for AI training. Freebuff or the provider may then keep submissions to develop, train, test, evaluate, fine-tune, and improve AI models or products, and Freebuff may share or sell them to third parties. Separately, Freebuff may use your data to train the models that choose and measure ads.

**How is my data used and stored?** Freebuff uses your messages, agent traces, and files to run the service and respond to your requests. We may analyze this data, and how you use the app, to personalize ads and train ad models. We do not give separately uploaded files, project files, or connected repositories to advertising providers. Restricted partners may evaluate repositories you connect to Freebuff Cloud, but cannot otherwise use, broadly share, or train on them. See the Privacy Policy for details.

See the [Privacy Policy](https://freebuff.com/privacy-policy) for complete details.



## Links

- [Documentation](https://codebuff.com/docs)
- [GitHub](https://github.com/CodebuffAI/codebuff)
- [Website](https://codebuff.com)

> Built on the [Codebuff](https://codebuff.com) platform.

---

### Overview (`scripts/tmux/tmux-viewer/README.md`)

# tmux-viewer

Interactive TUI for viewing tmux session logs. Designed to work for **both humans and AIs**.

## Usage

```bash
# Interactive TUI (for humans)
bun scripts/tmux/tmux-viewer/index.tsx <session-name>

# Start in replay mode (auto-plays through captures like a video)
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --replay

# JSON output (for AIs)
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --json

# Export as animated GIF
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --export-gif output.gif

# Export with custom frame delay (default: 1500ms)
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --export-gif output.gif --frame-delay 2000

# Export with custom font size (default: 14px)
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --export-gif output.gif --font-size 16

# List available sessions
bun scripts/tmux/tmux-viewer/index.tsx --list

# View most recent session (if no session specified)
bun scripts/tmux/tmux-viewer/index.tsx
```

Or using the npm script:

```bash
cd scripts/tmux && bun run view-session <session-name>
```

## Layout

The TUI uses a vertical layout designed for clarity:

```
┌─────────────────────────────────────────────────────────────────┐
│ Session: my-session  120x30           5 cmds  10 captures       │  ← Header
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│                    ┌──────────────────┐                         │
│                    │ [terminal output │                         │  ← Capture
│                    │  centered in     │                         │     View
│                    │  muted border]   │                         │
│                    └──────────────────┘                         │
│                                                                 │
├─ ⏸ Paused ──────────────────────────────────────────────────────┤
│  ┌─○ [1] 12:00:00─┐ ┌─▶ [2] 12:00:05─┐ ┌─○ [3] 12:00:10─┐       │  ← Timeline
│  │ initial-state  │ │ after-command  │ │ final-state    │       │     Cards
│  │ $ codebuff...  │ │ $ /help        │ │ $ /quit        │       │
│  └────────────────┘ └────────────────┘ └────────────────┘       │
├─────────────────────────────────────────────────────────────────┤
│ ▶ 2/10 @1.5s   space: play/pause  +/-: speed  ←→: navigate      │  ← Footer
└─────────────────────────────────────────────────────────────────┘
```

- **Header**: Session name, dimensions, command/capture counts
- **Capture View**: Terminal output centered with a muted border showing exact capture dimensions
- **Timeline**: Horizontal card-style navigation at the bottom, selected card stays centered
- **Footer**: Playback status, position, speed, and keyboard shortcuts

## Features

### For Humans (Interactive TUI)
- **Capture view**: Terminal output centered with visible boundary
- **Timeline panel**: Card-style navigation at the bottom with label and triggering command
- **Auto-centering**: Selected timeline card stays centered in view
- **Metadata display**: Session info, dimensions, command count
- **Replay mode**: Auto-play through captures like a video player
- **Keyboard shortcuts**:
  - `←` / `→` or `h` / `l`: Navigate between captures
  - `Space`: Play/pause replay
  - `+` / `-`: Adjust playback speed (faster/slower)
  - `r`: Restart from beginning
  - `q` or Ctrl+C: Quit
  - Use the `--json` flag on the CLI entrypoint for JSON output

### Replay Mode

Replay mode auto-advances through captures chronologically, like a video player:

```bash
# Start replay immediately
bun scripts/tmux/tmux-viewer/index.tsx my-session --replay

# Or press Space in the TUI to start/stop replay
```

**Playback controls:**
- `Space` - Toggle play/pause
- `+` or `=` - Speed up (shorter interval between captures)
- `-` or `_` - Slow down (longer interval between captures)
- `r` - Restart from the first capture
- `←` / `→` - Navigate captures (automatically pauses replay)

**Available speeds:** 0.5s, 1.0s, 1.5s (default), 2.0s, 3.0s, 5.0s per capture

The timeline panel title shows `▶ Playing` or `⏸ Paused`, and the footer shows current position (e.g., `2/10`), playback speed (e.g., `@1.5s`), and controls.

### For AIs (JSON Output)
Use the `--json` flag to get structured output:

```json
{
  "session": {
    "session": "cli-test-1234567890",
    "started": "2025-01-01T12:00:00Z",
    "dimensions": { "width": 120, "height": 30 },
    "status": "active"
  },
  "commands": [
    { "timestamp": "...", "type": "text", "input": "/help", "auto_enter": true }
  ],
  "captures": [
    {
      "sequence": 1,
      "label": "initial-state",
      "timestamp": "...",
      "after_command": null,
      "dimensions": { "width": 120, "height": 30 },
      "path": "debug/tmux-sessions/.../capture-001-initial-state.txt",
      "content": "[terminal output]"
    }
  ],
  "timeline": [
    { "timestamp": "...", "type": "command", "data": {...} },
    { "timestamp": "...", "type": "capture", "data": {...} }
  ]
}
```

## Data Format

The viewer reads YAML-formatted session data from `debug/tmux-sessions/{session}/`:

- `session-info.yaml` - Session metadata
- `commands.yaml` - Array of commands sent
- `capture-*.txt` - Capture files with YAML front-matter

### Session Info (session-info.yaml)
```yaml
session: cli-test-1234567890
started: 2025-01-01T12:00:00Z
started_local: Wed Jan 1 12:00:00 PST 2025
dimensions:
  width: 120
  height: 30
status: active
```

### Commands (commands.yaml)
```yaml
- timestamp: 2025-01-01T12:00:05Z
  type: text
  input: "/help"
  auto_enter: true
```

### Capture Files (capture-001-label.txt)
```yaml
---
sequence: 1
label: initial-state
timestamp: 2025-01-01T12:00:30Z
after_command: null
dimensions:
  width: 120
  height: 30
---
[terminal content here]
```

## Integration with cli-tester

The `@cli-tester` agent can use this viewer to inspect session data:

```typescript
// In cli-tester output
{
  captures: [
    { path: "debug/tmux-sessions/cli-test-123/capture-001-initial.txt", label: "initial" }
  ]
}

// Parent agent can view the session
// bun scripts/tmux/tmux-viewer/index.tsx cli-test-123 --json
```

## GIF Export

The `--export-gif` flag renders the session replay as an animated GIF, perfect for:
- Sharing CLI demonstrations
- Embedding in documentation
- Bug reports and issue tracking
- Creating tutorials

### GIF Export Options

| Option | Description | Default |
|--------|-------------|--------|
| `--export-gif [path]` | Output file path | `<session>-<timestamp>.gif` |
| `--frame-delay <ms>` | Delay between frames in milliseconds | `1500` |
| `--font-size <px>` | Font size for terminal text | `14` |

### Examples

```bash
# Basic export (auto-names the file)
bun scripts/tmux/tmux-viewer/index.tsx my-session --export-gif

# Specify output path
bun scripts/tmux/tmux-viewer/index.tsx my-session --export-gif demo.gif

# Fast playback (500ms per frame)
bun scripts/tmux/tmux-viewer/index.tsx my-session --export-gif fast.gif --frame-delay 500

# Larger text for readability
bun scripts/tmux/tmux-viewer/index.tsx my-session --export-gif large.gif --font-size 18
```

### GIF Output

The exported GIF includes:
- Terminal content rendered as monospace text
- Frame labels showing capture sequence number and label
- Timestamps for each frame
- Dark terminal-style background
- Automatic sizing based on terminal dimensions

## Development

```bash
# Typecheck
cd scripts/tmux/tmux-viewer && bun x tsc --noEmit

# Run directly
bun scripts/tmux/tmux-viewer/index.tsx --list
```

---

### Overview (`packages/agent-runtime/src/templates/README.md`)

# Agent Template Override System

This system allows users to customize agent behavior by placing override configuration files in their project's `.agents/templates/` directory.

## How It Works

1. **File Detection**: The system automatically detects `.agents/templates/*.json` and `.md` files and loads them into a dedicated `agentTemplates` field in the project file context
2. **Override Processing**: When an agent is spawned, the system checks for matching override files and applies them to the base template
3. **Dynamic Application**: Overrides are applied at runtime, so changes take effect immediately without restarting

## Override File Format

Override files should be JSON files with the following structure:

```json
{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "model": "anthropic/claude-sonnet-4",
    "systemPrompt": {
      "type": "append",
      "content": "Additional system instructions"
    },
    "spawnableAgents": {
      "type": "append",
      "content": ["thinker"]
    }
  }
}
```

## Configuration Options

### Basic Properties

- `type`: Agent type identifier (e.g., "CodebuffAI/reviewer" maps to "reviewer")
- `version`: Version identifier (currently informational)
- `model`: Override the model used by the agent

### Prompt Overrides

Each prompt type (`systemPrompt`, `instructionsPrompt`, `stepPrompt`) supports:

- `type`: How to apply the override
  - `"append"`: Add content after the base prompt
  - `"prepend"`: Add content before the base prompt
  - `"replace"`: Replace the entire base prompt
- `content`: Inline content to use
- `path`: Path to external file (relative to the override file)

### Array Overrides

For arrays like `spawnableAgents` and `toolNames`:

- `type`: How to apply the override
  - `"append"`: Add items to the existing array
  - `"replace"`: Replace the entire array
- `content`: String or array of strings to add/replace

## External Files

You can reference external files for prompt content:

```json
{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "systemPrompt": {
      "type": "append",
      "path": "./system-prompt.md"
    }
  }
}
```

The path is resolved relative to the override file's directory.

## Example Usage

### Basic Model Override

`.agents/templates/reviewer.json`:

```json
{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "model": "anthropic/claude-sonnet-4"
  }
}
```

### System Prompt Enhancement

`.agents/templates/reviewer.json`:

```json
{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "systemPrompt": {
      "type": "append",
      "content": "\n\nAdditional instructions: Focus on security vulnerabilities and performance issues."
    }
  }
}
```

### External File Reference

`.agents/templates/reviewer.json`:

```json
{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "systemPrompt": {
      "type": "append",
      "path": "./custom-review-instructions.md"
    }
  }
}
```

`.agents/templates/custom-review-instructions.md`:

```markdown
## Custom Review Guidelines

- Check for proper error handling
- Verify input validation
- Look for potential security issues
```

## Implementation Details

- Override processing happens in `packages/agent-runtime/src/templates/agent-overrides.ts`
- Files are loaded into `ProjectFileContext.agentTemplates` (separate from knowledge files)
- Agent template files are loaded in `cli/src/project-files.ts`
- Path resolution uses `path.posix.join()` for cross-platform compatibility
- Errors are logged but don't break agent execution (graceful fallback to base template)

## Error Handling

The system is designed to be robust:

- Invalid JSON files are logged and ignored
- Missing external files are logged and treated as empty content
- Non-matching agent types are ignored
- Processing errors fall back to the base template

This ensures that agent execution continues even if override files have issues.

---

### Overview (`packages/code-map/src/tree-sitter-queries/readme.md`)

# Credits

Codebuff uses modified versions of the tags.scm files from these open source tree-sitter language implementations:

- [https://github.com/tree-sitter/tree-sitter-c](https://github.com/tree-sitter/tree-sitter-c) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-c-sharp](https://github.com/tree-sitter/tree-sitter-c-sharp) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-cpp](https://github.com/tree-sitter/tree-sitter-cpp) — licensed under the MIT License.
- [https://github.com/Wilfred/tree-sitter-elisp](https://github.com/Wilfred/tree-sitter-elisp) — licensed under the MIT License.
- [https://github.com/elixir-lang/tree-sitter-elixir](https://github.com/elixir-lang/tree-sitter-elixir) — licensed under the Apache License, Version 2.0.
- [https://github.com/elm-tooling/tree-sitter-elm](https://github.com/elm-tooling/tree-sitter-elm) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-go](https://github.com/tree-sitter/tree-sitter-go) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-java](https://github.com/tree-sitter/tree-sitter-java) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-ocaml](https://github.com/tree-sitter/tree-sitter-ocaml) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-php](https://github.com/tree-sitter/tree-sitter-php) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-python](https://github.com/tree-sitter/tree-sitter-python) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-ql](https://github.com/tree-sitter/tree-sitter-ql) — licensed under the MIT License.
- [https://github.com/r-lib/tree-sitter-r](https://github.com/r-lib/tree-sitter-r) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-ruby](https://github.com/tree-sitter/tree-sitter-ruby) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-rust](https://github.com/tree-sitter/tree-sitter-rust) — licensed under the MIT License.
- [https://github.com/tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript) — licensed under the MIT License.

This collection of queries was originally put together in the project aider: https://github.com/paul-gauthier/aider/tree/main/aider/queries

---

### Agents And Tools (`docs/agents-and-tools.md`)

# Agents and Tools

## Agents

- Prompt/programmatic agents live in `.agents/` (programmatic agents use `handleSteps` generators).
- Generator functions are NOT sandboxed: the runtime `eval`s a `handleSteps` source string in the process that runs the agent (`packages/agent-runtime/src/run-programmatic-step.ts`); agent templates define tool access and subagents.

### Registry agents with executable `handleSteps` require publisher trust

A template fetched from the public agent registry (`--agent publisher/agent`, a
registry id in a local agent's `spawnableAgents`, or a bare id falling back to
`codebuff/<id>`) is executable code when it carries `handleSteps`: the row's
`handleSteps` is a source string, validation only checks that it starts with
`function*`, and the runtime evals it with no isolation on the user's machine.
The registry GET is public, any account can create a publisher and publish
without review, and `latest` is unpinned. So the SDK's `fetchAgentFromDatabase`
refuses a registry template with a string `handleSteps` unless its publisher is
trusted (`sdk/src/agent-publisher-trust.ts`), and throws an
`UntrustedAgentPublisherError` whose message names `publisher/agent@version`
and the knob to set. Trusted publishers are `codebuff` (our own, also bundled)
plus the comma-separated `CODEBUFF_TRUSTED_AGENT_PUBLISHERS` env var plus the
`trustedAgentPublishers` option on `CodebuffClient` / `run()`. Data-only
registry templates (no `handleSteps`) load as before, and local `.agents` files
and SDK `agentDefinitions` are never gated — they are code the user already
chose to run. This is a trust floor, not isolation: sandboxing the eval,
signing templates and pinning `latest` are separate work.

Clients released before this gate (`codebuff` and `@codebuff/sdk` on npm) eval
whatever the registry returns, so the registry GET routes refuse (403) a
non-`codebuff` template with `handleSteps` unless the request carries
`AGENT_TRUST_GATE_HEADER` (`common/src/constants/agents.ts`), which only the
gated SDK sends (`untrustedHandleStepsRefusal` in
`web/src/app/api/agents/_registry-guard.ts`). The server cannot see the
caller's trusted set, so the header defers that decision to the client.
`mcpServers` from non-`codebuff` publishers are refused outright, at publish
and at GET, whatever the client.

### Trust gate for repository `.agents` directories

`loadLocalAgents` dynamically imports every `.ts`/`.tsx`/`.js`/`.mjs`/`.cjs`
file under `{cwd}/.agents` and `{cwd}/../.agents`, and `loadMCPConfigSync`
reads an `mcp.json` there whose stdio servers are spawned on the first prompt
with `$VAR` env values filled from the CLI's own environment. Both execute
repository-supplied code, so cloning a hostile repo and running the CLI inside
it must not run anything without consent. The CLI therefore gates those two
repository-scoped directories (`cli/src/utils/agent-dir-trust.ts`):

- **What needs trust**: a directory containing at least one executable agent
  file (the loader's extension filter, minus `.d.ts` and tests) or an
  `mcp.json`. A directory holding only `skills/` markdown does not, and
  `~/.agents` never does (it is the user's own). Skills still load from an
  untrusted directory; only agents and `mcp.json` are withheld.
- **The prompt**: on an interactive launch (stdin and stdout are TTYs) the CLI
  prints a plain-terminal prompt before the TUI mounts, listing the directory,
  its agent files (capped at 10) and the commands or URLs its `mcp.json` would
  start, and asks `Load and run these? [y/N]`. `y` records the directory;
  anything else skips it for this run and prints how to trust it later.
- **The store**: `<configDir>/trusted-agent-dirs.json` (next to
  `credentials.json`, written with mode `0600`), a map of the normalized
  absolute directory path to `{ "trustedAt": "<ISO>" }`. Delete an entry to
  be asked again.
- **Non-interactive runs** (no TTY, CI) never prompt: the directory is skipped
  with a warning unless `CODEBUFF_TRUST_AGENT_DIRS=1` or `--trust-agents` is
  passed, which trusts every repository directory for that run only and writes
  nothing to the store.
- **SDK semantics are unchanged**: `loadLocalAgents({ agentDirs })` and
  `loadMCPConfig{,Sync}({ configDirs })` are opt-in options; omitting them
  keeps the default search, so Desktop and external SDK consumers behave as
  before. `getDefaultAgentDirs()` and `listLocalAgentFiles()` are exported so a
  host can build its own gate on the loader's own filter.

Trust is per directory, not per file content: a trusted repository that later
pulls a malicious agent file is loaded without a new prompt, the same trade-off
other coding agents make for their project-level config.

### Shell Shims

Direct commands without `codebuff` prefix:

```bash
codebuff shims install codebuff/base-lite@1.0.0
eval "$(codebuff shims env)"
base-lite "fix this bug"
```

## Tools

- Tool definitions live in `common/src/tools` and are executed via the SDK helpers + agent-runtime.

### Cut-off file edits are refused, never repaired

`packages/agent-runtime/src/util/truncated-write-guard.ts` stops two shapes of
a file edit that was cut off mid-generation:

- **Arguments that never closed.** When a response hits the output limit (or
  loses its connection) inside a tool call, the OpenAI-compatible provider
  still flushes the call with whatever argument text arrived, and OpenRouter
  reports the step as `finish_reason: "tool_calls"` (the real reason is only in
  `native_finish_reason: "max_output_tokens"`). `parseRawToolCall` recognises
  a truncated JSON prefix and answers with an explicit "nothing was written"
  error instead of the generic re-issue hint, and does not echo the partial
  content back. Do not add a JSON-repair step for mutating tools: completing a
  cut-off `content` string is exactly how a file gets truncated on disk.
- **A cut-off copy of an existing file.** `processFileBlock` refuses a
  `write_file` that is under 60% of an existing file of 30+ lines AND leaves
  brackets open where the current file balances them. Both refusals log
  `axiomEvent: file_write_guard` (`metric` says which).

The finish reason is deliberately not the signal: complete arguments in a
length-cut step are a complete call, and the OpenRouter shape above hides the
length cut anyway.

### Console-free terminal command broker

`run_terminal_command` separates process ownership from terminal UI ownership:

- `sdk/src/tools/run-terminal-command.ts` owns output buffering, timeouts,
  cancellation escalation, results, and process diagnostics. Headless SDK
  consumers use its direct process-group runner.
- Interactive hosts provide `terminalCommandBroker` in `CodebuffClientOptions`
  (or directly to `runTerminalCommand`). Each call synchronously starts an
  isolated helper and returns a handle for its complete process tree. A startup
  failure prevents the shell from running; there is no direct-console fallback.
- The CLI's tiny `src/entry.ts` handles private broker mode before importing
  React or OpenTUI. The detached, hidden helper receives one spawn request over
  stdin, starts the shell without a console or interactive stdin, relays only
  stdout/stderr pipes, and reports completion through a constrained one-shot
  file in the OS temp directory. It deliberately uses only the three standard
  stdio channels: Bun's custom child-process pipes can fail their Windows
  `node:net` handshake outside the `ChildProcess` error event and terminate the
  CLI as an unhandled rejection. The broker remains the process-group root and
  self-reaps the tree if its parent disappears, detected by polling the parent
  PID rather than holding another pipe open.
- Mouse and focus protocols stay enabled while commands run. The
  `TerminalProtocolController` only parses focus events; it has no command
  lifecycle state to synchronize or restore.

Thread the broker capability through every interactive command entry point.
Do not bypass it with a direct `spawn`, add command-active terminal state, or
fall back to the TUI process when broker startup fails.

---

### Testing (`docs/testing.md`)

# Testing

- Prefer dependency injection over module mocking; define contracts in `common/src/types/contracts/`.
- Use `spyOn()` only for globals / legacy seams.
- Avoid `mock.module()` for functions; use `@codebuff/common/testing/mock-modules.ts` helpers for constants only. A module mock lasts for the rest of the `bun test` process, and `mock.restore()` does not undo it (see "the FILE ORDER").

CLI hook testing note: React 19 + Bun + RTL `renderHook()` is unreliable; prefer integration tests via components for hook behavior.

## Test env must come from a fixture, not the developer's `.env`

`@codebuff/common`'s `env.ts` validates the `NEXT_PUBLIC_*` vars at **import** time and throws, and Bun loads `.env` files from the process cwd — so a package-local `bun test` sees none of the repo-root env even when the root run is green. A whole test file then dies with `Invalid environment configuration` before a single test runs, which bun reports as an unhandled error rather than a failure, so the suite silently stops covering that file.

Every package must therefore have a `bunfig.toml` preloading `sdk/test/setup-env.ts` — the one shared fixture, which supplies placeholder values for every var the schemas require. This is not just a local-dev nicety: CI runs `cd <package> && bun test`, i.e. exactly the package-local mode. Placeholders only — tests must never need real credentials, so `bun test` means the same thing in a fresh worktree and in a provisioned checkout.

The unit-test matrix therefore never loads Infisical, and neither does `flake-hunt.yml`, which runs the same suites in the same job environment. This is a correctness boundary, not merely a secrets optimization: production rollout switches are mutable operational state, so importing them makes unrelated commits fail as soon as an operator changes a flag. Tests for non-default modes must set the specific variable in the test or inject the mode explicitly. Secret-bearing integration jobs remain separate.

The shared preload deletes `FREEBUFF_ADS_SLACK_WEBHOOK_URL`, including when
integration CI or a local shell supplies it. Campaign fixtures otherwise send
real review notifications to `#ads-notifs` on every suite run and retry. The
Slack sender also refuses its default transport whenever `NODE_ENV`, `BUN_ENV`
or `NEXT_PUBLIC_CB_ENVIRONMENT` is `test`. Notification tests inject `fetchImpl`
or `postMessage`; the production webhook stays configured for real submissions.

The same rule covers generated inputs. `cli/src/agents/bundled-agents.generated.ts` is gitignored, and importing it at module scope wiped 17 files (~371 tests) in a fresh worktree; `cli/test/setup-agents-artifact.ts` now builds it on demand instead of relying on everyone knowing to run `bun run prebuild:agents`.

Tests that need a **service** rather than a variable should skip cleanly and say why, but never in CI. `@codebuff/internal/testing/test-db` probes Postgres once, skips the DB suites locally with the docker command to fix it, and throws when `CODEBUFF_GITHUB_ACTIONS=true` — otherwise a broken CI service container would read as a pass. Gate on **reachability**, never on `!process.env.DATABASE_URL`: the fixtures supply a placeholder URL, so presence stopped meaning availability.

A unit test that leaks a database call gets the same answer locally as in CI. `test/setup-db-leak-guard.ts`, the first preload in the bunfig of every package whose tests can reach Postgres (the root, `web`, `freebuff/web`, `packages/{internal,billing,auth,jobs}`, and `scripts`, which preloads nothing else), answers CI's `must-not-reach-a-database` URLs with an instant Postgres error that names the leak. When none of `DATABASE_URL`, `DIRECT_DATABASE_URL` and `SCRIPTS_DATABASE_URL` is set, it supplies that URL itself. Without it, a local run fell through to `setup-env.ts`'s localhost placeholder, where postgres-js's reconnect backoff let a swallowed leak time out some later test (2026-10-13). It must stay ahead of `setup-env.ts` in those lists, and `scripts/__tests__/db-leak-guard.test.ts` fails if it does not. Set any of the three, as an integration run does, and it leaves them all alone. `probeTestDb` reports the sentinel as unreachable without connecting, so integration files skip without tripping the guard.

Tests that spawn a server child should race readiness against `proc.exited` and report the child's captured output (see `freebuff-desktop/test/launch-bootstrap.smoke.test.ts`). Polling a dead port surfaces only as "a hook timed out", which names neither the cause nor the process that failed.

### The CI guard against disappearing tests

CI does not run `bun test` directly — it goes through `scripts/ci/test-with-guard.ts`, which fails the build on any of:

1. any error outside a test body — an `Unhandled error between tests` marker (a file that crashed at import) or a non-zero `N errors` in bun's summary,
2. a test or file count **below** the baseline in `.github/test-baselines.json`, including the degenerate case where the caller's glob selects **no files at all** (a renamed directory would otherwise print "no tests found" and pass).

Growth never fails the build, so adding tests needs no baseline change; the guard just notes the baseline is stale. Deleting tests on purpose means re-recording: re-run the job's command with `--update`.

**Refresh the baselines periodically, and act on the "stale" line.** Because growth never fails, a baseline nobody touches quietly becomes a floor far below reality — which is a guard that no longer guards. When this rot was first measured, `freebuff-desktop` was recorded at 971 tests while actually running 1278, so **307 tests could have stopped running and CI would have stayed green** (393 across all suites). Re-record from a real CI run, not locally.

Expect to do it more than once. Desktop moved 1278 → 1282 → 1339 over the two days it took to land that refresh, so on an active package the floor starts sliding immediately. That is fine — a floor 50 tests low still catches the failure this exists for, which is a suite losing *hundreds* at once. It is a floor 300 low that is worthless. Treat the "stale" line as a nudge, not an emergency.

A suite that selects **no files and has no baseline** is not a failure — it can legitimately have no tests yet — but it now emits a `Suite ran no tests` warning annotation rather than passing in silence. `test:.agents`, `test-integration:common` and `test-integration:packages/agent-runtime` sat in that state on every run until they were dropped from `ci.yml`, along with the `test:files*` scripts that existed only for them. Adding a first test there now fails `check-test-coverage` until the package is listed in a matrix again, so it cannot go unrun silently. When the warning fires, either give the suite tests or drop it, but decide rather than letting it read as green.

**A skipped suite and a running suite do not report the same total.** Skipped tests are counted, so it is tempting to assume a baseline recorded locally transfers to CI — it does not when a whole `describe` is skipped. Measured on the billing DB suites: each skipped file reported exactly **two more** than it did when it ran (47 vs 41 across three files), and the guard duly failed a healthy CI run. So a baseline for a suite that skips locally but runs in CI — anything DB-backed — must be taken from a real CI run. Read the observed counts out of the job log and write them into `.github/test-baselines.json`.

Re-record in a CI-equivalent state, which means running `cd sdk && bun run build` first. A few cli tests register a placeholder test *only when* `sdk/dist` is missing, so a baseline seeded without it is inflated and the guard then fails a perfectly healthy CI run. (The guard caught exactly this while being built.)

This exists because every check bun gives you is blind to the failure mode that actually happened here three times: a file stops contributing tests and the summary still reads as mostly-green.

**`[test].exclude` in `bunfig.toml` does nothing on bun 1.3.14.** Verified by pointing it at an ordinary test file, which still ran. That is why the repo-root `exclude` does not keep `*.integration.test.*` out of a root run, and why CI filters them with `find ... ! -name '*.integration.test.ts'`. Gate on a runtime condition instead of trusting the key.

**Never hand a bare postgres.js query to `expect(...).rejects`.** A query is lazy: it runs only when something calls its `.then()`, and bun 1.3.14's `.rejects` never does, so the test waits forever. The per-test timeout does not fire either, so the whole job runs until its 15-minute step timeout. On 2026-09-30, `region-verifier-runs.integration.test.ts` hung `test-integration-packages/internal` this way on every PR until #4595. Await the query inside a function and assert on what it throws (see that test's `rejects` helper), or use ``expect((async () => await client`...`)()).rejects``.

## Cloud Agent smoke test (no credentials)

A Cloud Agent — or any checkout without Infisical — can confirm the workspace can run a package-local unit test from the placeholder fixture alone. This is one file in `@codebuff/internal`. A pass does not validate the rest of the application, CI, or production runtime.

The repo pins Bun **1.3.14** (`.bun-version`, `package.json` `engines.bun` / `packageManager`) and Node **24.21.0** (`.node-version`, which Render applies to every repo-root service; only `web` runs on Node, the rest start Bun). The smoke test runs under Bun, not Node. If the VM has no Bun, install that pin from the official installer without changing tracked runtime files:

```bash
curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.14"
```

Do not use the repo `.bin/bun` wrapper for this. `.envrc` puts `.bin` first on `PATH`, and the wrapper tries Infisical; with `.env.example` present and no `.env.local` it exits 1 before invoking the real binary.

Resolve the installed Bun **before** `.envrc` prepends `.bin`. If bun is already on `PATH` (mise, a system package, a prior install), capture that path. Use `$HOME/.bun/bin/bun` only when you just ran the official installer above.

```bash
# Before direnv / .envrc, if bun is already installed:
REAL_BUN="$(command -v bun)"
# Only after the official installer, if bun was missing:
# REAL_BUN="$HOME/.bun/bin/bun"

cd packages/internal
"$REAL_BUN" test src/ad-targeting/__tests__/iso-3166-alpha2.test.ts
```

`bun install` is not required for this file. The package `bunfig.toml` preloads `sdk/test/setup-env.ts`. Expect 3 passing tests.

## The production-runtime smoke in `build-web`

Everything else in CI runs under Bun. Production `web` runs under Node (Render's
default, pinned in `.node-version` so CI and Render read the same file), and
the two disagree on which JS built-ins exist: on 2026-09-11 a
`Promise.withResolvers()` in the completions handler was green everywhere and
threw on every request in production (#3218). `scripts/check-node-runtime-apis.ts`
catches the known names; `web/scripts/prod-smoke.ts` catches the class. After
the Next build, `build-web` starts `next start` on the pinned Node against a
migrated Postgres container and drives one real client journey — session
admission, agent-run start, a streamed chat completion — with every external
HTTP call stubbed. A request that cannot complete end to end fails the job.

Both smokes are thin: `packages/internal/src/prod-smoke/harness.ts` owns the
loopback-only database guard, the Bun-shim refusal, the seeded fixture (ad-data
lifecycle row, user, session), staging a build outside the repo, the server
spawn with log relay, `/api/healthz` polling with the optional runner gate, the
authenticated request helper and teardown; `upstream-stub.cjs` beside it is the
one table of stubbed upstream endpoints. An app's `scripts/prod-smoke.ts`
supplies only its port, launch command, environment, auth header and journey. The `web` Render service deploys
manually, so this makes `main` red before someone deploys it rather than
blocking the deploy itself.

Run it locally after `bun run build` in `web/` with `DATABASE_URL` pointing at
a loopback Postgres that has the migrations applied (the script seeds rows, so
it refuses any other host):

```bash
cd web && DATABASE_URL=postgresql://postgres:postgres@127.0.0.1:5432/testdb bun run smoke:prod
```

`build-freebuff-web` has the same gate for the builder app,
`freebuff/web/scripts/prod-smoke.ts`, and it matters more there: `freebuff-web`
auto-deploys from `main`. Two differences. The pin is that service's own
`freebuff/web/.node-version`, not the repo root's, because Render applies the
root file to every service and a root pin meant for `web` moved freebuff-web
from 22.x to 20.15.1 on 2026-09-13, breaking its build on a Node 20 flag while
CI stayed green; the job now installs that Node _before_ the compile, since the
app's build script runs `node` itself. And the journey is a chat turn behind a
seeded database session cookie rather than an API-key completion, because that
is the path that `import()`s `@codebuff/sdk` out of the standalone tree at
request time and runs the agent in-process; the SDK's calls to the backend
(`/api/v1/me`, the run ledger, the model) all land in the stub's table.
Hiding `@codebuff/sdk` from the standalone tree makes it fail, which is the
proof that it exercises what the build's import check only approximates. The
embedded agent runner is made _required_ (as on a deployed instance) with
placeholder Convex credentials, so a runner that cannot boot is a 503 from
`/api/healthz` and a fast red rather than a warning; note that Next polyfills
`WebSocket` on its server, so the runner's "Node >= 22 required" bail does not
fire on Node 20 under this app — a Node downgrade is caught by building and
smoking on the pinned version, not by that check.

```bash
cd freebuff/web && DATABASE_URL=postgresql://postgres:postgres@127.0.0.1:5432/testdb bun run smoke:prod
```

## What CI actually spends its time on

`ci.yml` is one flat fan-out: `typecheck`, `build-web`, and three test matrices,
with **no `needs:` edges between them** (all of them wait only on the short
`changes` job, below). Before that, every test job waited on a
combined `build-and-check` gate, and measuring a run showed how badly that
misread the cost:

| | before | after |
|---|---|---|
| **Total run** | **10m40s** | **5m29s** |
| the gate / its replacements | `build-and-check` **441s** | `typecheck` 326s ‖ `build-web` 194s |
| web build's own compile | 330s (`in 5.5min`) | **73s** |
| all test jobs' `bun test` steps, added together | 184s | **110s** |
| `test-common`'s actual test run | **1s** (after 55s of setup) | 1s |

The gate was ~70% of the run, and the thing it gated took under two minutes of
real work. Nothing a test imports is produced by the typecheck or the Next
build, so the edges bought nothing.

Splitting typecheck from the web build was worth more than un-gating: the Next
compile went 330s → 73s purely by not sharing two cores with tsc. Note what did
**not** change — Next still logs `using 1 worker` for page-data collection and
static generation, before and after. On a 2-vCPU runner it always will. The win
came from the webpack compile phase, not from more workers; don't chase the
worker count.

The corollary for anyone optimizing this: **individual tests are not the
problem, and deleting them will not help.** Per-job setup (dependency-cache
restore, SDK build) costs far more than the tests in almost every job.

### Typecheck runs in lanes

`bun --filter='*' run typecheck` scheduled `@codebuff/freebuff-web` last: it did
not start until 114s into a 227s step, then ran ~113s alone while all 17 other
packages had finished and the runner idled on one tsc. So the typecheck job is a
matrix over *lanes* — `freebuff-web` gets its own runner, `rest` takes the other
17 — defined in `scripts/ci/typecheck-lanes.ts`.

The `freebuff/web` typecheck allows an 8 GiB Node heap. A cold check measured
6,239,321 KiB after the October 2026 Desktop/Web changes; the previous 6 GiB
limit exhausted the heap on both `main` and PR runners. Its dedicated
`ubicloud-standard-4` runner has 16 GiB RAM, leaving room for Node and the
runner outside the heap. Keep the full check; incremental caches only reduce
the work when the prior build info is usable.

Membership lives in that script rather than in `ci.yml` because both obvious
ways to write it in YAML fail silently:

- **`--filter='*' --filter='!@codebuff/freebuff-web'` does not exclude
  anything.** bun 1.3.14 ignores the negation and runs all 18 packages
  (verified), so the `rest` lane would re-run the package it exists to hand off
  and the split would buy nothing while looking correct.
- **Listing the other 17 by hand** means a newly added package matches no lane
  and is typechecked by nobody — a green build with a hole in it.

So lanes are derived from the workspace at runtime and the derivation is
checked: every package with a `typecheck` script must land in exactly one lane
or the script exits 1. Adding a package needs no change; only moving one
between lanes does.

Everything else in that script exists to make the remaining ways it can quietly
check nothing loud. Each of these **exits 0** on its own:

- **A lane declared in the script but missing from `ci.yml`'s matrix.** Its
  packages leave `rest` and are then run by nobody, while every lane that *does*
  run still passes. This is why the step passes `--expect-lanes`: the two lists
  must match or the build fails. Keep them in sync when adding a lane.
- **A filter that matches nothing, next to filters that do.** Measured on bun
  1.3.14: `--filter=@codebuff/common --filter=@codebuff/nope` runs `common`,
  ignores the bad one and **exits 0**. A lone non-matching filter exits 1, so
  only the multi-package case is silent — which is exactly what `rest` is. The
  script therefore asserts every lane member actually reported a result.
  (Relatedly, write `--filter=NAME`; as separate argv entries the value does not
  bind and bun matches nothing at all.)
- **A lane naming a package** that was renamed or lost its `typecheck` script.
- **`Bun.Glob` skipping dot-directories.** Without `dot: true` the workspace
  scan finds 20 manifests instead of 21 and `.agents` silently leaves every
  lane.

**Beware retry-inflated numbers when you profile this.** The first pass at the
table above recorded `freebuff-desktop` at 130s, which was wrong: the job had
silently failed once (`Boundary result-checkpointed was not reached`, 69.55s)
and passed on attempt 2 (48.78s), and `nick-fields/retry` reported success. Read
bun's own `Ran N tests across M files [Xs]` line, not the step duration, and
grep the log for `Attempt N failed` before trusting any per-job figure.

Unit suites run **once**; integration suites retry and end with a step that
raises a `Flaky suite` warning when a retry rescued them.

The split is measured, not assumed. Across 100 CI runs / 2000 test-job instances
(2026-07-31 → 08-02) the retry rescued a unit suite **zero** times; both real
rescues were `test-integration-packages/internal`, i.e. a Postgres service
container. Retrying infrastructure is what a retry is for; retrying test logic
just moves the bill to whoever hits it next. Unit suites therefore used to keep
three attempts only to fail a pass-on-retry as "flaky", which meant a red unit
suite ran three times and could never come back green: on 2026-10-07 failing
`test-freebuff/web` jobs took 736s against 275s green. Since then a flaky unit
case fails on its first attempt like any other failure, and its `(fail)` line is
the last in the log. Tell "flaky" from "broken" locally with
`scripts/flake-hunt.ts`.

### `sdk/dist` is cached, not rebuilt per job

`./.github/actions/build-sdk` restores `sdk/dist` from a cache keyed on every
input the bundler reads (sdk sources/scripts/vendor, the workspace packages it
inlines via tsconfig `paths`, and `bun.lock`). It has **no `restore-keys`** on
purpose: a prefix match would hand back a dist built from different sources and
every downstream test would silently run against a stale SDK. A miss just costs
the ~18s build.

`freebuff/web`'s `prepare:workspace` shells out to that same SDK build, and it
was the tail of the typecheck job — freebuff/web finishes last, and spent its
first ~65s rebuilding what the job had already restored. It now honours
`SDK_ALREADY_BUILT`, which **only** CI's typecheck job sets, immediately after
`build-sdk` has run. Unset everywhere else, so local runs and Render deploys
still build normally. If you add a job that runs `freebuff/web`'s typecheck or
build, either let it rebuild or set the flag *after* `build-sdk` — never before.

### Small suites share a runner

Per-job setup (checkout, a cold `bun install`, the SDK cache) is about 45s on
Ubicloud and 65s with Postgres. In 40 runs measured on 2026-10-06, seventeen
unit suites took 0.8–1.0 min each as separate jobs, running about 25s of tests
between them. So `ci.yml`'s test matrices are lists of **lanes**, each running
one or more packages (`packages`) one after another. Every package still runs
under its own `test-with-guard` key and baseline, a lane reports every package
that failed rather than stopping at the first, and `check-test-coverage.ts`
reads the packages from the lanes. In the database lane, each package after
the first gets an empty database of its own (`scripts/ci/fresh-test-database.ts`).

Give a suite its own lane when its runtime is a real fraction of the critical
path (the big four: `freebuff/web`, `freebuff-desktop`, `web`, `cli`), and add
it to `small` otherwise. The three Windows areas are one `test-windows` job
for the same reason: setup was 75–95% of each.

### Big suites are sharded

The opposite case: a suite too slow for one runner gets several lanes, one per
`shard: i/n`, each naming the same single package. `test-with-guard --shard i/n`
runs the files `scripts/ci/test-shard.ts` assigns to that shard and guards them
under the key `<key>#i/n`. Today that is the `freebuff/web` and
`freebuff-desktop` unit suites and `web`'s DB integration suite, two shards
each.

- **Assignment is a hash of the file's path** (FNV-1a over the path
  `test:files` prints, modulo n). Adding or deleting a file changes one shard's
  count and no other. A split by position in the sorted list would move files
  between shards on every addition, and a healthy shard could fall below its
  file floor.
- **Each shard has its own baseline**, recorded from a CI log like any other
  (never `--update` locally). Changing n, or the hash, moves files between
  shards, so every shard of that package needs re-recording from the first CI
  run that has the change. `scripts/__tests__/test-shard.test.ts` pins two
  assignments so a hash change cannot happen by accident.
- **The shards must be one complete partition.** `check-test-coverage.ts`
  fails if a sharded package does not declare every index 1..n of one n exactly
  once, appears unsharded in the same job as well, or has a file that hashes to
  no declared shard or to two. Without that check, dropping the `2/2` lane would
  lose half the suite while every remaining lane passed.
- **Some files must share a shard.** `SHARD_TOGETHER` in `test-shard.ts`
  hashes a directory as one unit. web's `src/server/desktop-cloud/` is one:
  its suites start and stop local R2 (workerd) in-process, and split by file,
  `store.integration.test.ts` hung for 60s after `org-invite-store`, `storage`
  and `github` ran before it, reproducibly in CI. With the rest of the
  directory between them, as in an unsharded run, it passes. Add a group only
  for a cross-file hazard you have reproduced, and fix the hazard where you can.
- **A new file order surfaces leaks.** Sharding's first CI run found two
  freebuff/web tests leaving a fake `window` behind (a later server render then
  threw on `window.location.href`) and a process-group test that read `/proc`
  before the killed process had finished exiting. Fix the leaking test; do
  not move files between shards to hide it.
- **Conditions that mean "this package" test `matrix.packages`**, not
  `matrix.lane`: a shard's lane is named `web (1/2)`, so `matrix.lane == 'web'`
  would quietly stop matching (and stop starting its Redis service).

Measured before sharding (8 green main runs, 2026-10-07; median job time):
`test-integration-web` 282s and `test-freebuff/web` 276s were the two slowest
jobs, with ~200s and ~225s of `bun test`. Per-file times from three of those
logs (the gap between consecutive file headers) split 102s/104s for
freebuff/web and, with desktop-cloud kept together, 78s/125s for web; web's
two slowest DB files (~33s and ~22s) land in different shards. With those two
split, `test-freebuff-desktop` (164–267s of tests on its pinned Bun, 255s
median job, 330s in the PR's run) was the slowest job, so it is split too,
96s/98s by per-file time.

Two shards each, not three: the jobs behind them sit at 220–265s
(`test-windows`, `build-freebuff-web`, `typecheck-freebuff-web`,
`typecheck-rest`), so a third shard would buy no wall clock for another ~70s of
setup. Shard a suite only when it is clearly the long pole. Each extra shard
costs a full job's setup (~50–70s of runner time), and on this pool runner
pickup was the larger delay in the slowest runs: up to ~350s before a job
started, which more concurrent jobs make no better.

### A pull request runs only the lanes it can reach

A small `changes` job runs `scripts/ci/affected-lanes.ts` on the PR's merge
commit, and every other job waits on it (the one `needs:` edge). It resolves
every file's imports and repo-path reads into a file-level graph, and a lane
runs when the PR touches its own packages or any file its tests, build or
typecheck can load. Shared config (lockfile, root tsconfig/bunfig, `ci.yml`,
`.github/actions/`, `scripts/ci/`, …), an unknown top-level path, or a failed
`changes` job runs everything; pushes to `main` always run everything, which is
the backstop for an edge the scanner misses.

- **Why did a lane run (or not)?** The `changes` job's step summary names the
  changed file that selected each lane. Locally:
  `bun scripts/ci/affected-lanes.ts --files <path>…`, and
  `AFFECTED_LANES_DEBUG=1` prints the import chain to each package.
- **A matrix row** is picked up from `ci.yml` by its `packages`; nothing to add.
  Shards share `packages`, so they run or skip together. In `test` and
  `test-db-integration` the rows are the JSON table inside the matrix's
  `include: ${{ fromJSON(needs.changes.outputs.…_matrix || '[…]') }}`: edit it
  there (no `}}` inside it). `changes` emits the selected rows, every key kept,
  so an unselected leg never starts a runner; a PR that selects none emits `[]`,
  which skips the job (an empty matrix is an error), and a failed `changes`
  emits nothing, so GitHub falls back to that table and every leg runs.
- **A new non-matrix job** that should be filtered needs an entry in
  `SINGLE_JOBS` in the script and its output in the `changes` job; a job with no
  `needs: changes` simply always runs.

### Known remaining cost

What is left on the critical path is not test time. `test-windows` used to
spend ~130s of its ~264s restoring the whole monorepo's `node_modules`; it now
installs only the workspaces its suites load (see "Windows installs" below).
`typecheck-freebuff-web` and `typecheck-rest` are one `tsc` each
(160–220s) behind ~45s of setup; the lever there is incremental `tsc` or a
larger runner, and both need their own measurement. `build-freebuff-web`'s
~110s Turbopack compile now starts from the previous run's persistent cache
(next section).

Measure test time on CI, not locally. freebuff-desktop's
`src/app/thread-engine.test.ts` once took ~101s on an M-series Mac against
~54s for the entire desktop suite on a runner, so local timings will send you
after the wrong thing.

### Cold Bun installs beat the dependency cache on Ubicloud

Do not cache `node_modules` on the Linux jobs. The first full CI run after the
Ubicloud migration gave a natural cold-cache comparison across all 31 Linux
jobs: cache lookup plus `bun install --frozen-lockfile` took 802 runner-seconds
in total, with a 26.0s median per job. The next three runs restored the same
1.185 GB archive and took 1,038–1,080 runner-seconds for restore plus install,
with 32.9–34.1s medians. A warm cache was therefore about 7–8s slower per job
and cost roughly four extra runner-minutes per full CI run.

The smaller observations agree with the full matrix. The since-deleted
Migration CI installed cold in 23s, while its warm restores took 18–41s. The abuse sweep installed in
17s on a miss and then spent another 24s compressing and saving the cache.
`setup-project` consequently skips dependency caching on Linux, and standalone
Ubicloud workflows use that action instead of carrying their own cache blocks.
The Windows and macOS cache remains until those runners have their own cold/warm
comparison; this measurement says nothing about their install performance.

### Windows installs

Measured on `windows-latest` on 2026-10-07 (Bun 1.4.2, Defender already off):

| Install | Time |
| --- | --- |
| `setup-project`, warm 1.05 GB cache (278k files) | ~110s restore, 95s of it tar unpacking, + 5s install |
| Whole monorepo, cold, Bun cache on its default C: | 647s |
| Whole monorepo, cold, Bun cache on D: | 187s |
| `test-windows`'s workspaces only, cold, cache on D: | 36–54s |
| The same, warm 400 MB cache (~51k files) | 24–27s restore + 5–7s install |

Two separate costs. File count dominates a restore, so `test-windows` installs
with `bun install --filter` for the workspaces its suites load. And the
checkout is on D: while Bun's install cache defaults to C:, so every package
is copied across volumes; pointing `BUN_INSTALL_CACHE_DIR` at `runner.temp`
(on D:) cut a cold install 3–4x. Every other Windows job still uses
`setup-project`, which leaves the cache on C:, so its cold path (a lockfile
change) is the 647s row.

Keep the caches whose transfer cost is materially smaller than the work they
avoid. `sdk/dist` is a roughly 17 MB restore instead of an ~18s build.
`build-freebuff-web` keeps Turbopack's persistent build cache
(`turbopackFileSystemCacheForBuild`, on in CI and off on Render, where the
download cost more than the compile it saved): `scripts/render-next-cache.mjs` stashes `.next/cache` in `$RUNNER_TEMP`
on GitHub Actions instead of letting the postbuild prune delete it, and main's
runs save it under a key scoped to `bun.lock` and `next.config.mjs`; pull
requests restore main's newest. The archive is ~1 GB (1.4 GB unpacked).
Measured 2026-10-07 on two runs of #5845: the compile went from 112s cold to
56s warm and the build step from 169s to 86s, against ~16s to restore and ~18s
to save. Past 3 GiB the stash is dropped, so the next run starts cold.
`build-web` has no Next cache: web turns the persistent cache off in
`web/next.config.mjs` because its cold compile is ~30s. `oven-sh/setup-bun`'s
own small runtime cache is independent of dependency installation and also
remains enabled.

## CLI tmux Testing

For testing CLI behavior via tmux, use the helper scripts in `scripts/tmux/`. These handle bracketed paste mode and session logging automatically. Session data is saved to `debug/tmux-sessions/` in YAML format and can be viewed with `bun scripts/tmux/tmux-viewer/index.tsx`. See `scripts/tmux/README.md` for details.

Useful workflow for agents:

```bash
# Start the dev CLI in a detached tmux session.
SESSION=$(./scripts/tmux/tmux-cli.sh start --name cli-check -w 160 -h 40 --wait 6)

# Capture the initial screen. Captures are written to debug/tmux-sessions/$SESSION/.
./scripts/tmux/tmux-cli.sh capture "$SESSION" --label initial

# Send a prompt. The helper uses bracketed paste so text is not dropped.
./scripts/tmux/tmux-cli.sh send "$SESSION" "Search for getAgentBaseName and report what you find" --wait-idle 4

# Capture after the run, then inspect the saved capture text.
./scripts/tmux/tmux-cli.sh capture "$SESSION" --label after-search --wait 2

# Clean up when finished.
./scripts/tmux/tmux-cli.sh stop "$SESSION"
```

If a change can be verified with a small local harness instead of a live model-backed CLI run, run that harness inside tmux too. This still checks terminal rendering and produces a capture:

```bash
SESSION=$(./scripts/tmux/tmux-cli.sh start \
  --name render-check \
  -w 160 -h 20 \
  --wait 1 \
  --command "bun .context/my-render-check.tsx")

./scripts/tmux/tmux-cli.sh capture "$SESSION" --label rendered
./scripts/tmux/tmux-cli.sh stop "$SESSION"
```

When verifying UI output, prefer checking the saved capture file for concrete strings that should and should not appear. For example, after expanding a code-searcher agent, check that the capture shows the search summary but not raw structured payload keys like `results:` or `stdout:`.

## Confirming a suspected flake

CI runs a failing test job up to three times (`nick-fields/retry` in `ci.yml`).
A unit suite that only passes on a later attempt now fails the build; an
integration one still passes with a warning. Either way the interesting question
is the same one, and CI cannot answer it: **CI never applies contention.** Each
package's suite gets its own runner, so the shared-state flakes live below its
resolution entirely. When a suite fails on a busy machine and passes on a quiet
one, the
temptation is to call it the machine's fault. Don't: **load is not a cause, it
is a magnifier.** What it magnifies are real defects that a single run on an
idle laptop cannot show.

`scripts/flake-hunt.ts` reproduces those conditions deliberately — N rounds,
each running the suite several times *simultaneously* while CPU busy-loops
compete for the cores — and names every test that fails.

```bash
# The default hunt: 3 rounds x 2 overlapping runs of the desktop suite, on the
# Bun that suite is tested on (see below).
bunx bun@$(cat freebuff-desktop/.bun-version) scripts/flake-hunt.ts

# Harder, and on a different package.
bun scripts/flake-hunt.ts --dir cli --rounds 5 --concurrency 3 --hogs 8

# Interrogate one suspect file: cheap enough to run 10 times.
bunx bun@$(cat freebuff-desktop/.bun-version) scripts/flake-hunt.ts \
  --cmd "bun test test/launch-bootstrap.smoke.test.ts" --rounds 10
```

It exits non-zero if any run reported a failing test, prints the load average
per round, and writes each run's full output to a log it names for you. If your
shell lacks the repo env (a worktree without direnv), pass it through the
command: `--cmd "bun --env-file=../.env.local test"`.

**freebuff-desktop is hunted on its own Bun.** CI tests it on the Bun the app
ships with (`freebuff-desktop/.bun-version`), not on the monorepo toolchain
(`.bun-version`) that `bun` on your PATH is, and the two are not
interchangeable. On 2026-09-26 a desktop hunt on the toolchain Bun (1.3.14)
failed `mcp-consent-bridge.test.ts` › "an asker that gives up closes its
dialog, so the next approval is not refused busy" on every run, loaded or solo,
while the pin (1.4.2) passed it every time: 1.3.14's `node:http` never emits
`close` on a response whose client hung up. It was a deterministic runtime
difference, and it read as a flake. So `flake-hunt.ts` now refuses a desktop
hunt when the `bun` its runs would get is not the pin, and says how to get it.
`bunx bun@<pin>` is the one-liner because it puts the pin first on PATH for
everything the hunt starts, and the suite spawns its orchestrator as `bun`
from PATH. Pass `--any-bun` to compare versions on purpose. The weekly workflow
and ci.yml's retry-flake annotation both follow the pin.

### When the hunter cannot reproduce it

The hunter is a magnifier, not an oracle, and one open flake is proof of that.
`useProjectSkills` failed on **three separate PRs, on three different cases**,
none of which touched `freebuff-desktop` — and never once under 24 hunted runs
at load 11-16, nor under 12 concurrent runs with CPU hogs, on the fixed and
unfixed versions alike.

Two concrete mechanisms were removed and it flaked again on a head that already
contained both. At that point further guessing costs other people's PRs, so the
file was skipped and tracked in **issue #1899** rather than patched a third time.

#1983 un-skipped it by moving it off the hand-rolled `mount()` entirely, onto
`@testing-library/react` + jsdom instead (the pattern already used by
`UpdatesPage.test.tsx`, `ConnectorsPanel.test.tsx`, …),
on the theory that the class of bug cannot exist there — React's own scheduler
owns the dispatcher for the whole render/effect/continuation lifecycle, instead
of a test file borrowing it by hand.

**That is a theory, not a diagnosis.** The original flake was never reproduced,
so nothing distinguishes "the dispatcher swap was the cause and is now gone"
from "the timing changed and the flake moved". Leave #1899 open until the file
has survived a stretch of CI; if it flakes again, the migration is evidence
*against* the dispatcher hypothesis and the search should widen, not repeat.
The other twelve files named below still use the swap either way.

Two things worth taking from it:

- **A green `gh pr checks` does not mean the job never failed.** CI retries, and
  a successful re-run overwrites the run conclusion. The failures above are only
  visible via `attempts/1/jobs` on the run. Check there before concluding a
  suite is healthy.
- **The suspect is usually shared, so the file that flaked is rarely the fix.**
  Here it is the hand-rolled `mount()` that swaps the process-global
  `React.__CLIENT_INTERNALS…H` dispatcher and restores it in a `finally` —
  while effects, and the fetch continuations they schedule, run *after* that
  restore. 13 desktop UI test files touch that global and at least six define
  their own `mount()` with the same shape, so fixing one file just moves the
  flake to a sibling — as it still can, for the twelve that remain.

**Three things to check before blaming the machine.** Each of these was a real
defect found by this harness on its first outing:

1. **Shared state between concurrent runs.** `server.test.ts` spawned its
   orchestrator on a hardcoded port *and probed that port for readiness*, so two
   overlapping runs did not collide — they merged. The loser's server died with
   `EADDRINUSE`, its readiness probe was answered by the winner's server, and
   its tests then drove a foreign engine with a different `$HOME` and a
   different open project. Two concurrent runs of that one file produced 6 and
   19 failures scattered across unrelated assertions; alone it passes 49/49.
   Spawn servers on port 0 and read back the port they actually got.

   Then check that the server honours 0. Vite's `server.listen()` reads
   `port: 0` as unset and binds 5173, and the repo config's `strictPort` makes
   it exit before listening if anything is already there: an overlapping run,
   or another project's dev server on Vite's default port. `terminal.test.ts`
   passed `port: 0` and still lost one of three overlapping suite runs in every
   hunted round, and 10 of 15 runs when only that file overlapped. It now calls
   `server.httpServer.listen(0)` itself and gives Vite a per-run `cacheDir`,
   because the deps optimizer otherwise commits into the package's shared
   `node_modules/.vite`. That failure also read `Vite exited before listening:`
   with nothing after the colon, because the child's stderr was piped and never
   read. A test that reports a child's death should carry its stderr and exit
   status.

2. **Waits that sample instead of listening.** A fixed count of 10ms polls is a
   budget with no relationship to how long the work takes, and it pays a full
   tick even when the work already finished. Await the signal the system already
   emits (the engine's event bus, a process's own stdout), with a wall-clock
   deadline that names what it was waiting for.

3. **Timers and promises that outlive their test.** A 200ms debounced save in
   the renderer store fired *inside a later test* that had swapped in its own
   `fetch`, landing a stray request in that test's recorded calls. Which test
   becomes the victim is decided by timing, so a loaded machine changes the
   answer. Cancel pending background work between cases.

A fourth pattern worth naming: a test that races two real timers against each
other (a 20ms cadence against a 40ms window) has a 2x margin that starvation
erases. Inject the clock and advance it explicitly rather than sleeping.

A variant of the second that no deadline fixes: **a wait that samples the
middle of the work.** `engines.test.ts` polled for an idle tab's
`node_modules` to vanish, then asserted that the tab's bootstrap marker was
gone. The strip removed the marker in the continuation *after* its async
`rm` resolved, though (the marker now goes first, which fixes a crash window,
not this), so a 5ms poll that woke in the ~0.1ms between the two failed the
test: 1.7% of runs on an idle laptop, 3.4% under load, and red on CI on
2026-09-26. Whatever you wait on has to be the last thing the work
does. For backgrounded work, await the work itself; `EngineRegistry` exposes
`backgroundSettled()` for exactly this.

A flake that rare will not show up in a six-run full-suite hunt; this one
never did. Narrow it to the test and multiply the samples instead:
`bun test <file> -t '<name>' --rerun-each 300` reproduced it 5/300 in about
a minute, and passing the same command as flake-hunt's `--cmd` adds the load.

One shape worth calling out separately, because no amount of isolating the test
fixes it: **a test whose subject is machine-global.** `orphan-reaper` kills any
process on the box stamped with a dead orchestrator — that is the feature — so
two copies of its real-process test reap each other's children, and the victim
fails on `Received: []` with its orphan already dead, for the module doing
exactly its job. Its temp dirs and pids were already unique; the shared state was
the process table itself. An atomic `mkdir` lock around the one section that
spawns stamped children serialises those runs against each other and costs
concurrency nowhere else.

**A fifth, and the reason some of these reports are unreadable rather than
merely red: a polling helper whose deadline is not shorter than the budget of
the test that called it.** Bun's default is 5s per test. A `waitFor` that gives
up after 10s (or exactly 5s) never gets to report anything — bun fails the test
at its own deadline first, moves on, and the helper's rejection surfaces later
with no test to attribute it to. It is printed as `unhandled error outside any
test` and counted *separately* from the failure that caused it, so one slow
shell shows up as two entries naming a file rather than one naming a case. Five
files had this inverted or equal (`terminal`, `loop-health`, `login`,
`orphan-reaper`, `smoke`). Rather than repeat the rule as a comment in each,
`test/support/budgets.ts` hands back the pair and refuses an inverted one at
import, so getting it wrong fails the file loudly instead of only when something
is slow enough for the difference to show:

```ts
const { slowTest, waitMs } = budgets(30_000, 15_000)
// slowTest declares tests carrying the 30s budget; every polling helper in the
// file gives up at waitMs, which is checked to be under it
```

Raising a test's budget is the right move *only* once you know what the test
actually spends it on. Every case above drives a real process — a pty, a bun
child, a `--version` probe — whose latency scales with machine load, so 5s was a
guess about machine speed rather than a statement about correctness. Where the
work was avoidable instead, it was removed: one claude-code test spawned four
stub binaries where one would do, paying a memoized-away `--version` per extra
binary, which made it the file's most expensive test by 2.5x and its first
casualty under load.

The first monorepo-wide sweep found the shared-state pattern twice more, both
in `cli` and both fixed the same way — give the run its own `mkdtemp` directory
instead of a name every run shares. One wrote to a fixed path *inside the
working tree* (`__dirname/temp-test-images`, deleted in `afterEach`, not
gitignored); the other to seven fixed `os.tmpdir()/codebuff-test-*` paths whose
`beforeEach` deleted them recursively. Overlapped they failed 6/10 and 4/10
runs; solo they always passed. Note the asymmetry that hid them: CI gives each
package's suite its own runner, so nothing there ever runs two copies at once —
these bit developers and this harness, not the pipeline.

Prefer these fixes over longer timeouts. A longer timeout makes the symptom
rarer without making the test deterministic, and it slows every honest run to
buy that. The one legitimate use is converting an *iteration count* into a
wall-clock deadline, since only the latter means "this is broken" rather than
"this machine is slow".

Two caveats when reading the results. A clean hunt is evidence, not proof —
absence of failure across six runs bounds the flake rate, it does not disprove
it. And a test of the form "sleep 20ms, then assert nothing has happened yet"
can only pass *vacuously* under load, so this harness will never flag it; those
need rewriting to assert ordering rather than absence.

The weekly `flake-hunt.yml` workflow runs this against `freebuff-desktop` — on
its pin, over the package's own `test:files` as ci.yml runs it — and reports
failures without blocking any PR.

**A sixth shape, and the only one the hunt cannot reproduce: the budget bun
applies to a HOOK.** `beforeEach`/`afterEach` are charged against the same
default a test gets (5s), and one that overruns is reported as a failing *test*
— `a beforeEach/afterEach hook timed out for this test` — so the case named is
never the case at fault, and the file's first test is what takes the hit. On
2026-08-11 that failed main: `engine-lifecycle`'s `beforeEach`, 3.7ms on a
laptop and 28ms on the retry, took 5.313s on a stalled runner and failed
`dispose > closes only this engine instance terminal scope`, a test that asserts
one recorded call. Three overlapping suite runs under four CPU hogs never
reproduced it, because there is nothing in the test to reproduce: the hook opens
a real sqlite database and `rm -rf`s it again, and its latency is the machine's.
Every hook in this package that builds a real fixture (`makeDb`, `makeRepo`,
`makeEngine`, and worktree.test.ts's inline equivalent — 23 files) now carries
`FIXTURE_HOOK_MS` from `test/support/budgets.ts`; the tests themselves keep the
5s default, and a hook that is genuinely stuck still fails at the step timeout.
The line is cost, not the helper: measured per hook pair, `makeRepo` is 77ms and
`makeDb`/`makeEngine` 4.7ms, against 0.1ms for the 60-odd hooks that only
`mkdtemp` and `rm -rf`. The stall that reddened main inflated a hook ~190x (and
the test after it ~42x, which is why it reads as contention decaying rather than
one fixed pause); at 0.1ms you would need ~10,000x, so those are left alone.

That flake also cost more to find than to fix, which is its own lesson: a suite
that fails and then passes leaves the failing attempt's output buried above the
passing one, and `nick-fields/retry` reports only a count. `test-with-guard`
therefore appends each attempt's failing test names to a file under the runner's
`$RUNNER_TEMP`, and ci.yml's flake annotation names them. Before that,
identifying the one `(fail)` line meant reading an 11,014-line log.
`scripts/ci/bun-test-failures.ts` parses the names back out of bun's output,
including the `^ …` reason line, and derives that path at both ends rather than
having ci.yml pass it: `${{ runner.temp }}` is not available in a job-level
`env:`, and a workflow that references it there fails validation and runs ZERO
jobs — which is exactly how this landed the first time.

**A seventh shape, and the one that does not care about load at all: the FILE
ORDER.** The five shapes above are magnified by a busy machine, so the flake hunt
finds them. This one is decided before any test runs, by which file bun loaded
first — and bun does **not** load files in the order the command line lists them
(`test:files` sorts; the CI log does not come out sorted). So the hunt is blind
to it, the full suite passes 3573/3573 on a laptop, and CI still fails with a
different set of tests every run.

What it looked like: `test-freebuff-desktop` red on four `main` commits in a row,
once on a commit with no file changes at all, naming ~20 tests across
`Tab.rename`, `ContextBar.menu`, `AccountMenu.menu`, `AgentPicker.menu` and
`QuotaBadge` (since deleted) — five files with nothing in common except that they were the only
five that mount React against a real DOM. Renderers came back as the empty
string, `fireEvent` was handed nothing to click, and the head of the cascade was
one test sitting on `await new Promise((r) => requestAnimationFrame(r))` until
bun's 5s budget.

The cause is `global-jsdom` deciding **once per process** which globals it
installs. Its `KEYS` list is built on the first call — every own property of the
jsdom window that is not already on `globalThis` — and cached at module scope for
every call after it. Two suites had left `requestAnimationFrame` on `globalThis`:
one assigned a `() => 0` stub at module scope and put back only `fetch`, the
other "restored" it with `Object.assign`, which writes `undefined` rather than
removing the property. `!(k in global)` cannot tell either from a real global, so
whichever mounted suite loaded first inherited a rAF that never fires — and so
did the other four, for the rest of the run.

Note what the loser is not: the file holding the stub passes either way. Fixing
this by chasing the failing tests, or by re-splitting a suite to isolate it (as
#1382 did for the quota timers), treats the victim.

The cascade is worth understanding separately from the poisoning, because the two
were fixed by different PRs. The other four files did not fail because they
needed rAF — they failed because `Tab.rename` was killed mid-`act()` at the 5s
budget and left React's state inconsistent for everything after it. #1569 drives
those frames by hand instead of awaiting a real one, which removes the hang and
therefore the cascade. That is the trigger. The mechanism below is what stays:
verified by probe on top of #1569, a stubbed `getComputedStyle` or
`MutationObserver` is still never replaced by jsdom's, in every mounted suite,
for the whole run. Nothing awaits those today, so nothing is red — which is
exactly the shape of a defect that reappears the next time someone writes a test
that does.

Two rules come out of it:

1. **Restore a global by DELETING the key when there was none.** Capture
   `Object.getOwnPropertyDescriptor` and `Reflect.deleteProperty` if it comes
   back undefined. Assigning the captured `undefined` leaves the property in
   place, which is all any `in` check — including global-jsdom's — looks at.
2. **Get a DOM only from `test/support/jsdom.ts`.** `installJsdom()` clears the
   DOM globals before handing over, so global-jsdom makes that one-shot decision
   against bun's own globals rather than against whatever ran first, and then
   asserts the keys React DOM needs really did come from jsdom. A stub it does
   not know about fails there, naming the key, instead of surfacing as a timeout
   in a file that never touched it.

The general form is worth keeping in mind beyond jsdom: **a library that caches a
decision at module scope turns "who ran first" into behaviour**, and bun's one
process per package makes every test file a candidate for first. Contention is
not required, so neither `flake-hunt.ts` nor a green local run says anything
about it. Reproduce it instead by putting the suspected leak in a `--preload`,
which is exactly "some earlier file did this", and running the affected files:

```bash
echo ';(globalThis as any).requestAnimationFrame = () => 0' > /tmp/leak.ts
cd freebuff-desktop && bun test \
  --preload ../test/setup-scm-loader.ts --preload ../sdk/test/setup-env.ts --preload /tmp/leak.ts \
  src/ui/shell/Tab.rename.test.tsx src/ui/agent/AgentPicker.menu.test.tsx
```

Our own components do it too. Freebuff Web's survey cards show once per page load
and keep that in module state behind a reset seam (`resetWorkspaceProfileSurveySession`,
`resetWorkspaceOnboardingSurveySession`). Any file that renders `WorkspaceApp` can
spend it. So a test that needs the card resets it in `beforeEach`, not only
`afterEach`. Otherwise its first test renders nothing whenever bun loads it after
`workspace-app.dom` or `workspace-queue.dom` (2026-10-13, `sponsored-survey.dom`).

**`mock.module()` is the same hazard, and nothing in bun undoes it.** Measured on
Bun 1.3.14 (2026-10-13):

- **`mock.restore()` does not restore a module mock.** It restores spies. A file
  that ends `afterAll(() => mock.restore())` leaves its module fakes in place for
  every file after it. Only `clearMockedModules()` from
  `@codebuff/common/testing/mock-modules.ts` puts a module back.
- **The fake behaves two ways, depending on whether anything loaded the module
  first.** If nothing did, later importers get only the factory's exports, and
  any file that imports another one fails to load:
  `SyntaxError: Export named 'db' not found in module '.../db/index.ts'`. If an
  earlier file did, bun patches the factory's exports into the loaded module in
  place and keeps the rest. Later files, and earlier importers through their live
  bindings, then run silently against the fake. A two-file run usually shows the
  first; the full suite usually the second.
- **Bun walks directories breadth-first.** Shallower directories come first;
  within a directory, files and subdirectories come in readdir order, which is
  fixed per tree on APFS and ext4 but differs between them. So
  `bun test leaker.test.ts victim.test.ts` runs the pair in that tree order, whatever
  the command line says (`--reporter=junit` lists it). A file can leak only into
  files bun loads after it: at the same depth, later in that order, or deeper.

So inject the dependency instead: a handler that takes its collaborators (the
`_post.ts` + `route.ts` split), or an optional `deps` argument. To reproduce a
suspected module-mock leak, run the leaking file with the victim. If bun orders
the victim first, put the leaking file's `mock.module` calls in a preload beside
it, so relative paths and `@/` resolve the same way. Then run the victim with
web's own preloads, since `--preload` on the command line replaces bunfig's list.
For the in-place case, `await import()` each module in that preload before
mocking it.

**An eighth shape, and one where the wait is fast and the CHECK is slow: a
failing matcher handed a mounted DOM node.** `expect(node).toBe(other)` and
`expect(node).toBeNull()` build their failure message by pretty-printing the
node, and bun's formatter does not stop at the element — it follows React's
`__reactFiber$…` expando through the whole fiber tree into jsdom's `Window`.
Measured on the Composer: one failed `expect(document.activeElement).toBe(input)`
built a **322 MB** message in ~1.7s of synchronous CPU. A passing matcher formats
nothing, so this costs nothing until a check fails, and **`waitFor` exists to
run checks that fail**: it calls its callback once synchronously before
yielding, so any condition that only becomes true after a frame or a fetch pays
that price on every run, with the event loop blocked the whole time (not even
`waitFor`'s own 1000ms timeout can fire).

That was `Composer.skill-submit` › "the Skills picker lets the user add
arguments before Send" (2026-09-24, #3835 and #3927): `placeInComposer` moves
focus in a `requestAnimationFrame` that fires in 17ms, and the test spent 1.7s
of its 5s budget on a laptop formatting the one failed check that preceded it —
and the whole budget on a loaded runner. The same pattern cost
`ProjectSidebar`'s archive/restore case 1.5s. Neither looks slow in the source,
and raising the timeout would only have hidden it.

Both packages now preload a hook that prints any jsdom node as its `outerHTML`
(capped at 500 chars) and a window or document as its URL:
`freebuff/web/test/support/jsdom-inspect.ts` (#5694) and its port
`freebuff-desktop/test/support/jsdom-inspect.ts`. With it, that failed
`expect(node).toBeNull()` is a one-line message built in about a millisecond.
A test run without the package's `bunfig.toml` (from another directory) has no
hook and still prints the whole Window, and the helpers' messages read better,
so the rule stands: **inside `waitFor` (or any retry), prefer
`expectFocused(el)` / `expectAbsent(query, what)` from
`freebuff-desktop/test/support/dom-expect.ts`** over a DOM node in a matcher. To
spot the shape, time the case with `bun test --reporter=junit
--reporter-outfile=…`: a mounted-React case taking over a second with no real
I/O is almost always this.

---

--- METRICS ---
- Files Extracted: 18
- Estimated Token Budget: ~37749 tokens
- Canonical Reference: https://codewiki.google/github.com/CodebuffAI/codebuff