codebuff Documentation

Technical Documentation: CodebuffAI/codebuff

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

Freebuff

Five free AI products for coding, building, and research. No subscription, credits, or API key required.

Freebuff brings specialized agents and a choice of leading models to your terminal, desktop, browser, and GitHub repositories. Text ads support access to the included models.

Choose your Freebuff

Product	What it does	Get started
Freebuff Desktop	Run parallel agents locally	Download for macOS, Windows, or Linux
Freebuff CLI	Code from your terminal	Install the CLI
Freebuff Web	Build and ship full-stack apps	Build an app
Freebuff Cloud	Run agents on any GitHub repository	Connect a repository
Freebuff Chat	Research and think with AI	Start a chat

Quick start

Run Freebuff in any project from your terminal:

bash

npm install freebuff
cd ~/my-project
freebuff

Then describe what you want. Freebuff finds the relevant files, makes changes, and runs the checks that matter for your project.

Models

Freebuff includes a curated model catalog. The regular picker currently offers:

Model	Access	Best for
MiMo v2.6 Flash Fast	Full and limited access	The default everywhere; balanced performance with image support
GLM 5.3 Flash	Full and limited access	Deepest reasoning, unmetered
DeepSeek V4.1 Flash	Full and limited access	Fast coding and tool use, unmetered
DeepSeek V4.1 Flash Fast	Full access; paid plans at limited access	Fast mode: the same model on DeepSeek's own API, splitting work across parallel subagents and verifying the result; priced on DeepSeek's peak and off-peak hours
GPT-6 Luna	Full access	Strong all-around with native images; runs on OpenAI flex capacity
MiMo 2.6 Pro	Full access	Xiaomi's stronger reasoning model, with image support
Solar Mini 4	Full and limited access	Upstage's fast, compact model; 524K context, text only
Solar Pro 4	Full and limited access	Upstage's larger, stronger model; 524K context, text only
Gemini 3.8 Flash	Paid plans	1M context, and the only model that accepts audio, video and PDF
Muse Spark 1.3	Paid plans	Meta's agentic coding model; 1M context. When it is busy or unavailable, it answers on DeepSeek V4.1 Flash rather than making you wait
Gemini 4 Argon	US, or paid plans elsewhere	Google's flagship reasoning model at a temporary promotional price, one session a day for every account; 1M context, images
GPT-6.1 Sol	US, or paid plans elsewhere	OpenAI's flagship, one session a day for every account; 1M context, images

Most models draw on your normal daily sessions rather than a separate limit. GLM 5.3 Flash, DeepSeek V4.1 Flash, MiMo v2.6 Flash Fast, Solar Mini 4 and Solar Pro 4 are unmetered at full access and cost no session at all. Models may still serve from a quantized (Q8_0) build.

DeepSeek V4 Pro was retired from the catalog; GLM 5.3 Flash replaces it as the deep-reasoning pick.

Beyond the regular picker:

Referrals and bounties earn extra sessions on top of the free daily allowance.
Gemini 3.1 Flash Lite powers specialist tasks such as file finding and research rather than appearing in the main picker.

Availability and limits depend on your access tier, product, and current capacity. Freebuff Desktop can also run locally installed Claude Code and Codex agents using your existing provider account; those connected models are separate from Freebuff's included catalog.

How Freebuff works

Freebuff uses specialized agents instead of sending every task through one model and one prompt. Depending on the task, agents gather context, plan, edit or research, run tools, and review the result.

Codebase context — File-finding agents map the relevant parts of a project before editing.
Implementation and review — Agents can divide work, make changes, run commands, and inspect the result.
Research and browser use — Agents can investigate documentation and test applications in a real browser.
Parallel local work — Desktop isolates concurrent agents in separate workspaces.
Hosted environments — Web and Cloud provide sandboxes, previews, terminals, and deployment workflows.

Free access

Freebuff is available in every country. Supported regions receive full access; other regions and VPN users receive limited access to GLM 5.3 Flash, DeepSeek V4.1 Flash, MiMo v2.6 Flash Fast, Solar Mini 4, and Solar Pro 4. Every model is paid for in hours at the multiplier shown in the picker: an hour of MiMo v2.6 Flash Slow uses one hour of balance, a 3x model uses three. The free limited-access allowance is 5 hours a day, or 4 on a VPN or proxy, spent first on every model, GLM included; earned hours go into your wallet and are used after it. (Balances read in hours since 2026-10-15; 1 hour = 5 Freebucks.)

Text ads support the included models. Freebuff shows the applicable session limits and any model-specific data-use notice before you start.

Is my data used to train AI? Only when a model or feature says data may be used for AI training. Freebuff or the provider may then keep submissions to develop, train, test, evaluate, fine-tune, and improve AI models or products, and Freebuff may share or sell them to third parties. Separately, Freebuff may use your data to train the models that choose and measure ads.

How is my data used and stored? Freebuff uses your messages, agent traces, and files to run the service and respond to your requests. We may analyze this data, and how you use the app, to personalize ads and train ad models. We do not give separately uploaded files, project files, or connected repositories to advertising providers. Restricted partners may evaluate repositories you connect to Freebuff Cloud, but cannot otherwise use, broadly share, or train on them. See the Privacy Policy for details.

See the Privacy Policy for complete details.

Contributing

Freebuff is a TypeScript monorepo built with Bun. Contributions to the products, agents, tools, documentation, and underlying runtime are welcome.

Local development requires Docker and a configured .env.local; see the
Contributing Guide before starting the services.

bash

git clone https://github.com/CodebuffAI/freebuff.git
cd freebuff
bun install
bun up

Start the CLI separately with:

bash

bun start-cli

See the Contributing Guide, development guide, and testing guide for environment setup and the checks to run before opening a pull request.

Built on Codebuff

Freebuff is built on Codebuff, the open multi-agent framework that powers its orchestration, tools, and SDK. To create custom agents or embed them in another application, see the Codebuff documentation and @codebuff/sdk.

Links

Overview (`cli/README.md`)

@codebuff/cli

A Terminal User Interface (TUI) package built with OpenTUI and React.

Installation

bash

bun install

Development

Run the TUI in development mode:

bash

bun run dev

Testing

Run the test suite:

bash

bun test

Interactive E2E Testing

For testing interactive CLI features, install tmux:

bash

# macOS
brew install tmux

# Ubuntu/Debian
sudo apt-get install tmux

# Windows (via WSL)
wsl --install
sudo apt-get install tmux

Then run the proof-of-concept:

bash

bun run test:tmux-poc

Note: When sending input to the CLI via tmux, you must use bracketed paste mode. Standard send-keys drops characters.

bash

# ❌ Broken: tmux send-keys -t session "hello"
# ✅ Works:  tmux send-keys -t session $'\e[200~hello\e[201~'

See tmux.knowledge.md for comprehensive tmux documentation and src/tests/README.md for testing documentation.

Build

Build the package:

bash

bun run build

Run

Run the built TUI:

bash

bun run start

Or use the binary directly:

bash

codebuff-tui

Features

Built with OpenTUI for modern terminal interfaces
Uses React for declarative component-based UI
TypeScript support out of the box

Overview (`freebuff/README.md`)

Freebuff

The free coding agent. No subscription. No configuration. Start in seconds.

An AI coding agent that runs in your terminal — describe what you want, and Freebuff edits your code.

Install

bash

npm install freebuff

Usage

bash

cd ~/my-project
freebuff

Project Structure

text

freebuff/
├── cli/       # CLI build & npm release files
└── web/       # Freebuff website

Building from Source

bash

# From the repo root
bun freebuff/cli/build.ts 1.0.0

For everything else — what Freebuff does, how it works, FAQ, and how it relates to Codebuff — see the repo root README. We keep that one up to date as the single source of truth.

License

MIT

Overview (`sdk/README.md`)

@codebuff/sdk

Official SDK for Codebuff - AI coding agent and framework

Installation

bash

npm install @codebuff/sdk

Prerequisites

Create a Codebuff account and get your Codebuff API key here.

Usage

Basic Example

typescript

import { CodebuffClient } from '@codebuff/sdk'

async function main() {
  const client = new CodebuffClient({
    // You need to pass in your own API key here.
    // Get one here: https://www.codebuff.com/api-keys
    apiKey: process.env.CODEBUFF_API_KEY,
    cwd: process.cwd(),
  })

  // First run
  const runState1 = await client.run({
    // The agent id. Any agent on the store (https://codebuff.com/store)
    agent: 'codebuff/[email protected]',
    prompt: 'Create a simple calculator class',
    handleEvent: (event) => {
      // All events that happen during the run: agent start/finish, tool calls/results, text responses, errors.
      console.log('Codebuff Event', JSON.stringify(event))
    },
  })

  // Continue the same session with a follow-up
  const runOrError2 = await client.run({
    agent: 'codebuff/[email protected]',
    prompt: 'Add unit tests for the calculator',
    previousRun: runState1, // <-- this is where your next run differs from the previous run
    handleEvent: (event) => {
      console.log('Codebuff Event', JSON.stringify(event))
    },
  })
}

main()

Example 2: Custom Agents and Tools

Here, we create a full agent and custom tools that can be reused between runs.

text

/* Detailed source-code truncated for AI context efficiency. */

API Reference

Knowledge Files

Knowledge files provide project context to the agent. The SDK auto-discovers:

Project files: knowledge.md, AGENTS.md, or CLAUDE.md in each directory (priority order)
User files: ~/.knowledge.md, ~/.AGENTS.md, or ~/.CLAUDE.md (case-insensitive)

Override with knowledgeFiles (replaces project files) or userKnowledgeFiles (merges with home directory files):

typescript

await client.run({
  agent: 'codebuff/[email protected]',
  prompt: 'Help me refactor',
  knowledgeFiles: { 'knowledge.md': '# Guidelines\n- Use TypeScript' },
  userKnowledgeFiles: { '~/.knowledge.md': '# Preferences\n- Be concise' },
})

File Filtering

The SDK always blocks .env and .env.* files case-insensitively, except for
the .env.example, .env.sample, and .env.template templates. Use
fileFilter to add project-specific rules:

typescript

const client = new CodebuffClient({
  apiKey: process.env.CODEBUFF_API_KEY,
  fileFilter: (filePath) => {
    if (filePath.endsWith('.pem')) return { status: 'blocked' }
    return { status: 'allow' }
  },
})

Statuses: 'blocked' (returns [BLOCKED]), 'allow-example' (prefixes content with [TEMPLATE]), 'allow' (normal read).

Default behavior: The built-in .env policy always applies. When no
fileFilter is provided, gitignore checking also applies automatically,
including for the allowed env templates. When a fileFilter is provided, the
caller owns any additional filtering, but env templates remain subject to
gitignore. This built-in policy is scoped to
SDK-mediated agent file reads (including read_files); it does not add
terminal-command or internal-edit restrictions.

Custom overrideTools.read_files implementations must preserve project
gitignore behavior for env templates. The built-in CLI, Desktop, Web, and Cloud
bridges already do this.

`loadLocalAgents(options)`

Loads agent definitions from .agents directories on disk.

typescript

import { loadLocalAgents, CodebuffClient } from '@codebuff/sdk'

// Load from default locations (.agents in cwd, parent, or home)
const agents = await loadLocalAgents({ verbose: true })

// Or load from a specific directory
// const agents = await loadLocalAgents({ agentsPath: './my-agents' })

// Or load and validate agents (invalid agents are filtered out)
// const agents = await loadLocalAgents({ validate: true, verbose: true })

// Access source file path for debugging
for (const agent of Object.values(agents)) {
  console.log(`${agent.id} loaded from ${agent._sourceFilePath}`)
}

// Use the loaded agents with client.run()
const client = new CodebuffClient({ apiKey: process.env.CODEBUFF_API_KEY })
const result = await client.run({
  agent: 'my-custom-agent',
  agentDefinitions: Object.values(agents),
  prompt: 'Hello',
})

Parameters

agentsPath (string, optional): Path to a specific agents directory. If omitted, searches in {cwd}/.agents, {cwd}/../.agents, and {homedir}/.agents.
verbose (boolean, optional): Whether to log errors during loading. Defaults to false.
validate (boolean, optional): Whether to validate agents after loading. Invalid agents are filtered out. Defaults to false.

Returns

Returns a Promise<LoadedAgents> - a Record<string, LoadedAgentDefinition> of agent definitions keyed by their ID.

Each LoadedAgentDefinition extends AgentDefinition with:

_sourceFilePath (string): The file path the agent was loaded from

Supported File Types

.ts, .tsx - TypeScript files (automatically transpiled)
.js, .mjs, .cjs - JavaScript files

Files ending in .d.ts or .test.ts are excluded.

`client.run(options)`

Runs a Codebuff agent with the specified options.

Parameters

agent (string, required): The agent to run. Use 'base' for the default agent, or specify a custom agent ID if you made your own agent definition (passed with the agentDefinitions param).
prompt (string, required): The user prompt describing what you want the agent to do.
params (object, optional): Additional parameters for the agent. Most agents don't use this, but some custom agents can take a JSON object as input in addition to the user prompt string.
handleEvent (function, optional): Callback function that receives every event during execution (assistant messages, tool calls, etc.). This allows you to stream the agent's progress in real-time. We will likely add a token-by-token streaming callback in the future.
previousRun (object, optional): JSON state returned from a previous run() call. Use this to continue a conversation or session with the agent, maintaining context from previous interactions.
projectFiles (object, optional): All the files in your project as a plain JavaScript object. Keys should be the full path from your current directory to each file, and values should be the string contents of the file. Example: { "src/index.ts": "console.log('hi')" }. This helps Codebuff pick good source files for context. Note: This parameter was previously named allFiles but has been renamed for clarity.
knowledgeFiles (object, optional): Knowledge files to inject into every run() call. Uses the same schema as projectFiles - keys are file paths and values are file contents. These files are added directly to the agent's context.
agentDefinitions (array, optional): Array of custom agent definitions. Each object should satisfy the AgentDefinition type.
customToolDefinitions (array, optional): Array of custom tool definitions that extend the agent's capabilities. Each tool definition includes a name, Zod schema for input validation, and a handler function. These tools can be called by the agent during execution.
maxAgentSteps (number, optional): Maximum number of steps the agent can take before stopping. Use this as a safety measure in case your agent starts going off the rails. A reasonable number is around 20.

Returns

Returns a Promise that resolves to either a "success" or a "failure" object.

The "success" object contains a RunState object which can be passed into subsequent runs via the previousRun parameter to resume the conversation.
The "failure" object contains an Error object with a name, message, and stack properties.

The RunState object contains:

sessionState: Internal state to be passed to the next run
output: The agent's output (text, error, or other types)

License

MIT

Overview (`cli/release-core/README.md`)

Release launcher core

This directory is the canonical implementation of the npm launchers used by
Codebuff, Codecane, and Freebuff. Each product keeps a small index.js in its
release package that supplies product-specific configuration to
createLauncher().

The npm packages must remain standalone, so their prepack scripts copy
launcher.js and http.js into the package directory. postpack removes those
generated, gitignored files again. No lifecycle scripts run when users install
or uninstall the packages.

Download integrity

The launcher downloads a .tar.gz from
https://codebuff.com/api/releases/download/<version>/<file> (a 302 to the
GitHub release asset) and verifies its sha256 before extracting it. The
expected hashes never come from the download origin:

For the wrapper's own version they are read from the package's own
package.json field binaryChecksums ({ "<target>": "<sha256 hex>" }).
For a self-update to latest (or any other version) they are read from the
npm registry's version document, which carries the same field.

That makes the npm registry (TLS + the package manifest) the trust root and
leaves the download host, and every redirect hop, unable to substitute a binary.
Verification fails closed: a missing or malformed checksum aborts the
install, so write-binary-checksums.js runs in each release workflow's npm
publish job — after the binaries are built — to stamp one hash per target into
the package manifest, and its verify mode fails the release if any target is
uncovered. Wrappers published before this field existed ignore it, so old
installs keep working.

Transport rules, in http.js / launcher.js: the NEXT_PUBLIC_CODEBUFF_APP_URL
override is honoured only over https: (or http: on localhost), redirects
never downgrade https: to http:, and a redirect may only land on the
original host, codebuff.com, freebuff.com, github.com (with www.
variants) or *.githubusercontent.com. Extraction admits only the binary and
tree-sitter.wasm as root-level plain files.

When changing launcher behavior, edit this directory and test all package
assemblies with:

bash

npm pack ./cli/release --dry-run
npm pack ./cli/release-staging --dry-run
npm pack ./freebuff/cli/release --dry-run

Overview (`cli/release-staging/README.md`)

🚀 Codecane - The most powerful coding agent (STAGING)

⚠️ This is a staging/beta release for testing purposes.

Codecane is a CLI tool that writes code for you.

Run codecane from your project directory
Tell it what to do
It will read and write to files and run commands to produce the code you want

Note: Codecane will run commands in your terminal as it deems necessary to fulfill your request.

Installation

To install Codecane (staging), run:

bash

npm install codecane@beta

(Use sudo if you get a permission error.)

Usage

After installation, you can start Codecane by running:

bash

codecane [project-directory]

If no project directory is specified, Codecane will use the current directory.

Once running, simply chat with Codecane to say what coding task you want done.

Features

Understands your whole codebase
Creates and edits multiple files based on your request
Can run your tests or type checker or linter; can install packages
It's powerful: ask Codecane to keep working until it reaches a condition and it will.

Our users regularly use Codecane to implement new features, write unit tests, refactor code, write scripts, or give advice.

Knowledge Files

To unlock the full benefits of modern LLMs, we recommend storing knowledge alongside your code. Add a knowledge.md file anywhere in your project to provide helpful context, guidance, and tips for the LLM as it performs tasks for you.

Codecane can fluently read and write files, so it will add knowledge as it goes. You don't need to write knowledge manually!

Some have said every change should be paired with a unit test. In 2024, every change should come with a knowledge update!

Tips

Type '/help' or just '/' to see available commands.
Create a knowledge.md file and collect specific points of advice. The assistant will use this knowledge to improve its responses.
Type undo or redo to revert or reapply file changes from the conversation.
Press Esc or Ctrl+C while Codecane is generating a response to stop it.

Troubleshooting

If you are getting permission errors during installation, try using sudo:

text

sudo npm install codecane@beta

If you still have errors, it's a good idea to reinstall Node.

Feedback

We value your input! Please email your feedback to [email protected]. Thank you for using Codecane!

Overview (`cli/release/README.md`)

The most powerful coding agent

Codebuff is a CLI tool that writes code for you.

Run codebuff from your project directory
Tell it what to do
It will read and write to files and run commands to produce the code you want

Note: Codebuff will run commands in your terminal as it deems necessary to fulfill your request.

Installation

To install Codebuff, run:

bash

npm install codebuff

(Use sudo if you get a permission error.)

Usage

After installation, you can start Codebuff by running:

bash

codebuff [project-directory]

If no project directory is specified, Codebuff will use the current directory.

Once running, simply chat with Codebuff to say what coding task you want done.

Features

Understands your whole codebase
Creates and edits multiple files based on your request
Can run your tests or type checker or linter; can install packages
It's powerful: ask Codebuff to keep working until it reaches a condition and it will.

Our users regularly use Codebuff to implement new features, write unit tests, refactor code,write scripts, or give advice.

Knowledge Files

Codebuff can fluently read and write files, so it will add knowledge as it goes. You don't need to write knowledge manually!

Some have said every change should be paired with a unit test. In 2024, every change should come with a knowledge update!

Tips

Type '/help' or just '/' to see available commands.
Create a knowledge.md file and collect specific points of advice. The assistant will use this knowledge to improve its responses.
Type undo or redo to revert or reapply file changes from the conversation.
Press Esc or Ctrl+C while Codebuff is generating a response to stop it.

Troubleshooting

Permission Errors

If you are getting permission errors during installation, try using sudo:

text

sudo npm install codebuff

If you still have errors, it's a good idea to reinstall Node.

Corporate Proxy / Firewall

If you see Failed to download codebuff: Request timeout or Failed to determine latest version, you may be behind a corporate proxy or firewall.

Codebuff respects standard proxy environment variables. Set HTTPS_PROXY to route traffic through your proxy:

Linux / macOS (bash/zsh):

bash

export HTTPS_PROXY=http://your-proxy-server:port
codebuff

Windows (PowerShell):

powershell

$env:HTTPS_PROXY = "http://your-proxy-server:port"
codebuff

Windows (CMD):

cmd

set HTTPS_PROXY=http://your-proxy-server:port
codebuff

To make it permanent, add the export or set line to your shell profile (e.g. ~/.bashrc, ~/.zshrc, or Windows System Environment Variables).

Supported environment variables:

Variable	Purpose
`HTTPS_PROXY` / `https_proxy`	Proxy for HTTPS requests (recommended)
`HTTP_PROXY` / `http_proxy`	Fallback proxy for HTTP requests
`NO_PROXY` / `no_proxy`	Comma-separated list of hostnames to bypass the proxy (port suffixes are ignored)

Both http:// and https:// proxy URLs are supported. Proxy authentication is supported via URL credentials (e.g. http://user:password@proxy:port).

Feedback

We value your input! Please email your feedback to [email protected]. Thank you for using Codebuff!

Overview (`evals/buffbench/README.md`)

BuffBench

BuffBench is Codebuff's evaluation framework for measuring AI coding agent performance through real-world git commit reconstruction tasks.

Overview

BuffBench evaluates coding agents by having them reconstruct actual git commits from open source repositories. The system uses AI-powered judging to assess both the quality of implementation and the agent's problem-solving process.

Key Features

Parallel Agent Comparison: Run multiple agents simultaneously on the same tasks for direct performance comparison
Trace Analysis: Deep analysis of agent execution traces to understand decision-making patterns and workflows
Meta-Analysis: Aggregate analysis across all tasks to identify consistent strengths, weaknesses, and patterns
Structured Output: Judges use structured output for more reliable and consistent scoring
Final Check Commands: Run custom validation commands (tests, lints, etc.) after agent implementations
Binary Installation: Support for installing required binaries in isolated environments
Enhanced Metrics: Track completion score, code quality score, and overall score separately

Architecture

System Components

Evaluation Orchestration (run-buffbench.ts)
- Manages the complete evaluation pipeline
- Handles parallel execution of multiple agents on multiple tasks
- Coordinates between agents, judges, and analyzers
- Saves comprehensive logs and traces
Agent Runner (agent-runner.ts)
- Executes coding agents in isolated test repositories
- Captures execution traces and metrics
- Extracts git diffs and context files
- Runs optional validation commands
Judging System (judge.ts)
- Uses AI (GPT-5) to score implementations
- Runs 2 parallel judges (GPT and Sonnet), averaging their scores and keeping the median judge's analysis; warns loudly if one fails, since a silently-dropped judge turns the panel into a single opinion
- Compares agent output against ground truth git diffs
- Provides detailed analysis with strengths and weaknesses
- Scores on three dimensions: completion, code quality, and overall
Trace Analyzer (trace-analyzer.ts)
- Analyzes agent execution traces after each task
- Compares how different agents approached the same problem
- Identifies process strengths, weaknesses, and recommendations
- Focuses on agent workflows and decision-making patterns
Meta Analyzer (meta-analyzer.ts)
- Performs aggregate analysis across all tasks
- Identifies consistent patterns in agent behavior
- Provides high-level insights and development priorities
- Compares agent configurations and their impact on performance
Evaluation Generation (gen-evals.ts, gen-repo-eval.ts)
- Generates evaluation tasks from git commits
- Creates natural language prompts and specifications
- Extracts file diffs and supplemental context
- Supports batch processing of multiple commits

Evaluation Workflow

mermaid

sequenceDiagram
    participant Orchestrator as BuffBench Orchestrator
    participant AgentRunner as Agent Runner
    participant Agent as Coding Agent
    participant Judge as AI Judge (3x)
    participant TraceAnalyzer as Trace Analyzer
    participant MetaAnalyzer as Meta Analyzer

    loop For each task
        Orchestrator->>AgentRunner: Run all agents in parallel
        
        par Agent 1
            AgentRunner->>Agent: Execute with prompt
            Agent->>AgentRunner: Return trace + changes
        and Agent 2
            AgentRunner->>Agent: Execute with prompt
            Agent->>AgentRunner: Return trace + changes
        end
        
        AgentRunner->>Judge: Judge each agent's output (3 judges)
        Judge->>AgentRunner: Return median scores + analysis
        
        AgentRunner->>TraceAnalyzer: Analyze all agent traces
        TraceAnalyzer->>AgentRunner: Return process insights
        
        Orchestrator->>Orchestrator: Save task results + analysis
    end
    
    Orchestrator->>MetaAnalyzer: Analyze all tasks
    MetaAnalyzer->>Orchestrator: Return aggregate insights

Key Features

Multi-Agent Comparison

Parallel Execution: Run multiple agents simultaneously on the same tasks
Fair Comparison: All agents work from identical repository states
Comprehensive Metrics: Compare scores, costs, duration, and success rates
Process Analysis: Understand how agents differ in their problem-solving approaches

Advanced Analysis

Trace Analysis (Per-Task)

After each task, the trace analyzer examines:

How agents approached the problem
Tool usage patterns and sequences
Context gathering strategies
Decision-making workflows
Relative performance comparison

Meta-Analysis (Across All Tasks)

After all tasks complete, the meta analyzer identifies:

Consistent strengths and weaknesses per agent
Performance trends and patterns
Cost vs quality trade-offs
Reliability and consistency metrics
Actionable development priorities

Comprehensive Scoring

The AI judge evaluates three dimensions:

Completion Score (0-10): How completely was the prompt addressed?
Code Quality Score (0-10): How well-structured and maintainable is the code?
Overall Score (0-10): Combined assessment of implementation quality

Validation Support

Final Check Commands: Run tests, lints, or other validation after agent execution
Binary Installation: Install required tools (e.g., linters, test runners) in isolated environments
Custom Environment: Set environment variables for evaluation runs

External CLI Agents

BuffBench supports running external CLI coding agents for comparison:

Claude Code: Use external:claude - requires claude CLI installed
Codex: Use external:codex - requires codex CLI installed
OpenCode: Use external:opencode - requires opencode CLI installed

Example comparing Codebuff vs Claude Code:

typescript

await runBuffBench({
  evalDataPath: 'evals/buffbench/eval-codebuff.json',
  agents: ['base2', 'external:claude'],
  taskConcurrency: 3,
})

Prerequisites for External Agents

Claude Code CLI:

bash

npm install @anthropic-ai/claude-code
# Set ANTHROPIC_API_KEY or CLAUDE_CODE_KEY environment variable

Codex CLI:

bash

npm install @openai/codex
# Set OPENAI_API_KEY environment variable

OpenCode CLI:

bash

# Install from https://opencode.ai/docs/install
# Set OPENCODE_API_KEY environment variable
# BuffBench uses opencode/kimi-k2.6 by default; override with OPENCODE_MODEL if needed.

Directory Structure

text

evals/buffbench/
├── run-buffbench.ts          # Main orchestrator
├── main.ts                   # Example usage
├── main-nightly.ts           # Nightly evaluation runner with email reporting
├── main-single-eval.ts       # Single evaluation runner
│
├── agent-runner.ts           # Executes agents in test repos
├── freestyle-sandbox.ts      # --freestyle: one Freestyle VM per agent run
├── freestyle-task.ts         # The in-VM entry point of a Freestyle run
├── judge.ts                  # AI judging system
├── trace-analyzer.ts         # Per-task trace analysis
├── meta-analyzer.ts          # Cross-task meta analysis
│
├── gen-evals.ts              # Generate eval tasks from commits
├── gen-repo-eval.ts          # End-to-end eval creation
├── eval-task-generator.ts    # Task prompt generation
│
├── format-output.ts          # Result formatting utilities
├── trace-utils.ts            # Trace processing utilities
├── lessons-extractor.ts      # Extract lessons from failures
│
├── types.ts                  # Type definitions
│
├── eval-codebuff.json        # Codebuff project evaluations
├── eval-manifold.json        # Manifold evaluations
├── eval-plane.json           # Plane project evaluations
└── eval-saleor.json          # Saleor e-commerce evaluations

Usage

Running Evaluations

Basic Example

typescript

import { runBuffBench } from './run-buffbench'

await runBuffBench({
  evalDataPath: 'eval-codebuff.json',
  agents: ['base2', 'base2-fast'],
  taskConcurrency: 3,
})

Command Line

bash

# Run evaluations
bun run evals/buffbench/main.ts

# Run single evaluation
bun run evals/buffbench/main-single-eval.ts

# Run nightly evaluation with email reporting
bun run evals/buffbench/main-nightly.ts

Running in Freestyle VMs (recommended)

Pass --freestyle to any main-*.ts entry point (or sandbox: 'freestyle' to runBuffBench) and every agent run gets its own Freestyle VM, so a laptop that sleeps delays the run without tainting it:

bash

cd evals
export NEXT_PUBLIC_CB_ENVIRONMENT=prod NEXT_PUBLIC_CODEBUFF_APP_URL=https://www.codebuff.com
PORT=4242 bun --env-file=../.env.local run buffbench/main-fast-harness.ts 5 all --freestyle

The VM receives this checkout's bench code (uncommitted edits included, never the eval-*.json files), runs the clone, agent and final checks as a local run does (freestyle-task.ts), and powers off; this machine polls and judges (freestyle-sandbox.ts). It needs FREESTYLE_API_KEY and a backend the VM can reach (not localhost). Running VMs = taskConcurrency × agents, which must fit the Freestyle plan. A run that ends without a result is an error carrying the VM's log.

Creating New Evaluations

1. Generate from Specific Commits

bash

bun run evals/buffbench/gen-evals.ts \
  https://github.com/user/repo \
  abc123 \
  def456 \
  ghi789

2. Generate from Repository (End-to-End)

bash

bun run evals/buffbench/gen-repo-eval.ts \
  https://github.com/user/repo

This will:

Clone the repository
Select high-quality commits using AI
Generate evaluation tasks for each commit
Save the complete evaluation file

Evaluation Data Format

Evaluation File Structure

typescript

interface EvalDataV2 {
  repoUrl: string              // Source repository URL
  testRepoName?: string        // Optional repo name override
  generationDate: string       // ISO timestamp of creation
  initCommand?: string         // Optional setup command
  binInstalls?: BinInstall[]   // Binaries to install
  env?: Record<string, string> // Environment variables
  finalCheckCommands?: string[] // Validation commands
  evalCommits: EvalCommitV2[]  // List of evaluation tasks
}

interface EvalCommitV2 {
  id: string                   // Unique task identifier
  sha: string                  // Target commit SHA
  parentSha: string            // Parent commit SHA
  spec: string                 // Technical specification
  prompt: string               // Natural language prompt
  supplementalFiles: string[]  // Context files
  fileDiffs: FileDiff[]        // Ground truth changes
}

interface FileDiff {
  path: string                 // File path
  status: 'modified' | 'added' | 'deleted' | 'renamed'
  oldPath?: string             // For renamed files
  diff: string                 // Unified diff
}

Results Format

typescript

interface AgentEvalResults {
  agentId: string
  runs: EvalRun[]
  averageScore: number               // Overall score across valid runs
  averageScoreExcludingFailures: number // Score excluding failures (≤1.0)
  averageCost: number
  averageDuration: number
}

interface EvalRun {
  commitSha: string
  prompt: string
  diff: string                       // Agent's changes
  judging: JudgingResult
  cost: number
  durationMs: number
  error?: string
  finalCheckOutputs?: FinalCheckOutput[]
}

interface JudgingResult {
  analysis: string                   // Detailed analysis
  strengths: string[]
  weaknesses: string[]
  completionScore: number            // 0-10
  codeQualityScore: number           // 0-10
  overallScore: number               // 0-10
}

Advanced Features

Binary Installation

Install required binaries in isolated environments:

json

{
  "binInstalls": [
    {
      "name": "ruff",
      "installScript": "curl -LsSf https://astral.sh/ruff/install.sh | sh -s -- --to $INSTALL_DIR",
      "binPath": "ruff"
    }
  ]
}

Final Check Commands

Run validation commands after agent execution:

json

{
  "finalCheckCommands": [
    "npm run typecheck",
    "npm test",
    "npm run lint"
  ]
}

Nightly Evaluations

Run automated nightly evaluations with email reporting:

The nightly evaluation runner automatically sends email reports with results.

bash

bun run evals/buffbench/main-nightly.ts

Output Files

BuffBench creates a timestamped logs directory for each run:

text

logs/YYYY-MM-DDTHH-MM_agent1_vs_agent2/
├── 1-task-id-agent1-abc123.json      # Agent 1 trace for task 1
├── 1-task-id-agent2-abc123.json      # Agent 2 trace for task 1
├── 1-task-id-ANALYSIS-abc123.json    # Trace analysis for task 1
├── 2-task-id-agent1-def456.json      # Agent 1 trace for task 2
├── 2-task-id-agent2-def456.json      # Agent 2 trace for task 2
├── 2-task-id-ANALYSIS-def456.json    # Trace analysis for task 2
└── FINAL_RESULTS.json                # Complete results + meta-analysis

FINAL_RESULTS.json Structure

json

{
  "metadata": {
    "timestamp": "2024-01-15T10:30:00.000Z",
    "evalDataPath": "eval-codebuff.json",
    "agentsTested": ["base2", "base2-fast"],
    "commitsEvaluated": 10,
    "logsDirectory": "logs/..."
  },
  "metaAnalysis": {
    "overallComparison": "...",
    "agentInsights": [...],
    "keyFindings": [...]
  },
  "base2": {
    "agentId": "base2",
    "runs": [...],
    "averageScore": 7.5,
    "averageCost": 0.0234,
    "averageDuration": 45000
  },
  "base2-fast": {...}
}

Configuration

Environment Variables

bash

# Required
CODEBUFF_API_KEY=your-api-key

# Optional
[email protected]  # For nightly email reports

Task Concurrency

Control parallel execution:

typescript

taskConcurrency: 1   // Sequential (safest, slowest)
taskConcurrency: 3   // Moderate parallelism
taskConcurrency: 10  // High parallelism (faster, more resources)

Examples

The evals/buffbench/ directory contains several example evaluation files:

eval-codebuff.json - Codebuff project evaluations
eval-manifold.json - Manifold prediction market evaluations
eval-plane.json - Plane project management evaluations
eval-saleor.json - Saleor e-commerce platform evaluations

These demonstrate the evaluation format and provide ready-to-use test cases.

Overview (`freebuff/e2e/README.md`)

Freebuff E2E Tests

End-to-end tests for the Freebuff CLI binary. Tests verify that the compiled binary works correctly by interacting with it via tmux.

Architecture

Two testing approaches are supported:

1. Direct tmux tests (fast, deterministic)

Use the FreebuffSession class to start the binary in tmux, send commands, capture output, and assert directly.

typescript

import { describe, test, expect, afterEach } from 'bun:test'
import { FreebuffSession, requireFreebuffBinary } from '../utils'

describe('My Feature', () => {
  let session: FreebuffSession | null = null

  afterEach(async () => {
    if (session) await session.stop()
    session = null
  })

  test('works correctly', async () => {
    const binary = requireFreebuffBinary()
    session = await FreebuffSession.start(binary)

    await session.send('/help')
    const output = await session.capture(2)

    expect(output).toContain('Shortcuts')
  }, 60_000)
})

2. SDK agent-driven tests (AI-powered verification)

Use the Codebuff SDK to run a testing agent that interacts with Freebuff via custom tmux tools. The agent reasons about the CLI output and verifies complex behaviors.

typescript

import { describe, test, expect, afterEach } from 'bun:test'
import { CodebuffClient } from '@codebuff/sdk'
import { freebuffTesterAgent } from '../agent/freebuff-tester'
import { createFreebuffTmuxTools, requireFreebuffBinary } from '../utils'

describe('Agent Test', () => {
  let cleanup: (() => Promise<void>) | null = null

  afterEach(async () => {
    if (cleanup) await cleanup()
    cleanup = null
  })

  test('verifies startup', async () => {
    const apiKey = process.env.CODEBUFF_API_KEY
    if (!apiKey) return // Skip if no API key

    const binary = requireFreebuffBinary()
    const tmuxTools = createFreebuffTmuxTools(binary)
    cleanup = tmuxTools.cleanup

    const client = new CodebuffClient({ apiKey })
    const result = await client.run({
      agent: freebuffTesterAgent.id,
      prompt: 'Start Freebuff and verify the branding is correct.',
      agentDefinitions: [freebuffTesterAgent],
      customToolDefinitions: tmuxTools.tools,
      handleEvent: () => {},
    })

    expect(result.output.type).not.toBe('error')
  }, 180_000)
})

3. Live smoke against production

tests/live-turn.e2e.test.ts is the one file here that talks to the real backend:
it navigates the landing picker to DeepSeek V4.1 Flash, starts a session, sends a
prompt, waits for the answer, then runs /end-session and checks the backend shows
no open session. It skips unless FREEBUFF_SMOKE_API_KEY (or
CODEBUFF_API_KEY) is set, and needs a binary built with the production public
env. It is not in freebuff-e2e.yml — .github/workflows/prod-smoke.yml
runs it on a schedule and before each release.

bash

NEXT_PUBLIC_CB_ENVIRONMENT=prod NEXT_PUBLIC_CODEBUFF_APP_URL=https://www.codebuff.com \
NEXT_PUBLIC_FREEBUFF_APP_URL=https://freebuff.com [email protected] \
NEXT_PUBLIC_POSTHOG_API_KEY=test NEXT_PUBLIC_POSTHOG_HOST_URL=http://127.0.0.1:9 \
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=test NEXT_PUBLIC_STRIPE_CUSTOMER_PORTAL=http://127.0.0.1:9 \
NEXT_PUBLIC_WEB_PORT=3000 bun freebuff/cli/build.ts 0.0.0-smoke
CODEBUFF_API_KEY=<your token> bun test freebuff/e2e/tests/live-turn.e2e.test.ts --timeout=300000

The CLI runs with its own FREEBUFF_CONFIG_DIR, so your real profile is untouched.

Prerequisites

tmux must be installed: brew install tmux (macOS) or sudo apt-get install tmux (Ubuntu)
Freebuff binary must be built: bun freebuff/cli/build.ts 0.0.0-dev
SDK built (for agent tests): cd sdk && bun run build
CODEBUFF_API_KEY (for agent tests only): Set this environment variable

Running Tests

Build the binary first

bash

bun freebuff/cli/build.ts 0.0.0-dev

Run all tests

bash

bun test freebuff/e2e/tests/

Run a specific test

bash

bun test freebuff/e2e/tests/version.e2e.test.ts
bun test freebuff/e2e/tests/startup.e2e.test.ts
bun test freebuff/e2e/tests/help-command.e2e.test.ts
bun test freebuff/e2e/tests/agent-startup.e2e.test.ts

Use a custom binary path

bash

FREEBUFF_BINARY=/path/to/freebuff bun test freebuff/e2e/tests/

Adding New Tests

Create a new file in freebuff/e2e/tests/ with the naming convention <feature>.e2e.test.ts
Add an e2e - <name> step to the build-and-e2e-freebuff job in
.github/workflows/freebuff-e2e.yml, with the same if: as its siblings so it
still runs when another suite fails. Keep it off the network: that job has no
credentials, so anything that needs a real backend belongs in prod-smoke.yml.

CI Workflow

The .github/workflows/freebuff-e2e.yml workflow:

Builds the Freebuff binary once (linux-x64) and smoke-boots it
Runs each suite as its own step in that same job, against that binary
Uploads tmux session logs on failure for debugging
Builds and smoke-boots on Windows in a separate job (no tmux there)

Triggers: pull requests to main or great-db-merge-v2, pushes to main, manual
workflow_dispatch, and freebuff-release.yml before each release.

Utilities Reference

`FreebuffSession`

Method	Description
`FreebuffSession.start(binaryPath)`	Start binary in tmux, returns session
`session.send(text)`	Send text input (presses Enter)
`session.sendKey(key)`	Send special key (e.g. `'C-c'`, `'Escape'`)
`session.capture(waitSec?)`	Capture terminal output
`session.captureLabeled(label, waitSec?)`	Capture and save to session logs
`session.waitForText(pattern, timeoutMs?)`	Poll until text appears
`session.stop()`	Stop session and clean up

`createFreebuffTmuxTools(binaryPath)`

Creates SDK custom tools for agent-driven testing:

start_freebuff - Launch the CLI
send_to_freebuff - Send text input
capture_freebuff_output - Capture terminal output
stop_freebuff - Stop and clean up

Helper functions

Function	Description
`requireFreebuffBinary()`	Get binary path, throws if not found
`getFreebuffBinaryPath()`	Get binary path (may not exist)

Overview (`scripts/tmux/README.md`)

tmux TUI Testing Scripts

Helper scripts for testing TUI applications (Terminal User Interface apps) using tmux. These scripts abstract away the complexity of tmux communication, particularly the bracketed paste mode requirement.

Features

Automatic bracketed paste mode - Input is wrapped in escape sequences so characters don't get dropped
Automatic session logs - Every capture is saved to debug/tmux-sessions/{session}/ for debugging
Works with any TUI app - Codebuff, Claude Code, Codex, or any custom TUI application
Paper trail - Both the subagent and parent agent can review what the app displayed

Why These Scripts?

Many TUI apps (using frameworks like OpenTUI, Ink, Textual, etc.) process keyboard input character-by-character. When tmux sends characters rapidly via send-keys, they get dropped or garbled. These scripts automatically wrap input in bracketed paste escape sequences (\e[200~...\e[201~), which tells the terminal to process the input atomically.

Without these scripts:

bash

# ❌ Characters get dropped!
tmux send-keys -t session "hello world"
# Result: Only "d" appears in the input!

With these scripts:

bash

# ✅ Works correctly
./scripts/tmux/tmux-send.sh session "hello world"
# Result: "hello world" appears correctly

Quick Start

Testing Any TUI App

bash

# Start a test session with any command
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "claude")
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "codex chat")
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "python my_tui.py")
echo "Started session: $SESSION"

# Send a command
./scripts/tmux/tmux-cli.sh send "$SESSION" "/help"

# Wait and capture output
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 2

# Clean up
./scripts/tmux/tmux-cli.sh stop "$SESSION"

Testing Codebuff (Default)

bash

# Start a test session (defaults to Codebuff dev server)
SESSION=$(./scripts/tmux/tmux-cli.sh start)
echo "Started session: $SESSION"

# Send a command
./scripts/tmux/tmux-cli.sh send "$SESSION" "/help"

# Wait and capture output
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 2

# Clean up
./scripts/tmux/tmux-cli.sh stop "$SESSION"

Testing Compiled Binaries

bash

# Build the binary first (Codebuff example)
cd cli && bun run build:binary

# Test with binary at default location (./cli/bin/codebuff)
SESSION=$(./scripts/tmux/tmux-cli.sh start --binary)

# Or specify a custom binary path
SESSION=$(./scripts/tmux/tmux-cli.sh start --binary /path/to/binary)

# Or use environment variable
CODEBUFF_BINARY=./cli/bin/codebuff ./scripts/tmux/tmux-cli.sh start

The session-info.yaml will record which mode was used (cli_mode: custom, cli_mode: binary, or cli_mode: dynamic).

Scripts

`tmux-cli.sh` (Unified Interface)

The main entry point with subcommands:

bash

./scripts/tmux/tmux-cli.sh start      # Start a session
./scripts/tmux/tmux-cli.sh send       # Send input
./scripts/tmux/tmux-cli.sh capture    # Capture output
./scripts/tmux/tmux-cli.sh stop       # Stop a session
./scripts/tmux/tmux-cli.sh list       # List sessions
./scripts/tmux/tmux-cli.sh help       # Show help

`tmux-start.sh`

Start a new tmux session with a TUI app.

bash

# Start with a custom command (any TUI app)
./scripts/tmux/tmux-start.sh --command "claude"
./scripts/tmux/tmux-start.sh --command "codex chat"
./scripts/tmux/tmux-start.sh --command "python my_app.py"

# Default settings (Codebuff dev server)
./scripts/tmux/tmux-start.sh
# Output: tui-test-1234567890

# Custom session name
./scripts/tmux/tmux-start.sh --name my-test

# Custom dimensions
./scripts/tmux/tmux-start.sh -w 160 -h 40

# Custom wait time for app initialization
./scripts/tmux/tmux-start.sh --wait 6

# Test a compiled binary (default location: ./cli/bin/codebuff)
./scripts/tmux/tmux-start.sh --binary

# Test a compiled binary at custom path
./scripts/tmux/tmux-start.sh --binary /path/to/binary

`tmux-send.sh`

Send input to a running session.

bash

# Send text (auto-presses Enter)
./scripts/tmux/tmux-send.sh SESSION "your prompt"

# Send without pressing Enter
./scripts/tmux/tmux-send.sh SESSION "partial" --no-enter

# Send special keys
./scripts/tmux/tmux-send.sh SESSION --key Escape
./scripts/tmux/tmux-send.sh SESSION --key C-c
./scripts/tmux/tmux-send.sh SESSION --key Enter

# Send and wait for CLI to finish streaming (for rapid message tests)
./scripts/tmux/tmux-send.sh SESSION "hello" --wait-idle 2

# Paste clipboard content and submit immediately
./scripts/tmux/tmux-send.sh SESSION --paste

# Paste clipboard but don't submit (useful for testing attachment UI)
# This lets you capture a screenshot of the attachment card before sending
./scripts/tmux/tmux-send.sh SESSION --paste --no-enter

`tmux-capture.sh`

Capture output from a session. Automatically saves captures to debug/tmux-sessions/{session}/.

bash

# Basic capture (auto-saves to session logs)
./scripts/tmux/tmux-capture.sh SESSION

# Capture with a descriptive label (recommended)
./scripts/tmux/tmux-capture.sh SESSION --label "after-help-command"
# Saves to: debug/tmux-sessions/SESSION/capture-{sequence}-after-help-command.txt

# Wait before capturing
./scripts/tmux/tmux-capture.sh SESSION --wait 3

# Capture without auto-saving to session logs
./scripts/tmux/tmux-capture.sh SESSION --no-save

# Preserve colors
./scripts/tmux/tmux-capture.sh SESSION --colors

# Save to specific file (disables auto-save)
./scripts/tmux/tmux-capture.sh SESSION -o output.txt

`tmux-stop.sh`

Stop/kill sessions.

bash

# Stop specific session
./scripts/tmux/tmux-stop.sh SESSION

# Stop all test sessions (tui-test-* and cli-test-*)
./scripts/tmux/tmux-stop.sh --all

# List sessions first, then stop
./scripts/tmux/tmux-stop.sh --list SESSION

Example: Full Test Workflow

bash

#!/bin/bash

# Start session with a TUI app
SESSION=$(./scripts/tmux/tmux-cli.sh start --command "my-tui-app" --name my-test)
echo "Testing in: $SESSION"

# Verify app started (capture auto-saved with label)
./scripts/tmux/tmux-cli.sh capture "$SESSION" --label "initial-state"

# Send /status command
./scripts/tmux/tmux-cli.sh send "$SESSION" "/status"
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 2 --label "after-status"

# Test a prompt
./scripts/tmux/tmux-cli.sh send "$SESSION" "hello world"
./scripts/tmux/tmux-cli.sh capture "$SESSION" --wait 5 --label "after-prompt"

# Clean up
./scripts/tmux/tmux-cli.sh stop "$SESSION"
echo "Test complete!"

# View all session logs
ls -la debug/tmux-sessions/$SESSION/

Session Logs Directory Structure

text

debug/tmux-sessions/
└── tui-test-1234567890/
    ├── session-info.yaml             # Session metadata (YAML format)
    ├── commands.yaml                 # Log of all commands sent (YAML array)
    ├── capture-001-initial-state.txt # Captures with YAML front-matter
    ├── capture-002-after-status.txt
    └── capture-003-after-prompt.txt

Session Info (session-info.yaml)

yaml

session: tui-test-1234567890
started: 2025-01-01T12:00:00Z
started_local: Wed Jan  1 12:00:00 PST 2025
dimensions:
  width: 120
  height: 30
cli_mode: custom  # or "binary" or "dynamic"
cli_command: claude  # or path to binary, or "bun --cwd=cli run dev"
status: active

Commands Log (commands.yaml)

The commands.yaml file records every input sent to the app as a YAML array:

yaml

- timestamp: 2025-01-01T12:00:05Z
  type: text
  input: "/help"
  auto_enter: true

- timestamp: 2025-01-01T12:00:10Z
  type: text
  input: "hello world"
  auto_enter: true

- timestamp: 2025-01-01T12:00:15Z
  type: key
  input: "Escape"

- timestamp: 2025-01-01T12:00:20Z
  type: text
  input: "partial input"
  auto_enter: false

Capture Files with YAML Front-Matter

Each capture file includes YAML front-matter with metadata:

yaml

---
sequence: 1
label: initial-state
timestamp: 2025-01-01T12:00:30Z
after_command: null
dimensions:
  width: 120
  height: 30
---
[actual terminal output below]

The front-matter provides:

sequence: Order of captures (1, 2, 3, ...)
label: Descriptive label if provided via --label
timestamp: ISO 8601 timestamp
after_command: The last command sent before this capture (or null)
dimensions: Terminal dimensions at capture time

This structured format enables both human reading and programmatic parsing (AI agents, viewers, etc.).

Debugging

Attach to a Session Interactively

bash

tmux attach -t SESSION_NAME
# Detach with Ctrl+B, D

List All Sessions

bash

./scripts/tmux/tmux-cli.sh list
# or
tmux list-sessions

Check If Session Exists

bash

tmux has-session -t SESSION_NAME && echo "exists" || echo "not found"

Prerequisites

tmux must be installed:
- macOS: brew install tmux
- Ubuntu: sudo apt-get install tmux
- Arch: sudo pacman -S tmux

Used By

These scripts are used by TUI testing agents:

@codebuff-tester - Tests the Codebuff CLI
@claude-code-tester - Tests Claude Code CLI
@codex-tester - Tests OpenAI Codex CLI
@gemini-tester - Tests Google Gemini CLI
Custom agents for other TUI apps can easily be created following the same pattern

Overview (`sdk/e2e/README.md`)

SDK Tests & Examples

This folder contains end-to-end tests, integration tests, unit tests, and runnable examples for the Codebuff SDK.

Directory Structure

text

sdk/e2e/
├── streaming/          # E2E tests for streaming behavior
├── workflows/          # E2E tests for multi-step workflows  
├── custom-agents/      # E2E tests for custom agent & tool integration
├── features/           # E2E tests for SDK feature coverage
├── integration/        # Integration tests (SDK + API mechanics)
├── examples/           # Runnable example scripts (not tests)
└── utils/              # Shared utilities
    ├── __tests__/      # Unit tests for utilities
    ├── event-collector.ts
    ├── get-api-key.ts
    └── test-fixtures.ts

Test Categories

E2E Tests (`test:e2e`)

Full end-to-end tests that exercise complete user workflows with real API calls.

bash

bun run test:e2e

Category	Description
`streaming/`	Subagent streaming, concurrent streams
`workflows/`	Multi-turn conversations, error recovery
`custom-agents/`	Custom agents with custom tools
`features/`	projectFiles, knowledgeFiles, maxAgentSteps

Integration Tests (`test:integration`)

Tests that verify SDK + API integration mechanics.

bash

bun run test:integration

Test	Description
`event-types`	Validates all PrintModeEvent types are emitted
`event-ordering`	Validates event sequence (start → content → finish)
`stream-chunks`	Tests handleStreamChunk callback
`connection-check`	Tests checkConnection() method

Unit Tests (`test:unit:e2e`)

Pure unit tests with no external dependencies.

bash

bun run test:unit:e2e

Test	Description
`event-collector.test.ts`	Tests EventCollector utility class

Examples

Runnable scripts demonstrating SDK usage patterns. Not tests - just examples!

bash

# Run an example
bun run sdk/e2e/examples/code-reviewer.example.ts

Example	Description
`code-reviewer.example.ts`	AI code review
`code-explainer.example.ts`	Explain code in plain English
`commit-message-generator.example.ts`	Generate commit messages from diffs
`sdk-lint.example.ts`	AI-powered linter
`sdk-refactor.example.ts`	Code refactoring
`sdk-test-gen.example.ts`	Unit test generation

Running All Tests

bash

# All E2E tests
bun run test:e2e

# All integration tests
bun run test:integration

# All unit tests
bun run test:unit:e2e

# Everything
bun run test:e2e && bun run test:integration && bun run test:unit:e2e

Prerequisites

API Key: Set CODEBUFF_API_KEY for E2E and integration tests
Opt-in: Set RUN_CODEBUFF_E2E=true for local live API runs (CI runs automatically)
Tests skip gracefully if API key is not set

Writing Tests

E2E Test Pattern

typescript

import { describe, test, expect, beforeAll } from 'bun:test'
import { CodebuffClient } from '../../src/client'
import { EventCollector, getApiKey, skipIfNoApiKey, isAuthError, DEFAULT_AGENT, DEFAULT_TIMEOUT } from '../utils'

describe('E2E: My Test', () => {
  let client: CodebuffClient

  beforeAll(() => {
    if (skipIfNoApiKey()) return
    client = new CodebuffClient({ apiKey: getApiKey() })
  })

  test('does something', async () => {
    if (skipIfNoApiKey()) return
    const collector = new EventCollector()
    
    const result = await client.run({
      agent: DEFAULT_AGENT,
      prompt: 'Test prompt',
      handleEvent: collector.handleEvent,
    })

    if (isAuthError(result.output)) return
    
    expect(result.output.type).not.toBe('error')
  }, DEFAULT_TIMEOUT)
})

Unit Test Pattern

typescript

import { describe, test, expect, beforeEach } from 'bun:test'
import { EventCollector } from '../event-collector'

describe('Unit: EventCollector', () => {
  let collector: EventCollector

  beforeEach(() => {
    collector = new EventCollector()
  })

  test('collects events', () => {
    collector.handleEvent({ type: 'start', messageHistoryLength: 0 })
    expect(collector.events).toHaveLength(1)
  })
})

Overview (`cli/src/tests/README.md`)

CLI Testing

Comprehensive testing suite for the Codebuff CLI using tmux for interactive terminal emulation.

Test Naming Convention

IMPORTANT: Follow these patterns for automatic tmux detection:

Unit tests: *.test.ts (e.g., cli-args.test.ts)
E2E tests: e2e-*.test.ts (e.g., e2e-cli.test.ts)
Integration tests: integration-*.test.ts (e.g., integration-tmux.test.ts)

Files matching *integration*.test.ts or *e2e*.test.ts trigger automatic tmux availability checking in .bin/bun.

Quick Start

bash

cd cli
bun test

Prerequisites

For Integration Tests

Install tmux for interactive CLI testing:

bash

# macOS
brew install tmux

# Ubuntu/Debian
sudo apt-get install tmux

# Windows (via WSL)
wsl --install
sudo apt-get install tmux

For E2E Tests

Build the SDK first:

bash

cd sdk
bun run build
cd ../cli

Running Tests

All Tests

bash

bun test

Specific Test Suites

bash

# Unit tests
bun test cli-args.test.ts

# E2E tests (requires SDK)
bun test e2e-cli.test.ts

# Integration tests (requires tmux)
bun test integration-tmux.test.ts

Manual tmux POC

bash

bun run test:tmux-poc

Automatic tmux Detection

The .bin/bun wrapper automatically checks for tmux when running integration/E2E tests:

Detects test files matching *integration*.test.ts or *e2e*.test.ts
Checks if tmux is installed
Shows installation instructions if missing
Skips tests gracefully if tmux unavailable

Benefits:

✅ Project-wide (works in any package)
✅ No hardcoded paths
✅ Clear test categorization
✅ Automatic dependency validation

Test Structure

Unit Tests

Test individual functions in isolation:

typescript

import { describe, test, expect } from 'bun:test'

describe('CLI Arguments', () => {
  test('parses --agent flag', () => {
    // Test implementation
  })
})

Integration Tests (tmux)

Test interactive CLI with full terminal emulation:

typescript

import { describe, test, expect } from 'bun:test'
import { isTmuxAvailable } from './test-utils'

const tmuxAvailable = isTmuxAvailable()

describe.skipIf(!tmuxAvailable)('CLI Integration Tests', () => {
  test('handles user input', async () => {
    // Create tmux session
    // Send commands
    // Verify output
  })
})

E2E Tests

Test complete CLI workflows:

typescript

import { describe, test, expect } from 'bun:test'
import { isSDKBuilt } from './test-utils'

const sdkBuilt = isSDKBuilt()

describe.skipIf(!sdkBuilt)('CLI E2E Tests', () => {
  test('runs --help command', async () => {
    // Test CLI behavior
  })
})

Test Utilities

Shared utilities in test-utils.ts:

typescript

import { isTmuxAvailable, isSDKBuilt, sleep } from './test-utils'

// Check for tmux
if (isTmuxAvailable()) {
  // Run tmux tests
}

// Check for SDK
if (isSDKBuilt()) {
  // Run E2E tests
}

// Async delay
await sleep(1000)

tmux Testing

See ../../tmux.knowledge.md for comprehensive tmux documentation, including:

Why standard send-keys doesn't work (must use bracketed paste mode)
Helper functions for Bash and TypeScript
Complete example scripts
Debugging and troubleshooting tips

Quick reference:

typescript

// ❌ Broken:
await tmux(['send-keys', '-t', session, 'hello'])

// ✅ Works:
await tmux(['send-keys', '-t', session, '-l', '\x1b[200~hello\x1b[201~'])

Debugging Tests

View Test Output

bash

# Verbose test output
bun test --verbose

# Watch mode
bun test --watch

Contributing

When adding new tests:

Follow naming convention (*integration*.test.ts or *e2e*.test.ts)
Use test-utils.ts for shared functionality
Add graceful skipping for missing dependencies
Clean up resources (tmux sessions, temp files)
Document test purpose clearly in test descriptions

Troubleshooting

tmux Not Found

text

⚠️  tmux not found but required for integration/E2E tests

Solution: Install tmux (see Prerequisites above)

SDK Not Built

text

✓ Build SDK for E2E tests: cd sdk && bun run build [skip]

Solution: Build the SDK first (see Prerequisites above)

Tests Hanging

Check tmux session isn't waiting for input
Ensure proper cleanup in finally blocks
Use timeouts for tmux operations

Session Already Exists

Use unique session names (e.g., timestamp suffix)
Clean up sessions in beforeEach/afterEach

Performance

Unit tests: ~100ms total
Integration tests: ~2-5s per test (tmux overhead)
E2E tests: ~3-10s per test (full CLI startup)

CI/CD

For CI environments:

yaml

# Install tmux in CI
- name: Install tmux
  run: |
    sudo apt-get update
    sudo apt-get install -y tmux

# Run tests
- name: Run tests
  run: bun test

Overview (`freebuff/cli/release/README.md`)

Freebuff

The free coding agent. No subscription. No configuration. Start in seconds.

An AI coding agent that runs in your terminal — describe what you want, and Freebuff edits your code.

Install

bash

npm install freebuff

Usage

bash

cd ~/my-project
freebuff

Older Intel Macs (without AVX2)

The npm launcher automatically switches to the darwin-x64-baseline build if
the standard Intel binary exits with an illegal-instruction error (SIGILL).
To select that build before the first launch:

bash

FREEBUFF_BINARY_TARGET=darwin-x64-baseline freebuff

For manual installation, download freebuff-darwin-x64-baseline.tar.gz from the
Freebuff CLI releases
and keep the extracted freebuff binary and tree-sitter.wasm together.

Why Freebuff?

Choice of leading models — The regular CLI picker includes MiMo v2.6 Flash Fast, GLM 5.3 Flash, DeepSeek, DeepSeek V4.1 Flash Fast, GPT-6 Luna, MiMo 2.6 Pro, Solar Mini 4, Solar Pro 4, Space Bunny Alpha, Gemini 3.8 Flash, Muse Spark 1.3, Gemini 4 Argon, and GPT-6.1 Sol, with MiMo v2.6 Flash Fast selected by default.

Fast — 5–10× speed up. 3–5× tokens per second compared to Claude, plus context gathering in seconds.

Loaded — Built-in web research, browser use, and more.

FAQ

How can it be free? Freebuff is supported by text ads.

What models do you use? In full mode, the regular picker includes MiMo v2.6 Flash Fast, GLM 5.3 Flash, DeepSeek V4.1 Flash, DeepSeek V4.1 Flash Fast, GPT-6 Luna, MiMo 2.6 Pro, Solar Mini 4, Solar Pro 4, Space Bunny Alpha, Gemini 3.8 Flash, Muse Spark 1.3, Gemini 4 Argon, and GPT-6.1 Sol. Space Bunny Alpha is a beta stealth model from an anonymous provider that retains prompts. Gemini 3.8 Flash and Muse Spark 1.3 require a paid plan: without one they are listed but locked, and selecting one opens the plans page. Gemini 4 Argon and GPT-6.1 Sol are free in the US and need a paid plan elsewhere, and every account, paid or not, gets one session a day on each; Gemini 4 Argon's price is a temporary promotion. GPT-6 Luna and MiMo 2.6 Pro are open to every full-access account. DeepSeek V4.1 Flash Fast is fast mode: the same model on DeepSeek's own API, splitting work across parallel subagents. Muse Spark 1.3 has a 1M-token context, and when it is busy or unavailable it answers on DeepSeek V4.1 Flash instead of leaving you waiting. MiMo v2.6 Flash Fast is the CLI default, and GLM 5.3 Flash is the cheapest model to run. Limited mode uses MiMo v2.6 Flash Fast, GLM 5.3 Flash, DeepSeek V4.1 Flash, Solar Mini 4, Solar Pro 4, and Space Bunny Alpha; the models a paid plan unlocks there are listed too, locked until you have one. Every model is paid for in hours, one multiplier per one-hour session, shown in the model picker: an hour of MiMo v2.6 Flash Slow uses one hour of balance, a 3x model uses three. Models may still serve from a quantized (Q8_0) build.

Which countries is Freebuff available in? All countries. Freebuff runs in "full" mode in the US, Canada, UK, EU, and other select countries, and in "limited" mode everywhere else (or while using a VPN). See freebuff.com for the full list.

What is limited mode? Limited mode lets you use Freebuff outside the full-access countries, or while using a VPN. It includes GLM 5.3 Flash, DeepSeek V4.1 Flash, MiMo v2.6 Flash Fast, Solar Mini 4, Solar Pro 4, and Space Bunny Alpha. The free limited-access allowance is 5 hours a day, or 4 on a VPN or proxy, and each model's multiplier per one-hour session is shown in the picker. Today's allowance is spent first, whichever model you pick, including GLM 5.3 Flash; Hours you earn on the Earn page go into your wallet and are used once today's allowance runs out.

See the Privacy Policy for complete details.

tmux-viewer

Interactive TUI for viewing tmux session logs. Designed to work for both humans and AIs.

Usage

bash

# Interactive TUI (for humans)
bun scripts/tmux/tmux-viewer/index.tsx <session-name>

# Start in replay mode (auto-plays through captures like a video)
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --replay

# JSON output (for AIs)
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --json

# Export as animated GIF
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --export-gif output.gif

# Export with custom frame delay (default: 1500ms)
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --export-gif output.gif --frame-delay 2000

# Export with custom font size (default: 14px)
bun scripts/tmux/tmux-viewer/index.tsx <session-name> --export-gif output.gif --font-size 16

# List available sessions
bun scripts/tmux/tmux-viewer/index.tsx --list

# View most recent session (if no session specified)
bun scripts/tmux/tmux-viewer/index.tsx

Or using the npm script:

bash

cd scripts/tmux && bun run view-session <session-name>

Layout

The TUI uses a vertical layout designed for clarity:

text

┌─────────────────────────────────────────────────────────────────┐
│ Session: my-session  120x30           5 cmds  10 captures       │  ← Header
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│                    ┌──────────────────┐                         │
│                    │ [terminal output │                         │  ← Capture
│                    │  centered in     │                         │     View
│                    │  muted border]   │                         │
│                    └──────────────────┘                         │
│                                                                 │
├─ ⏸ Paused ──────────────────────────────────────────────────────┤
│  ┌─○ [1] 12:00:00─┐ ┌─▶ [2] 12:00:05─┐ ┌─○ [3] 12:00:10─┐       │  ← Timeline
│  │ initial-state  │ │ after-command  │ │ final-state    │       │     Cards
│  │ $ codebuff...  │ │ $ /help        │ │ $ /quit        │       │
│  └────────────────┘ └────────────────┘ └────────────────┘       │
├─────────────────────────────────────────────────────────────────┤
│ ▶ 2/10 @1.5s   space: play/pause  +/-: speed  ←→: navigate      │  ← Footer
└─────────────────────────────────────────────────────────────────┘

Header: Session name, dimensions, command/capture counts
Capture View: Terminal output centered with a muted border showing exact capture dimensions
Timeline: Horizontal card-style navigation at the bottom, selected card stays centered
Footer: Playback status, position, speed, and keyboard shortcuts

Features

For Humans (Interactive TUI)

Capture view: Terminal output centered with visible boundary
Timeline panel: Card-style navigation at the bottom with label and triggering command
Auto-centering: Selected timeline card stays centered in view
Metadata display: Session info, dimensions, command count
Replay mode: Auto-play through captures like a video player
Keyboard shortcuts:
- ← / → or h / l: Navigate between captures
- Space: Play/pause replay
- + / -: Adjust playback speed (faster/slower)
- r: Restart from beginning
- q or Ctrl+C: Quit
- Use the --json flag on the CLI entrypoint for JSON output

Replay Mode

Replay mode auto-advances through captures chronologically, like a video player:

bash

# Start replay immediately
bun scripts/tmux/tmux-viewer/index.tsx my-session --replay

# Or press Space in the TUI to start/stop replay

Playback controls:

Space - Toggle play/pause
+ or = - Speed up (shorter interval between captures)
- or _ - Slow down (longer interval between captures)
r - Restart from the first capture
← / → - Navigate captures (automatically pauses replay)

Available speeds: 0.5s, 1.0s, 1.5s (default), 2.0s, 3.0s, 5.0s per capture

The timeline panel title shows ▶ Playing or ⏸ Paused, and the footer shows current position (e.g., 2/10), playback speed (e.g., @1.5s), and controls.

For AIs (JSON Output)

Use the --json flag to get structured output:

json

{
  "session": {
    "session": "cli-test-1234567890",
    "started": "2025-01-01T12:00:00Z",
    "dimensions": { "width": 120, "height": 30 },
    "status": "active"
  },
  "commands": [
    { "timestamp": "...", "type": "text", "input": "/help", "auto_enter": true }
  ],
  "captures": [
    {
      "sequence": 1,
      "label": "initial-state",
      "timestamp": "...",
      "after_command": null,
      "dimensions": { "width": 120, "height": 30 },
      "path": "debug/tmux-sessions/.../capture-001-initial-state.txt",
      "content": "[terminal output]"
    }
  ],
  "timeline": [
    { "timestamp": "...", "type": "command", "data": {...} },
    { "timestamp": "...", "type": "capture", "data": {...} }
  ]
}

Data Format

The viewer reads YAML-formatted session data from debug/tmux-sessions/{session}/:

session-info.yaml - Session metadata
commands.yaml - Array of commands sent
capture-*.txt - Capture files with YAML front-matter

Session Info (session-info.yaml)

yaml

session: cli-test-1234567890
started: 2025-01-01T12:00:00Z
started_local: Wed Jan 1 12:00:00 PST 2025
dimensions:
  width: 120
  height: 30
status: active

Commands (commands.yaml)

yaml

- timestamp: 2025-01-01T12:00:05Z
  type: text
  input: "/help"
  auto_enter: true

Capture Files (capture-001-label.txt)

yaml

---
sequence: 1
label: initial-state
timestamp: 2025-01-01T12:00:30Z
after_command: null
dimensions:
  width: 120
  height: 30
---
[terminal content here]

Integration with cli-tester

The @cli-tester agent can use this viewer to inspect session data:

typescript

// In cli-tester output
{
  captures: [
    { path: "debug/tmux-sessions/cli-test-123/capture-001-initial.txt", label: "initial" }
  ]
}

// Parent agent can view the session
// bun scripts/tmux/tmux-viewer/index.tsx cli-test-123 --json

GIF Export

The --export-gif flag renders the session replay as an animated GIF, perfect for:

Sharing CLI demonstrations
Embedding in documentation
Bug reports and issue tracking
Creating tutorials

GIF Export Options

Option	Description	Default
`--export-gif [path]`	Output file path	`<session>-<timestamp>.gif`
`--frame-delay <ms>`	Delay between frames in milliseconds	`1500`
`--font-size`	Font size for terminal text	`14`

Examples

bash

# Basic export (auto-names the file)
bun scripts/tmux/tmux-viewer/index.tsx my-session --export-gif

# Specify output path
bun scripts/tmux/tmux-viewer/index.tsx my-session --export-gif demo.gif

# Fast playback (500ms per frame)
bun scripts/tmux/tmux-viewer/index.tsx my-session --export-gif fast.gif --frame-delay 500

# Larger text for readability
bun scripts/tmux/tmux-viewer/index.tsx my-session --export-gif large.gif --font-size 18

GIF Output

The exported GIF includes:

Terminal content rendered as monospace text
Frame labels showing capture sequence number and label
Timestamps for each frame
Dark terminal-style background
Automatic sizing based on terminal dimensions

Development

bash

# Typecheck
cd scripts/tmux/tmux-viewer && bun x tsc --noEmit

# Run directly
bun scripts/tmux/tmux-viewer/index.tsx --list

Overview (`packages/agent-runtime/src/templates/README.md`)

Agent Template Override System

This system allows users to customize agent behavior by placing override configuration files in their project's .agents/templates/ directory.

How It Works

File Detection: The system automatically detects .agents/templates/*.json and .md files and loads them into a dedicated agentTemplates field in the project file context
Override Processing: When an agent is spawned, the system checks for matching override files and applies them to the base template
Dynamic Application: Overrides are applied at runtime, so changes take effect immediately without restarting

Override File Format

Override files should be JSON files with the following structure:

json

{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "model": "anthropic/claude-sonnet-4",
    "systemPrompt": {
      "type": "append",
      "content": "Additional system instructions"
    },
    "spawnableAgents": {
      "type": "append",
      "content": ["thinker"]
    }
  }
}

Configuration Options

Basic Properties

type: Agent type identifier (e.g., "CodebuffAI/reviewer" maps to "reviewer")
version: Version identifier (currently informational)
model: Override the model used by the agent

Prompt Overrides

Each prompt type (systemPrompt, instructionsPrompt, stepPrompt) supports:

type: How to apply the override
- "append": Add content after the base prompt
- "prepend": Add content before the base prompt
- "replace": Replace the entire base prompt
content: Inline content to use
path: Path to external file (relative to the override file)

Array Overrides

For arrays like spawnableAgents and toolNames:

type: How to apply the override
- "append": Add items to the existing array
- "replace": Replace the entire array
content: String or array of strings to add/replace

External Files

You can reference external files for prompt content:

json

{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "systemPrompt": {
      "type": "append",
      "path": "./system-prompt.md"
    }
  }
}

The path is resolved relative to the override file's directory.

Example Usage

Basic Model Override

.agents/templates/reviewer.json:

json

{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "model": "anthropic/claude-sonnet-4"
  }
}

System Prompt Enhancement

.agents/templates/reviewer.json:

json

{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "systemPrompt": {
      "type": "append",
      "content": "\n\nAdditional instructions: Focus on security vulnerabilities and performance issues."
    }
  }
}

External File Reference

.agents/templates/reviewer.json:

json

{
  "override": {
    "type": "CodebuffAI/reviewer",
    "version": "0.1.7",
    "systemPrompt": {
      "type": "append",
      "path": "./custom-review-instructions.md"
    }
  }
}

.agents/templates/custom-review-instructions.md:

markdown

## Custom Review Guidelines

- Check for proper error handling
- Verify input validation
- Look for potential security issues

Implementation Details

Override processing happens in packages/agent-runtime/src/templates/agent-overrides.ts
Files are loaded into ProjectFileContext.agentTemplates (separate from knowledge files)
Agent template files are loaded in cli/src/project-files.ts
Path resolution uses path.posix.join() for cross-platform compatibility
Errors are logged but don't break agent execution (graceful fallback to base template)

Error Handling

The system is designed to be robust:

Invalid JSON files are logged and ignored
Missing external files are logged and treated as empty content
Non-matching agent types are ignored
Processing errors fall back to the base template

This ensures that agent execution continues even if override files have issues.

Overview (`packages/code-map/src/tree-sitter-queries/readme.md`)

Credits

Codebuff uses modified versions of the tags.scm files from these open source tree-sitter language implementations:

https://github.com/tree-sitter/tree-sitter-c — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-c-sharp — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-cpp — licensed under the MIT License.
https://github.com/Wilfred/tree-sitter-elisp — licensed under the MIT License.
https://github.com/elixir-lang/tree-sitter-elixir — licensed under the Apache License, Version 2.0.
https://github.com/elm-tooling/tree-sitter-elm — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-go — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-java — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-javascript — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-ocaml — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-php — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-python — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-ql — licensed under the MIT License.
https://github.com/r-lib/tree-sitter-r — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-ruby — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-rust — licensed under the MIT License.
https://github.com/tree-sitter/tree-sitter-typescript — licensed under the MIT License.

This collection of queries was originally put together in the project aider: https://github.com/paul-gauthier/aider/tree/main/aider/queries

Agents And Tools (`docs/agents-and-tools.md`)

Agents and Tools

Agents

Prompt/programmatic agents live in .agents/ (programmatic agents use handleSteps generators).
Generator functions are NOT sandboxed: the runtime evals a handleSteps source string in the process that runs the agent (packages/agent-runtime/src/run-programmatic-step.ts); agent templates define tool access and subagents.

Registry agents with executable `handleSteps` require publisher trust

A template fetched from the public agent registry (--agent publisher/agent, a
registry id in a local agent's spawnableAgents, or a bare id falling back to
codebuff/) is executable code when it carries handleSteps: the row's
handleSteps is a source string, validation only checks that it starts with
function*, and the runtime evals it with no isolation on the user's machine.
The registry GET is public, any account can create a publisher and publish
without review, and latest is unpinned. So the SDK's fetchAgentFromDatabase
refuses a registry template with a string handleSteps unless its publisher is
trusted (sdk/src/agent-publisher-trust.ts), and throws an
UntrustedAgentPublisherError whose message names publisher/agent@version
and the knob to set. Trusted publishers are codebuff (our own, also bundled)
plus the comma-separated CODEBUFF_TRUSTED_AGENT_PUBLISHERS env var plus the
trustedAgentPublishers option on CodebuffClient / run(). Data-only
registry templates (no handleSteps) load as before, and local .agents files
and SDK agentDefinitions are never gated — they are code the user already
chose to run. This is a trust floor, not isolation: sandboxing the eval,
signing templates and pinning latest are separate work.

Clients released before this gate (codebuff and @codebuff/sdk on npm) eval
whatever the registry returns, so the registry GET routes refuse (403) a
non-codebuff template with handleSteps unless the request carries
AGENT_TRUST_GATE_HEADER (common/src/constants/agents.ts), which only the
gated SDK sends (untrustedHandleStepsRefusal in
web/src/app/api/agents/_registry-guard.ts). The server cannot see the
caller's trusted set, so the header defers that decision to the client.
mcpServers from non-codebuff publishers are refused outright, at publish
and at GET, whatever the client.

Trust gate for repository `.agents` directories

loadLocalAgents dynamically imports every .ts/.tsx/.js/.mjs/.cjs
file under {cwd}/.agents and {cwd}/../.agents, and loadMCPConfigSync
reads an mcp.json there whose stdio servers are spawned on the first prompt
with $VAR env values filled from the CLI's own environment. Both execute
repository-supplied code, so cloning a hostile repo and running the CLI inside
it must not run anything without consent. The CLI therefore gates those two
repository-scoped directories (cli/src/utils/agent-dir-trust.ts):

What needs trust: a directory containing at least one executable agent
file (the loader's extension filter, minus .d.ts and tests) or an
mcp.json. A directory holding only skills/ markdown does not, and
~/.agents never does (it is the user's own). Skills still load from an
untrusted directory; only agents and mcp.json are withheld.
The prompt: on an interactive launch (stdin and stdout are TTYs) the CLI
prints a plain-terminal prompt before the TUI mounts, listing the directory,
its agent files (capped at 10) and the commands or URLs its mcp.json would
start, and asks Load and run these? [y/N]. y records the directory;
anything else skips it for this run and prints how to trust it later.
The store: <configDir>/trusted-agent-dirs.json (next to
credentials.json, written with mode 0600), a map of the normalized
absolute directory path to { "trustedAt": "" }. Delete an entry to
be asked again.
Non-interactive runs (no TTY, CI) never prompt: the directory is skipped
with a warning unless CODEBUFF_TRUST_AGENT_DIRS=1 or --trust-agents is
passed, which trusts every repository directory for that run only and writes
nothing to the store.
SDK semantics are unchanged: loadLocalAgents({ agentDirs }) and
loadMCPConfig{,Sync}({ configDirs }) are opt-in options; omitting them
keeps the default search, so Desktop and external SDK consumers behave as
before. getDefaultAgentDirs() and listLocalAgentFiles() are exported so a
host can build its own gate on the loader's own filter.

Trust is per directory, not per file content: a trusted repository that later
pulls a malicious agent file is loaded without a new prompt, the same trade-off
other coding agents make for their project-level config.

Shell Shims

Direct commands without codebuff prefix:

bash

codebuff shims install codebuff/[email protected]
eval "$(codebuff shims env)"
base-lite "fix this bug"

Tools

Tool definitions live in common/src/tools and are executed via the SDK helpers + agent-runtime.

Cut-off file edits are refused, never repaired

packages/agent-runtime/src/util/truncated-write-guard.ts stops two shapes of
a file edit that was cut off mid-generation:

Arguments that never closed. When a response hits the output limit (or
loses its connection) inside a tool call, the OpenAI-compatible provider
still flushes the call with whatever argument text arrived, and OpenRouter
reports the step as finish_reason: "tool_calls" (the real reason is only in
native_finish_reason: "max_output_tokens"). parseRawToolCall recognises
a truncated JSON prefix and answers with an explicit "nothing was written"
error instead of the generic re-issue hint, and does not echo the partial
content back. Do not add a JSON-repair step for mutating tools: completing a
cut-off content string is exactly how a file gets truncated on disk.
A cut-off copy of an existing file. processFileBlock refuses a
write_file that is under 60% of an existing file of 30+ lines AND leaves
brackets open where the current file balances them. Both refusals log
axiomEvent: file_write_guard (metric says which).

The finish reason is deliberately not the signal: complete arguments in a
length-cut step are a complete call, and the OpenRouter shape above hides the
length cut anyway.

Console-free terminal command broker

run_terminal_command separates process ownership from terminal UI ownership:

sdk/src/tools/run-terminal-command.ts owns output buffering, timeouts,
cancellation escalation, results, and process diagnostics. Headless SDK
consumers use its direct process-group runner.
Interactive hosts provide terminalCommandBroker in CodebuffClientOptions
(or directly to runTerminalCommand). Each call synchronously starts an
isolated helper and returns a handle for its complete process tree. A startup
failure prevents the shell from running; there is no direct-console fallback.
The CLI's tiny src/entry.ts handles private broker mode before importing
React or OpenTUI. The detached, hidden helper receives one spawn request over
stdin, starts the shell without a console or interactive stdin, relays only
stdout/stderr pipes, and reports completion through a constrained one-shot
file in the OS temp directory. It deliberately uses only the three standard
stdio channels: Bun's custom child-process pipes can fail their Windows
node:net handshake outside the ChildProcess error event and terminate the
CLI as an unhandled rejection. The broker remains the process-group root and
self-reaps the tree if its parent disappears, detected by polling the parent
PID rather than holding another pipe open.
Mouse and focus protocols stay enabled while commands run. The
TerminalProtocolController only parses focus events; it has no command
lifecycle state to synchronize or restore.

Thread the broker capability through every interactive command entry point.
Do not bypass it with a direct spawn, add command-active terminal state, or
fall back to the TUI process when broker startup fails.

Testing (`docs/testing.md`)

Testing

Prefer dependency injection over module mocking; define contracts in common/src/types/contracts/.
Use spyOn() only for globals / legacy seams.
Avoid mock.module() for functions; use @codebuff/common/testing/mock-modules.ts helpers for constants only. A module mock lasts for the rest of the bun test process, and mock.restore() does not undo it (see "the FILE ORDER").

CLI hook testing note: React 19 + Bun + RTL renderHook() is unreliable; prefer integration tests via components for hook behavior.

Test env must come from a fixture, not the developer's `.env`

@codebuff/common's env.ts validates the NEXT_PUBLIC_* vars at import time and throws, and Bun loads .env files from the process cwd — so a package-local bun test sees none of the repo-root env even when the root run is green. A whole test file then dies with Invalid environment configuration before a single test runs, which bun reports as an unhandled error rather than a failure, so the suite silently stops covering that file.

Every package must therefore have a bunfig.toml preloading sdk/test/setup-env.ts — the one shared fixture, which supplies placeholder values for every var the schemas require. This is not just a local-dev nicety: CI runs cd && bun test, i.e. exactly the package-local mode. Placeholders only — tests must never need real credentials, so bun test means the same thing in a fresh worktree and in a provisioned checkout.

The unit-test matrix therefore never loads Infisical, and neither does flake-hunt.yml, which runs the same suites in the same job environment. This is a correctness boundary, not merely a secrets optimization: production rollout switches are mutable operational state, so importing them makes unrelated commits fail as soon as an operator changes a flag. Tests for non-default modes must set the specific variable in the test or inject the mode explicitly. Secret-bearing integration jobs remain separate.

The shared preload deletes FREEBUFF_ADS_SLACK_WEBHOOK_URL, including when
integration CI or a local shell supplies it. Campaign fixtures otherwise send
real review notifications to #ads-notifs on every suite run and retry. The
Slack sender also refuses its default transport whenever NODE_ENV, BUN_ENV
or NEXT_PUBLIC_CB_ENVIRONMENT is test. Notification tests inject fetchImpl
or postMessage; the production webhook stays configured for real submissions.

The same rule covers generated inputs. cli/src/agents/bundled-agents.generated.ts is gitignored, and importing it at module scope wiped 17 files (~371 tests) in a fresh worktree; cli/test/setup-agents-artifact.ts now builds it on demand instead of relying on everyone knowing to run bun run prebuild:agents.

Tests that need a service rather than a variable should skip cleanly and say why, but never in CI. @codebuff/internal/testing/test-db probes Postgres once, skips the DB suites locally with the docker command to fix it, and throws when CODEBUFF_GITHUB_ACTIONS=true — otherwise a broken CI service container would read as a pass. Gate on reachability, never on !process.env.DATABASE_URL: the fixtures supply a placeholder URL, so presence stopped meaning availability.

A unit test that leaks a database call gets the same answer locally as in CI. test/setup-db-leak-guard.ts, the first preload in the bunfig of every package whose tests can reach Postgres (the root, web, freebuff/web, packages/{internal,billing,auth,jobs}, and scripts, which preloads nothing else), answers CI's must-not-reach-a-database URLs with an instant Postgres error that names the leak. When none of DATABASE_URL, DIRECT_DATABASE_URL and SCRIPTS_DATABASE_URL is set, it supplies that URL itself. Without it, a local run fell through to setup-env.ts's localhost placeholder, where postgres-js's reconnect backoff let a swallowed leak time out some later test (2026-10-13). It must stay ahead of setup-env.ts in those lists, and scripts/__tests__/db-leak-guard.test.ts fails if it does not. Set any of the three, as an integration run does, and it leaves them all alone. probeTestDb reports the sentinel as unreachable without connecting, so integration files skip without tripping the guard.

Tests that spawn a server child should race readiness against proc.exited and report the child's captured output (see freebuff-desktop/test/launch-bootstrap.smoke.test.ts). Polling a dead port surfaces only as "a hook timed out", which names neither the cause nor the process that failed.

The CI guard against disappearing tests

CI does not run bun test directly — it goes through scripts/ci/test-with-guard.ts, which fails the build on any of:

any error outside a test body — an Unhandled error between tests marker (a file that crashed at import) or a non-zero N errors in bun's summary,
a test or file count below the baseline in .github/test-baselines.json, including the degenerate case where the caller's glob selects no files at all (a renamed directory would otherwise print "no tests found" and pass).

Growth never fails the build, so adding tests needs no baseline change; the guard just notes the baseline is stale. Deleting tests on purpose means re-recording: re-run the job's command with --update.

Refresh the baselines periodically, and act on the "stale" line. Because growth never fails, a baseline nobody touches quietly becomes a floor far below reality — which is a guard that no longer guards. When this rot was first measured, freebuff-desktop was recorded at 971 tests while actually running 1278, so 307 tests could have stopped running and CI would have stayed green (393 across all suites). Re-record from a real CI run, not locally.

Expect to do it more than once. Desktop moved 1278 → 1282 → 1339 over the two days it took to land that refresh, so on an active package the floor starts sliding immediately. That is fine — a floor 50 tests low still catches the failure this exists for, which is a suite losing hundreds at once. It is a floor 300 low that is worthless. Treat the "stale" line as a nudge, not an emergency.

A suite that selects no files and has no baseline is not a failure — it can legitimately have no tests yet — but it now emits a Suite ran no tests warning annotation rather than passing in silence. test:.agents, test-integration:common and test-integration:packages/agent-runtime sat in that state on every run until they were dropped from ci.yml, along with the test:files* scripts that existed only for them. Adding a first test there now fails check-test-coverage until the package is listed in a matrix again, so it cannot go unrun silently. When the warning fires, either give the suite tests or drop it, but decide rather than letting it read as green.

A skipped suite and a running suite do not report the same total. Skipped tests are counted, so it is tempting to assume a baseline recorded locally transfers to CI — it does not when a whole describe is skipped. Measured on the billing DB suites: each skipped file reported exactly two more than it did when it ran (47 vs 41 across three files), and the guard duly failed a healthy CI run. So a baseline for a suite that skips locally but runs in CI — anything DB-backed — must be taken from a real CI run. Read the observed counts out of the job log and write them into .github/test-baselines.json.

Re-record in a CI-equivalent state, which means running cd sdk && bun run build first. A few cli tests register a placeholder test only when sdk/dist is missing, so a baseline seeded without it is inflated and the guard then fails a perfectly healthy CI run. (The guard caught exactly this while being built.)

This exists because every check bun gives you is blind to the failure mode that actually happened here three times: a file stops contributing tests and the summary still reads as mostly-green.

[test].exclude in bunfig.toml does nothing on bun 1.3.14. Verified by pointing it at an ordinary test file, which still ran. That is why the repo-root exclude does not keep *.integration.test.* out of a root run, and why CI filters them with find ... ! -name '*.integration.test.ts'. Gate on a runtime condition instead of trusting the key.

Never hand a bare postgres.js query to expect(...).rejects. A query is lazy: it runs only when something calls its .then(), and bun 1.3.14's .rejects never does, so the test waits forever. The per-test timeout does not fire either, so the whole job runs until its 15-minute step timeout. On 2026-09-30, region-verifier-runs.integration.test.ts hung test-integration-packages/internal this way on every PR until #4595. Await the query inside a function and assert on what it throws (see that test's rejects helper), or use expect((async () => await client`...`)()).rejects.

Cloud Agent smoke test (no credentials)

A Cloud Agent — or any checkout without Infisical — can confirm the workspace can run a package-local unit test from the placeholder fixture alone. This is one file in @codebuff/internal. A pass does not validate the rest of the application, CI, or production runtime.

The repo pins Bun 1.3.14 (.bun-version, package.json engines.bun / packageManager) and Node 24.21.0 (.node-version, which Render applies to every repo-root service; only web runs on Node, the rest start Bun). The smoke test runs under Bun, not Node. If the VM has no Bun, install that pin from the official installer without changing tracked runtime files:

bash

curl -fsSL https://bun.sh/install | bash -s "bun-v1.3.14"

Do not use the repo .bin/bun wrapper for this. .envrc puts .bin first on PATH, and the wrapper tries Infisical; with .env.example present and no .env.local it exits 1 before invoking the real binary.

Resolve the installed Bun before .envrc prepends .bin. If bun is already on PATH (mise, a system package, a prior install), capture that path. Use $HOME/.bun/bin/bun only when you just ran the official installer above.

bash

# Before direnv / .envrc, if bun is already installed:
REAL_BUN="$(command -v bun)"
# Only after the official installer, if bun was missing:
# REAL_BUN="$HOME/.bun/bin/bun"

cd packages/internal
"$REAL_BUN" test src/ad-targeting/__tests__/iso-3166-alpha2.test.ts

bun install is not required for this file. The package bunfig.toml preloads sdk/test/setup-env.ts. Expect 3 passing tests.

The production-runtime smoke in `build-web`

Everything else in CI runs under Bun. Production web runs under Node (Render's
default, pinned in .node-version so CI and Render read the same file), and
the two disagree on which JS built-ins exist: on 2026-09-11 a
Promise.withResolvers() in the completions handler was green everywhere and
threw on every request in production (#3218). scripts/check-node-runtime-apis.ts
catches the known names; web/scripts/prod-smoke.ts catches the class. After
the Next build, build-web starts next start on the pinned Node against a
migrated Postgres container and drives one real client journey — session
admission, agent-run start, a streamed chat completion — with every external
HTTP call stubbed. A request that cannot complete end to end fails the job.

Both smokes are thin: packages/internal/src/prod-smoke/harness.ts owns the
loopback-only database guard, the Bun-shim refusal, the seeded fixture (ad-data
lifecycle row, user, session), staging a build outside the repo, the server
spawn with log relay, /api/healthz polling with the optional runner gate, the
authenticated request helper and teardown; upstream-stub.cjs beside it is the
one table of stubbed upstream endpoints. An app's scripts/prod-smoke.ts
supplies only its port, launch command, environment, auth header and journey. The web Render service deploys
manually, so this makes main red before someone deploys it rather than
blocking the deploy itself.

Run it locally after bun run build in web/ with DATABASE_URL pointing at
a loopback Postgres that has the migrations applied (the script seeds rows, so
it refuses any other host):

bash

cd web && DATABASE_URL=postgresql://postgres:[email protected]:5432/testdb bun run smoke:prod

build-freebuff-web has the same gate for the builder app,
freebuff/web/scripts/prod-smoke.ts, and it matters more there: freebuff-web
auto-deploys from main. Two differences. The pin is that service's own
freebuff/web/.node-version, not the repo root's, because Render applies the
root file to every service and a root pin meant for web moved freebuff-web
from 22.x to 20.15.1 on 2026-09-13, breaking its build on a Node 20 flag while
CI stayed green; the job now installs that Node before the compile, since the
app's build script runs node itself. And the journey is a chat turn behind a
seeded database session cookie rather than an API-key completion, because that
is the path that import()s @codebuff/sdk out of the standalone tree at
request time and runs the agent in-process; the SDK's calls to the backend
(/api/v1/me, the run ledger, the model) all land in the stub's table.
Hiding @codebuff/sdk from the standalone tree makes it fail, which is the
proof that it exercises what the build's import check only approximates. The
embedded agent runner is made required (as on a deployed instance) with
placeholder Convex credentials, so a runner that cannot boot is a 503 from
/api/healthz and a fast red rather than a warning; note that Next polyfills
WebSocket on its server, so the runner's "Node >= 22 required" bail does not
fire on Node 20 under this app — a Node downgrade is caught by building and
smoking on the pinned version, not by that check.

bash

cd freebuff/web && DATABASE_URL=postgresql://postgres:[email protected]:5432/testdb bun run smoke:prod

What CI actually spends its time on

ci.yml is one flat fan-out: typecheck, build-web, and three test matrices,
with no needs: edges between them (all of them wait only on the short
changes job, below). Before that, every test job waited on a
combined build-and-check gate, and measuring a run showed how badly that
misread the cost:

	before	after
Total run	10m40s	5m29s
the gate / its replacements	`build-and-check` 441s	`typecheck` 326s ‖ `build-web` 194s
web build's own compile	330s (`in 5.5min`)	73s
all test jobs' `bun test` steps, added together	184s	110s
`test-common`'s actual test run	1s (after 55s of setup)	1s

The gate was ~70% of the run, and the thing it gated took under two minutes of
real work. Nothing a test imports is produced by the typecheck or the Next
build, so the edges bought nothing.

Splitting typecheck from the web build was worth more than un-gating: the Next
compile went 330s → 73s purely by not sharing two cores with tsc. Note what did
not change — Next still logs using 1 worker for page-data collection and
static generation, before and after. On a 2-vCPU runner it always will. The win
came from the webpack compile phase, not from more workers; don't chase the
worker count.

The corollary for anyone optimizing this: individual tests are not the
problem, and deleting them will not help. Per-job setup (dependency-cache
restore, SDK build) costs far more than the tests in almost every job.

Typecheck runs in lanes

bun --filter='*' run typecheck scheduled @codebuff/freebuff-web last: it did
not start until 114s into a 227s step, then ran ~113s alone while all 17 other
packages had finished and the runner idled on one tsc. So the typecheck job is a
matrix over lanes — freebuff-web gets its own runner, rest takes the other
17 — defined in scripts/ci/typecheck-lanes.ts.

The freebuff/web typecheck allows an 8 GiB Node heap. A cold check measured
6,239,321 KiB after the October 2026 Desktop/Web changes; the previous 6 GiB
limit exhausted the heap on both main and PR runners. Its dedicated
ubicloud-standard-4 runner has 16 GiB RAM, leaving room for Node and the
runner outside the heap. Keep the full check; incremental caches only reduce
the work when the prior build info is usable.

Membership lives in that script rather than in ci.yml because both obvious
ways to write it in YAML fail silently:

--filter='*' --filter='!@codebuff/freebuff-web' does not exclude
anything. bun 1.3.14 ignores the negation and runs all 18 packages
(verified), so the rest lane would re-run the package it exists to hand off
and the split would buy nothing while looking correct.
Listing the other 17 by hand means a newly added package matches no lane
and is typechecked by nobody — a green build with a hole in it.

So lanes are derived from the workspace at runtime and the derivation is
checked: every package with a typecheck script must land in exactly one lane
or the script exits 1. Adding a package needs no change; only moving one
between lanes does.

Everything else in that script exists to make the remaining ways it can quietly
check nothing loud. Each of these exits 0 on its own:

A lane declared in the script but missing from ci.yml's matrix. Its
packages leave rest and are then run by nobody, while every lane that does
run still passes. This is why the step passes --expect-lanes: the two lists
must match or the build fails. Keep them in sync when adding a lane.
A filter that matches nothing, next to filters that do. Measured on bun
1.3.14: --filter=@codebuff/common --filter=@codebuff/nope runs common,
ignores the bad one and exits 0. A lone non-matching filter exits 1, so
only the multi-package case is silent — which is exactly what rest is. The
script therefore asserts every lane member actually reported a result.
(Relatedly, write --filter=NAME; as separate argv entries the value does not
bind and bun matches nothing at all.)
A lane naming a package that was renamed or lost its typecheck script.
Bun.Glob skipping dot-directories. Without dot: true the workspace
scan finds 20 manifests instead of 21 and .agents silently leaves every
lane.

Beware retry-inflated numbers when you profile this. The first pass at the
table above recorded freebuff-desktop at 130s, which was wrong: the job had
silently failed once (Boundary result-checkpointed was not reached, 69.55s)
and passed on attempt 2 (48.78s), and nick-fields/retry reported success. Read
bun's own Ran N tests across M files [Xs] line, not the step duration, and
grep the log for Attempt N failed before trusting any per-job figure.

Unit suites run once; integration suites retry and end with a step that
raises a Flaky suite warning when a retry rescued them.

The split is measured, not assumed. Across 100 CI runs / 2000 test-job instances
(2026-07-31 → 08-02) the retry rescued a unit suite zero times; both real
rescues were test-integration-packages/internal, i.e. a Postgres service
container. Retrying infrastructure is what a retry is for; retrying test logic
just moves the bill to whoever hits it next. Unit suites therefore used to keep
three attempts only to fail a pass-on-retry as "flaky", which meant a red unit
suite ran three times and could never come back green: on 2026-10-07 failing
test-freebuff/web jobs took 736s against 275s green. Since then a flaky unit
case fails on its first attempt like any other failure, and its (fail) line is
the last in the log. Tell "flaky" from "broken" locally with
scripts/flake-hunt.ts.

`sdk/dist` is cached, not rebuilt per job

./.github/actions/build-sdk restores sdk/dist from a cache keyed on every
input the bundler reads (sdk sources/scripts/vendor, the workspace packages it
inlines via tsconfig paths, and bun.lock). It has no restore-keys on
purpose: a prefix match would hand back a dist built from different sources and
every downstream test would silently run against a stale SDK. A miss just costs
the ~18s build.

freebuff/web's prepare:workspace shells out to that same SDK build, and it
was the tail of the typecheck job — freebuff/web finishes last, and spent its
first ~65s rebuilding what the job had already restored. It now honours
SDK_ALREADY_BUILT, which only CI's typecheck job sets, immediately after
build-sdk has run. Unset everywhere else, so local runs and Render deploys
still build normally. If you add a job that runs freebuff/web's typecheck or
build, either let it rebuild or set the flag after build-sdk — never before.

Per-job setup (checkout, a cold bun install, the SDK cache) is about 45s on
Ubicloud and 65s with Postgres. In 40 runs measured on 2026-10-06, seventeen
unit suites took 0.8–1.0 min each as separate jobs, running about 25s of tests
between them. So ci.yml's test matrices are lists of lanes, each running
one or more packages (packages) one after another. Every package still runs
under its own test-with-guard key and baseline, a lane reports every package
that failed rather than stopping at the first, and check-test-coverage.ts
reads the packages from the lanes. In the database lane, each package after
the first gets an empty database of its own (scripts/ci/fresh-test-database.ts).

Give a suite its own lane when its runtime is a real fraction of the critical
path (the big four: freebuff/web, freebuff-desktop, web, cli), and add
it to small otherwise. The three Windows areas are one test-windows job
for the same reason: setup was 75–95% of each.

Big suites are sharded

The opposite case: a suite too slow for one runner gets several lanes, one per
shard: i/n, each naming the same single package. test-with-guard --shard i/n
runs the files scripts/ci/test-shard.ts assigns to that shard and guards them
under the key <key>#i/n. Today that is the freebuff/web and
freebuff-desktop unit suites and web's DB integration suite, two shards
each.

Assignment is a hash of the file's path (FNV-1a over the path
test:files prints, modulo n). Adding or deleting a file changes one shard's
count and no other. A split by position in the sorted list would move files
between shards on every addition, and a healthy shard could fall below its
file floor.
Each shard has its own baseline, recorded from a CI log like any other
(never --update locally). Changing n, or the hash, moves files between
shards, so every shard of that package needs re-recording from the first CI
run that has the change. scripts/__tests__/test-shard.test.ts pins two
assignments so a hash change cannot happen by accident.
The shards must be one complete partition. check-test-coverage.ts
fails if a sharded package does not declare every index 1..n of one n exactly
once, appears unsharded in the same job as well, or has a file that hashes to
no declared shard or to two. Without that check, dropping the 2/2 lane would
lose half the suite while every remaining lane passed.
Some files must share a shard. SHARD_TOGETHER in test-shard.ts
hashes a directory as one unit. web's src/server/desktop-cloud/ is one:
its suites start and stop local R2 (workerd) in-process, and split by file,
store.integration.test.ts hung for 60s after org-invite-store, storage
and github ran before it, reproducibly in CI. With the rest of the
directory between them, as in an unsharded run, it passes. Add a group only
for a cross-file hazard you have reproduced, and fix the hazard where you can.
A new file order surfaces leaks. Sharding's first CI run found two
freebuff/web tests leaving a fake window behind (a later server render then
threw on window.location.href) and a process-group test that read /proc
before the killed process had finished exiting. Fix the leaking test; do
not move files between shards to hide it.
Conditions that mean "this package" test matrix.packages, not
matrix.lane: a shard's lane is named web (1/2), so matrix.lane == 'web'
would quietly stop matching (and stop starting its Redis service).

Measured before sharding (8 green main runs, 2026-10-07; median job time):
test-integration-web 282s and test-freebuff/web 276s were the two slowest
jobs, with ~200s and ~225s of bun test. Per-file times from three of those
logs (the gap between consecutive file headers) split 102s/104s for
freebuff/web and, with desktop-cloud kept together, 78s/125s for web; web's
two slowest DB files (~33s and ~22s) land in different shards. With those two
split, test-freebuff-desktop (164–267s of tests on its pinned Bun, 255s
median job, 330s in the PR's run) was the slowest job, so it is split too,
96s/98s by per-file time.

Two shards each, not three: the jobs behind them sit at 220–265s
(test-windows, build-freebuff-web, typecheck-freebuff-web,
typecheck-rest), so a third shard would buy no wall clock for another ~70s of
setup. Shard a suite only when it is clearly the long pole. Each extra shard
costs a full job's setup (~50–70s of runner time), and on this pool runner
pickup was the larger delay in the slowest runs: up to ~350s before a job
started, which more concurrent jobs make no better.

A pull request runs only the lanes it can reach

A small changes job runs scripts/ci/affected-lanes.ts on the PR's merge
commit, and every other job waits on it (the one needs: edge). It resolves
every file's imports and repo-path reads into a file-level graph, and a lane
runs when the PR touches its own packages or any file its tests, build or
typecheck can load. Shared config (lockfile, root tsconfig/bunfig, ci.yml,
.github/actions/, scripts/ci/, …), an unknown top-level path, or a failed
changes job runs everything; pushes to main always run everything, which is
the backstop for an edge the scanner misses.

Why did a lane run (or not)? The changes job's step summary names the
changed file that selected each lane. Locally:
bun scripts/ci/affected-lanes.ts --files …, and
AFFECTED_LANES_DEBUG=1 prints the import chain to each package.
A matrix row is picked up from ci.yml by its packages; nothing to add.
Shards share packages, so they run or skip together. In test and
test-db-integration the rows are the JSON table inside the matrix's
include: ${{ fromJSON(needs.changes.outputs.…_matrix || '[…]') }}: edit it
there (no }} inside it). changes emits the selected rows, every key kept,
so an unselected leg never starts a runner; a PR that selects none emits [],
which skips the job (an empty matrix is an error), and a failed changes
emits nothing, so GitHub falls back to that table and every leg runs.
A new non-matrix job that should be filtered needs an entry in
SINGLE_JOBS in the script and its output in the changes job; a job with no
needs: changes simply always runs.

Known remaining cost

What is left on the critical path is not test time. test-windows used to
spend ~130s of its ~264s restoring the whole monorepo's node_modules; it now
installs only the workspaces its suites load (see "Windows installs" below).
typecheck-freebuff-web and typecheck-rest are one tsc each
(160–220s) behind ~45s of setup; the lever there is incremental tsc or a
larger runner, and both need their own measurement. build-freebuff-web's
~110s Turbopack compile now starts from the previous run's persistent cache
(next section).

Measure test time on CI, not locally. freebuff-desktop's
src/app/thread-engine.test.ts once took ~101s on an M-series Mac against
~54s for the entire desktop suite on a runner, so local timings will send you
after the wrong thing.

Cold Bun installs beat the dependency cache on Ubicloud

Do not cache node_modules on the Linux jobs. The first full CI run after the
Ubicloud migration gave a natural cold-cache comparison across all 31 Linux
jobs: cache lookup plus bun install --frozen-lockfile took 802 runner-seconds
in total, with a 26.0s median per job. The next three runs restored the same
1.185 GB archive and took 1,038–1,080 runner-seconds for restore plus install,
with 32.9–34.1s medians. A warm cache was therefore about 7–8s slower per job
and cost roughly four extra runner-minutes per full CI run.

The smaller observations agree with the full matrix. The since-deleted
Migration CI installed cold in 23s, while its warm restores took 18–41s. The abuse sweep installed in
17s on a miss and then spent another 24s compressing and saving the cache.
setup-project consequently skips dependency caching on Linux, and standalone
Ubicloud workflows use that action instead of carrying their own cache blocks.
The Windows and macOS cache remains until those runners have their own cold/warm
comparison; this measurement says nothing about their install performance.

Windows installs

Measured on windows-latest on 2026-10-07 (Bun 1.4.2, Defender already off):

Install	Time
`setup-project`, warm 1.05 GB cache (278k files)	~110s restore, 95s of it tar unpacking, + 5s install
Whole monorepo, cold, Bun cache on its default C:	647s
Whole monorepo, cold, Bun cache on D:	187s
`test-windows`'s workspaces only, cold, cache on D:	36–54s
The same, warm 400 MB cache (~51k files)	24–27s restore + 5–7s install

Two separate costs. File count dominates a restore, so test-windows installs
with bun install --filter for the workspaces its suites load. And the
checkout is on D: while Bun's install cache defaults to C:, so every package
is copied across volumes; pointing BUN_INSTALL_CACHE_DIR at runner.temp
(on D:) cut a cold install 3–4x. Every other Windows job still uses
setup-project, which leaves the cache on C:, so its cold path (a lockfile
change) is the 647s row.

Keep the caches whose transfer cost is materially smaller than the work they
avoid. sdk/dist is a roughly 17 MB restore instead of an ~18s build.
build-freebuff-web keeps Turbopack's persistent build cache
(turbopackFileSystemCacheForBuild, on in CI and off on Render, where the
download cost more than the compile it saved): scripts/render-next-cache.mjs stashes .next/cache in $RUNNER_TEMP
on GitHub Actions instead of letting the postbuild prune delete it, and main's
runs save it under a key scoped to bun.lock and next.config.mjs; pull
requests restore main's newest. The archive is ~1 GB (1.4 GB unpacked).
Measured 2026-10-07 on two runs of #5845: the compile went from 112s cold to
56s warm and the build step from 169s to 86s, against ~16s to restore and ~18s
to save. Past 3 GiB the stash is dropped, so the next run starts cold.
build-web has no Next cache: web turns the persistent cache off in
web/next.config.mjs because its cold compile is ~30s. oven-sh/setup-bun's
own small runtime cache is independent of dependency installation and also
remains enabled.

CLI tmux Testing

For testing CLI behavior via tmux, use the helper scripts in scripts/tmux/. These handle bracketed paste mode and session logging automatically. Session data is saved to debug/tmux-sessions/ in YAML format and can be viewed with bun scripts/tmux/tmux-viewer/index.tsx. See scripts/tmux/README.md for details.

Useful workflow for agents:

bash

# Start the dev CLI in a detached tmux session.
SESSION=$(./scripts/tmux/tmux-cli.sh start --name cli-check -w 160 -h 40 --wait 6)

# Capture the initial screen. Captures are written to debug/tmux-sessions/$SESSION/.
./scripts/tmux/tmux-cli.sh capture "$SESSION" --label initial

# Send a prompt. The helper uses bracketed paste so text is not dropped.
./scripts/tmux/tmux-cli.sh send "$SESSION" "Search for getAgentBaseName and report what you find" --wait-idle 4

# Capture after the run, then inspect the saved capture text.
./scripts/tmux/tmux-cli.sh capture "$SESSION" --label after-search --wait 2

# Clean up when finished.
./scripts/tmux/tmux-cli.sh stop "$SESSION"

If a change can be verified with a small local harness instead of a live model-backed CLI run, run that harness inside tmux too. This still checks terminal rendering and produces a capture:

bash

SESSION=$(./scripts/tmux/tmux-cli.sh start \
  --name render-check \
  -w 160 -h 20 \
  --wait 1 \
  --command "bun .context/my-render-check.tsx")

./scripts/tmux/tmux-cli.sh capture "$SESSION" --label rendered
./scripts/tmux/tmux-cli.sh stop "$SESSION"

When verifying UI output, prefer checking the saved capture file for concrete strings that should and should not appear. For example, after expanding a code-searcher agent, check that the capture shows the search summary but not raw structured payload keys like results: or stdout:.

Confirming a suspected flake

CI runs a failing test job up to three times (nick-fields/retry in ci.yml).
A unit suite that only passes on a later attempt now fails the build; an
integration one still passes with a warning. Either way the interesting question
is the same one, and CI cannot answer it: CI never applies contention. Each
package's suite gets its own runner, so the shared-state flakes live below its
resolution entirely. When a suite fails on a busy machine and passes on a quiet
one, the
temptation is to call it the machine's fault. Don't: load is not a cause, it
is a magnifier. What it magnifies are real defects that a single run on an
idle laptop cannot show.

scripts/flake-hunt.ts reproduces those conditions deliberately — N rounds,
each running the suite several times simultaneously while CPU busy-loops
compete for the cores — and names every test that fails.

bash

# The default hunt: 3 rounds x 2 overlapping runs of the desktop suite, on the
# Bun that suite is tested on (see below).
bunx bun@$(cat freebuff-desktop/.bun-version) scripts/flake-hunt.ts

# Harder, and on a different package.
bun scripts/flake-hunt.ts --dir cli --rounds 5 --concurrency 3 --hogs 8

# Interrogate one suspect file: cheap enough to run 10 times.
bunx bun@$(cat freebuff-desktop/.bun-version) scripts/flake-hunt.ts \
  --cmd "bun test test/launch-bootstrap.smoke.test.ts" --rounds 10

It exits non-zero if any run reported a failing test, prints the load average
per round, and writes each run's full output to a log it names for you. If your
shell lacks the repo env (a worktree without direnv), pass it through the
command: --cmd "bun --env-file=../.env.local test".

freebuff-desktop is hunted on its own Bun. CI tests it on the Bun the app
ships with (freebuff-desktop/.bun-version), not on the monorepo toolchain
(.bun-version) that bun on your PATH is, and the two are not
interchangeable. On 2026-09-26 a desktop hunt on the toolchain Bun (1.3.14)
failed mcp-consent-bridge.test.ts › "an asker that gives up closes its
dialog, so the next approval is not refused busy" on every run, loaded or solo,
while the pin (1.4.2) passed it every time: 1.3.14's node:http never emits
close on a response whose client hung up. It was a deterministic runtime
difference, and it read as a flake. So flake-hunt.ts now refuses a desktop
hunt when the bun its runs would get is not the pin, and says how to get it.
bunx bun@ is the one-liner because it puts the pin first on PATH for
everything the hunt starts, and the suite spawns its orchestrator as bun
from PATH. Pass --any-bun to compare versions on purpose. The weekly workflow
and ci.yml's retry-flake annotation both follow the pin.

When the hunter cannot reproduce it

The hunter is a magnifier, not an oracle, and one open flake is proof of that.
useProjectSkills failed on three separate PRs, on three different cases,
none of which touched freebuff-desktop — and never once under 24 hunted runs
at load 11-16, nor under 12 concurrent runs with CPU hogs, on the fixed and
unfixed versions alike.

Two concrete mechanisms were removed and it flaked again on a head that already
contained both. At that point further guessing costs other people's PRs, so the
file was skipped and tracked in issue #1899 rather than patched a third time.

#1983 un-skipped it by moving it off the hand-rolled mount() entirely, onto
@testing-library/react + jsdom instead (the pattern already used by
UpdatesPage.test.tsx, ConnectorsPanel.test.tsx, …),
on the theory that the class of bug cannot exist there — React's own scheduler
owns the dispatcher for the whole render/effect/continuation lifecycle, instead
of a test file borrowing it by hand.

That is a theory, not a diagnosis. The original flake was never reproduced,
so nothing distinguishes "the dispatcher swap was the cause and is now gone"
from "the timing changed and the flake moved". Leave #1899 open until the file
has survived a stretch of CI; if it flakes again, the migration is evidence
against the dispatcher hypothesis and the search should widen, not repeat.
The other twelve files named below still use the swap either way.

Two things worth taking from it:

A green gh pr checks does not mean the job never failed. CI retries, and
a successful re-run overwrites the run conclusion. The failures above are only
visible via attempts/1/jobs on the run. Check there before concluding a
suite is healthy.
The suspect is usually shared, so the file that flaked is rarely the fix.
Here it is the hand-rolled mount() that swaps the process-global
React.__CLIENT_INTERNALS…H dispatcher and restores it in a finally —
while effects, and the fetch continuations they schedule, run after that
restore. 13 desktop UI test files touch that global and at least six define
their own mount() with the same shape, so fixing one file just moves the
flake to a sibling — as it still can, for the twelve that remain.

Three things to check before blaming the machine. Each of these was a real
defect found by this harness on its first outing:

Shared state between concurrent runs. server.test.ts spawned its
orchestrator on a hardcoded port and probed that port for readiness, so two
overlapping runs did not collide — they merged. The loser's server died with
EADDRINUSE, its readiness probe was answered by the winner's server, and
its tests then drove a foreign engine with a different $HOME and a
different open project. Two concurrent runs of that one file produced 6 and
19 failures scattered across unrelated assertions; alone it passes 49/49.
Spawn servers on port 0 and read back the port they actually got.

Then check that the server honours 0. Vite's server.listen() reads
port: 0 as unset and binds 5173, and the repo config's strictPort makes
it exit before listening if anything is already there: an overlapping run,
or another project's dev server on Vite's default port. terminal.test.ts
passed port: 0 and still lost one of three overlapping suite runs in every
hunted round, and 10 of 15 runs when only that file overlapped. It now calls
server.httpServer.listen(0) itself and gives Vite a per-run cacheDir,
because the deps optimizer otherwise commits into the package's shared
node_modules/.vite. That failure also read Vite exited before listening:
with nothing after the colon, because the child's stderr was piped and never
read. A test that reports a child's death should carry its stderr and exit
status.
Waits that sample instead of listening. A fixed count of 10ms polls is a
budget with no relationship to how long the work takes, and it pays a full
tick even when the work already finished. Await the signal the system already
emits (the engine's event bus, a process's own stdout), with a wall-clock
deadline that names what it was waiting for.
Timers and promises that outlive their test. A 200ms debounced save in
the renderer store fired inside a later test that had swapped in its own
fetch, landing a stray request in that test's recorded calls. Which test
becomes the victim is decided by timing, so a loaded machine changes the
answer. Cancel pending background work between cases.

A fourth pattern worth naming: a test that races two real timers against each
other (a 20ms cadence against a 40ms window) has a 2x margin that starvation
erases. Inject the clock and advance it explicitly rather than sleeping.

A variant of the second that no deadline fixes: a wait that samples the
middle of the work. engines.test.ts polled for an idle tab's
node_modules to vanish, then asserted that the tab's bootstrap marker was
gone. The strip removed the marker in the continuation after its async
rm resolved, though (the marker now goes first, which fixes a crash window,
not this), so a 5ms poll that woke in the ~0.1ms between the two failed the
test: 1.7% of runs on an idle laptop, 3.4% under load, and red on CI on
2026-09-26. Whatever you wait on has to be the last thing the work
does. For backgrounded work, await the work itself; EngineRegistry exposes
backgroundSettled() for exactly this.

A flake that rare will not show up in a six-run full-suite hunt; this one
never did. Narrow it to the test and multiply the samples instead:
bun test <file> -t '<name>' --rerun-each 300 reproduced it 5/300 in about
a minute, and passing the same command as flake-hunt's --cmd adds the load.

One shape worth calling out separately, because no amount of isolating the test
fixes it: a test whose subject is machine-global. orphan-reaper kills any
process on the box stamped with a dead orchestrator — that is the feature — so
two copies of its real-process test reap each other's children, and the victim
fails on Received: [] with its orphan already dead, for the module doing
exactly its job. Its temp dirs and pids were already unique; the shared state was
the process table itself. An atomic mkdir lock around the one section that
spawns stamped children serialises those runs against each other and costs
concurrency nowhere else.

A fifth, and the reason some of these reports are unreadable rather than
merely red: a polling helper whose deadline is not shorter than the budget of
the test that called it. Bun's default is 5s per test. A waitFor that gives
up after 10s (or exactly 5s) never gets to report anything — bun fails the test
at its own deadline first, moves on, and the helper's rejection surfaces later
with no test to attribute it to. It is printed as unhandled error outside any test and counted separately from the failure that caused it, so one slow
shell shows up as two entries naming a file rather than one naming a case. Five
files had this inverted or equal (terminal, loop-health, login,
orphan-reaper, smoke). Rather than repeat the rule as a comment in each,
test/support/budgets.ts hands back the pair and refuses an inverted one at
import, so getting it wrong fails the file loudly instead of only when something
is slow enough for the difference to show:

const { slowTest, waitMs } = budgets(30_000, 15_000)
// slowTest declares tests carrying the 30s budget; every polling helper in the
// file gives up at waitMs, which is checked to be under it

Raising a test's budget is the right move only once you know what the test
actually spends it on. Every case above drives a real process — a pty, a bun
child, a --version probe — whose latency scales with machine load, so 5s was a
guess about machine speed rather than a statement about correctness. Where the
work was avoidable instead, it was removed: one claude-code test spawned four
stub binaries where one would do, paying a memoized-away --version per extra
binary, which made it the file's most expensive test by 2.5x and its first
casualty under load.

The first monorepo-wide sweep found the shared-state pattern twice more, both
in cli and both fixed the same way — give the run its own mkdtemp directory
instead of a name every run shares. One wrote to a fixed path inside the
working tree (__dirname/temp-test-images, deleted in afterEach, not
gitignored); the other to seven fixed os.tmpdir()/codebuff-test-* paths whose
beforeEach deleted them recursively. Overlapped they failed 6/10 and 4/10
runs; solo they always passed. Note the asymmetry that hid them: CI gives each
package's suite its own runner, so nothing there ever runs two copies at once —
these bit developers and this harness, not the pipeline.

Prefer these fixes over longer timeouts. A longer timeout makes the symptom
rarer without making the test deterministic, and it slows every honest run to
buy that. The one legitimate use is converting an iteration count into a
wall-clock deadline, since only the latter means "this is broken" rather than
"this machine is slow".

Two caveats when reading the results. A clean hunt is evidence, not proof —
absence of failure across six runs bounds the flake rate, it does not disprove
it. And a test of the form "sleep 20ms, then assert nothing has happened yet"
can only pass vacuously under load, so this harness will never flag it; those
need rewriting to assert ordering rather than absence.

The weekly flake-hunt.yml workflow runs this against freebuff-desktop — on
its pin, over the package's own test:files as ci.yml runs it — and reports
failures without blocking any PR.

A sixth shape, and the only one the hunt cannot reproduce: the budget bun
applies to a HOOK. beforeEach/afterEach are charged against the same
default a test gets (5s), and one that overruns is reported as a failing test
— a beforeEach/afterEach hook timed out for this test — so the case named is
never the case at fault, and the file's first test is what takes the hit. On
2026-08-11 that failed main: engine-lifecycle's beforeEach, 3.7ms on a
laptop and 28ms on the retry, took 5.313s on a stalled runner and failed
dispose > closes only this engine instance terminal scope, a test that asserts
one recorded call. Three overlapping suite runs under four CPU hogs never
reproduced it, because there is nothing in the test to reproduce: the hook opens
a real sqlite database and rm -rfs it again, and its latency is the machine's.
Every hook in this package that builds a real fixture (makeDb, makeRepo,
makeEngine, and worktree.test.ts's inline equivalent — 23 files) now carries
FIXTURE_HOOK_MS from test/support/budgets.ts; the tests themselves keep the
5s default, and a hook that is genuinely stuck still fails at the step timeout.
The line is cost, not the helper: measured per hook pair, makeRepo is 77ms and
makeDb/makeEngine 4.7ms, against 0.1ms for the 60-odd hooks that only
mkdtemp and rm -rf. The stall that reddened main inflated a hook ~190x (and
the test after it ~42x, which is why it reads as contention decaying rather than
one fixed pause); at 0.1ms you would need ~10,000x, so those are left alone.

That flake also cost more to find than to fix, which is its own lesson: a suite
that fails and then passes leaves the failing attempt's output buried above the
passing one, and nick-fields/retry reports only a count. test-with-guard
therefore appends each attempt's failing test names to a file under the runner's
$RUNNER_TEMP, and ci.yml's flake annotation names them. Before that,
identifying the one (fail) line meant reading an 11,014-line log.
scripts/ci/bun-test-failures.ts parses the names back out of bun's output,
including the ^ … reason line, and derives that path at both ends rather than
having ci.yml pass it: ${{ runner.temp }} is not available in a job-level
env:, and a workflow that references it there fails validation and runs ZERO
jobs — which is exactly how this landed the first time.

A seventh shape, and the one that does not care about load at all: the FILE
ORDER. The five shapes above are magnified by a busy machine, so the flake hunt
finds them. This one is decided before any test runs, by which file bun loaded
first — and bun does not load files in the order the command line lists them
(test:files sorts; the CI log does not come out sorted). So the hunt is blind
to it, the full suite passes 3573/3573 on a laptop, and CI still fails with a
different set of tests every run.

What it looked like: test-freebuff-desktop red on four main commits in a row,
once on a commit with no file changes at all, naming ~20 tests across
Tab.rename, ContextBar.menu, AccountMenu.menu, AgentPicker.menu and
QuotaBadge (since deleted) — five files with nothing in common except that they were the only
five that mount React against a real DOM. Renderers came back as the empty
string, fireEvent was handed nothing to click, and the head of the cascade was
one test sitting on await new Promise((r) => requestAnimationFrame(r)) until
bun's 5s budget.

The cause is global-jsdom deciding once per process which globals it
installs. Its KEYS list is built on the first call — every own property of the
jsdom window that is not already on globalThis — and cached at module scope for
every call after it. Two suites had left requestAnimationFrame on globalThis:
one assigned a () => 0 stub at module scope and put back only fetch, the
other "restored" it with Object.assign, which writes undefined rather than
removing the property. !(k in global) cannot tell either from a real global, so
whichever mounted suite loaded first inherited a rAF that never fires — and so
did the other four, for the rest of the run.

Note what the loser is not: the file holding the stub passes either way. Fixing
this by chasing the failing tests, or by re-splitting a suite to isolate it (as
#1382 did for the quota timers), treats the victim.

The cascade is worth understanding separately from the poisoning, because the two
were fixed by different PRs. The other four files did not fail because they
needed rAF — they failed because Tab.rename was killed mid-act() at the 5s
budget and left React's state inconsistent for everything after it. #1569 drives
those frames by hand instead of awaiting a real one, which removes the hang and
therefore the cascade. That is the trigger. The mechanism below is what stays:
verified by probe on top of #1569, a stubbed getComputedStyle or
MutationObserver is still never replaced by jsdom's, in every mounted suite,
for the whole run. Nothing awaits those today, so nothing is red — which is
exactly the shape of a defect that reappears the next time someone writes a test
that does.

Two rules come out of it:

Restore a global by DELETING the key when there was none. Capture
Object.getOwnPropertyDescriptor and Reflect.deleteProperty if it comes
back undefined. Assigning the captured undefined leaves the property in
place, which is all any in check — including global-jsdom's — looks at.
Get a DOM only from test/support/jsdom.ts. installJsdom() clears the
DOM globals before handing over, so global-jsdom makes that one-shot decision
against bun's own globals rather than against whatever ran first, and then
asserts the keys React DOM needs really did come from jsdom. A stub it does
not know about fails there, naming the key, instead of surfacing as a timeout
in a file that never touched it.

The general form is worth keeping in mind beyond jsdom: a library that caches a
decision at module scope turns "who ran first" into behaviour, and bun's one
process per package makes every test file a candidate for first. Contention is
not required, so neither flake-hunt.ts nor a green local run says anything
about it. Reproduce it instead by putting the suspected leak in a --preload,
which is exactly "some earlier file did this", and running the affected files:

bash

echo ';(globalThis as any).requestAnimationFrame = () => 0' > /tmp/leak.ts
cd freebuff-desktop && bun test \
  --preload ../test/setup-scm-loader.ts --preload ../sdk/test/setup-env.ts --preload /tmp/leak.ts \
  src/ui/shell/Tab.rename.test.tsx src/ui/agent/AgentPicker.menu.test.tsx

Our own components do it too. Freebuff Web's survey cards show once per page load
and keep that in module state behind a reset seam (resetWorkspaceProfileSurveySession,
resetWorkspaceOnboardingSurveySession). Any file that renders WorkspaceApp can
spend it. So a test that needs the card resets it in beforeEach, not only
afterEach. Otherwise its first test renders nothing whenever bun loads it after
workspace-app.dom or workspace-queue.dom (2026-10-13, sponsored-survey.dom).

mock.module() is the same hazard, and nothing in bun undoes it. Measured on
Bun 1.3.14 (2026-10-13):

mock.restore() does not restore a module mock. It restores spies. A file
that ends afterAll(() => mock.restore()) leaves its module fakes in place for
every file after it. Only clearMockedModules() from
@codebuff/common/testing/mock-modules.ts puts a module back.
The fake behaves two ways, depending on whether anything loaded the module
first. If nothing did, later importers get only the factory's exports, and
any file that imports another one fails to load:
SyntaxError: Export named 'db' not found in module '.../db/index.ts'. If an
earlier file did, bun patches the factory's exports into the loaded module in
place and keeps the rest. Later files, and earlier importers through their live
bindings, then run silently against the fake. A two-file run usually shows the
first; the full suite usually the second.
Bun walks directories breadth-first. Shallower directories come first;
within a directory, files and subdirectories come in readdir order, which is
fixed per tree on APFS and ext4 but differs between them. So
bun test leaker.test.ts victim.test.ts runs the pair in that tree order, whatever
the command line says (--reporter=junit lists it). A file can leak only into
files bun loads after it: at the same depth, later in that order, or deeper.

So inject the dependency instead: a handler that takes its collaborators (the
_post.ts + route.ts split), or an optional deps argument. To reproduce a
suspected module-mock leak, run the leaking file with the victim. If bun orders
the victim first, put the leaking file's mock.module calls in a preload beside
it, so relative paths and @/ resolve the same way. Then run the victim with
web's own preloads, since --preload on the command line replaces bunfig's list.
For the in-place case, await import() each module in that preload before
mocking it.

An eighth shape, and one where the wait is fast and the CHECK is slow: a
failing matcher handed a mounted DOM node. expect(node).toBe(other) and
expect(node).toBeNull() build their failure message by pretty-printing the
node, and bun's formatter does not stop at the element — it follows React's
__reactFiber$… expando through the whole fiber tree into jsdom's Window.
Measured on the Composer: one failed expect(document.activeElement).toBe(input)
built a 322 MB message in ~1.7s of synchronous CPU. A passing matcher formats
nothing, so this costs nothing until a check fails, and waitFor exists to
run checks that fail: it calls its callback once synchronously before
yielding, so any condition that only becomes true after a frame or a fetch pays
that price on every run, with the event loop blocked the whole time (not even
waitFor's own 1000ms timeout can fire).

That was Composer.skill-submit › "the Skills picker lets the user add
arguments before Send" (2026-09-24, #3835 and #3927): placeInComposer moves
focus in a requestAnimationFrame that fires in 17ms, and the test spent 1.7s
of its 5s budget on a laptop formatting the one failed check that preceded it —
and the whole budget on a loaded runner. The same pattern cost
ProjectSidebar's archive/restore case 1.5s. Neither looks slow in the source,
and raising the timeout would only have hidden it.

Both packages now preload a hook that prints any jsdom node as its outerHTML
(capped at 500 chars) and a window or document as its URL:
freebuff/web/test/support/jsdom-inspect.ts (#5694) and its port
freebuff-desktop/test/support/jsdom-inspect.ts. With it, that failed
expect(node).toBeNull() is a one-line message built in about a millisecond.
A test run without the package's bunfig.toml (from another directory) has no
hook and still prints the whole Window, and the helpers' messages read better,
so the rule stands: inside waitFor (or any retry), prefer
expectFocused(el) / expectAbsent(query, what) from
freebuff-desktop/test/support/dom-expect.ts over a DOM node in a matcher. To
spot the shape, time the case with bun test --reporter=junit --reporter-outfile=…: a mounted-React case taking over a second with no real
I/O is almost always this.

--- METRICS ---

Files Extracted: 18
Estimated Token Budget: ~37749 tokens
Canonical Reference: https://codewiki.google/github.com/CodebuffAI/codebuff

Technical Documentation: CodebuffAI/codebuff

Overview (`README.md`) Copy Section

Freebuff

Choose your Freebuff Copy Chapter

Quick start Copy Chapter

Models Copy Chapter

How Freebuff works Copy Chapter

Free access Copy Chapter

Contributing Copy Chapter

Built on Codebuff Copy Chapter

Links Copy Chapter

Overview (`cli/README.md`) Copy Section

@codebuff/cli

Installation Copy Chapter

Development Copy Chapter

Testing Copy Chapter

Interactive E2E Testing Copy Section

Build Copy Chapter

Run Copy Chapter

Features Copy Chapter

Overview (`freebuff/README.md`) Copy Section

Freebuff

Install Copy Chapter

Usage Copy Chapter

Project Structure Copy Chapter

Building from Source Copy Chapter

License Copy Chapter

Overview (`sdk/README.md`) Copy Section

@codebuff/sdk

Installation Copy Chapter

Prerequisites Copy Chapter

Usage Copy Chapter

Basic Example Copy Section

Example 2: Custom Agents and Tools Copy Section

API Reference Copy Chapter

Knowledge Files Copy Section

File Filtering Copy Section

`loadLocalAgents(options)` Copy Section

Parameters

Returns

Supported File Types

`client.run(options)` Copy Section

Parameters

Returns

License Copy Chapter

Overview (`cli/release-core/README.md`) Copy Section

Release launcher core

Download integrity Copy Chapter

Overview (`cli/release-staging/README.md`) Copy Section

🚀 Codecane - The most powerful coding agent (STAGING)

Installation Copy Chapter

Usage Copy Chapter

Features Copy Chapter

Knowledge Files Copy Chapter

Tips Copy Chapter

Troubleshooting Copy Chapter

Feedback Copy Chapter

Overview (`cli/release/README.md`) Copy Section

The most powerful coding agent

Installation Copy Chapter

Usage Copy Chapter

Features Copy Chapter

Knowledge Files Copy Chapter

Tips Copy Chapter

Troubleshooting Copy Chapter

Permission Errors Copy Section

Corporate Proxy / Firewall Copy Section

Feedback Copy Chapter

Overview (`evals/buffbench/README.md`) Copy Section

BuffBench

Overview Copy Chapter

Key Features Copy Section

Architecture Copy Chapter

System Components Copy Section

Evaluation Workflow Copy Section

Key Features Copy Chapter

Multi-Agent Comparison Copy Section

Advanced Analysis Copy Section

Trace Analysis (Per-Task)

Meta-Analysis (Across All Tasks)

Overview (`README.md`)

Choose your Freebuff

Quick start

Models

How Freebuff works

Free access

Contributing

Built on Codebuff

Links

Overview (`cli/README.md`)

Installation

Development

Testing

Interactive E2E Testing

Build

Run

Features

Overview (`freebuff/README.md`)

Install

Usage

Project Structure

Building from Source

License

Overview (`sdk/README.md`)

Installation

Prerequisites

Usage

Basic Example

Example 2: Custom Agents and Tools

API Reference

Knowledge Files

File Filtering

`loadLocalAgents(options)`

`client.run(options)`

License

Overview (`cli/release-core/README.md`)

Download integrity

Overview (`cli/release-staging/README.md`)

Installation

Usage

Features

Knowledge Files

Tips

Troubleshooting

Feedback

Overview (`cli/release/README.md`)

Installation

Usage

Features

Knowledge Files

Tips

Troubleshooting

Permission Errors

Corporate Proxy / Firewall

Feedback

Overview (`evals/buffbench/README.md`)

Overview

Key Features

Architecture

System Components

Evaluation Workflow

Key Features

Multi-Agent Comparison

Advanced Analysis

Comprehensive Scoring

Validation Support

External CLI Agents

Prerequisites for External Agents

Directory Structure

Usage

Running Evaluations

Creating New Evaluations

Evaluation Data Format

Evaluation File Structure

Results Format

Advanced Features

Binary Installation

Final Check Commands

Nightly Evaluations

Output Files