cocoindex Agent Skills & AI Rules

# AGENTS.md

This file provides guidance to coding agents working with code in this repository.
Claude Code compatibility is kept through `CLAUDE.md`, which imports this file.

## Build and Test Commands

This project uses [uv](https://docs.astral.sh/uv/) for Python project management.

### Building

```bash
uv run maturin develop   # Build Rust code and install Python package (required after Rust changes)
```

### Testing

```bash
cargo test               # Run Rust tests
uv run mypy              # Type check Python code
uv run pytest python/    # Run Python tests (use after both Rust and Python changes)
```

### Code Formatting and Linting

Python linting and formatting go through `prek`, which runs the `ruff-check` and
`ruff-format` hooks from `.pre-commit-config.yaml` with the exact ruff version
and file set CI uses. A `ruff` from `uv run`, `uvx`, or Homebrew is usually a
different version than the hook pins, so run the hooks rather than `ruff` itself.

```bash
uv tool install prek                         # one-time; some hooks also need `protoc` (brew install protobuf)
prek run ruff-check ruff-format --all-files  # Lint (applying safe fixes), then format Python code (same as CI)
cargo fmt                                    # Format Rust code
./dev/run_cargo_clippy.sh                    # Lint Rust code (same as CI)
```

### Pre-submission Checks

CI enforces the full pre-commit hook suite from `.pre-commit-config.yaml`, which
covers more than the commands above — notably regenerating the CLI docs
(`docs/src/content/docs/cli.mdx`) whenever `cli.py` changes. Run it locally
before pushing:

```bash
prek run --all-files
```

### Pull Requests and CI

Open a pull request as a draft and mark it ready once the checks above pass:
drafts run no CI. When `main` has a merge queue, a ready pull request runs only
the fast checks, and the queue runs the full suite on the commit that lands.
For a full run on a branch before then, use `gh workflow run CI.yml --ref <branch>`.
The CI tiers, the merge queue and stacked pull requests are described in
[CI.md](https://github.com/cocoindex-io/.github/blob/main/CI.md).

### Workflow Summary

| Change Type | Commands to Run |
|-------------|-----------------|
| Rust code only | `uv run maturin develop && cargo test` |
| Python code only | `uv run mypy && uv run pytest python/` |
| Both Rust and Python | Run all commands from both categories above |
| Python lint and format | `prek run ruff-check ruff-format --all-files` |
| Before pushing | `prek run --all-files` |

After changing a core engine API (`rust/core`), run `cargo check --workspace --all-targets` to catch call sites in every workspace crate — both `rust/py` and the Rust SDK under `rust/sdk/` call core directly, and the SDK is easy to miss when grepping. The `examples/rust/*` and `benchmarks/*/rust` crates are standalone workspaces that consume the SDK; the `prek` suite's `cargo check (rust examples + benchmarks)` hook covers those.

## Code Structure

```
cocoindex/
├── rust/                       # Rust crates (workspace)
│   ├── core/                   # Core engine crate
│   │   └── src/
│   │       ├── engine/         # Core engine
│   │       ├── state/          # States of the core engine
│   │       └── inspect/        # Database inspection utilities
│   ├── py/                     # Python bindings (PyO3); calls core engine APIs
│   ├── py_utils/               # Python-Rust utility helpers (error, convert, future)
│   ├── utils/                  # General utilities: error, batching, fingerprint, etc.
│   ├── code_ast/               # Tree-sitter foundation: language registry (links all grammars), CodeSource, positions
│   ├── ops_text/               # Text processing operations (splitter, language detection)
│   ├── code_match/             # Structural code pattern matching over tree-sitter ASTs
│   └── sdk/
│       ├── cocoindex/          # Rust SDK (`cocoindex` crate); calls core engine APIs directly, like py/
│       └── cocoindex_macros/   # Proc macros for the Rust SDK (`#[function]`)
│
├── python/
│   ├── cocoindex/              # Python package
│   │   ├── __init__.py         # Package entry point
│   │   ├── cli.py              # CLI commands
│   │   ├── _internal/          # Internal implementation for the core engine
│   │   │   ├── api.py          # Public API: mount, use_mount, mount_each, map, mount_target, App, fn, start/stop
│   │   │   ├── app.py          # App base implementation
│   │   │   ├── context_keys.py # ContextKey and ContextProvider
│   │   │   ├── environment.py  # Environment and lifespan handling
│   │   │   ├── function.py     # @coco.fn decorator implementation
│   │   │   ├── component_ctx.py # ComponentContext and component_subpath
│   │   │   ├── target_state.py # Target state implementation
│   │   │   └── core.pyi        # Type stubs for the Rust extension module (update when PyO3 APIs change)
│   │   ├── connectors/         # External system connectors (localfs, postgres, qdrant, lancedb, google_drive)
│   │   ├── connectorkits/      # Connector building utilities
│   │   ├── resources/          # Abstractions: file.py (FileLike), chunk.py (Chunk), schema.py
│   │   └── ops/                # Operations: text.py (RecursiveSplitter), sentence_transformers.py
│   └── tests/                  # Python tests
│
├── examples/                   # Example applications
├── docs/                       # Documentation
└── dev/                        # Development utilities
```

## Key Concepts

### Mental model: declarative data pipelines

CocoIndex uses a **declarative** programming model — you specify *what* your output should look like (target states), not *how* to incrementally update it. The engine handles change detection and applies minimal updates automatically.

Think of it like:

* **React**: declare UI as function of state → React re-renders what changed
* **Spreadsheets**: declare formulas → cells recompute when inputs change
* **CocoIndex**: declare target states as function of source → engine syncs what changed

### Core concepts

**App** — The top-level runnable unit. Bundles a main function with its arguments. When you call `app.update()`, the main function runs as the root processing component.

**Processing Component** — The unit of execution that owns a set of target states. Created by `mount()` or `use_mount()` at a specific component path. When a component finishes, its target states sync atomically to external systems.

**Component Path** — Stable identifier for a processing component across runs. Created via `coco.component_subpath("process", filename)`. CocoIndex uses component paths to:

* Match components to their previous runs for change detection
* Determine ownership of target states (if a path disappears, its target states are cleaned up)

**Target State** — What you want to exist in an external system (a file, a database row, a table). You *declare* target states; CocoIndex keeps them in sync — creating, updating, or removing as needed.

**Target** — The API object used to declare target states (e.g., `DirTarget`, `TableTarget`). Targets can be nested: a container target state (directory/table) provides a Target for declaring child target states (files/rows).

**Function** — A Python function decorated with `@coco.fn`. Use `memo=True` to enable memoization (skip execution when inputs and code are unchanged).

**Context** — React-style provider mechanism for sharing resources. Define keys with `ContextKey[T]`, provide values in lifespan via `builder.provide()`, use in functions via `coco.use_context(key)`.

### Key APIs

```python
# Mounting processing components (subpath auto-derived from fn.__name__)
await coco.mount(fn, *args, **kw)                                       # child runs independently
result = await coco.use_mount(fn, *args, **kw)                          # returns value directly

# Explicit subpath (for multi-part paths or multiple mounts of same function)
await coco.mount(coco.component_subpath("process", filename), fn, *args, **kw)
result = await coco.use_mount(coco.component_subpath("name"), fn, *args, **kw)

# Component subpath composition
subpath = coco.component_subpath("process", filename)  # multiple parts
subpath = coco.component_subpath("a") / "b" / "c"      # chaining with /

# Using component_subpath as context manager (applies to all nested mount calls)
with coco.component_subpath("process"):
    for f in files:
        await coco.mount(coco.component_subpath(str(f.relative_path)), process_file, f, target)

# Declaring target states (typically via Target methods)
dir_target.declare_file(filename=name, content=data)
table_target.declare_row(row=MyRow(...))

# Using context values
db = coco.use_context(PG_DB)  # retrieve value provided in lifespan

# Explicit context management (for ThreadPoolExecutor)
ctx = coco.get_component_context()
with ctx.attach():
    # coco APIs work correctly in this thread
    coco.mount(...)
```

**Mount handles:**

* `mount()` → `ComponentMountHandle`: call `await handle.ready()` to wait until target states are synced
* `use_mount()` → returns the result value directly (awaitable)

### How syncing works

When a processing component finishes, CocoIndex compares its declared target states with those from the previous run at the same component path:

* New target states → create (insert row, create file)
* Changed target states → update
* Missing target states → delete

Changes are applied atomically per component. If a source item is deleted (path no longer mounted), all its target states are cleaned up automatically.

### Example

```python
import pathlib

import cocoindex as coco
from cocoindex.connectors import localfs
from cocoindex.resources.file import FileLike, PatternFilePathMatcher

@coco.fn(memo=True)
async def process_file(file: FileLike, target: localfs.DirTarget) -> None:
    html = _markdown_it.render(await file.read_text())
    outname = "__".join(file.file_path.path.parts) + ".html"
    target.declare_file(filename=outname, content=html)

@coco.fn
async def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:
    target = await coco.use_mount(localfs.declare_dir_target, outdir)

    files = localfs.walk_dir(
        sourcedir,
        recursive=True,
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.md"]),
    )
    await coco.mount_each(process_file, files.items(), target)


app = coco.App(
    coco.AppConfig(name="FilesTransform"),
    app_main,
    sourcedir=pathlib.Path("./docs"),
    outdir=pathlib.Path("./out"),
)
app.update_blocking(report_to_stdout=True)
```

## Code Conventions

### Internal vs External Modules

We distinguish between **internal modules** (under packages with `_` prefix, e.g. `_internal.*` or `connectors.*._source`) and **external modules** (which users can directly import).

**External modules** (user-facing, e.g. `cocoindex/ops/sentence_transformers.py`):

* Be strict about not leaking implementation details
* Use `__all__` to explicitly list public exports
* Prefix ALL non-public symbols with `_`, including:
  * Standard library imports: `import threading as _threading`, `import typing as _typing`
  * Third-party imports: `import numpy as _np`, `from numpy.typing import NDArray as _NDArray`
  * Internal package imports: `from cocoindex.resources import schema as _schema`
* Exception: `TYPE_CHECKING` imports for type hints don't need prefixing

**Internal modules** (e.g. `cocoindex/_internal/component_ctx.py`, `**/_target.py`):

* Less strict since users shouldn't import these directly
* Standard library and internal imports don't need underscore prefix
* Only prefix symbols that are truly private to the module itself (e.g. `_context_var` for a module-private ContextVar)

### Minimize API surface until deemed necessary

When adding a new public API (function, class, kwarg, configuration option), prefer the smallest surface that solves the concrete need in front of you. Do not pre-expose tunables, hooks, or alternatives "in case someone needs them." Adding a kwarg later is straightforward and backwards-compatible; removing or renaming one is disruptive. If a knob is currently a hardcoded constant inside the implementation, leave it there until a real use case demands it — at which point promoting it to a kwarg is mechanical. This applies equally to optional parameters, callback hooks, parser/transformer plug-points, and configuration keys.

### General principles (also covered by `/review-changes`)

- **Top-level imports.** Defer to in-function only for a real circular dependency or a heavy import that isn't always needed.
- **Specific types over `Any`.** Use concrete types — including concrete types from third-party libraries. `Any` only when the type is truly generic and no downstream code needs to downcast it.
- **Validate and exchange early.** When a value enters as a weaker form (`str`, `Any`, raw identifier), convert to the strong type at the earliest point. Don't propagate the weak form. Concrete example in this codebase: connector handlers receive a `ContextKey` string as part of a target-state key. The parent handler resolves the key once (when constructing the child handler) and passes the typed connection directly — the child stores `_pool: asyncpg.Pool`, not `_db_key: str`.
- **`NamedTuple`/small dataclass for multi-value returns.** Access fields by name (`result.can_reuse`) at call sites.
- **Exceptions for exceptional situations only.** Reserve exceptions (Python) and errors (Rust) for truly unexpected failures — not for end-of-iteration, "not found", or other expected state transitions. Use explicit return values (sentinel, enum variant, `None`) for those.
- **Single source of truth, delete dead code, honest names.** When the same value/logic appears in multiple places, consolidate. When changes make code unreachable, delete it (and its tests, and any dead config knobs). Function and field names should describe what the implementation does today.

- Rust code follows [dev/rust-guidelines.md](dev/rust-guidelines.md).

### Testing Guidelines

We prefer end-to-end tests on user-facing APIs, over unit tests on smaller internal functions. With this said, there're cases where unit tests are necessary, e.g. for internal logic with various situations and edge cases, in which case it's usually easier to cover various scenarios with unit tests.

#### Test Environment Setup

Use `common.create_test_env(__file__)` to create a CocoIndex `Environment` for tests. It derives a unique `db_path` from the test file path and picks up the current event loop automatically.

* **Sync tests** (module-level creation): Call at module level — the Environment creates a background loop for async callbacks.

  ```python
  coco_env = common.create_test_env(__file__)
  ```

* **Async tests with async resources** (e.g., asyncpg pools): Call inside an async fixture so the Environment binds to the test's running event loop (same loop the pool is on). Use the `suffix` parameter when each test needs its own Environment:

  ```python
  @pytest_asyncio.fixture
  async def pg_env(request: pytest.FixtureRequest) -> Any:
      pool = await asyncpg.create_pool(dsn)
      coco_env = common.create_test_env(__file__, suffix=request.node.name)
      coco_env.context_provider.provide(DB_KEY, pool)
      yield pool, coco_env
      await pool.close()
  ```

  The `suffix` ensures each test gets a unique `db_path`, avoiding "environment already open" errors.

#### Testcontainers for Database Tests

Use `testcontainers[postgres]` (in the `build-test` dependency group) to spin up real database instances automatically — no manual setup or environment variables needed. Use a module-scoped sync fixture for the container and a function-scoped async fixture for per-test resources:

```python
_PG_IMAGE = "postgres:16-alpine"

pytestmark = [pytest.mark.requires_docker(_PG_IMAGE)]

@pytest.fixture(scope="module")
def pg_dsn() -> Any:
    with PostgresContainer(_PG_IMAGE) as pg:
        dsn = pg.get_connection_url().replace("postgresql+psycopg2://", "postgresql://")
        yield dsn

@pytest_asyncio.fixture
async def pool(pg_dsn: str) -> Any:
    p = await asyncpg.create_pool(pg_dsn)
    yield p
    await p.close()
```

List every image a test's containers use in its `requires_docker(...)` marker. `python/tests/conftest.py` pulls missing images after collection, before the first test runs; otherwise testcontainers pulls inside the fixture, and a cold-cache download counts against the first test's `pytest-timeout`.

### LMDB write paths

- All LMDB writes must go through `Storage::run_txn` (uses the single-writer batcher).
- Do not open a heed write txn directly or wrap the env in a separate mutex/semaphore — bypassing the batcher loses fsync coalescing and regresses concurrent-submit throughput by 10-100×.
- LMDB has no savepoints. If a sub-operation needs to "abort" without failing its caller, handle it at the body level (e.g. return a sentinel result without writing). A body that returns `Err` fails only its own caller: the batcher drops its writes by aborting the txn and re-running the bodies ahead of it, so every body must be safe to re-run.
- An LMDB write txn must begin and end (commit or abort) on the same OS thread: the writer lock is thread-owned and LMDB ignores a failed release, so a txn that migrates between runtime workers blocks every later writer for good. heed marks `RwTxn` as `Send` regardless, so the compiler won't catch it. `Storage::run_txn` runs each batch on one blocking thread for this reason; never hold a write txn across an `.await` on the multi-thread runtime.

### Sync vs Async

The Rust core (`rust/core`, `rust/utils`) uses **async-first** design with Tokio. The `rust/py` crate bridges Rust async to Python, offering both sync and async APIs:

* Rust core exposes async functions
* `rust/py` provides sync wrappers that use `block_on()` to call async Rust from sync Python
* Python's `cocoindex` API is **async-first**: mount APIs (`mount`, `use_mount`, `mount_each`, `map`) are all `async def`; `App.update()`/`App.drop()` are async; sync entry points (`App.update_blocking()`, `App.drop_blocking()`, `start_blocking()`, `stop_blocking()`) are available for CLI and blocking contexts

When adding new functionality that involves I/O or concurrency:

* Implement async in Rust
* Bridge to Python via `rust/py`, providing both sync and async variants if needed

## Versioning

The current codebase is for CocoIndex v1, which is a fundamental redesign from CocoIndex v0. CocoIndex v1 lives on the `main` branch; the legacy v0 code is preserved on the `v0` branch.

## Docs diagrams

All diagrams embedded in docs pages are built as inline Astro components under `docs/src/components/diagrams/`, not exported SVG files. See [docs/src/components/diagrams/README.md](docs/src/components/diagrams/README.md) for the palette, primitives, shared CSS classes, animation conventions, and step-by-step authoring guidelines. Follow that guide for any new or updated diagram.

## Agent Tooling

### Semantic code search over this repo (`cocoindex-code`)

To find where a concept lives, prefer semantic search over blind `grep`. This repo keeps a local
`cocoindex-code` index; query it from any agent's shell with the `ccc` CLI:

```bash
ccc search "where is X handled"      # semantic search; add --refresh to reindex first
```

If `ccc search` reports no index, set it up once:

```bash
command -v ccc >/dev/null || uv tool install 'cocoindex-code[full]'   # per machine; local embeddings, no API key
ccc init -f && ccc index                                             # repo-scoped index (first build ~1-2 min)
```

Everything is local — SQLite index, local embeddings, no keys — and a background daemon keeps it
fresh. On a brand-new machine `ccc init` asks once for an embedding provider; choose the local
default. The `-f` matters: if a parent directory is also a `ccc` project, plain `ccc init` refuses
and search silently uses the parent's index, so run `ccc status` to confirm the root is this repo.
We register no MCP server on purpose — an `ccc mcp` mode exists, but the CLI costs no context until
invoked and behaves the same across agents. Source:
[github.com/cocoindex-io/cocoindex-code](https://github.com/cocoindex-io/cocoindex-code).

## Agent Playbooks

Reusable task-specific agent guidance lives under `dev/agent-skills/`. These playbooks use a `SKILL.md` format so Claude Code can load them directly from `.claude/skills/`, but the content is intended to be readable by any coding agent.

- For docs diagrams, read `dev/agent-skills/cocoindex-diagrams/SKILL.md`.
- For new target connectors, read `dev/agent-skills/target-connector/SKILL.md`.
- For upgrading example package versions, read `dev/agent-skills/upgrade-examples/SKILL.md`.

Agent Skills-compatible clients, including Codex, discover the checked-in symlinks under `.agents/skills/`. The public CocoIndex skill is exposed there as `cocoindex`, and contributor playbooks are exposed there from `dev/agent-skills/`.

Claude Code compatibility remains under `.claude/`: hooks/settings live there, and `.claude/skills/` mirrors the contributor playbooks with symlinks. Do not duplicate skill content under `.agents/skills/` or `.claude/skills/`; edit the source directories (`skills/cocoindex/` for the public skill, `dev/agent-skills/` for contributor playbooks) and keep dot-directory entries as symlinks.

Portable agent checks live under `dev/agent-checks/`. Claude Code hooks in `.claude/hooks/` are thin adapters that call these scripts when relevant files changed.