{"owner":"cocoindex-io","repo":"cocoindex","hasSkills":true,"totalSkillsCount":12,"totalTokensCount":25278,"categories":["anthropic-skill","root-instruction","claude-rule"],"hasMcp":false,"mcpConfig":null,"found":["skills/cocoindex/references/connectors.md","skills/cocoindex/references/patterns.md","skills/cocoindex/references/setup_database.md","skills/cocoindex/references/setup_project.md","skills/cocoindex/references/api_reference.md","AGENTS.md","CLAUDE.md","skills/cocoindex/SKILL.md","examples/AGENTS.md","dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md","dev/agent-skills/cocoindex-diagrams/references/pitfalls.md","dev/agent-skills/cocoindex-diagrams/references/workflow.md"],"skills":{"skills/cocoindex/references/connectors.md":"# CocoIndex Connectors Reference\n\nComprehensive reference for all CocoIndex connectors.\n\n## Common Patterns\n\n### Target Setup\n\nAll target connectors follow this pattern:\n\n1. **Provide** connection via `ContextKey` in lifespan\n2. **Mount** target using `mount_*_target(ContextKey, ...)` convenience method\n3. **Declare** rows/points/files on the returned target object\n\nTarget connectors take a `ContextKey` (not the connection object directly) as their first argument. This enables CocoIndex to track the identity of the target across runs.\n\n### Vector Support\n\nMost connectors support vector embeddings via `VectorSchemaProvider`:\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions from ContextKey\n```\n\n---\n\n## PostgreSQL\n\n**Import**: `from cocoindex.connectors import postgres`\n\n**Capabilities**: Source + Target | **Vector**: pgvector\n\n### Connection Setup\n\n```python\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n yield\n```\n\n### As Target\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Annotated\nfrom numpy.typing import NDArray\n\n@dataclass\nclass Embedding:\n id: int\n text: str\n vector: Annotated[NDArray, EMBEDDER]\n\n# Mount table target\ntarget_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"embeddings\",\n table_schema=await postgres.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n pg_schema_name=\"my_schema\", # Optional, default \"public\"\n)\n\n# Optional: declare vector index\ntarget_table.declare_vector_index(column=\"vector\")\n\n# Declare rows\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### As Source\n\n```python\n@dataclass\nclass Record:\n id: int\n name: str\n\nsource = postgres.PgTableSource(\n coco.use_context(PG_DB),\n table_name=\"my_table\",\n row_type=Record,\n)\n\n# Iterate (async)\nfetcher = source.fetch_rows()\nasync for record in fetcher:\n ...\n\n# Keyed iteration for mount_each\nawait coco.mount_each(process_record, source.fetch_rows().items(key=lambda r: r.id), table)\n```\n\n### Type Mapping\n\n| Python | PostgreSQL |\n|--------|-----------|\n| `bool` | `boolean` |\n| `int` | `bigint` |\n| `float` | `double precision` |\n| `str` | `text` |\n| `bytes` | `bytea` |\n| `UUID` | `uuid` |\n| `datetime` | `timestamp with time zone` |\n| `list`, `dict` | `jsonb` |\n| `NDArray` + vector schema | `vector(n)` or `halfvec(n)` |\n\n### SQL Command Attachments\n\n```python\ntarget_table.declare_sql_command_attachment(\n name=\"my_index\",\n setup_sql=\"CREATE INDEX ...\",\n teardown_sql=\"DROP INDEX ...\", # Optional\n)\n```\n\n---\n\n## SQLite\n\n**Import**: `from cocoindex.connectors import sqlite`\n\n**Capabilities**: Target only | **Vector**: sqlite-vec\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import sqlite\n\nSQLITE_DB = coco.ContextKey[sqlite.ManagedConnection](\"sqlite_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n conn = sqlite.connect(\"./data.db\", load_vec=\"auto\")\n builder.provide(SQLITE_DB, conn)\n yield\n conn.close()\n```\n\n### As Target\n\n```python\ntarget_table = await sqlite.mount_table_target(\n SQLITE_DB,\n table_name=\"embeddings\",\n table_schema=await sqlite.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n)\n\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### Type Mapping\n\n| Python | SQLite |\n|--------|--------|\n| `bool` | `INTEGER` (0/1) |\n| `int` | `INTEGER` |\n| `float` | `REAL` |\n| `str` | `TEXT` |\n| `bytes` | `BLOB` |\n| `datetime` | `TEXT` (ISO format) |\n| `list`, `dict` | `TEXT` (JSON) |\n| `NDArray` + vector schema | `BLOB` (sqlite-vec) |\n\n---\n\n## LanceDB\n\n**Import**: `from cocoindex.connectors import lancedb`\n\n**Capabilities**: Target only | **Vector**: Native | **Storage**: Local or cloud (S3, GCS)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import lancedb\n\nLANCE_DB = coco.ContextKey[lancedb.LanceAsyncConnection](\"lance_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n conn = await lancedb.connect_async(\"./lancedb_data\") # or \"s3://bucket/path\"\n builder.provide(LANCE_DB, conn)\n yield\n```\n\n### As Target\n\n```python\ntarget_table = await lancedb.mount_table_target(\n LANCE_DB,\n table_name=\"embeddings\",\n table_schema=await lancedb.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n)\n\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### Type Mapping\n\n| Python | PyArrow |\n|--------|---------|\n| `bool` | `bool` |\n| `int` | `int64` |\n| `float` | `float64` |\n| `str` | `string` |\n| `bytes` | `binary` |\n| `list`, `dict` | `string` (JSON) |\n| `NDArray` + vector schema | `fixed_size_list` |\n\n---\n\n## Qdrant\n\n**Import**: `from cocoindex.connectors import qdrant`\n\n**Capabilities**: Target only | **Vector**: Native | **Model**: Point-oriented\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import qdrant\n\nfrom qdrant_client import QdrantClient\nQDRANT_DB = coco.ContextKey[QdrantClient](\"qdrant_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n client = qdrant.create_client(\"http://localhost:6333\")\n builder.provide(QDRANT_DB, client)\n yield\n```\n\n### As Target (Single Vector)\n\n```python\ncollection = await qdrant.mount_collection_target(\n QDRANT_DB,\n collection_name=\"embeddings\",\n schema=await qdrant.CollectionSchema.create(\n vectors=qdrant.QdrantVectorDef(schema=EMBEDDER, distance=\"cosine\"),\n ),\n)\n\ncollection.declare_point(point=qdrant.PointStruct(\n id=\"point-1\",\n vector=embedding_array.tolist(),\n payload={\"text\": \"hello\"},\n))\n```\n\n### As Target (Named Vectors)\n\n```python\nschema = await qdrant.CollectionSchema.create(\n vectors={\n \"text\": qdrant.QdrantVectorDef(schema=text_embedder_key, distance=\"cosine\"),\n \"image\": qdrant.QdrantVectorDef(schema=image_embedder_key, distance=\"cosine\"),\n },\n)\ncollection = await qdrant.mount_collection_target(QDRANT_DB, \"multimodal\", schema=schema)\n\ncollection.declare_point(point=qdrant.PointStruct(\n id=\"point-1\",\n vector={\"text\": text_vec.tolist(), \"image\": image_vec.tolist()},\n payload={\"title\": \"example\"},\n))\n```\n\n### Distance Metrics\n\n- `\"cosine\"` -- Cosine similarity (default)\n- `\"dot\"` -- Dot product\n- `\"euclid\"` -- Euclidean distance (L2)\n\n---\n\n## SurrealDB\n\n**Import**: `from cocoindex.connectors import surrealdb`\n\n**Capabilities**: Target only | **Vector**: Native | **Model**: Graph + document\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import surrealdb\n\nSURREAL_DB = coco.ContextKey[surrealdb.ConnectionFactory](\"surreal_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n factory = surrealdb.ConnectionFactory(\n url=\"ws://localhost:8000/rpc\",\n namespace=\"my_ns\",\n database=\"my_db\",\n credentials={\"username\": \"root\", \"password\": \"root\"},\n )\n builder.provide(SURREAL_DB, factory)\n yield\n```\n\n### As Target (Table)\n\n```python\ntarget_table = await surrealdb.mount_table_target(\n SURREAL_DB,\n table_name=\"records\",\n table_schema=await surrealdb.TableSchema.from_class(MyRecord),\n)\n\ntarget_table.declare_row(row=MyRecord(id=\"rec1\", name=\"example\"))\n```\n\n### As Target (Relation)\n\n```python\nrelation_target = await surrealdb.mount_relation_target(\n SURREAL_DB,\n table_name=\"likes\",\n from_table=users_table,\n to_table=items_table,\n table_schema=await surrealdb.TableSchema.from_class(MyRelation),\n)\nrelation_target.declare_relation(from_id=\"user:1\", to_id=\"item:1\", record=MyRelation(...))\n```\n\n---\n\n## Neo4j\n\n**Import**: `from cocoindex.connectors import neo4j`\n\n**Capabilities**: Target only | **Vector**: Native (vector index) | **Model**: Knowledge graph (nodes + relationships)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import neo4j\n\nNEO4J_DB = coco.ContextKey[neo4j.ConnectionFactory](\"neo4j_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n neo4j_db = neo4j.ConnectionFactory(\n uri=\"bolt://localhost:7687\",\n auth=(\"neo4j\", \"cocoindex\"),\n database=\"neo4j\", # Optional, default \"neo4j\"\n )\n builder.provide(NEO4J_DB, neo4j_db)\n yield\n await neo4j_db.close() # closes the driver the factory opened on this loop\n```\n\n### As Target (Nodes + Relationships)\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass Person:\n name: str\n\n@dataclass\nclass Task:\n description: str\n\n@dataclass\nclass AssignedRel:\n priority: int\n\n# Mount node tables (each label is a table)\nperson_table = await neo4j.mount_table_target(\n NEO4J_DB,\n \"Person\",\n await neo4j.TableSchema.from_class(Person, primary_key=\"name\"),\n primary_key=\"name\",\n)\ntask_table = await neo4j.mount_table_target(\n NEO4J_DB,\n \"Task\",\n await neo4j.TableSchema.from_class(Task, primary_key=\"description\"),\n primary_key=\"description\",\n)\n\n# Mount a relationship target between two node tables\nassigned_rel = await neo4j.mount_relation_target(\n NEO4J_DB, \"ASSIGNED_TO\", person_table, task_table,\n await neo4j.TableSchema.from_class(AssignedRel),\n)\n\n# Declare nodes\nperson_table.declare_record(row=Person(name=\"Alice\")) # alias: declare_row\ntask_table.declare_record(row=Task(description=\"ship v1\"))\n\n# Declare edges (record optional; PK auto-derived from endpoints when omitted)\nassigned_rel.declare_relation(\n from_id=\"Alice\", to_id=\"ship v1\", record=AssignedRel(priority=1),\n)\n```\n\n### Vector Index\n\n```python\nperson_table.declare_vector_index(\n field=\"embedding\",\n dimension=384,\n metric=\"cosine\", # or \"euclidean\"\n)\n```\n\n### Type Override\n\n```python\nfrom typing import Annotated\n\n@dataclass\nclass Row:\n id: str\n score: Annotated[float, neo4j.Neo4jType(\"decimal\", encoder=str)]\n```\n\n---\n\n## FalkorDB\n\n**Import**: `from cocoindex.connectors import falkordb`\n\n**Capabilities**: Target only | **Vector**: Native (vector index) | **Model**: Knowledge graph (nodes + relationships)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import falkordb\n\nFALKOR_DB = coco.ContextKey[falkordb.ConnectionFactory](\"falkor_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n falkor_db = falkordb.ConnectionFactory(\n uri=\"falkor://localhost:6379\",\n graph=\"knowledge_graph\", # Optional, default \"default\"\n )\n builder.provide(FALKOR_DB, falkor_db)\n yield\n await falkor_db.close() # closes the client the factory opened on this loop\n```\n\n### As Target (Nodes + Relationships)\n\n```python\nperson_table = await falkordb.mount_table_target(\n FALKOR_DB,\n \"Person\",\n await falkordb.TableSchema.from_class(Person, primary_key=\"name\"),\n primary_key=\"name\",\n)\ntask_table = await falkordb.mount_table_target(\n FALKOR_DB,\n \"Task\",\n await falkordb.TableSchema.from_class(Task, primary_key=\"description\"),\n primary_key=\"description\",\n)\n\n# Mount relationship target (no schema -> PK auto-derived from endpoints)\nassigned_rel = await falkordb.mount_relation_target(\n FALKOR_DB, \"ASSIGNED_TO\", person_table, task_table,\n)\n\n# Declare nodes\nperson_table.declare_record(row=Person(name=\"Alice\")) # alias: declare_row\ntask_table.declare_record(row=Task(description=\"ship v1\"))\n\n# Declare edges\nassigned_rel.declare_relation(from_id=\"Alice\", to_id=\"ship v1\")\n```\n\n### Vector Index\n\n```python\nperson_table.declare_vector_index(\n field=\"embedding\",\n dimension=384,\n metric=\"cosine\", # or \"euclidean\", \"ip\"\n)\n```\n\n### Type Override\n\n```python\nfrom typing import Annotated\n\n@dataclass\nclass Row:\n id: str\n score: Annotated[float, falkordb.FalkorType(\"decimal\", encoder=str)]\n```\n\n---\n\n## LocalFS\n\n**Import**: `from cocoindex.connectors import localfs`\n\n**Capabilities**: Source + Target\n\n### As Source\n\n```python\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import PatternFilePathMatcher\n\nfiles = localfs.walk_dir(\n pathlib.Path(\"./data\"),\n recursive=True,\n path_matcher=PatternFilePathMatcher(\n included_patterns=[\"**/*.py\", \"**/*.md\"],\n excluded_patterns=[\".*/**\", \"__pycache__/**\"],\n ),\n live=True, # Enable file watching for live mode\n)\n\n# Process each file\nawait coco.mount_each(process_file, files.items(), target_table)\n```\n\n### As Target (Single File)\n\n```python\nlocalfs.declare_file(\n outdir / \"output.txt\",\n \"file content\",\n create_parent_dirs=True,\n)\n```\n\n### As Target (Directory)\n\n```python\ndir_target = await localfs.mount_dir_target(pathlib.Path(\"./output\"))\ndir_target.declare_file(\"file1.txt\", \"content 1\")\ndir_target.declare_file(\"subdir/file2.txt\", \"content 2\")\n```\n\n### Stable File Paths\n\nUse `localfs.FilePath` for stable memoization when the base directory might move:\n\n```python\nfiles = localfs.walk_dir(localfs.FilePath(path=\"./data\"), ...)\n```\n\n---\n\n## Amazon S3\n\n**Import**: `from cocoindex.connectors import amazon_s3`\n\n**Capabilities**: Source only\n\n### Connection Setup\n\n```python\nimport aiobotocore.session\nfrom aiobotocore.client import AioBaseClient\n\nS3_CLIENT = coco.ContextKey[AioBaseClient](\"s3_client\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n session = aiobotocore.session.get_session()\n async with session.create_client(\"s3\") as s3_client:\n builder.provide(S3_CLIENT, s3_client)\n yield\n```\n\n### Reading Objects\n\n```python\nfrom cocoindex.connectors import amazon_s3\n\nclient = coco.use_context(S3_CLIENT)\n\n# List and iterate objects\nwalker = amazon_s3.list_objects(\n client, \"my-bucket\",\n prefix=\"data/\",\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.json\"]),\n max_file_size=10_000_000,\n)\n\nawait coco.mount_each(process_file, walker.items(), target)\n\n# Single object\nfile = await amazon_s3.get_object(client, \"s3://my-bucket/path/to/file.json\")\ncontent = await amazon_s3.read(client, \"s3://my-bucket/path/to/file.json\")\n```\n\n---\n\n## Kafka\n\n**Import**: `from cocoindex.connectors import kafka`\n\n**Capabilities**: Source + Target\n\n### As Source (Live)\n\n```python\nfrom confluent_kafka.aio import AIOConsumer\nfrom cocoindex.connectors import kafka\n\nconsumer = AIOConsumer({\n \"bootstrap.servers\": \"localhost:9092\",\n \"group.id\": \"my-group\",\n \"enable.auto.commit\": \"false\",\n \"auto.offset.reset\": \"earliest\",\n})\n\nitems = kafka.topic_as_map(consumer, [\"my-topic\"])\nawait coco.mount_each(process_message, items, target_table)\n```\n\n`topic_as_map()` returns a `LiveMapFeed`, which `mount_each()` auto-detects for live mode.\n\n### As Target\n\n```python\nfrom confluent_kafka.aio import AIOProducer\n\nKAFKA_PRODUCER = coco.ContextKey[AIOProducer](\"kafka_producer\")\n\ntopic_target = await kafka.mount_kafka_topic_target(KAFKA_PRODUCER, \"my-topic\")\ntopic_target.declare_target_state(key=\"msg-key\", value=json.dumps(data))\n```\n\n---\n\n## Apache Doris\n\n**Import**: `from cocoindex.connectors import doris`\n\n**Capabilities**: Target only | **Vector**: Native | **Features**: Vector indexes, inverted indexes, Stream Load\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import doris\n\nDORIS_DB = coco.ContextKey[doris.ManagedConnection](\"doris_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n config = doris.DorisConnectionConfig(\n fe_host=\"localhost\",\n database=\"my_db\",\n fe_http_port=8080,\n query_port=9030,\n username=\"root\",\n password=\"\",\n )\n conn = doris.connect(config)\n builder.provide(DORIS_DB, conn)\n yield\n```\n\n### As Target\n\n```python\ntarget_table = await doris.mount_table_target(\n DORIS_DB,\n table_name=\"embeddings\",\n table_schema=await doris.TableSchema.from_class(MyRecord, primary_key=[\"id\"]),\n vector_indexes=[doris.VectorIndexDef(field_name=\"embedding\", metric_type=\"cosine_distance\")],\n)\n\ntarget_table.declare_row(row=MyRecord(id=1, text=\"hello\", embedding=vec))\n```\n\n---\n\n## Google Drive\n\n**Import**: `from cocoindex.connectors import google_drive`\n\n**Capabilities**: Source only\n\n```python\nfrom cocoindex.connectors import google_drive\n\nsource = google_drive.GoogleDriveSource(\n service_account_credential_path=\"credentials.json\",\n root_folder_ids=[\"folder-id\"],\n)\nawait coco.mount_each(process_file, source.items(), target)\n```\n\n---\n\n## Connector Comparison\n\n| Connector | Source | Target | Vectors | Best For |\n|-----------|--------|--------|---------|----------|\n| **PostgreSQL** | Y | Y | pgvector | Production SQL + vectors |\n| **SQLite** | - | Y | sqlite-vec | Local SQL + vectors |\n| **LanceDB** | - | Y | Native | Cloud-native vector DB |\n| **Qdrant** | - | Y | Native | Specialized vector search |\n| **SurrealDB** | - | Y | Native | Graph + document DB |\n| **Neo4j** | - | Y | Native | Knowledge graphs |\n| **FalkorDB** | - | Y | Native | Knowledge graphs (Redis) |\n| **Apache Doris** | - | Y | Native | Analytical DB + vectors |\n| **LocalFS** | Y | Y | N/A | File-based pipelines |\n| **Amazon S3** | Y | - | N/A | Cloud object storage |\n| **Kafka** | Y | Y | N/A | Streaming pipelines |\n| **Google Drive** | Y | - | N/A | Cloud file source |\n\n---\n\n## See Also\n\n- [API Reference](./api_reference.md)\n- [Patterns](./patterns.md)\n- [Setup Database](./setup_database.md)\n","skills/cocoindex/references/patterns.md":"# CocoIndex Common Patterns\n\nCommon patterns and workflows for building CocoIndex pipelines.\n\n## Core Pattern: TargetState = Transform(SourceState)\n\nAll CocoIndex pipelines follow this declarative pattern:\n\n1. Read source state\n2. Transform\n3. Declare target state\n\nCocoIndex handles the incremental sync automatically.\n\n---\n\n## Pattern 1: File Transformation Pipeline\n\n**Use case**: Transform files from one format to another (e.g., Markdown -> HTML, PDF -> Markdown)\n\n```python\nimport pathlib\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom markdown_it import MarkdownIt\n\n_markdown_it = MarkdownIt(\"gfm-like\")\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, outdir: pathlib.Path) -> None:\n html = _markdown_it.render(await file.read_text())\n outname = \"__\".join(file.file_path.path.parts) + \".html\"\n localfs.declare_file(outdir / outname, html, create_parent_dirs=True)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n files = localfs.walk_dir(\n sourcedir,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n live=True, # Enable live file watching\n )\n await coco.mount_each(process_file, files.items(), outdir)\n\napp = coco.App(\n coco.AppConfig(name=\"FilesTransform\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n outdir=pathlib.Path(\"./output_html\"),\n)\n```\n\n**Key points:**\n- `memo=True` -- Skip reprocessing unchanged files\n- `mount_each()` -- One component per file; keys from `files.items()` become component subpaths\n- `live=True` on `walk_dir()` -- Enables file watching for live mode\n- Auto-cleanup -- Deleting source file automatically removes output file\n\n---\n\n## Pattern 2: Vector Embedding Pipeline\n\n**Use case**: Chunk and embed documents for semantic search\n\n```python\nimport pathlib\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator, Annotated\n\nimport asyncpg\nfrom numpy.typing import NDArray\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs, postgres\nfrom cocoindex.ops.text import RecursiveSplitter\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\nfrom cocoindex.resources.chunk import Chunk\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom cocoindex.resources.id import IdGenerator\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n_splitter = RecursiveSplitter()\n\n@dataclass\nclass DocEmbedding:\n id: int\n filename: str\n chunk_start: int\n chunk_end: int\n text: str\n embedding: Annotated[NDArray, EMBEDDER]\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n@coco.fn\nasync def process_chunk(\n chunk: Chunk, filename: pathlib.PurePath,\n id_gen: IdGenerator, table: postgres.TableTarget[DocEmbedding],\n) -> None:\n table.declare_row(row=DocEmbedding(\n id=await id_gen.next_id(chunk.text),\n filename=str(filename),\n chunk_start=chunk.start.char_offset,\n chunk_end=chunk.end.char_offset,\n text=chunk.text,\n embedding=await coco.use_context(EMBEDDER).embed(chunk.text),\n ))\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, table: postgres.TableTarget[DocEmbedding]) -> None:\n text = await file.read_text()\n chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language=\"markdown\")\n id_gen = IdGenerator()\n await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n target_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"doc_embeddings\",\n table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=[\"id\"]),\n )\n target_table.declare_vector_index(column=\"embedding\")\n\n files = localfs.walk_dir(sourcedir, recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]))\n await coco.mount_each(process_file, files.items(), target_table)\n\napp = coco.App(coco.AppConfig(name=\"TextEmbedding\"), app_main,\n sourcedir=pathlib.Path(\"./markdown_files\"))\n```\n\n**Key points:**\n- `mount_table_target(PG_DB, ...)` -- Takes `ContextKey` as first argument\n- `Annotated[NDArray, EMBEDDER]` -- Vector annotation uses ContextKey for auto dimension inference\n- `map()` -- Concurrent execution within a component (no child components created)\n- `IdGenerator` -- Generates stable unique IDs for chunks across incremental updates\n- `memo=True` on `process_file` -- Skips unchanged files entirely\n\n---\n\n## Pattern 3: Database Source -> Transform -> Database Target\n\n**Use case**: Transform data from one database to another\n\n```python\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator\n\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nSOURCE_DB_URL = \"postgres://localhost/source_db\"\nTARGET_DB_URL = \"postgres://localhost/target_db\"\n\nSOURCE_DB = coco.ContextKey[asyncpg.Pool](\"source_db\")\nTARGET_DB = coco.ContextKey[asyncpg.Pool](\"target_db\")\n\n@dataclass\nclass SourceRecord:\n id: int\n name: str\n value: float\n\n@dataclass\nclass TargetRecord:\n id: int\n name: str\n value: float\n processed: bool\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with (\n await asyncpg.create_pool(SOURCE_DB_URL) as source_pool,\n await asyncpg.create_pool(TARGET_DB_URL) as target_pool,\n ):\n builder.provide(SOURCE_DB, source_pool)\n builder.provide(TARGET_DB, target_pool)\n yield\n\n@coco.fn(memo=True)\nasync def process_record(record: SourceRecord, target_table: postgres.TableTarget[TargetRecord]) -> None:\n target_table.declare_row(row=TargetRecord(\n id=record.id, name=record.name.upper(),\n value=record.value * 2, processed=True,\n ))\n\n@coco.fn\nasync def app_main() -> None:\n target_table = await postgres.mount_table_target(\n TARGET_DB,\n table_name=\"target_records\",\n table_schema=await postgres.TableSchema.from_class(TargetRecord, primary_key=[\"id\"]),\n )\n\n source = postgres.PgTableSource(\n coco.use_context(SOURCE_DB),\n table_name=\"source_records\",\n row_type=SourceRecord,\n )\n await coco.mount_each(\n coco.component_subpath(\"record\"),\n process_record,\n source.fetch_rows().items(key=lambda r: r.id),\n target_table,\n )\n\napp = coco.App(coco.AppConfig(name=\"DatabaseTransform\"), app_main)\n```\n\n---\n\n## Pattern 4: LLM-Based Extraction Pipeline\n\n**Use case**: Extract structured data using LLMs\n\n```python\nimport instructor\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator\nfrom pydantic import BaseModel\nfrom litellm import acompletion\n\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\n\n_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)\n\nclass ExtractedTopic(BaseModel):\n name: str\n description: str\n\nclass ExtractionResult(BaseModel):\n title: str\n topics: list[ExtractedTopic]\n\n@dataclass\nclass Message:\n id: int\n title: str\n content: str\n\n@dataclass\nclass Topic:\n message_id: int\n name: str\n description: str\n\n@coco.fn(memo=True)\nasync def extract_and_store(\n content: str, message_id: int,\n messages_table: postgres.TableTarget[Message],\n topics_table: postgres.TableTarget[Topic],\n) -> None:\n result = await _instructor_client.chat.completions.create(\n model=\"gpt-4\",\n response_model=ExtractionResult,\n messages=[{\"role\": \"user\", \"content\": f\"Extract topics:\\n\\n{content}\"}],\n )\n messages_table.declare_row(row=Message(id=message_id, title=result.title, content=content))\n for topic in result.topics:\n topics_table.declare_row(row=Topic(\n message_id=message_id, name=topic.name, description=topic.description,\n ))\n\n@coco.fn\nasync def app_main(input_texts: list[str]) -> None:\n messages_table = await postgres.mount_table_target(\n PG_DB, table_name=\"messages\",\n table_schema=await postgres.TableSchema.from_class(Message, primary_key=[\"id\"]),\n )\n topics_table = await postgres.mount_table_target(\n PG_DB, table_name=\"topics\",\n table_schema=await postgres.TableSchema.from_class(Topic, primary_key=[\"message_id\", \"name\"]),\n )\n for idx, text in enumerate(input_texts):\n await coco.mount(\n coco.component_subpath(\"text\", idx),\n extract_and_store, text, idx, messages_table, topics_table,\n )\n\napp = coco.App(coco.AppConfig(name=\"LLMExtraction\"), app_main, input_texts=[\"text1...\", \"text2...\"])\n```\n\n**Key points:**\n- `memo=True` on extraction -- Avoids re-calling LLM for unchanged inputs\n- Multiple target tables -- Declare rows in multiple tables from single component\n- Pydantic models -- For structured LLM outputs\n\n---\n\n## Pattern 5: Kafka Streaming Pipeline\n\n**Use case**: Process Kafka messages and write to a database\n\n```python\nimport json\nfrom collections.abc import AsyncIterator\nfrom dataclasses import dataclass\n\nfrom confluent_kafka import Message\nfrom confluent_kafka.aio import AIOConsumer\n\nimport cocoindex as coco\nfrom cocoindex.connectors import kafka, lancedb\n\nLANCE_DB = coco.ContextKey[lancedb.LanceAsyncConnection](\"lance_db\")\n\n@dataclass\nclass Product:\n sku: str\n name: str\n category: str\n price: float\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n conn = await lancedb.connect_async(\"./lancedb_data\")\n builder.provide(LANCE_DB, conn)\n yield\n\n@coco.fn\nasync def process_message(msg: Message, table: lancedb.TableTarget[Product]) -> None:\n value = msg.value()\n if value is None:\n return\n row = json.loads(value.decode() if isinstance(value, bytes) else value)\n table.declare_row(row=Product(**{**row, \"price\": float(row[\"price\"])}))\n\n@coco.fn\nasync def app_main() -> None:\n products_table = await lancedb.mount_table_target(\n LANCE_DB, table_name=\"products\",\n table_schema=await lancedb.TableSchema.from_class(Product, primary_key=[\"sku\"]),\n )\n\n consumer = AIOConsumer({\n \"bootstrap.servers\": \"localhost:9092\",\n \"group.id\": \"my-group\",\n \"enable.auto.commit\": \"false\",\n \"auto.offset.reset\": \"earliest\",\n })\n items = kafka.topic_as_map(consumer, [\"products-topic\"])\n await coco.mount_each(process_message, items, products_table)\n\napp = coco.App(coco.AppConfig(name=\"KafkaToLanceDB\"), app_main)\n```\n\n**Key points:**\n- `kafka.topic_as_map()` returns a `LiveMapFeed` -- `mount_each()` auto-detects for live mode\n- Pipeline runs continuously in live mode, processing new messages as they arrive\n\n---\n\n## Pattern 6: Context Management for Shared Resources\n\n**Use case**: Share expensive resources across components\n\n```python\nimport cocoindex as coco\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\n\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\nCONFIG = coco.ContextKey[dict](\"config\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n builder.provide(CONFIG, {\"chunk_size\": 1000, \"overlap\": 200})\n yield\n\n@coco.fn\nasync def process_item(text: str) -> None:\n embedder = coco.use_context(EMBEDDER)\n config = coco.use_context(CONFIG)\n embedding = await embedder.embed(text)\n ...\n```\n\n**Key points:**\n- `detect_change=True` -- Opt in to invalidate memos when value changes (use for models/config affecting output)\n- `detect_change=False` (default) -- For resources not affecting computation (DB connections, loggers)\n- `ContextKey` name is stable identity -- avoid renaming across runs\n\n---\n\n## Common Anti-Patterns to Avoid\n\n### Missing `@coco.fn` Decorator\n\n```python\n# BAD: Missing decorator\nasync def process_file(file, table):\n table.declare_row(...)\n\n# GOOD:\n@coco.fn\nasync def process_file(file, table):\n table.declare_row(...)\n```\n\n### Reprocessing Everything\n\n```python\n# BAD: No memoization\n@coco.fn # Missing memo=True\nasync def process_file(file, table):\n embedding = await embedder.embed(await file.read_text()) # Expensive!\n\n# GOOD:\n@coco.fn(memo=True)\nasync def process_file(file, table):\n ...\n```\n\n### Unstable Component Paths\n\n```python\n# BAD: Using object references or indices\nfor file in files:\n await coco.mount(coco.component_subpath(file), ...) # Object ref\nfor idx, item in enumerate(items):\n await coco.mount(coco.component_subpath(idx), ...) # Index changes\n\n# GOOD: Use stable identifiers\nawait coco.mount_each(process_file, files.items(), table) # Keys from items()\n```\n\n### Loading Resources Per Component\n\n```python\n# BAD: Loading model in every component\n@coco.fn\nasync def process(text):\n model = SentenceTransformer(\"model\") # Loaded repeatedly!\n\n# GOOD: Load once in lifespan, use via context\nembedder = coco.use_context(EMBEDDER)\n```\n\n### Mixing Target State with Side Effects\n\n```python\n# BAD: Side effects not detected\n@coco.fn\nasync def process(data):\n requests.post(\"https://api.example.com\", json=data) # Not detected!\n\n# GOOD: Only declare target states\ntable.declare_row(row=result)\n```\n\n---\n\n## See Also\n\n- [Connectors Reference](./connectors.md)\n- [API Reference](./api_reference.md)\n- [Setup Project](./setup_project.md)\n","skills/cocoindex/references/setup_database.md":"# Database Setup Guide\n\nSetup instructions for databases used with CocoIndex.\n\n## PostgreSQL with pgvector\n\n### Using Docker (Recommended)\n\n```bash\ncat > docker-compose.yml < docker-compose.yml <=1.0.0\",\n \"sentence-transformers\",\n \"asyncpg\",\n]\n```\n\n### PostgreSQL Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"asyncpg\",\n]\n```\n\n### SQLite Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"sqlite-vec\",\n]\n```\n\n### LanceDB Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"lancedb\",\n]\n```\n\n### Qdrant Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"qdrant-client\",\n]\n```\n\n### Kafka Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"confluent-kafka\",\n]\n```\n\n### LLM-Based Extraction\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"litellm\",\n \"instructor\",\n \"pydantic>=2.0\",\n \"asyncpg\",\n]\n```\n\n---\n\n## Environment Configuration\n\n### `.env` File\n\nThe `cocoindex` CLI automatically loads `.env` from the current directory (via `find_dotenv`).\n\n```bash\n# CocoIndex internal database (optional fallback).\n# Only used if the lifespan does not set builder.settings.db_path.\n# The `cocoindex init` template sets db_path in the lifespan instead, so this is not needed there.\nCOCOINDEX_DB=./cocoindex.db\n\n# PostgreSQL (if using)\nPOSTGRES_URL=postgres://user:pass@localhost/db\n\n# Qdrant (if using)\nQDRANT_URL=http://localhost:6333\n\n# API keys (if using LLM extraction)\nOPENAI_API_KEY=your-openai-api-key\nANTHROPIC_API_KEY=your-anthropic-api-key\n```\n\n### Manual Settings (in lifespan)\n\n```python\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n builder.settings.db_path = pathlib.Path(\"./custom.db\")\n yield\n```\n\n---\n\n## Running Your Pipeline\n\n```bash\nuv sync # Install dependencies\nuv run cocoindex update main.py # Run pipeline\nuv run cocoindex update main.py -L # Run in live mode\nuv run cocoindex show main.py # Show component paths\nuv run cocoindex drop main.py -f # Reset everything\n```\n\nIf `pyproject.toml` has `[tool.uv.sources]` or `[[tool.uv.index]]` tables, install with uv. pip ignores those tables, so it would fetch the packages they pin from PyPI instead.\n\n---\n\n## Common Issues\n\n### Import Errors\n\n```bash\nuv sync\n```\n\n### Database Connection Errors\n\nVerify database is running and `.env` has correct URLs. See [setup_database.md](./setup_database.md).\n\n---\n\n## See Also\n\n- [Database Setup](./setup_database.md)\n- [Patterns](./patterns.md)\n- [API Reference](./api_reference.md)\n","skills/cocoindex/references/api_reference.md":"# CocoIndex API Reference\n\nQuick reference for the most commonly used CocoIndex APIs.\n\n## Import Convention\n\n```python\nimport cocoindex as coco\n```\n\nAll APIs live directly on the `coco` module. There is no separate `cocoindex.asyncio` module.\n\n---\n\n## `@coco.fn` Decorator\n\nMark a function as a CocoIndex processing function.\n\n```python\n@coco.fn\nasync def my_function(arg1: str) -> None: ...\n\n@coco.fn(memo=True, version=1)\nasync def expensive_fn(data: str) -> Result: ...\n\n# Force async interface for a sync function (useful for batching)\n@coco.fn.as_async(memo=True, batching=True)\ndef batch_embed(texts: list[str]) -> list[NDArray]: ...\n```\n\n**Parameters:**\n- `memo: bool = False` -- Enable memoization (skip if inputs/code unchanged)\n- `version: int | None = None` -- Explicit version bump to force re-execution\n- `logic_tracking: \"full\" | \"self\" | None = \"full\"` -- How code changes are tracked\n- `batching: bool = False` -- Auto-batch concurrent calls (async only)\n- `max_batch_size: int | None = None` -- Max batch size\n- `runner: Runner | None = None` -- e.g. `coco.GPU` for serialized GPU execution\n\n---\n\n## Mount APIs (all async)\n\nAll mount APIs accept an optional `ComponentSubpath` as their first argument. When omitted, the subpath is auto-derived from `Symbol(fn.__name__)`. Provide an explicit subpath when mounting the same function multiple times, using multi-part paths, or needing a specific path name.\n\n### `coco.mount()`\n\nMount a processing component in the background.\n\n```python\n# Subpath auto-derived from fn.__name__\nhandle = await coco.mount(processor_fn, *args, **kwargs)\n\n# Explicit subpath\nhandle = await coco.mount(coco.component_subpath(\"name\"), processor_fn, *args, **kwargs)\n\nawait handle.ready() # Optional: wait until component finishes\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `processor_fn` -- Function (or LiveComponent class) to run.\n- `*args, **kwargs` -- Arguments passed to the function.\n\n**Returns:** `ComponentMountHandle`\n\n### `coco.use_mount()`\n\nMount a dependent component and return its result. Parent depends on the child.\n\n```python\n# Subpath auto-derived from fn.__name__\nresult = await coco.use_mount(init_fn, *args, **kwargs)\n\n# Explicit subpath\nresult = await coco.use_mount(coco.component_subpath(\"setup\"), init_fn, *args, **kwargs)\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `processor_fn` -- Function to run.\n- `*args, **kwargs` -- Arguments passed to the function.\n\n**Returns:** The return value of `processor_fn`.\n\n### `coco.mount_each()`\n\nMount one component per item in a keyed iterable. Preferred for processing lists.\n\n```python\n# Subpath auto-derived from fn.__name__\nawait coco.mount_each(process_file, files.items(), *extra_args)\n\n# Explicit subpath\nawait coco.mount_each(coco.component_subpath(\"process\"), process_file, files.items(), table)\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `fn` -- Function to run per item. Item value is passed as first argument.\n- `items` -- Keyed iterable of `(StableKey, T)` pairs, or a `LiveMapFeed` for live mode.\n- `*args, **kwargs` -- Additional arguments passed to `fn` after the item.\n\n**Returns:** `ComponentMountHandle`\n\n### `coco.mount_target()`\n\nMount a target state, ensuring the container is applied before returning the child provider.\n\n```python\nprovider = await coco.mount_target(target_state)\n```\n\nPrefer connector convenience methods (`postgres.mount_table_target()`, etc.) which call this internally.\n\n### `coco.map()`\n\nRun a function concurrently on each item. No processing components are created -- pure concurrent execution within the current component.\n\n```python\nresults = await coco.map(process_chunk, chunks, *extra_args)\n```\n\n**Parameters:**\n- `fn` -- Async function to apply. Item is passed as first argument.\n- `items` -- Iterable or async iterable.\n- `*args, **kwargs` -- Additional arguments passed to `fn` after the item.\n\n**Returns:** `list[T]`\n\n### `coco.component_subpath()`\n\nCreate a stable component path for mounting.\n\n```python\ncoco.component_subpath(\"setup\")\ncoco.component_subpath(\"file\", str(file_path))\ncoco.component_subpath(\"record\", record.id)\n\n# Chaining with /\nsubpath = coco.component_subpath(\"a\") / \"b\" / \"c\"\n\n# Context manager form (applies to all nested mount calls)\nwith coco.component_subpath(\"process\"):\n for f in files:\n await coco.mount(coco.component_subpath(str(f.path)), process_file, f)\n```\n\n**StableKey types:** `str | int | bool | bytes | uuid.UUID | Symbol | tuple[StableKey, ...]`\n\n---\n\n## Context System\n\n### `coco.ContextKey[T]`\n\nType-safe key for sharing resources. The `key` string is the stable identity across runs.\n\n```python\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n```\n\n- `detect_change=True` -- Opt in to auto-invalidate dependent memos when value changes (models, configs)\n- `detect_change=False` (default) -- For resources not affecting computation (DB connections, loggers)\n\n### `builder.provide()`\n\nRegister a resource in context (used in lifespan).\n\n```python\nbuilder.provide(PG_DB, pool)\nbuilder.provide_with(KEY, context_manager) # Sync CM\nawait builder.provide_async_with(KEY, async_cm) # Async CM\n```\n\n### `coco.use_context()`\n\nRetrieve a resource from context inside a processing function.\n\n```python\npool = coco.use_context(PG_DB)\nembedder = coco.use_context(EMBEDDER)\n```\n\n---\n\n## Lifespan\n\n### `@coco.lifespan`\n\nDefine environment setup/teardown. Registered to the default environment.\n\n```python\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(MODEL))\n yield\n```\n\nCan also be sync:\n\n```python\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n builder.settings.db_path = pathlib.Path(\"./my.db\")\n yield\n```\n\n---\n\n## App\n\n### `coco.App`\n\n```python\napp = coco.App(\n coco.AppConfig(name=\"MyApp\"),\n main_fn,\n **params,\n)\n\n# Async\nawait app.update()\nhandle = app.update(live=True) # Live mode\nawait app.drop()\n\n# Sync (blocking)\napp.update_blocking(report_to_stdout=True)\napp.drop_blocking()\n```\n\n### `coco.AppConfig`\n\n```python\ncoco.AppConfig(\n name=\"MyApp\", # Required\n environment=env, # Optional: custom Environment\n max_inflight_components=1024, # Optional: concurrency limit\n)\n```\n\n### Start/Stop (for programmatic usage)\n\n```python\n# Context manager (preferred)\nasync with coco.runtime():\n await app.update()\n\n# Or manually\nawait coco.start()\ntry:\n await app.update()\nfinally:\n await coco.stop()\n\n# Sync variants\nwith coco.runtime():\n app.update_blocking()\n```\n\n---\n\n## Exception Handlers\n\n### Global (in lifespan)\n\n```python\nbuilder.set_exception_handler(my_handler)\n```\n\n### Scoped (in processing functions)\n\n```python\nasync with coco.exception_handler(my_handler):\n await coco.mount_each(process_file, files.items(), table)\n```\n\n### Handler Signature\n\n```python\nasync def my_handler(exc: BaseException, ctx: coco.ExceptionContext) -> None:\n logger.error(f\"Error in {ctx.stable_path}: {exc}\")\n```\n\n`ExceptionContext` provides: `env_name`, `stable_path`, `processor_name`, `mount_kind`, `parent_stable_path`, `is_background`, `source`, `original_exception`.\n\n---\n\n## Live Mode\n\nRun the pipeline continuously, streaming changes from sources:\n\n```python\n# CLI\ncocoindex update main.py -L\n\n# Programmatic\nhandle = app.update(live=True)\nasync for snapshot in handle.watch():\n print(snapshot.stats)\n```\n\nSources that support live mode:\n- `localfs.walk_dir(..., live=True)` -- File watching\n- `kafka.topic_as_map(consumer, topics)` -- Kafka topic consumption\n\n---\n\n## CLI Commands\n\n```bash\ncocoindex init [PROJECT_NAME] # Create new project\ncocoindex update APP_TARGET # Run app once\ncocoindex update APP_TARGET -L # Run in live mode\ncocoindex update APP_TARGET --full-reprocess # Force reprocess all\ncocoindex update APP_TARGET --reset # Reset state before running\ncocoindex drop APP_TARGET [-f] # Drop app and all state\ncocoindex ls [APP_TARGET] [--db PATH] # List apps\ncocoindex show APP_TARGET [--tree] # Show component paths\n```\n\n**APP_TARGET format:** `main.py`, `main.py:app_name`, `my_module:app_name`\n\n---\n\n## Text Operations\n\n**Import:** `from cocoindex.ops.text import ...`\n\n### `detect_code_language()`\n\n```python\nlanguage = detect_code_language(filename=\"example.py\") # -> \"python\"\n```\n\n### `RecursiveSplitter`\n\n```python\nsplitter = RecursiveSplitter()\nchunks = splitter.split(\n text,\n chunk_size=1000,\n chunk_overlap=200,\n min_chunk_size=300,\n language=\"python\", # Syntax-aware splitting\n)\n# Each chunk: Chunk(text=..., start=TextPosition(...), end=TextPosition(...))\n```\n\n### `SeparatorSplitter`\n\n```python\nsplitter = SeparatorSplitter(separators_regex=[r\"\\n\\n\", r\"\\n\"])\nchunks = splitter.split(text)\n```\n\n---\n\n## Embedding Operations\n\n**Import:** `from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder`\n\n```python\nembedder = SentenceTransformerEmbedder(\"sentence-transformers/all-MiniLM-L6-v2\")\nembedding = await embedder.embed(text) # -> NDArray (float32)\n```\n\n**As VectorSchemaProvider:**\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions\n```\n\n---\n\n## File Resources\n\n**Import:** `from cocoindex.resources.file import ...`\n\n### `FileLike` (async file object)\n\n```python\ntext = await file.read_text()\ndata = await file.read() # bytes, lazy/cached\nfp = await file.content_fingerprint()\n```\n\n### `PatternFilePathMatcher`\n\n```python\nmatcher = PatternFilePathMatcher(\n included_patterns=[\"**/*.py\", \"**/*.md\"],\n excluded_patterns=[\".*/**\", \"__pycache__/**\"],\n)\n```\n\n---\n\n## ID Generation\n\n**Import:** `from cocoindex.resources.id import ...`\n\n```python\n# Deterministic: same dep -> same ID\nchunk_id = await generate_id(chunk.text)\nchunk_uuid = generate_uuid(chunk.text)\n\n# Distinct per call (even with same dep)\nid_gen = IdGenerator()\nchunk_id = await id_gen.next_id(chunk.text)\n\nuuid_gen = UuidGenerator()\nchunk_uuid = uuid_gen.next_uuid(chunk.text)\n```\n\n---\n\n## Vector Schema\n\n**Import:** `from cocoindex.resources.schema import VectorSchema`\n\n```python\n# Via ContextKey (preferred -- auto-infer from embedder)\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER]\n\n# Explicit dimensions\nschema = VectorSchema(dtype=np.dtype(np.float32), size=384)\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, schema]\n```\n\n---\n\n## See Also\n\n- [Connectors Reference](./connectors.md) -- Database and system connectors\n- [Patterns Reference](./patterns.md) -- Common pipeline patterns\n- [Setup Project](./setup_project.md) -- Project setup guide\n- [Setup Database](./setup_database.md) -- Database setup guide\n","AGENTS.md":"# AGENTS.md\n\nThis file provides guidance to coding agents working with code in this repository.\nClaude Code compatibility is kept through `CLAUDE.md`, which imports this file.\n\n## Build and Test Commands\n\nThis project uses [uv](https://docs.astral.sh/uv/) for Python project management.\n\n### Building\n\n```bash\nuv run maturin develop # Build Rust code and install Python package (required after Rust changes)\n```\n\n### Testing\n\n```bash\ncargo test # Run Rust tests\nuv run mypy # Type check Python code\nuv run pytest python/ # Run Python tests (use after both Rust and Python changes)\n```\n\n### Code Formatting and Linting\n\nPython linting and formatting go through `prek`, which runs the `ruff-check` and\n`ruff-format` hooks from `.pre-commit-config.yaml` with the exact ruff version\nand file set CI uses. A `ruff` from `uv run`, `uvx`, or Homebrew is usually a\ndifferent version than the hook pins, so run the hooks rather than `ruff` itself.\n\n```bash\nuv tool install prek # one-time; some hooks also need `protoc` (brew install protobuf)\nprek run ruff-check ruff-format --all-files # Lint (applying safe fixes), then format Python code (same as CI)\ncargo fmt # Format Rust code\n./dev/run_cargo_clippy.sh # Lint Rust code (same as CI)\n```\n\n### Pre-submission Checks\n\nCI enforces the full pre-commit hook suite from `.pre-commit-config.yaml`, which\ncovers more than the commands above — notably regenerating the CLI docs\n(`docs/src/content/docs/cli.mdx`) whenever `cli.py` changes. Run it locally\nbefore pushing:\n\n```bash\nprek run --all-files\n```\n\n### Pull Requests and CI\n\nOpen a pull request as a draft and mark it ready once the checks above pass:\ndrafts run no CI. When `main` has a merge queue, a ready pull request runs only\nthe fast checks, and the queue runs the full suite on the commit that lands.\nFor a full run on a branch before then, use `gh workflow run CI.yml --ref `.\nThe CI tiers, the merge queue and stacked pull requests are described in\n[CI.md](https://github.com/cocoindex-io/.github/blob/main/CI.md).\n\n### Workflow Summary\n\n| Change Type | Commands to Run |\n|-------------|-----------------|\n| Rust code only | `uv run maturin develop && cargo test` |\n| Python code only | `uv run mypy && uv run pytest python/` |\n| Both Rust and Python | Run all commands from both categories above |\n| Python lint and format | `prek run ruff-check ruff-format --all-files` |\n| Before pushing | `prek run --all-files` |\n\nAfter changing a core engine API (`rust/core`), run `cargo check --workspace --all-targets` to catch call sites in every workspace crate — both `rust/py` and the Rust SDK under `rust/sdk/` call core directly, and the SDK is easy to miss when grepping. The `examples/rust/*` and `benchmarks/*/rust` crates are standalone workspaces that consume the SDK; the `prek` suite's `cargo check (rust examples + benchmarks)` hook covers those.\n\n## Code Structure\n\n```\ncocoindex/\n├── rust/ # Rust crates (workspace)\n│ ├── core/ # Core engine crate\n│ │ └── src/\n│ │ ├── engine/ # Core engine\n│ │ ├── state/ # States of the core engine\n│ │ └── inspect/ # Database inspection utilities\n│ ├── py/ # Python bindings (PyO3); calls core engine APIs\n│ ├── py_utils/ # Python-Rust utility helpers (error, convert, future)\n│ ├── utils/ # General utilities: error, batching, fingerprint, etc.\n│ ├── code_ast/ # Tree-sitter foundation: language registry (links all grammars), CodeSource, positions\n│ ├── ops_text/ # Text processing operations (splitter, language detection)\n│ ├── code_match/ # Structural code pattern matching over tree-sitter ASTs\n│ └── sdk/\n│ ├── cocoindex/ # Rust SDK (`cocoindex` crate); calls core engine APIs directly, like py/\n│ └── cocoindex_macros/ # Proc macros for the Rust SDK (`#[function]`)\n│\n├── python/\n│ ├── cocoindex/ # Python package\n│ │ ├── __init__.py # Package entry point\n│ │ ├── cli.py # CLI commands\n│ │ ├── _internal/ # Internal implementation for the core engine\n│ │ │ ├── api.py # Public API: mount, use_mount, mount_each, map, mount_target, App, fn, start/stop\n│ │ │ ├── app.py # App base implementation\n│ │ │ ├── context_keys.py # ContextKey and ContextProvider\n│ │ │ ├── environment.py # Environment and lifespan handling\n│ │ │ ├── function.py # @coco.fn decorator implementation\n│ │ │ ├── component_ctx.py # ComponentContext and component_subpath\n│ │ │ ├── target_state.py # Target state implementation\n│ │ │ └── core.pyi # Type stubs for the Rust extension module (update when PyO3 APIs change)\n│ │ ├── connectors/ # External system connectors (localfs, postgres, qdrant, lancedb, google_drive)\n│ │ ├── connectorkits/ # Connector building utilities\n│ │ ├── resources/ # Abstractions: file.py (FileLike), chunk.py (Chunk), schema.py\n│ │ └── ops/ # Operations: text.py (RecursiveSplitter), sentence_transformers.py\n│ └── tests/ # Python tests\n│\n├── examples/ # Example applications\n├── docs/ # Documentation\n└── dev/ # Development utilities\n```\n\n## Key Concepts\n\n### Mental model: declarative data pipelines\n\nCocoIndex uses a **declarative** programming model — you specify *what* your output should look like (target states), not *how* to incrementally update it. The engine handles change detection and applies minimal updates automatically.\n\nThink of it like:\n\n* **React**: declare UI as function of state → React re-renders what changed\n* **Spreadsheets**: declare formulas → cells recompute when inputs change\n* **CocoIndex**: declare target states as function of source → engine syncs what changed\n\n### Core concepts\n\n**App** — The top-level runnable unit. Bundles a main function with its arguments. When you call `app.update()`, the main function runs as the root processing component.\n\n**Processing Component** — The unit of execution that owns a set of target states. Created by `mount()` or `use_mount()` at a specific component path. When a component finishes, its target states sync atomically to external systems.\n\n**Component Path** — Stable identifier for a processing component across runs. Created via `coco.component_subpath(\"process\", filename)`. CocoIndex uses component paths to:\n\n* Match components to their previous runs for change detection\n* Determine ownership of target states (if a path disappears, its target states are cleaned up)\n\n**Target State** — What you want to exist in an external system (a file, a database row, a table). You *declare* target states; CocoIndex keeps them in sync — creating, updating, or removing as needed.\n\n**Target** — The API object used to declare target states (e.g., `DirTarget`, `TableTarget`). Targets can be nested: a container target state (directory/table) provides a Target for declaring child target states (files/rows).\n\n**Function** — A Python function decorated with `@coco.fn`. Use `memo=True` to enable memoization (skip execution when inputs and code are unchanged).\n\n**Context** — React-style provider mechanism for sharing resources. Define keys with `ContextKey[T]`, provide values in lifespan via `builder.provide()`, use in functions via `coco.use_context(key)`.\n\n### Key APIs\n\n```python\n# Mounting processing components (subpath auto-derived from fn.__name__)\nawait coco.mount(fn, *args, **kw) # child runs independently\nresult = await coco.use_mount(fn, *args, **kw) # returns value directly\n\n# Explicit subpath (for multi-part paths or multiple mounts of same function)\nawait coco.mount(coco.component_subpath(\"process\", filename), fn, *args, **kw)\nresult = await coco.use_mount(coco.component_subpath(\"name\"), fn, *args, **kw)\n\n# Component subpath composition\nsubpath = coco.component_subpath(\"process\", filename) # multiple parts\nsubpath = coco.component_subpath(\"a\") / \"b\" / \"c\" # chaining with /\n\n# Using component_subpath as context manager (applies to all nested mount calls)\nwith coco.component_subpath(\"process\"):\n for f in files:\n await coco.mount(coco.component_subpath(str(f.relative_path)), process_file, f, target)\n\n# Declaring target states (typically via Target methods)\ndir_target.declare_file(filename=name, content=data)\ntable_target.declare_row(row=MyRow(...))\n\n# Using context values\ndb = coco.use_context(PG_DB) # retrieve value provided in lifespan\n\n# Explicit context management (for ThreadPoolExecutor)\nctx = coco.get_component_context()\nwith ctx.attach():\n # coco APIs work correctly in this thread\n coco.mount(...)\n```\n\n**Mount handles:**\n\n* `mount()` → `ComponentMountHandle`: call `await handle.ready()` to wait until target states are synced\n* `use_mount()` → returns the result value directly (awaitable)\n\n### How syncing works\n\nWhen a processing component finishes, CocoIndex compares its declared target states with those from the previous run at the same component path:\n\n* New target states → create (insert row, create file)\n* Changed target states → update\n* Missing target states → delete\n\nChanges are applied atomically per component. If a source item is deleted (path no longer mounted), all its target states are cleaned up automatically.\n\n### Example\n\n```python\nimport pathlib\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, target: localfs.DirTarget) -> None:\n html = _markdown_it.render(await file.read_text())\n outname = \"__\".join(file.file_path.path.parts) + \".html\"\n target.declare_file(filename=outname, content=html)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n target = await coco.use_mount(localfs.declare_dir_target, outdir)\n\n files = localfs.walk_dir(\n sourcedir,\n recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n )\n await coco.mount_each(process_file, files.items(), target)\n\n\napp = coco.App(\n coco.AppConfig(name=\"FilesTransform\"),\n app_main,\n sourcedir=pathlib.Path(\"./docs\"),\n outdir=pathlib.Path(\"./out\"),\n)\napp.update_blocking(report_to_stdout=True)\n```\n\n## Code Conventions\n\n### Internal vs External Modules\n\nWe distinguish between **internal modules** (under packages with `_` prefix, e.g. `_internal.*` or `connectors.*._source`) and **external modules** (which users can directly import).\n\n**External modules** (user-facing, e.g. `cocoindex/ops/sentence_transformers.py`):\n\n* Be strict about not leaking implementation details\n* Use `__all__` to explicitly list public exports\n* Prefix ALL non-public symbols with `_`, including:\n * Standard library imports: `import threading as _threading`, `import typing as _typing`\n * Third-party imports: `import numpy as _np`, `from numpy.typing import NDArray as _NDArray`\n * Internal package imports: `from cocoindex.resources import schema as _schema`\n* Exception: `TYPE_CHECKING` imports for type hints don't need prefixing\n\n**Internal modules** (e.g. `cocoindex/_internal/component_ctx.py`, `**/_target.py`):\n\n* Less strict since users shouldn't import these directly\n* Standard library and internal imports don't need underscore prefix\n* Only prefix symbols that are truly private to the module itself (e.g. `_context_var` for a module-private ContextVar)\n\n### Minimize API surface until deemed necessary\n\nWhen adding a new public API (function, class, kwarg, configuration option), prefer the smallest surface that solves the concrete need in front of you. Do not pre-expose tunables, hooks, or alternatives \"in case someone needs them.\" Adding a kwarg later is straightforward and backwards-compatible; removing or renaming one is disruptive. If a knob is currently a hardcoded constant inside the implementation, leave it there until a real use case demands it — at which point promoting it to a kwarg is mechanical. This applies equally to optional parameters, callback hooks, parser/transformer plug-points, and configuration keys.\n\n### General principles (also covered by `/review-changes`)\n\n- **Top-level imports.** Defer to in-function only for a real circular dependency or a heavy import that isn't always needed.\n- **Specific types over `Any`.** Use concrete types — including concrete types from third-party libraries. `Any` only when the type is truly generic and no downstream code needs to downcast it.\n- **Validate and exchange early.** When a value enters as a weaker form (`str`, `Any`, raw identifier), convert to the strong type at the earliest point. Don't propagate the weak form. Concrete example in this codebase: connector handlers receive a `ContextKey` string as part of a target-state key. The parent handler resolves the key once (when constructing the child handler) and passes the typed connection directly — the child stores `_pool: asyncpg.Pool`, not `_db_key: str`.\n- **`NamedTuple`/small dataclass for multi-value returns.** Access fields by name (`result.can_reuse`) at call sites.\n- **Exceptions for exceptional situations only.** Reserve exceptions (Python) and errors (Rust) for truly unexpected failures — not for end-of-iteration, \"not found\", or other expected state transitions. Use explicit return values (sentinel, enum variant, `None`) for those.\n- **Single source of truth, delete dead code, honest names.** When the same value/logic appears in multiple places, consolidate. When changes make code unreachable, delete it (and its tests, and any dead config knobs). Function and field names should describe what the implementation does today.\n\n- Rust code follows [dev/rust-guidelines.md](dev/rust-guidelines.md).\n\n### Testing Guidelines\n\nWe prefer end-to-end tests on user-facing APIs, over unit tests on smaller internal functions. With this said, there're cases where unit tests are necessary, e.g. for internal logic with various situations and edge cases, in which case it's usually easier to cover various scenarios with unit tests.\n\n#### Test Environment Setup\n\nUse `common.create_test_env(__file__)` to create a CocoIndex `Environment` for tests. It derives a unique `db_path` from the test file path and picks up the current event loop automatically.\n\n* **Sync tests** (module-level creation): Call at module level — the Environment creates a background loop for async callbacks.\n\n ```python\n coco_env = common.create_test_env(__file__)\n ```\n\n* **Async tests with async resources** (e.g., asyncpg pools): Call inside an async fixture so the Environment binds to the test's running event loop (same loop the pool is on). Use the `suffix` parameter when each test needs its own Environment:\n\n ```python\n @pytest_asyncio.fixture\n async def pg_env(request: pytest.FixtureRequest) -> Any:\n pool = await asyncpg.create_pool(dsn)\n coco_env = common.create_test_env(__file__, suffix=request.node.name)\n coco_env.context_provider.provide(DB_KEY, pool)\n yield pool, coco_env\n await pool.close()\n ```\n\n The `suffix` ensures each test gets a unique `db_path`, avoiding \"environment already open\" errors.\n\n#### Testcontainers for Database Tests\n\nUse `testcontainers[postgres]` (in the `build-test` dependency group) to spin up real database instances automatically — no manual setup or environment variables needed. Use a module-scoped sync fixture for the container and a function-scoped async fixture for per-test resources:\n\n```python\n_PG_IMAGE = \"postgres:16-alpine\"\n\npytestmark = [pytest.mark.requires_docker(_PG_IMAGE)]\n\n@pytest.fixture(scope=\"module\")\ndef pg_dsn() -> Any:\n with PostgresContainer(_PG_IMAGE) as pg:\n dsn = pg.get_connection_url().replace(\"postgresql+psycopg2://\", \"postgresql://\")\n yield dsn\n\n@pytest_asyncio.fixture\nasync def pool(pg_dsn: str) -> Any:\n p = await asyncpg.create_pool(pg_dsn)\n yield p\n await p.close()\n```\n\nList every image a test's containers use in its `requires_docker(...)` marker. `python/tests/conftest.py` pulls missing images after collection, before the first test runs; otherwise testcontainers pulls inside the fixture, and a cold-cache download counts against the first test's `pytest-timeout`.\n\n### LMDB write paths\n\n- All LMDB writes must go through `Storage::run_txn` (uses the single-writer batcher).\n- Do not open a heed write txn directly or wrap the env in a separate mutex/semaphore — bypassing the batcher loses fsync coalescing and regresses concurrent-submit throughput by 10-100×.\n- LMDB has no savepoints. If a sub-operation needs to \"abort\" without failing its caller, handle it at the body level (e.g. return a sentinel result without writing). A body that returns `Err` fails only its own caller: the batcher drops its writes by aborting the txn and re-running the bodies ahead of it, so every body must be safe to re-run.\n- An LMDB write txn must begin and end (commit or abort) on the same OS thread: the writer lock is thread-owned and LMDB ignores a failed release, so a txn that migrates between runtime workers blocks every later writer for good. heed marks `RwTxn` as `Send` regardless, so the compiler won't catch it. `Storage::run_txn` runs each batch on one blocking thread for this reason; never hold a write txn across an `.await` on the multi-thread runtime.\n\n### Sync vs Async\n\nThe Rust core (`rust/core`, `rust/utils`) uses **async-first** design with Tokio. The `rust/py` crate bridges Rust async to Python, offering both sync and async APIs:\n\n* Rust core exposes async functions\n* `rust/py` provides sync wrappers that use `block_on()` to call async Rust from sync Python\n* Python's `cocoindex` API is **async-first**: mount APIs (`mount`, `use_mount`, `mount_each`, `map`) are all `async def`; `App.update()`/`App.drop()` are async; sync entry points (`App.update_blocking()`, `App.drop_blocking()`, `start_blocking()`, `stop_blocking()`) are available for CLI and blocking contexts\n\nWhen adding new functionality that involves I/O or concurrency:\n\n* Implement async in Rust\n* Bridge to Python via `rust/py`, providing both sync and async variants if needed\n\n## Versioning\n\nThe current codebase is for CocoIndex v1, which is a fundamental redesign from CocoIndex v0. CocoIndex v1 lives on the `main` branch; the legacy v0 code is preserved on the `v0` branch.\n\n## Docs diagrams\n\nAll diagrams embedded in docs pages are built as inline Astro components under `docs/src/components/diagrams/`, not exported SVG files. See [docs/src/components/diagrams/README.md](docs/src/components/diagrams/README.md) for the palette, primitives, shared CSS classes, animation conventions, and step-by-step authoring guidelines. Follow that guide for any new or updated diagram.\n\n## Agent Tooling\n\n### Semantic code search over this repo (`cocoindex-code`)\n\nTo find where a concept lives, prefer semantic search over blind `grep`. This repo keeps a local\n`cocoindex-code` index; query it from any agent's shell with the `ccc` CLI:\n\n```bash\nccc search \"where is X handled\" # semantic search; add --refresh to reindex first\n```\n\nIf `ccc search` reports no index, set it up once:\n\n```bash\ncommand -v ccc >/dev/null || uv tool install 'cocoindex-code[full]' # per machine; local embeddings, no API key\nccc init -f && ccc index # repo-scoped index (first build ~1-2 min)\n```\n\nEverything is local — SQLite index, local embeddings, no keys — and a background daemon keeps it\nfresh. On a brand-new machine `ccc init` asks once for an embedding provider; choose the local\ndefault. The `-f` matters: if a parent directory is also a `ccc` project, plain `ccc init` refuses\nand search silently uses the parent's index, so run `ccc status` to confirm the root is this repo.\nWe register no MCP server on purpose — an `ccc mcp` mode exists, but the CLI costs no context until\ninvoked and behaves the same across agents. Source:\n[github.com/cocoindex-io/cocoindex-code](https://github.com/cocoindex-io/cocoindex-code).\n\n## Agent Playbooks\n\nReusable task-specific agent guidance lives under `dev/agent-skills/`. These playbooks use a `SKILL.md` format so Claude Code can load them directly from `.claude/skills/`, but the content is intended to be readable by any coding agent.\n\n- For docs diagrams, read `dev/agent-skills/cocoindex-diagrams/SKILL.md`.\n- For new target connectors, read `dev/agent-skills/target-connector/SKILL.md`.\n- For upgrading example package versions, read `dev/agent-skills/upgrade-examples/SKILL.md`.\n\nAgent Skills-compatible clients, including Codex, discover the checked-in symlinks under `.agents/skills/`. The public CocoIndex skill is exposed there as `cocoindex`, and contributor playbooks are exposed there from `dev/agent-skills/`.\n\nClaude Code compatibility remains under `.claude/`: hooks/settings live there, and `.claude/skills/` mirrors the contributor playbooks with symlinks. Do not duplicate skill content under `.agents/skills/` or `.claude/skills/`; edit the source directories (`skills/cocoindex/` for the public skill, `dev/agent-skills/` for contributor playbooks) and keep dot-directory entries as symlinks.\n\nPortable agent checks live under `dev/agent-checks/`. Claude Code hooks in `.claude/hooks/` are thin adapters that call these scripts when relevant files changed.\n","CLAUDE.md":"# CLAUDE.md\n\n@AGENTS.md\n","skills/cocoindex/SKILL.md":"---\nname: cocoindex\ndescription: This skill should be used when building data processing pipelines with CocoIndex, a Python library for incremental data transformation. Use when the task involves processing files/data into databases, creating vector embeddings, building knowledge graphs, ETL workflows, or any data pipeline requiring automatic change detection and incremental updates. CocoIndex is Python-native (supports any Python types), has no DSL, and uses version 1.0.0 or later.\n---\n\n# CocoIndex\n\nCocoIndex is a Python library for building incremental data processing pipelines with declarative target states. Think spreadsheets or React for data pipelines: declare what the output should look like based on current input, and CocoIndex automatically handles incremental updates, change detection, and syncing to external systems.\n\n## Overview\n\nCocoIndex enables building data pipelines that:\n\n- **Automatically handle incremental updates**: Only reprocess changed data\n- **Use declarative target states**: Declare what should exist, not how to update\n- **Support any Python types**: No custom DSL -- use dataclasses, Pydantic, NamedTuple\n- **Provide function memoization**: Skip expensive operations when inputs/code unchanged\n- **Sync to multiple targets**: PostgreSQL, SQLite, LanceDB, Qdrant, SurrealDB, Apache Doris, file systems, Kafka\n\n**Key principle**: `TargetState = Transform(SourceState)`\n\n## When to Use This Skill\n\nUse this skill when building pipelines that involve:\n\n- **Document processing**: PDF/Markdown conversion, text extraction, chunking\n- **Vector embeddings**: Embedding documents/code for semantic search\n- **Database transformations**: ETL from source DB to target DB\n- **Knowledge graphs**: Extract entities and relationships from data\n- **LLM-based extraction**: Structured data extraction using LLMs\n- **File-based pipelines**: Transform files from one format to another\n- **Incremental indexing**: Keep search indexes up-to-date with source changes\n- **Streaming pipelines**: Kafka-based real-time data processing\n\n## Quick Start: Creating a New Project\n\n### Initialize Project\n\n```bash\ncocoindex init my-project\ncd my-project\n```\n\nThis creates: `main.py`, `pyproject.toml`, `README.md`. The generated `main.py` sets the database location in its lifespan via `builder.settings.db_path = pathlib.Path(\"./cocoindex.db\")`.\n\n### Add Dependencies\n\n```toml\n# For vector embeddings with PostgreSQL\ndependencies = [\"cocoindex>=1.0.0\", \"sentence-transformers\", \"asyncpg\"]\n\n# For LLM extraction\ndependencies = [\"cocoindex>=1.0.0\", \"litellm\", \"instructor\", \"pydantic>=2.0\"]\n```\n\nSee [references/setup_project.md](references/setup_project.md) for complete examples.\n\n### Run the Pipeline\n\n```bash\nuv run cocoindex update main.py # installs dependencies, then runs\n```\n\n## Core Concepts\n\n### 1. Apps\n\nAn **App** is the top-level executable that binds a main function with parameters:\n\n```python\nimport cocoindex as coco\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n ...\n\napp = coco.App(\n coco.AppConfig(name=\"MyApp\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n)\n```\n\n### 2. Functions (`@coco.fn`)\n\nThe `@coco.fn` decorator marks functions as CocoIndex processing functions. Add `memo=True` to skip re-execution when inputs/code are unchanged:\n\n```python\n@coco.fn(memo=True)\nasync def expensive_operation(data: str) -> Result:\n # LLM call, embedding generation, heavy computation\n return await expensive_transform(data)\n```\n\n**Key parameters:**\n- `memo=True` -- Enable memoization (skip if inputs/code unchanged)\n- `version=1` -- Explicit version bump to force re-execution\n- `batching=True` -- Auto-batch concurrent calls (async only)\n- `runner=coco.GPU` -- Serialize GPU-bound execution\n\n### 3. Processing Components\n\nA **processing component** groups an item's processing with its target states.\n\n**Mount components** with `mount_each()` (preferred for lists) or `mount()`:\n\n```python\n# One component per item (preferred for lists)\nawait coco.mount_each(process_file, files.items(), target_table)\n\n# Single component (subpath auto-derived from fn.__name__)\nawait coco.mount(setup_fn, arg1)\n\n# Dependent component (blocks until result returned)\nresult = await coco.use_mount(init_fn)\n\n# Explicit subpath (when you need a specific path, e.g. in loops)\nawait coco.mount(coco.component_subpath(\"item\", item_id), process_item, item)\n```\n\n**Key points:**\n- All mount APIs are `async`\n- `mount()`, `use_mount()`, and `mount_each()` auto-derive subpath from `fn.__name__`; optional explicit subpath as first arg\n- Use `use_mount()` when you need the return value\n- Use stable component paths for proper memoization and cleanup\n\n### 4. Target States\n\n**Declare** what should exist -- CocoIndex handles creation/update/deletion:\n\n```python\n# Database row target\ntable.declare_row(row=MyRecord(id=1, name=\"example\"))\n\n# File target\nlocalfs.declare_file(outdir / \"output.txt\", content, create_parent_dirs=True)\n\n# Kafka message target\ntopic_target.declare_target_state(key=\"msg-1\", value=json.dumps(data))\n```\n\n### 5. Context for Shared Resources\n\nUse `ContextKey` to share expensive resources (DB connections, models) across components:\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n# In processing functions:\nembedder = coco.use_context(EMBEDDER)\n```\n\nThe `@coco.lifespan` decorator registers the function to the default CocoIndex environment, which is shared among all apps by default. `ContextKey` also serves as the stable identity for sources/targets -- the `key` string must remain stable across runs.\n\n### 6. ID Generation\n\nGenerate stable, unique identifiers that persist across incremental updates:\n\n```python\nfrom cocoindex.resources.id import generate_id, IdGenerator\n\n# Deterministic: same dep -> same ID\nchunk_id = await generate_id(chunk.text)\n\n# Always distinct: each call -> new ID, even with same dep\nid_gen = IdGenerator()\nfor chunk in chunks:\n chunk_id = await id_gen.next_id(chunk.text)\n```\n\n### 7. Catch-Up vs Live Mode\n\nBy default, `app.update()` runs in **catch-up mode**: it scans all sources, processes what changed since the last run (memoized components are skipped), syncs target states, and returns. Each call still has to scan sources to discover changes.\n\n**Live mode** keeps the app running after catch-up and lets components stream changes continuously from their sources (e.g., file watcher, Kafka consumer), applying them with very low latency.\n\n```python\n# Enable live mode\napp.update_blocking(live=True)\n# Or: cocoindex update main.py -L\n```\n\nTwo things are needed: (1) enable live mode on the app, and (2) use a source that supports live updates.\n\n- **`LiveMapView`** sources (e.g., `localfs.walk_dir(..., live=True)`) scan current state first, then watch for changes. They also work in catch-up mode -- write the pipeline once, choose mode at run time.\n- **`LiveMapFeed`** sources (e.g., `kafka.topic_as_map()`) only stream changes with no initial snapshot. `mount_each()` auto-detects these and creates a live component internally.\n\n```python\n# LocalFS with live watching\nfiles = localfs.walk_dir(sourcedir, live=True, ...)\nawait coco.mount_each(process_file, files.items(), target)\n\n# Kafka -- inherently live\nitems = kafka.topic_as_map(consumer, [\"my-topic\"])\nawait coco.mount_each(process_message, items, target)\n```\n\n## Common Pipeline Patterns\n\n### Pattern 1: File Transformation\n\n```python\nimport pathlib\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, outdir: pathlib.Path) -> None:\n content = await file.read_text()\n transformed = transform_content(content)\n outname = file.file_path.path.stem + \".out\"\n localfs.declare_file(outdir / outname, transformed, create_parent_dirs=True)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n files = localfs.walk_dir(\n sourcedir,\n recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n )\n await coco.mount_each(process_file, files.items(), outdir)\n\napp = coco.App(\n coco.AppConfig(name=\"Transform\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n outdir=pathlib.Path(\"./out\"),\n)\n```\n\n### Pattern 2: Vector Embedding Pipeline\n\n```python\nimport pathlib\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator, Annotated\n\nimport asyncpg\nfrom numpy.typing import NDArray\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs, postgres\nfrom cocoindex.ops.text import RecursiveSplitter\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\nfrom cocoindex.resources.chunk import Chunk\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom cocoindex.resources.id import IdGenerator\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n_splitter = RecursiveSplitter()\n\n@dataclass\nclass DocEmbedding:\n id: int\n filename: str\n text: str\n embedding: Annotated[NDArray, EMBEDDER] # Dimensions inferred from ContextKey\n chunk_start: int\n chunk_end: int\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n@coco.fn\nasync def process_chunk(\n chunk: Chunk, filename: pathlib.PurePath,\n id_gen: IdGenerator, table: postgres.TableTarget[DocEmbedding],\n) -> None:\n table.declare_row(row=DocEmbedding(\n id=await id_gen.next_id(chunk.text),\n filename=str(filename),\n text=chunk.text,\n embedding=await coco.use_context(EMBEDDER).embed(chunk.text),\n chunk_start=chunk.start.char_offset,\n chunk_end=chunk.end.char_offset,\n ))\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, table: postgres.TableTarget[DocEmbedding]) -> None:\n text = await file.read_text()\n chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500)\n id_gen = IdGenerator()\n await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n target_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"embeddings\",\n table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=[\"id\"]),\n )\n target_table.declare_vector_index(column=\"embedding\")\n\n files = localfs.walk_dir(sourcedir, recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]))\n await coco.mount_each(process_file, files.items(), target_table)\n\napp = coco.App(coco.AppConfig(name=\"Embedding\"), app_main, sourcedir=pathlib.Path(\"./data\"))\n```\n\n### Pattern 3: LLM-Based Extraction\n\n```python\nimport instructor\nfrom pydantic import BaseModel\nfrom litellm import acompletion\n\n_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)\n\nclass ExtractionResult(BaseModel):\n title: str\n topics: list[str]\n\n@coco.fn(memo=True) # Memo avoids re-calling LLM\nasync def extract_and_store(content: str, message_id: int, table) -> None:\n result = await _instructor_client.chat.completions.create(\n model=\"gpt-4\",\n response_model=ExtractionResult,\n messages=[{\"role\": \"user\", \"content\": f\"Extract topics: {content}\"}],\n )\n table.declare_row(row=Message(id=message_id, title=result.title, content=content))\n```\n\n## Connectors and Operations\n\nCocoIndex provides connectors for reading from and writing to external systems:\n\n| Connector | Source | Target | Vectors | Use Case |\n|-----------|--------|--------|---------|----------|\n| **PostgreSQL** | Y | Y | pgvector | Production SQL + vectors |\n| **SQLite** | - | Y | sqlite-vec | Local SQL + vectors |\n| **LanceDB** | - | Y | Y | Cloud-native vector DB |\n| **Qdrant** | - | Y | Y | Specialized vector DB |\n| **SurrealDB** | - | Y | Y | Graph + document DB |\n| **Apache Doris** | - | Y | Y | Analytical DB + vectors |\n| **LocalFS** | Y | Y | N/A | File-based pipelines |\n| **Amazon S3** | Y | - | N/A | Cloud object storage |\n| **Kafka** | Y | Y | N/A | Streaming pipelines |\n| **Google Drive** | Y | - | N/A | Cloud file source |\n\nFor detailed connector documentation, see [references/connectors.md](references/connectors.md).\n\n## Text and Embedding Operations\n\n### Text Splitting\n\n```python\nfrom cocoindex.ops.text import RecursiveSplitter, detect_code_language\n\nsplitter = RecursiveSplitter()\nlanguage = detect_code_language(filename=\"example.py\")\nchunks = splitter.split(text, chunk_size=1000, chunk_overlap=200, language=language)\n```\n\n### Embeddings\n\n```python\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\n\nembedder = SentenceTransformerEmbedder(\"sentence-transformers/all-MiniLM-L6-v2\")\nembedding = await embedder.embed(text) # Returns NDArray\n```\n\n## CLI Commands\n\n```bash\ncocoindex init my-project # Create new project\ncocoindex update main.py # Run app\ncocoindex update main.py:my_app # Run specific app\ncocoindex update main.py -L # Run in live mode (continuous)\ncocoindex update main.py --full-reprocess # Reprocess everything\ncocoindex drop main.py [-f] # Drop and reset all state\ncocoindex ls [main.py] # List apps\ncocoindex show main.py [--tree] # Show component paths\n```\n\n## Best Practices\n\n### 1. Use `@coco.fn` on All Processing Functions\n\nEvery function that participates in the pipeline (declares target states, calls mount APIs, etc.) must be decorated with `@coco.fn`.\n\n### 2. Add Memoization for Expensive Operations\n\n```python\n@coco.fn(memo=True) # Skip re-execution when inputs/code unchanged\nasync def process_chunk(chunk, table):\n embedding = await embedder.embed(chunk.text) # Expensive!\n table.declare_row(...)\n```\n\n### 3. Use Stable Component Paths\n\n```python\n# Good: Stable identifiers\ncoco.component_subpath(\"file\", str(file.file_path.path))\ncoco.component_subpath(\"record\", record.id)\n\n# Bad: Unstable identifiers\ncoco.component_subpath(\"file\", file) # Object reference\ncoco.component_subpath(\"idx\", idx) # Index changes\n```\n\n### 4. Use Context for Shared Resources\n\n```python\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n yield\n```\n\n### 5. Use `Annotated[NDArray, CONTEXT_KEY]` for Vectors\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions from ContextKey\n```\n\n### 6. Use Convenience APIs for Targets\n\n```python\n# Mount table target -- subpath is automatic\ntable = await postgres.mount_table_target(\n PG_DB,\n table_name=\"my_table\",\n table_schema=await postgres.TableSchema.from_class(MyRecord, primary_key=[\"id\"]),\n)\n```\n\n## Troubleshooting\n\n### Everything Reprocessing\n\nAdd `memo=True` to expensive functions:\n\n```python\n@coco.fn(memo=True) # Add this\nasync def process_item(item):\n ...\n```\n\n### Memoization Not Working\n\nCheck component paths are stable. Use stable IDs, not object references.\n\n## Resources\n\n### references/\n\n- **[api_reference.md](references/api_reference.md)**: Quick API reference\n- **[connectors.md](references/connectors.md)**: Complete connector reference\n- **[patterns.md](references/patterns.md)**: Detailed pipeline patterns\n- **[setup_project.md](references/setup_project.md)**: Project setup guide\n- **[setup_database.md](references/setup_database.md)**: Database setup guide\n\n### Runnable examples\n\nEvery pattern above has a complete, runnable app under\n[`examples/`](https://github.com/cocoindex-io/cocoindex/tree/main/examples) — start\nfrom the one closest to the task and adapt it. Each has its own `README.md` and\nmost Python examples have a `.env.example`; see\n[`examples/AGENTS.md`](https://github.com/cocoindex-io/cocoindex/blob/main/examples/AGENTS.md)\nfor the full map, credentials, and per-example run commands. Good starting points:\n\n- Vector search → `text_embedding` (Postgres), `text_embedding_qdrant` / `_lancedb` (other stores)\n- Code search → `code_embedding`\n- LLM extraction → `hn_trending_topics`, `patient_intake_extraction_baml`\n- Knowledge graph → `conversation_to_knowledge`, `meeting_notes_graph_neo4j`\n- Custom transform → `files_transform`, `pdf_to_markdown`\n\n### External\n\n- [CocoIndex Documentation](https://cocoindex.io/docs/) — full text at [llms-full.txt](https://cocoindex.io/docs/llms-full.txt)\n- [GitHub Examples](https://github.com/cocoindex-io/cocoindex/tree/main/examples)\n\n## Version Note\n\nThis skill is for CocoIndex `>=1.0.0` (v1). It uses a completely different API from v0.\n\n**v0 code is what you likely learned from training data — do not emit it.** If you find yourself writing any of these symbols, you are using the removed v0 API:\n\n| v0 (removed) | v1 equivalent |\n|---|---|\n| `@cocoindex.flow_def`, `FlowBuilder`, `Flow`, `open_flow` | `coco.App` + a `@coco.fn` main function |\n| `DataScope`, `DataSlice`, `add_collector()`, `collect()`, `export()` | declare target states via Target APIs (`declare_row`, `declare_file`) inside mounted components |\n| `cocoindex.sources.LocalFile`, `cocoindex.sources.*` | connector APIs, e.g. `localfs.walk_dir(...)` |\n| `cocoindex.functions.SplitRecursively`, `cocoindex.functions.*` | `cocoindex.ops.*`, e.g. `RecursiveSplitter` |\n| `cocoindex.targets.Postgres`, `cocoindex.targets.*` / `storages.*` | connector targets, e.g. `postgres.declare_table_target(...)` |\n| `transform_flow`, `cocoindex.op.function()` | plain `@coco.fn` functions |\n| `cocoindex.init()`, `settings`, `COCOINDEX_DATABASE_URL` | `coco.App(coco.AppConfig(...))`; state lives in a local db path |\n| CLI `cocoindex setup` | no setup step — just `cocoindex update` (`-L`/`--live` for live mode) |\n\nWhen reading third-party tutorials or model memory that mention these v0 symbols, disregard them and use the patterns in this skill and `references/api_reference.md` instead.\n","examples/AGENTS.md":"# AGENTS.md — CocoIndex examples\n\nGuidance for AI coding agents (Claude Code, Cursor, etc.) working in this `examples/`\ndirectory. Most top-level Python subfolders are self-contained, runnable\nCocoIndex **v1** apps; `rust/` contains Rust ports with per-example READMEs.\n\n## Before you write CocoIndex code: install the skill\n\nCocoIndex v1 is a fundamental redesign from v0. Without context, LLMs tend to\nhallucinate the v0 flow-builder DSL and deprecated decorators. Install the\nofficial skill first — it teaches the correct v1 API. Quickest portable path\nfor Codex and other Agent Skills-compatible clients is the hosted skill:\n\n```sh\nmkdir -p .agents/skills/cocoindex/references\ncurl -fsSL https://cocoindex.io/docs/skill.md -o .agents/skills/cocoindex/SKILL.md\nfor f in api_reference connectors patterns setup_database setup_project; do\n curl -fsSL https://cocoindex.io/docs/references/$f.md -o .agents/skills/cocoindex/references/$f.md\ndone\n```\n\nOr clone the repo and copy the folder:\n\n```sh\ngit clone --depth=1 https://github.com/cocoindex-io/cocoindex.git /tmp/cocoindex-skill\nmkdir -p .agents/skills && cp -r /tmp/cocoindex-skill/skills/cocoindex .agents/skills/\n```\n\nFor Claude Code's native project path, use `.claude/skills/cocoindex` instead\nof `.agents/skills/cocoindex`; the skill format is the same.\n\nThe skill itself lives at [`skills/cocoindex/`](../skills/cocoindex) (SKILL.md +\n`references/`). For Cursor, copy `SKILL.md` into `.cursor/rules/`. Full machine-readable\ndocs: (index) and\n (everything, including example\nwalkthroughs). Regular docs pages also have raw Markdown twins by replacing the\ntrailing slash with `.md` (e.g.\n`https://cocoindex.io/docs/programming_guide/core_concepts.md`).\n\n## The v1 mental model\n\n`target_state = transform(source_state)`. You declare what the target should look\nlike; the Rust engine keeps it in sync, reprocessing only what changed (state is\ntracked in a local LMDB store — **no database is required for the engine itself**,\nonly when an example writes to one). Key APIs: `@coco.fn`, `mount` / `use_mount` /\n`mount_each`, `ContextKey`, target-state declarations. See the skill for details.\n\n## Running examples\n\nMost Python examples are standalone projects with their own `pyproject.toml`:\n\n```sh\ncd \ncp .env.example .env # if present — fill in the blanks (see below)\npip install -e . # or: uv pip install -e .\ncocoindex update main # catch-up: scan sources, sync, exit\ncocoindex update -L main # live mode: catch up, then watch for changes (where supported)\n```\n\nUse the example's README as the source of truth. Known exceptions:\n\n- `multi_codebase_summarization`, `audio_to_text`,\n `patient_intake_extraction_baml`, `patient_intake_extraction_dspy`:\n `cocoindex update main.py`\n- `conversation_to_knowledge`: `cocoindex update conv_knowledge.app`\n- `image_search`, `image_search_colpali`: start the API with\n `python -m uvicorn api:app --reload --host 0.0.0.0 --port 8000`; it runs the\n CocoIndex app in live mode, then start the frontend from `frontend/`.\n- `csv_to_kafka`, `kafka_to_lancedb`: `cocoindex update -L main.py`\n- `rust/`: follow that example's README. Many use\n `cargo run -- index` for indexing and `cargo run -- query \"...\"` for search;\n a few take custom paths or service-specific subcommands.\n\nSome examples expose a query/CLI demo via `python main.py \"\"`; check the\nexample's `README.md`. Examples that need extra services or a code-gen step\n(e.g. `baml generate`) say so in their README.\n\n## Environment / credentials\n\nWhen an example needs credentials or service configuration, required env vars\nare templated in that example's **`.env.example`** — `cp` it to `.env` and fill\nin the blanks; both `python main.py` and the `cocoindex` CLI load `.env`\nautomatically. Common ones:\n\n- `POSTGRES_URL` — for Postgres/pgvector targets. Local instance:\n `docker compose -f ../../dev/postgres.yaml up -d` from inside an example\n directory.\n- `OPENAI_API_KEY` / `GEMINI_API_KEY` — for examples that call an LLM.\n- Service-specific (`QDRANT_URL`, `LANCEDB_URI`, `NEO4J_*`, `KAFKA_*`,\n `GOOGLE_SERVICE_ACCOUNT_CREDENTIAL`, …) — see that example's `.env.example`.\n\nExamples with no `.env.example` (e.g. `files_transform`, `pdf_to_markdown`) run\nfully locally with no credentials.\n\n**Never commit secrets.** The `.env` files tracked in this repo hold only\nnon-secret defaults (`COCOINDEX_DB`, local service URLs); keep API keys and\ncredentials in your local `.env` edits and out of commits.\n\n## The examples\n\nA walkthrough URL means there's a step-by-step guide at\n`https://cocoindex.io/docs/examples//`; otherwise start from the example's README.\n\n### Vector indexes (embed → store → search by meaning)\n- `text_embedding` — Markdown → pgvector; the simplest end-to-end index. *(walkthrough: text-embedding)*\n- `code_embedding` — repo → Tree-sitter chunks → pgvector; query code in English. *(walkthrough: index-codebase)*\n- `text_embedding_qdrant` / `text_embedding_lancedb` / `text_embedding_turbopuffer` — same flow, different vector store.\n- `code_embedding_lancedb` — code chunks → LanceDB.\n- `pdf_embedding` — PDFs → markdown → chunks → pgvector.\n- `paper_metadata` — extract title/authors/abstract from PDFs → Postgres + embeddings.\n- `amazon_s3_embedding` / `gdrive_text_embedding` / `oci_object_storage_embedding` — same flow, remote source (S3 / Google Drive / OCI).\n- `postgres_source` — read from an existing Postgres table as the source.\n- `entire_session_search` — semantic search over AI coding sessions captured by Entire.\n- `sec_edgar_analytics` — multi-format SEC filings → Apache Doris with a vector **and** a full-text index; hybrid (semantic + keyword) RRF search. *(walkthrough: sec-edgar-analytics)*\n\n### Multimodal\n- `image_search` — CLIP embeddings + Qdrant, queried via FastAPI + React.\n- `image_search_colpali` — ColPali multi-vector model + Qdrant MaxSim.\n- `multi_format_indexing` — PDFs + images as page screenshots → ColPali → Qdrant; no OCR, no chunking. *(walkthrough: multi-format-indexing)*\n- `face_recognition` — detect faces (dlib) → 128-d embeddings → Qdrant face search. *(walkthrough: face-recognition)*\n- `audio_to_text` — transcribe audio with LiteLLM → Postgres.\n- `slides_to_speech` — slides → vision-LLM notes → Pocket TTS narration → LanceDB semantic search. *(walkthrough: slides-to-speech)*\n\n### Structured extraction (LLM / BAML / DSPy)\n- `multi_codebase_summarization` — LLM per-file summaries across many repos. *(walkthrough: multi-codebase-summarization)*\n- `hn_trending_topics` — scrape HackerNews → LLM topic extraction → Postgres.\n- `manuals_llm_extraction` — PDF manuals → Markdown (docling) → typed module records → Postgres. *(walkthrough: manuals-llm-extraction)*\n- `patient_intake_extraction_baml` / `patient_intake_extraction_dspy` — structured PDF extraction with BAML / DSPy (Gemini vision).\n\n### Knowledge graphs\n- `conversation_to_knowledge` — YouTube podcasts → SurrealDB knowledge graph. *(walkthrough: podcast-to-knowledge-graph)*\n- `docs_to_knowledge_graph` — Markdown docs → Neo4j concept graph of LLM-extracted triples. *(walkthrough: docs-to-knowledge-graph)*\n- `product_recommendation` — product catalog → LLM taxonomy extraction → Neo4j recommendation graph. *(walkthrough: product-recommendation)*\n- `meeting_notes_graph_neo4j` / `meeting_notes_graph_falkordb` — Google Drive meeting notes → Neo4j / FalkorDB graph.\n- `meeting_notes_graph_surrealdb` — local Markdown meeting notes → SurrealDB graph.\n\n### Custom sources / targets / streaming\n- `pdf_to_markdown` — incremental PDF → Markdown with docling (local, no services). *(walkthrough: pdf-to-markdown)*\n- `files_transform` — watch Markdown files → HTML, live mode (local, no services).\n- `csv_to_kafka` — watch CSVs → publish rows to Kafka.\n- `kafka_to_lancedb` — consume Kafka → route to LanceDB tables.\n\n### Rust\n- `rust/` — Rust ports of many of the above, using the CocoIndex Rust API.\n\n## Conventions for edits\n\n- Keep each Python example self-contained: its own `pyproject.toml` and\n `README.md`; add `.env.example` when credentials or configurable services are\n required. When you add an example, also add a line to `EXAMPLE_CATALOG` in the\n docs repo (`docs/src/data/examples.ts`) so it appears in `/docs/llms.txt`.\n- Match the surrounding code's low comment density.\n- Don't commit generated artifacts (`cocoindex.db`, `__pycache__`, build output) —\n they're already git-ignored.\n","dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md)","dev/agent-skills/cocoindex-diagrams/references/pitfalls.md":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/pitfalls.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/pitfalls.md)","dev/agent-skills/cocoindex-diagrams/references/workflow.md":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/workflow.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/workflow.md)"},"files":{"skills/cocoindex/references/connectors.md":"# CocoIndex Connectors Reference\n\nComprehensive reference for all CocoIndex connectors.\n\n## Common Patterns\n\n### Target Setup\n\nAll target connectors follow this pattern:\n\n1. **Provide** connection via `ContextKey` in lifespan\n2. **Mount** target using `mount_*_target(ContextKey, ...)` convenience method\n3. **Declare** rows/points/files on the returned target object\n\nTarget connectors take a `ContextKey` (not the connection object directly) as their first argument. This enables CocoIndex to track the identity of the target across runs.\n\n### Vector Support\n\nMost connectors support vector embeddings via `VectorSchemaProvider`:\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions from ContextKey\n```\n\n---\n\n## PostgreSQL\n\n**Import**: `from cocoindex.connectors import postgres`\n\n**Capabilities**: Source + Target | **Vector**: pgvector\n\n### Connection Setup\n\n```python\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n yield\n```\n\n### As Target\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Annotated\nfrom numpy.typing import NDArray\n\n@dataclass\nclass Embedding:\n id: int\n text: str\n vector: Annotated[NDArray, EMBEDDER]\n\n# Mount table target\ntarget_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"embeddings\",\n table_schema=await postgres.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n pg_schema_name=\"my_schema\", # Optional, default \"public\"\n)\n\n# Optional: declare vector index\ntarget_table.declare_vector_index(column=\"vector\")\n\n# Declare rows\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### As Source\n\n```python\n@dataclass\nclass Record:\n id: int\n name: str\n\nsource = postgres.PgTableSource(\n coco.use_context(PG_DB),\n table_name=\"my_table\",\n row_type=Record,\n)\n\n# Iterate (async)\nfetcher = source.fetch_rows()\nasync for record in fetcher:\n ...\n\n# Keyed iteration for mount_each\nawait coco.mount_each(process_record, source.fetch_rows().items(key=lambda r: r.id), table)\n```\n\n### Type Mapping\n\n| Python | PostgreSQL |\n|--------|-----------|\n| `bool` | `boolean` |\n| `int` | `bigint` |\n| `float` | `double precision` |\n| `str` | `text` |\n| `bytes` | `bytea` |\n| `UUID` | `uuid` |\n| `datetime` | `timestamp with time zone` |\n| `list`, `dict` | `jsonb` |\n| `NDArray` + vector schema | `vector(n)` or `halfvec(n)` |\n\n### SQL Command Attachments\n\n```python\ntarget_table.declare_sql_command_attachment(\n name=\"my_index\",\n setup_sql=\"CREATE INDEX ...\",\n teardown_sql=\"DROP INDEX ...\", # Optional\n)\n```\n\n---\n\n## SQLite\n\n**Import**: `from cocoindex.connectors import sqlite`\n\n**Capabilities**: Target only | **Vector**: sqlite-vec\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import sqlite\n\nSQLITE_DB = coco.ContextKey[sqlite.ManagedConnection](\"sqlite_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n conn = sqlite.connect(\"./data.db\", load_vec=\"auto\")\n builder.provide(SQLITE_DB, conn)\n yield\n conn.close()\n```\n\n### As Target\n\n```python\ntarget_table = await sqlite.mount_table_target(\n SQLITE_DB,\n table_name=\"embeddings\",\n table_schema=await sqlite.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n)\n\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### Type Mapping\n\n| Python | SQLite |\n|--------|--------|\n| `bool` | `INTEGER` (0/1) |\n| `int` | `INTEGER` |\n| `float` | `REAL` |\n| `str` | `TEXT` |\n| `bytes` | `BLOB` |\n| `datetime` | `TEXT` (ISO format) |\n| `list`, `dict` | `TEXT` (JSON) |\n| `NDArray` + vector schema | `BLOB` (sqlite-vec) |\n\n---\n\n## LanceDB\n\n**Import**: `from cocoindex.connectors import lancedb`\n\n**Capabilities**: Target only | **Vector**: Native | **Storage**: Local or cloud (S3, GCS)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import lancedb\n\nLANCE_DB = coco.ContextKey[lancedb.LanceAsyncConnection](\"lance_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n conn = await lancedb.connect_async(\"./lancedb_data\") # or \"s3://bucket/path\"\n builder.provide(LANCE_DB, conn)\n yield\n```\n\n### As Target\n\n```python\ntarget_table = await lancedb.mount_table_target(\n LANCE_DB,\n table_name=\"embeddings\",\n table_schema=await lancedb.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n)\n\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### Type Mapping\n\n| Python | PyArrow |\n|--------|---------|\n| `bool` | `bool` |\n| `int` | `int64` |\n| `float` | `float64` |\n| `str` | `string` |\n| `bytes` | `binary` |\n| `list`, `dict` | `string` (JSON) |\n| `NDArray` + vector schema | `fixed_size_list` |\n\n---\n\n## Qdrant\n\n**Import**: `from cocoindex.connectors import qdrant`\n\n**Capabilities**: Target only | **Vector**: Native | **Model**: Point-oriented\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import qdrant\n\nfrom qdrant_client import QdrantClient\nQDRANT_DB = coco.ContextKey[QdrantClient](\"qdrant_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n client = qdrant.create_client(\"http://localhost:6333\")\n builder.provide(QDRANT_DB, client)\n yield\n```\n\n### As Target (Single Vector)\n\n```python\ncollection = await qdrant.mount_collection_target(\n QDRANT_DB,\n collection_name=\"embeddings\",\n schema=await qdrant.CollectionSchema.create(\n vectors=qdrant.QdrantVectorDef(schema=EMBEDDER, distance=\"cosine\"),\n ),\n)\n\ncollection.declare_point(point=qdrant.PointStruct(\n id=\"point-1\",\n vector=embedding_array.tolist(),\n payload={\"text\": \"hello\"},\n))\n```\n\n### As Target (Named Vectors)\n\n```python\nschema = await qdrant.CollectionSchema.create(\n vectors={\n \"text\": qdrant.QdrantVectorDef(schema=text_embedder_key, distance=\"cosine\"),\n \"image\": qdrant.QdrantVectorDef(schema=image_embedder_key, distance=\"cosine\"),\n },\n)\ncollection = await qdrant.mount_collection_target(QDRANT_DB, \"multimodal\", schema=schema)\n\ncollection.declare_point(point=qdrant.PointStruct(\n id=\"point-1\",\n vector={\"text\": text_vec.tolist(), \"image\": image_vec.tolist()},\n payload={\"title\": \"example\"},\n))\n```\n\n### Distance Metrics\n\n- `\"cosine\"` -- Cosine similarity (default)\n- `\"dot\"` -- Dot product\n- `\"euclid\"` -- Euclidean distance (L2)\n\n---\n\n## SurrealDB\n\n**Import**: `from cocoindex.connectors import surrealdb`\n\n**Capabilities**: Target only | **Vector**: Native | **Model**: Graph + document\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import surrealdb\n\nSURREAL_DB = coco.ContextKey[surrealdb.ConnectionFactory](\"surreal_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n factory = surrealdb.ConnectionFactory(\n url=\"ws://localhost:8000/rpc\",\n namespace=\"my_ns\",\n database=\"my_db\",\n credentials={\"username\": \"root\", \"password\": \"root\"},\n )\n builder.provide(SURREAL_DB, factory)\n yield\n```\n\n### As Target (Table)\n\n```python\ntarget_table = await surrealdb.mount_table_target(\n SURREAL_DB,\n table_name=\"records\",\n table_schema=await surrealdb.TableSchema.from_class(MyRecord),\n)\n\ntarget_table.declare_row(row=MyRecord(id=\"rec1\", name=\"example\"))\n```\n\n### As Target (Relation)\n\n```python\nrelation_target = await surrealdb.mount_relation_target(\n SURREAL_DB,\n table_name=\"likes\",\n from_table=users_table,\n to_table=items_table,\n table_schema=await surrealdb.TableSchema.from_class(MyRelation),\n)\nrelation_target.declare_relation(from_id=\"user:1\", to_id=\"item:1\", record=MyRelation(...))\n```\n\n---\n\n## Neo4j\n\n**Import**: `from cocoindex.connectors import neo4j`\n\n**Capabilities**: Target only | **Vector**: Native (vector index) | **Model**: Knowledge graph (nodes + relationships)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import neo4j\n\nNEO4J_DB = coco.ContextKey[neo4j.ConnectionFactory](\"neo4j_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n neo4j_db = neo4j.ConnectionFactory(\n uri=\"bolt://localhost:7687\",\n auth=(\"neo4j\", \"cocoindex\"),\n database=\"neo4j\", # Optional, default \"neo4j\"\n )\n builder.provide(NEO4J_DB, neo4j_db)\n yield\n await neo4j_db.close() # closes the driver the factory opened on this loop\n```\n\n### As Target (Nodes + Relationships)\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass Person:\n name: str\n\n@dataclass\nclass Task:\n description: str\n\n@dataclass\nclass AssignedRel:\n priority: int\n\n# Mount node tables (each label is a table)\nperson_table = await neo4j.mount_table_target(\n NEO4J_DB,\n \"Person\",\n await neo4j.TableSchema.from_class(Person, primary_key=\"name\"),\n primary_key=\"name\",\n)\ntask_table = await neo4j.mount_table_target(\n NEO4J_DB,\n \"Task\",\n await neo4j.TableSchema.from_class(Task, primary_key=\"description\"),\n primary_key=\"description\",\n)\n\n# Mount a relationship target between two node tables\nassigned_rel = await neo4j.mount_relation_target(\n NEO4J_DB, \"ASSIGNED_TO\", person_table, task_table,\n await neo4j.TableSchema.from_class(AssignedRel),\n)\n\n# Declare nodes\nperson_table.declare_record(row=Person(name=\"Alice\")) # alias: declare_row\ntask_table.declare_record(row=Task(description=\"ship v1\"))\n\n# Declare edges (record optional; PK auto-derived from endpoints when omitted)\nassigned_rel.declare_relation(\n from_id=\"Alice\", to_id=\"ship v1\", record=AssignedRel(priority=1),\n)\n```\n\n### Vector Index\n\n```python\nperson_table.declare_vector_index(\n field=\"embedding\",\n dimension=384,\n metric=\"cosine\", # or \"euclidean\"\n)\n```\n\n### Type Override\n\n```python\nfrom typing import Annotated\n\n@dataclass\nclass Row:\n id: str\n score: Annotated[float, neo4j.Neo4jType(\"decimal\", encoder=str)]\n```\n\n---\n\n## FalkorDB\n\n**Import**: `from cocoindex.connectors import falkordb`\n\n**Capabilities**: Target only | **Vector**: Native (vector index) | **Model**: Knowledge graph (nodes + relationships)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import falkordb\n\nFALKOR_DB = coco.ContextKey[falkordb.ConnectionFactory](\"falkor_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n falkor_db = falkordb.ConnectionFactory(\n uri=\"falkor://localhost:6379\",\n graph=\"knowledge_graph\", # Optional, default \"default\"\n )\n builder.provide(FALKOR_DB, falkor_db)\n yield\n await falkor_db.close() # closes the client the factory opened on this loop\n```\n\n### As Target (Nodes + Relationships)\n\n```python\nperson_table = await falkordb.mount_table_target(\n FALKOR_DB,\n \"Person\",\n await falkordb.TableSchema.from_class(Person, primary_key=\"name\"),\n primary_key=\"name\",\n)\ntask_table = await falkordb.mount_table_target(\n FALKOR_DB,\n \"Task\",\n await falkordb.TableSchema.from_class(Task, primary_key=\"description\"),\n primary_key=\"description\",\n)\n\n# Mount relationship target (no schema -> PK auto-derived from endpoints)\nassigned_rel = await falkordb.mount_relation_target(\n FALKOR_DB, \"ASSIGNED_TO\", person_table, task_table,\n)\n\n# Declare nodes\nperson_table.declare_record(row=Person(name=\"Alice\")) # alias: declare_row\ntask_table.declare_record(row=Task(description=\"ship v1\"))\n\n# Declare edges\nassigned_rel.declare_relation(from_id=\"Alice\", to_id=\"ship v1\")\n```\n\n### Vector Index\n\n```python\nperson_table.declare_vector_index(\n field=\"embedding\",\n dimension=384,\n metric=\"cosine\", # or \"euclidean\", \"ip\"\n)\n```\n\n### Type Override\n\n```python\nfrom typing import Annotated\n\n@dataclass\nclass Row:\n id: str\n score: Annotated[float, falkordb.FalkorType(\"decimal\", encoder=str)]\n```\n\n---\n\n## LocalFS\n\n**Import**: `from cocoindex.connectors import localfs`\n\n**Capabilities**: Source + Target\n\n### As Source\n\n```python\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import PatternFilePathMatcher\n\nfiles = localfs.walk_dir(\n pathlib.Path(\"./data\"),\n recursive=True,\n path_matcher=PatternFilePathMatcher(\n included_patterns=[\"**/*.py\", \"**/*.md\"],\n excluded_patterns=[\".*/**\", \"__pycache__/**\"],\n ),\n live=True, # Enable file watching for live mode\n)\n\n# Process each file\nawait coco.mount_each(process_file, files.items(), target_table)\n```\n\n### As Target (Single File)\n\n```python\nlocalfs.declare_file(\n outdir / \"output.txt\",\n \"file content\",\n create_parent_dirs=True,\n)\n```\n\n### As Target (Directory)\n\n```python\ndir_target = await localfs.mount_dir_target(pathlib.Path(\"./output\"))\ndir_target.declare_file(\"file1.txt\", \"content 1\")\ndir_target.declare_file(\"subdir/file2.txt\", \"content 2\")\n```\n\n### Stable File Paths\n\nUse `localfs.FilePath` for stable memoization when the base directory might move:\n\n```python\nfiles = localfs.walk_dir(localfs.FilePath(path=\"./data\"), ...)\n```\n\n---\n\n## Amazon S3\n\n**Import**: `from cocoindex.connectors import amazon_s3`\n\n**Capabilities**: Source only\n\n### Connection Setup\n\n```python\nimport aiobotocore.session\nfrom aiobotocore.client import AioBaseClient\n\nS3_CLIENT = coco.ContextKey[AioBaseClient](\"s3_client\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n session = aiobotocore.session.get_session()\n async with session.create_client(\"s3\") as s3_client:\n builder.provide(S3_CLIENT, s3_client)\n yield\n```\n\n### Reading Objects\n\n```python\nfrom cocoindex.connectors import amazon_s3\n\nclient = coco.use_context(S3_CLIENT)\n\n# List and iterate objects\nwalker = amazon_s3.list_objects(\n client, \"my-bucket\",\n prefix=\"data/\",\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.json\"]),\n max_file_size=10_000_000,\n)\n\nawait coco.mount_each(process_file, walker.items(), target)\n\n# Single object\nfile = await amazon_s3.get_object(client, \"s3://my-bucket/path/to/file.json\")\ncontent = await amazon_s3.read(client, \"s3://my-bucket/path/to/file.json\")\n```\n\n---\n\n## Kafka\n\n**Import**: `from cocoindex.connectors import kafka`\n\n**Capabilities**: Source + Target\n\n### As Source (Live)\n\n```python\nfrom confluent_kafka.aio import AIOConsumer\nfrom cocoindex.connectors import kafka\n\nconsumer = AIOConsumer({\n \"bootstrap.servers\": \"localhost:9092\",\n \"group.id\": \"my-group\",\n \"enable.auto.commit\": \"false\",\n \"auto.offset.reset\": \"earliest\",\n})\n\nitems = kafka.topic_as_map(consumer, [\"my-topic\"])\nawait coco.mount_each(process_message, items, target_table)\n```\n\n`topic_as_map()` returns a `LiveMapFeed`, which `mount_each()` auto-detects for live mode.\n\n### As Target\n\n```python\nfrom confluent_kafka.aio import AIOProducer\n\nKAFKA_PRODUCER = coco.ContextKey[AIOProducer](\"kafka_producer\")\n\ntopic_target = await kafka.mount_kafka_topic_target(KAFKA_PRODUCER, \"my-topic\")\ntopic_target.declare_target_state(key=\"msg-key\", value=json.dumps(data))\n```\n\n---\n\n## Apache Doris\n\n**Import**: `from cocoindex.connectors import doris`\n\n**Capabilities**: Target only | **Vector**: Native | **Features**: Vector indexes, inverted indexes, Stream Load\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import doris\n\nDORIS_DB = coco.ContextKey[doris.ManagedConnection](\"doris_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n config = doris.DorisConnectionConfig(\n fe_host=\"localhost\",\n database=\"my_db\",\n fe_http_port=8080,\n query_port=9030,\n username=\"root\",\n password=\"\",\n )\n conn = doris.connect(config)\n builder.provide(DORIS_DB, conn)\n yield\n```\n\n### As Target\n\n```python\ntarget_table = await doris.mount_table_target(\n DORIS_DB,\n table_name=\"embeddings\",\n table_schema=await doris.TableSchema.from_class(MyRecord, primary_key=[\"id\"]),\n vector_indexes=[doris.VectorIndexDef(field_name=\"embedding\", metric_type=\"cosine_distance\")],\n)\n\ntarget_table.declare_row(row=MyRecord(id=1, text=\"hello\", embedding=vec))\n```\n\n---\n\n## Google Drive\n\n**Import**: `from cocoindex.connectors import google_drive`\n\n**Capabilities**: Source only\n\n```python\nfrom cocoindex.connectors import google_drive\n\nsource = google_drive.GoogleDriveSource(\n service_account_credential_path=\"credentials.json\",\n root_folder_ids=[\"folder-id\"],\n)\nawait coco.mount_each(process_file, source.items(), target)\n```\n\n---\n\n## Connector Comparison\n\n| Connector | Source | Target | Vectors | Best For |\n|-----------|--------|--------|---------|----------|\n| **PostgreSQL** | Y | Y | pgvector | Production SQL + vectors |\n| **SQLite** | - | Y | sqlite-vec | Local SQL + vectors |\n| **LanceDB** | - | Y | Native | Cloud-native vector DB |\n| **Qdrant** | - | Y | Native | Specialized vector search |\n| **SurrealDB** | - | Y | Native | Graph + document DB |\n| **Neo4j** | - | Y | Native | Knowledge graphs |\n| **FalkorDB** | - | Y | Native | Knowledge graphs (Redis) |\n| **Apache Doris** | - | Y | Native | Analytical DB + vectors |\n| **LocalFS** | Y | Y | N/A | File-based pipelines |\n| **Amazon S3** | Y | - | N/A | Cloud object storage |\n| **Kafka** | Y | Y | N/A | Streaming pipelines |\n| **Google Drive** | Y | - | N/A | Cloud file source |\n\n---\n\n## See Also\n\n- [API Reference](./api_reference.md)\n- [Patterns](./patterns.md)\n- [Setup Database](./setup_database.md)\n","skills/cocoindex/references/patterns.md":"# CocoIndex Common Patterns\n\nCommon patterns and workflows for building CocoIndex pipelines.\n\n## Core Pattern: TargetState = Transform(SourceState)\n\nAll CocoIndex pipelines follow this declarative pattern:\n\n1. Read source state\n2. Transform\n3. Declare target state\n\nCocoIndex handles the incremental sync automatically.\n\n---\n\n## Pattern 1: File Transformation Pipeline\n\n**Use case**: Transform files from one format to another (e.g., Markdown -> HTML, PDF -> Markdown)\n\n```python\nimport pathlib\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom markdown_it import MarkdownIt\n\n_markdown_it = MarkdownIt(\"gfm-like\")\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, outdir: pathlib.Path) -> None:\n html = _markdown_it.render(await file.read_text())\n outname = \"__\".join(file.file_path.path.parts) + \".html\"\n localfs.declare_file(outdir / outname, html, create_parent_dirs=True)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n files = localfs.walk_dir(\n sourcedir,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n live=True, # Enable live file watching\n )\n await coco.mount_each(process_file, files.items(), outdir)\n\napp = coco.App(\n coco.AppConfig(name=\"FilesTransform\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n outdir=pathlib.Path(\"./output_html\"),\n)\n```\n\n**Key points:**\n- `memo=True` -- Skip reprocessing unchanged files\n- `mount_each()` -- One component per file; keys from `files.items()` become component subpaths\n- `live=True` on `walk_dir()` -- Enables file watching for live mode\n- Auto-cleanup -- Deleting source file automatically removes output file\n\n---\n\n## Pattern 2: Vector Embedding Pipeline\n\n**Use case**: Chunk and embed documents for semantic search\n\n```python\nimport pathlib\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator, Annotated\n\nimport asyncpg\nfrom numpy.typing import NDArray\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs, postgres\nfrom cocoindex.ops.text import RecursiveSplitter\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\nfrom cocoindex.resources.chunk import Chunk\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom cocoindex.resources.id import IdGenerator\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n_splitter = RecursiveSplitter()\n\n@dataclass\nclass DocEmbedding:\n id: int\n filename: str\n chunk_start: int\n chunk_end: int\n text: str\n embedding: Annotated[NDArray, EMBEDDER]\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n@coco.fn\nasync def process_chunk(\n chunk: Chunk, filename: pathlib.PurePath,\n id_gen: IdGenerator, table: postgres.TableTarget[DocEmbedding],\n) -> None:\n table.declare_row(row=DocEmbedding(\n id=await id_gen.next_id(chunk.text),\n filename=str(filename),\n chunk_start=chunk.start.char_offset,\n chunk_end=chunk.end.char_offset,\n text=chunk.text,\n embedding=await coco.use_context(EMBEDDER).embed(chunk.text),\n ))\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, table: postgres.TableTarget[DocEmbedding]) -> None:\n text = await file.read_text()\n chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language=\"markdown\")\n id_gen = IdGenerator()\n await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n target_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"doc_embeddings\",\n table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=[\"id\"]),\n )\n target_table.declare_vector_index(column=\"embedding\")\n\n files = localfs.walk_dir(sourcedir, recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]))\n await coco.mount_each(process_file, files.items(), target_table)\n\napp = coco.App(coco.AppConfig(name=\"TextEmbedding\"), app_main,\n sourcedir=pathlib.Path(\"./markdown_files\"))\n```\n\n**Key points:**\n- `mount_table_target(PG_DB, ...)` -- Takes `ContextKey` as first argument\n- `Annotated[NDArray, EMBEDDER]` -- Vector annotation uses ContextKey for auto dimension inference\n- `map()` -- Concurrent execution within a component (no child components created)\n- `IdGenerator` -- Generates stable unique IDs for chunks across incremental updates\n- `memo=True` on `process_file` -- Skips unchanged files entirely\n\n---\n\n## Pattern 3: Database Source -> Transform -> Database Target\n\n**Use case**: Transform data from one database to another\n\n```python\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator\n\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nSOURCE_DB_URL = \"postgres://localhost/source_db\"\nTARGET_DB_URL = \"postgres://localhost/target_db\"\n\nSOURCE_DB = coco.ContextKey[asyncpg.Pool](\"source_db\")\nTARGET_DB = coco.ContextKey[asyncpg.Pool](\"target_db\")\n\n@dataclass\nclass SourceRecord:\n id: int\n name: str\n value: float\n\n@dataclass\nclass TargetRecord:\n id: int\n name: str\n value: float\n processed: bool\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with (\n await asyncpg.create_pool(SOURCE_DB_URL) as source_pool,\n await asyncpg.create_pool(TARGET_DB_URL) as target_pool,\n ):\n builder.provide(SOURCE_DB, source_pool)\n builder.provide(TARGET_DB, target_pool)\n yield\n\n@coco.fn(memo=True)\nasync def process_record(record: SourceRecord, target_table: postgres.TableTarget[TargetRecord]) -> None:\n target_table.declare_row(row=TargetRecord(\n id=record.id, name=record.name.upper(),\n value=record.value * 2, processed=True,\n ))\n\n@coco.fn\nasync def app_main() -> None:\n target_table = await postgres.mount_table_target(\n TARGET_DB,\n table_name=\"target_records\",\n table_schema=await postgres.TableSchema.from_class(TargetRecord, primary_key=[\"id\"]),\n )\n\n source = postgres.PgTableSource(\n coco.use_context(SOURCE_DB),\n table_name=\"source_records\",\n row_type=SourceRecord,\n )\n await coco.mount_each(\n coco.component_subpath(\"record\"),\n process_record,\n source.fetch_rows().items(key=lambda r: r.id),\n target_table,\n )\n\napp = coco.App(coco.AppConfig(name=\"DatabaseTransform\"), app_main)\n```\n\n---\n\n## Pattern 4: LLM-Based Extraction Pipeline\n\n**Use case**: Extract structured data using LLMs\n\n```python\nimport instructor\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator\nfrom pydantic import BaseModel\nfrom litellm import acompletion\n\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\n\n_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)\n\nclass ExtractedTopic(BaseModel):\n name: str\n description: str\n\nclass ExtractionResult(BaseModel):\n title: str\n topics: list[ExtractedTopic]\n\n@dataclass\nclass Message:\n id: int\n title: str\n content: str\n\n@dataclass\nclass Topic:\n message_id: int\n name: str\n description: str\n\n@coco.fn(memo=True)\nasync def extract_and_store(\n content: str, message_id: int,\n messages_table: postgres.TableTarget[Message],\n topics_table: postgres.TableTarget[Topic],\n) -> None:\n result = await _instructor_client.chat.completions.create(\n model=\"gpt-4\",\n response_model=ExtractionResult,\n messages=[{\"role\": \"user\", \"content\": f\"Extract topics:\\n\\n{content}\"}],\n )\n messages_table.declare_row(row=Message(id=message_id, title=result.title, content=content))\n for topic in result.topics:\n topics_table.declare_row(row=Topic(\n message_id=message_id, name=topic.name, description=topic.description,\n ))\n\n@coco.fn\nasync def app_main(input_texts: list[str]) -> None:\n messages_table = await postgres.mount_table_target(\n PG_DB, table_name=\"messages\",\n table_schema=await postgres.TableSchema.from_class(Message, primary_key=[\"id\"]),\n )\n topics_table = await postgres.mount_table_target(\n PG_DB, table_name=\"topics\",\n table_schema=await postgres.TableSchema.from_class(Topic, primary_key=[\"message_id\", \"name\"]),\n )\n for idx, text in enumerate(input_texts):\n await coco.mount(\n coco.component_subpath(\"text\", idx),\n extract_and_store, text, idx, messages_table, topics_table,\n )\n\napp = coco.App(coco.AppConfig(name=\"LLMExtraction\"), app_main, input_texts=[\"text1...\", \"text2...\"])\n```\n\n**Key points:**\n- `memo=True` on extraction -- Avoids re-calling LLM for unchanged inputs\n- Multiple target tables -- Declare rows in multiple tables from single component\n- Pydantic models -- For structured LLM outputs\n\n---\n\n## Pattern 5: Kafka Streaming Pipeline\n\n**Use case**: Process Kafka messages and write to a database\n\n```python\nimport json\nfrom collections.abc import AsyncIterator\nfrom dataclasses import dataclass\n\nfrom confluent_kafka import Message\nfrom confluent_kafka.aio import AIOConsumer\n\nimport cocoindex as coco\nfrom cocoindex.connectors import kafka, lancedb\n\nLANCE_DB = coco.ContextKey[lancedb.LanceAsyncConnection](\"lance_db\")\n\n@dataclass\nclass Product:\n sku: str\n name: str\n category: str\n price: float\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n conn = await lancedb.connect_async(\"./lancedb_data\")\n builder.provide(LANCE_DB, conn)\n yield\n\n@coco.fn\nasync def process_message(msg: Message, table: lancedb.TableTarget[Product]) -> None:\n value = msg.value()\n if value is None:\n return\n row = json.loads(value.decode() if isinstance(value, bytes) else value)\n table.declare_row(row=Product(**{**row, \"price\": float(row[\"price\"])}))\n\n@coco.fn\nasync def app_main() -> None:\n products_table = await lancedb.mount_table_target(\n LANCE_DB, table_name=\"products\",\n table_schema=await lancedb.TableSchema.from_class(Product, primary_key=[\"sku\"]),\n )\n\n consumer = AIOConsumer({\n \"bootstrap.servers\": \"localhost:9092\",\n \"group.id\": \"my-group\",\n \"enable.auto.commit\": \"false\",\n \"auto.offset.reset\": \"earliest\",\n })\n items = kafka.topic_as_map(consumer, [\"products-topic\"])\n await coco.mount_each(process_message, items, products_table)\n\napp = coco.App(coco.AppConfig(name=\"KafkaToLanceDB\"), app_main)\n```\n\n**Key points:**\n- `kafka.topic_as_map()` returns a `LiveMapFeed` -- `mount_each()` auto-detects for live mode\n- Pipeline runs continuously in live mode, processing new messages as they arrive\n\n---\n\n## Pattern 6: Context Management for Shared Resources\n\n**Use case**: Share expensive resources across components\n\n```python\nimport cocoindex as coco\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\n\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\nCONFIG = coco.ContextKey[dict](\"config\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n builder.provide(CONFIG, {\"chunk_size\": 1000, \"overlap\": 200})\n yield\n\n@coco.fn\nasync def process_item(text: str) -> None:\n embedder = coco.use_context(EMBEDDER)\n config = coco.use_context(CONFIG)\n embedding = await embedder.embed(text)\n ...\n```\n\n**Key points:**\n- `detect_change=True` -- Opt in to invalidate memos when value changes (use for models/config affecting output)\n- `detect_change=False` (default) -- For resources not affecting computation (DB connections, loggers)\n- `ContextKey` name is stable identity -- avoid renaming across runs\n\n---\n\n## Common Anti-Patterns to Avoid\n\n### Missing `@coco.fn` Decorator\n\n```python\n# BAD: Missing decorator\nasync def process_file(file, table):\n table.declare_row(...)\n\n# GOOD:\n@coco.fn\nasync def process_file(file, table):\n table.declare_row(...)\n```\n\n### Reprocessing Everything\n\n```python\n# BAD: No memoization\n@coco.fn # Missing memo=True\nasync def process_file(file, table):\n embedding = await embedder.embed(await file.read_text()) # Expensive!\n\n# GOOD:\n@coco.fn(memo=True)\nasync def process_file(file, table):\n ...\n```\n\n### Unstable Component Paths\n\n```python\n# BAD: Using object references or indices\nfor file in files:\n await coco.mount(coco.component_subpath(file), ...) # Object ref\nfor idx, item in enumerate(items):\n await coco.mount(coco.component_subpath(idx), ...) # Index changes\n\n# GOOD: Use stable identifiers\nawait coco.mount_each(process_file, files.items(), table) # Keys from items()\n```\n\n### Loading Resources Per Component\n\n```python\n# BAD: Loading model in every component\n@coco.fn\nasync def process(text):\n model = SentenceTransformer(\"model\") # Loaded repeatedly!\n\n# GOOD: Load once in lifespan, use via context\nembedder = coco.use_context(EMBEDDER)\n```\n\n### Mixing Target State with Side Effects\n\n```python\n# BAD: Side effects not detected\n@coco.fn\nasync def process(data):\n requests.post(\"https://api.example.com\", json=data) # Not detected!\n\n# GOOD: Only declare target states\ntable.declare_row(row=result)\n```\n\n---\n\n## See Also\n\n- [Connectors Reference](./connectors.md)\n- [API Reference](./api_reference.md)\n- [Setup Project](./setup_project.md)\n","skills/cocoindex/references/setup_database.md":"# Database Setup Guide\n\nSetup instructions for databases used with CocoIndex.\n\n## PostgreSQL with pgvector\n\n### Using Docker (Recommended)\n\n```bash\ncat > docker-compose.yml < docker-compose.yml <=1.0.0\",\n \"sentence-transformers\",\n \"asyncpg\",\n]\n```\n\n### PostgreSQL Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"asyncpg\",\n]\n```\n\n### SQLite Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"sqlite-vec\",\n]\n```\n\n### LanceDB Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"lancedb\",\n]\n```\n\n### Qdrant Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"qdrant-client\",\n]\n```\n\n### Kafka Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"confluent-kafka\",\n]\n```\n\n### LLM-Based Extraction\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"litellm\",\n \"instructor\",\n \"pydantic>=2.0\",\n \"asyncpg\",\n]\n```\n\n---\n\n## Environment Configuration\n\n### `.env` File\n\nThe `cocoindex` CLI automatically loads `.env` from the current directory (via `find_dotenv`).\n\n```bash\n# CocoIndex internal database (optional fallback).\n# Only used if the lifespan does not set builder.settings.db_path.\n# The `cocoindex init` template sets db_path in the lifespan instead, so this is not needed there.\nCOCOINDEX_DB=./cocoindex.db\n\n# PostgreSQL (if using)\nPOSTGRES_URL=postgres://user:pass@localhost/db\n\n# Qdrant (if using)\nQDRANT_URL=http://localhost:6333\n\n# API keys (if using LLM extraction)\nOPENAI_API_KEY=your-openai-api-key\nANTHROPIC_API_KEY=your-anthropic-api-key\n```\n\n### Manual Settings (in lifespan)\n\n```python\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n builder.settings.db_path = pathlib.Path(\"./custom.db\")\n yield\n```\n\n---\n\n## Running Your Pipeline\n\n```bash\nuv sync # Install dependencies\nuv run cocoindex update main.py # Run pipeline\nuv run cocoindex update main.py -L # Run in live mode\nuv run cocoindex show main.py # Show component paths\nuv run cocoindex drop main.py -f # Reset everything\n```\n\nIf `pyproject.toml` has `[tool.uv.sources]` or `[[tool.uv.index]]` tables, install with uv. pip ignores those tables, so it would fetch the packages they pin from PyPI instead.\n\n---\n\n## Common Issues\n\n### Import Errors\n\n```bash\nuv sync\n```\n\n### Database Connection Errors\n\nVerify database is running and `.env` has correct URLs. See [setup_database.md](./setup_database.md).\n\n---\n\n## See Also\n\n- [Database Setup](./setup_database.md)\n- [Patterns](./patterns.md)\n- [API Reference](./api_reference.md)\n","skills/cocoindex/references/api_reference.md":"# CocoIndex API Reference\n\nQuick reference for the most commonly used CocoIndex APIs.\n\n## Import Convention\n\n```python\nimport cocoindex as coco\n```\n\nAll APIs live directly on the `coco` module. There is no separate `cocoindex.asyncio` module.\n\n---\n\n## `@coco.fn` Decorator\n\nMark a function as a CocoIndex processing function.\n\n```python\n@coco.fn\nasync def my_function(arg1: str) -> None: ...\n\n@coco.fn(memo=True, version=1)\nasync def expensive_fn(data: str) -> Result: ...\n\n# Force async interface for a sync function (useful for batching)\n@coco.fn.as_async(memo=True, batching=True)\ndef batch_embed(texts: list[str]) -> list[NDArray]: ...\n```\n\n**Parameters:**\n- `memo: bool = False` -- Enable memoization (skip if inputs/code unchanged)\n- `version: int | None = None` -- Explicit version bump to force re-execution\n- `logic_tracking: \"full\" | \"self\" | None = \"full\"` -- How code changes are tracked\n- `batching: bool = False` -- Auto-batch concurrent calls (async only)\n- `max_batch_size: int | None = None` -- Max batch size\n- `runner: Runner | None = None` -- e.g. `coco.GPU` for serialized GPU execution\n\n---\n\n## Mount APIs (all async)\n\nAll mount APIs accept an optional `ComponentSubpath` as their first argument. When omitted, the subpath is auto-derived from `Symbol(fn.__name__)`. Provide an explicit subpath when mounting the same function multiple times, using multi-part paths, or needing a specific path name.\n\n### `coco.mount()`\n\nMount a processing component in the background.\n\n```python\n# Subpath auto-derived from fn.__name__\nhandle = await coco.mount(processor_fn, *args, **kwargs)\n\n# Explicit subpath\nhandle = await coco.mount(coco.component_subpath(\"name\"), processor_fn, *args, **kwargs)\n\nawait handle.ready() # Optional: wait until component finishes\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `processor_fn` -- Function (or LiveComponent class) to run.\n- `*args, **kwargs` -- Arguments passed to the function.\n\n**Returns:** `ComponentMountHandle`\n\n### `coco.use_mount()`\n\nMount a dependent component and return its result. Parent depends on the child.\n\n```python\n# Subpath auto-derived from fn.__name__\nresult = await coco.use_mount(init_fn, *args, **kwargs)\n\n# Explicit subpath\nresult = await coco.use_mount(coco.component_subpath(\"setup\"), init_fn, *args, **kwargs)\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `processor_fn` -- Function to run.\n- `*args, **kwargs` -- Arguments passed to the function.\n\n**Returns:** The return value of `processor_fn`.\n\n### `coco.mount_each()`\n\nMount one component per item in a keyed iterable. Preferred for processing lists.\n\n```python\n# Subpath auto-derived from fn.__name__\nawait coco.mount_each(process_file, files.items(), *extra_args)\n\n# Explicit subpath\nawait coco.mount_each(coco.component_subpath(\"process\"), process_file, files.items(), table)\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `fn` -- Function to run per item. Item value is passed as first argument.\n- `items` -- Keyed iterable of `(StableKey, T)` pairs, or a `LiveMapFeed` for live mode.\n- `*args, **kwargs` -- Additional arguments passed to `fn` after the item.\n\n**Returns:** `ComponentMountHandle`\n\n### `coco.mount_target()`\n\nMount a target state, ensuring the container is applied before returning the child provider.\n\n```python\nprovider = await coco.mount_target(target_state)\n```\n\nPrefer connector convenience methods (`postgres.mount_table_target()`, etc.) which call this internally.\n\n### `coco.map()`\n\nRun a function concurrently on each item. No processing components are created -- pure concurrent execution within the current component.\n\n```python\nresults = await coco.map(process_chunk, chunks, *extra_args)\n```\n\n**Parameters:**\n- `fn` -- Async function to apply. Item is passed as first argument.\n- `items` -- Iterable or async iterable.\n- `*args, **kwargs` -- Additional arguments passed to `fn` after the item.\n\n**Returns:** `list[T]`\n\n### `coco.component_subpath()`\n\nCreate a stable component path for mounting.\n\n```python\ncoco.component_subpath(\"setup\")\ncoco.component_subpath(\"file\", str(file_path))\ncoco.component_subpath(\"record\", record.id)\n\n# Chaining with /\nsubpath = coco.component_subpath(\"a\") / \"b\" / \"c\"\n\n# Context manager form (applies to all nested mount calls)\nwith coco.component_subpath(\"process\"):\n for f in files:\n await coco.mount(coco.component_subpath(str(f.path)), process_file, f)\n```\n\n**StableKey types:** `str | int | bool | bytes | uuid.UUID | Symbol | tuple[StableKey, ...]`\n\n---\n\n## Context System\n\n### `coco.ContextKey[T]`\n\nType-safe key for sharing resources. The `key` string is the stable identity across runs.\n\n```python\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n```\n\n- `detect_change=True` -- Opt in to auto-invalidate dependent memos when value changes (models, configs)\n- `detect_change=False` (default) -- For resources not affecting computation (DB connections, loggers)\n\n### `builder.provide()`\n\nRegister a resource in context (used in lifespan).\n\n```python\nbuilder.provide(PG_DB, pool)\nbuilder.provide_with(KEY, context_manager) # Sync CM\nawait builder.provide_async_with(KEY, async_cm) # Async CM\n```\n\n### `coco.use_context()`\n\nRetrieve a resource from context inside a processing function.\n\n```python\npool = coco.use_context(PG_DB)\nembedder = coco.use_context(EMBEDDER)\n```\n\n---\n\n## Lifespan\n\n### `@coco.lifespan`\n\nDefine environment setup/teardown. Registered to the default environment.\n\n```python\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(MODEL))\n yield\n```\n\nCan also be sync:\n\n```python\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n builder.settings.db_path = pathlib.Path(\"./my.db\")\n yield\n```\n\n---\n\n## App\n\n### `coco.App`\n\n```python\napp = coco.App(\n coco.AppConfig(name=\"MyApp\"),\n main_fn,\n **params,\n)\n\n# Async\nawait app.update()\nhandle = app.update(live=True) # Live mode\nawait app.drop()\n\n# Sync (blocking)\napp.update_blocking(report_to_stdout=True)\napp.drop_blocking()\n```\n\n### `coco.AppConfig`\n\n```python\ncoco.AppConfig(\n name=\"MyApp\", # Required\n environment=env, # Optional: custom Environment\n max_inflight_components=1024, # Optional: concurrency limit\n)\n```\n\n### Start/Stop (for programmatic usage)\n\n```python\n# Context manager (preferred)\nasync with coco.runtime():\n await app.update()\n\n# Or manually\nawait coco.start()\ntry:\n await app.update()\nfinally:\n await coco.stop()\n\n# Sync variants\nwith coco.runtime():\n app.update_blocking()\n```\n\n---\n\n## Exception Handlers\n\n### Global (in lifespan)\n\n```python\nbuilder.set_exception_handler(my_handler)\n```\n\n### Scoped (in processing functions)\n\n```python\nasync with coco.exception_handler(my_handler):\n await coco.mount_each(process_file, files.items(), table)\n```\n\n### Handler Signature\n\n```python\nasync def my_handler(exc: BaseException, ctx: coco.ExceptionContext) -> None:\n logger.error(f\"Error in {ctx.stable_path}: {exc}\")\n```\n\n`ExceptionContext` provides: `env_name`, `stable_path`, `processor_name`, `mount_kind`, `parent_stable_path`, `is_background`, `source`, `original_exception`.\n\n---\n\n## Live Mode\n\nRun the pipeline continuously, streaming changes from sources:\n\n```python\n# CLI\ncocoindex update main.py -L\n\n# Programmatic\nhandle = app.update(live=True)\nasync for snapshot in handle.watch():\n print(snapshot.stats)\n```\n\nSources that support live mode:\n- `localfs.walk_dir(..., live=True)` -- File watching\n- `kafka.topic_as_map(consumer, topics)` -- Kafka topic consumption\n\n---\n\n## CLI Commands\n\n```bash\ncocoindex init [PROJECT_NAME] # Create new project\ncocoindex update APP_TARGET # Run app once\ncocoindex update APP_TARGET -L # Run in live mode\ncocoindex update APP_TARGET --full-reprocess # Force reprocess all\ncocoindex update APP_TARGET --reset # Reset state before running\ncocoindex drop APP_TARGET [-f] # Drop app and all state\ncocoindex ls [APP_TARGET] [--db PATH] # List apps\ncocoindex show APP_TARGET [--tree] # Show component paths\n```\n\n**APP_TARGET format:** `main.py`, `main.py:app_name`, `my_module:app_name`\n\n---\n\n## Text Operations\n\n**Import:** `from cocoindex.ops.text import ...`\n\n### `detect_code_language()`\n\n```python\nlanguage = detect_code_language(filename=\"example.py\") # -> \"python\"\n```\n\n### `RecursiveSplitter`\n\n```python\nsplitter = RecursiveSplitter()\nchunks = splitter.split(\n text,\n chunk_size=1000,\n chunk_overlap=200,\n min_chunk_size=300,\n language=\"python\", # Syntax-aware splitting\n)\n# Each chunk: Chunk(text=..., start=TextPosition(...), end=TextPosition(...))\n```\n\n### `SeparatorSplitter`\n\n```python\nsplitter = SeparatorSplitter(separators_regex=[r\"\\n\\n\", r\"\\n\"])\nchunks = splitter.split(text)\n```\n\n---\n\n## Embedding Operations\n\n**Import:** `from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder`\n\n```python\nembedder = SentenceTransformerEmbedder(\"sentence-transformers/all-MiniLM-L6-v2\")\nembedding = await embedder.embed(text) # -> NDArray (float32)\n```\n\n**As VectorSchemaProvider:**\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions\n```\n\n---\n\n## File Resources\n\n**Import:** `from cocoindex.resources.file import ...`\n\n### `FileLike` (async file object)\n\n```python\ntext = await file.read_text()\ndata = await file.read() # bytes, lazy/cached\nfp = await file.content_fingerprint()\n```\n\n### `PatternFilePathMatcher`\n\n```python\nmatcher = PatternFilePathMatcher(\n included_patterns=[\"**/*.py\", \"**/*.md\"],\n excluded_patterns=[\".*/**\", \"__pycache__/**\"],\n)\n```\n\n---\n\n## ID Generation\n\n**Import:** `from cocoindex.resources.id import ...`\n\n```python\n# Deterministic: same dep -> same ID\nchunk_id = await generate_id(chunk.text)\nchunk_uuid = generate_uuid(chunk.text)\n\n# Distinct per call (even with same dep)\nid_gen = IdGenerator()\nchunk_id = await id_gen.next_id(chunk.text)\n\nuuid_gen = UuidGenerator()\nchunk_uuid = uuid_gen.next_uuid(chunk.text)\n```\n\n---\n\n## Vector Schema\n\n**Import:** `from cocoindex.resources.schema import VectorSchema`\n\n```python\n# Via ContextKey (preferred -- auto-infer from embedder)\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER]\n\n# Explicit dimensions\nschema = VectorSchema(dtype=np.dtype(np.float32), size=384)\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, schema]\n```\n\n---\n\n## See Also\n\n- [Connectors Reference](./connectors.md) -- Database and system connectors\n- [Patterns Reference](./patterns.md) -- Common pipeline patterns\n- [Setup Project](./setup_project.md) -- Project setup guide\n- [Setup Database](./setup_database.md) -- Database setup guide\n","AGENTS.md":"# AGENTS.md\n\nThis file provides guidance to coding agents working with code in this repository.\nClaude Code compatibility is kept through `CLAUDE.md`, which imports this file.\n\n## Build and Test Commands\n\nThis project uses [uv](https://docs.astral.sh/uv/) for Python project management.\n\n### Building\n\n```bash\nuv run maturin develop # Build Rust code and install Python package (required after Rust changes)\n```\n\n### Testing\n\n```bash\ncargo test # Run Rust tests\nuv run mypy # Type check Python code\nuv run pytest python/ # Run Python tests (use after both Rust and Python changes)\n```\n\n### Code Formatting and Linting\n\nPython linting and formatting go through `prek`, which runs the `ruff-check` and\n`ruff-format` hooks from `.pre-commit-config.yaml` with the exact ruff version\nand file set CI uses. A `ruff` from `uv run`, `uvx`, or Homebrew is usually a\ndifferent version than the hook pins, so run the hooks rather than `ruff` itself.\n\n```bash\nuv tool install prek # one-time; some hooks also need `protoc` (brew install protobuf)\nprek run ruff-check ruff-format --all-files # Lint (applying safe fixes), then format Python code (same as CI)\ncargo fmt # Format Rust code\n./dev/run_cargo_clippy.sh # Lint Rust code (same as CI)\n```\n\n### Pre-submission Checks\n\nCI enforces the full pre-commit hook suite from `.pre-commit-config.yaml`, which\ncovers more than the commands above — notably regenerating the CLI docs\n(`docs/src/content/docs/cli.mdx`) whenever `cli.py` changes. Run it locally\nbefore pushing:\n\n```bash\nprek run --all-files\n```\n\n### Pull Requests and CI\n\nOpen a pull request as a draft and mark it ready once the checks above pass:\ndrafts run no CI. When `main` has a merge queue, a ready pull request runs only\nthe fast checks, and the queue runs the full suite on the commit that lands.\nFor a full run on a branch before then, use `gh workflow run CI.yml --ref `.\nThe CI tiers, the merge queue and stacked pull requests are described in\n[CI.md](https://github.com/cocoindex-io/.github/blob/main/CI.md).\n\n### Workflow Summary\n\n| Change Type | Commands to Run |\n|-------------|-----------------|\n| Rust code only | `uv run maturin develop && cargo test` |\n| Python code only | `uv run mypy && uv run pytest python/` |\n| Both Rust and Python | Run all commands from both categories above |\n| Python lint and format | `prek run ruff-check ruff-format --all-files` |\n| Before pushing | `prek run --all-files` |\n\nAfter changing a core engine API (`rust/core`), run `cargo check --workspace --all-targets` to catch call sites in every workspace crate — both `rust/py` and the Rust SDK under `rust/sdk/` call core directly, and the SDK is easy to miss when grepping. The `examples/rust/*` and `benchmarks/*/rust` crates are standalone workspaces that consume the SDK; the `prek` suite's `cargo check (rust examples + benchmarks)` hook covers those.\n\n## Code Structure\n\n```\ncocoindex/\n├── rust/ # Rust crates (workspace)\n│ ├── core/ # Core engine crate\n│ │ └── src/\n│ │ ├── engine/ # Core engine\n│ │ ├── state/ # States of the core engine\n│ │ └── inspect/ # Database inspection utilities\n│ ├── py/ # Python bindings (PyO3); calls core engine APIs\n│ ├── py_utils/ # Python-Rust utility helpers (error, convert, future)\n│ ├── utils/ # General utilities: error, batching, fingerprint, etc.\n│ ├── code_ast/ # Tree-sitter foundation: language registry (links all grammars), CodeSource, positions\n│ ├── ops_text/ # Text processing operations (splitter, language detection)\n│ ├── code_match/ # Structural code pattern matching over tree-sitter ASTs\n│ └── sdk/\n│ ├── cocoindex/ # Rust SDK (`cocoindex` crate); calls core engine APIs directly, like py/\n│ └── cocoindex_macros/ # Proc macros for the Rust SDK (`#[function]`)\n│\n├── python/\n│ ├── cocoindex/ # Python package\n│ │ ├── __init__.py # Package entry point\n│ │ ├── cli.py # CLI commands\n│ │ ├── _internal/ # Internal implementation for the core engine\n│ │ │ ├── api.py # Public API: mount, use_mount, mount_each, map, mount_target, App, fn, start/stop\n│ │ │ ├── app.py # App base implementation\n│ │ │ ├── context_keys.py # ContextKey and ContextProvider\n│ │ │ ├── environment.py # Environment and lifespan handling\n│ │ │ ├── function.py # @coco.fn decorator implementation\n│ │ │ ├── component_ctx.py # ComponentContext and component_subpath\n│ │ │ ├── target_state.py # Target state implementation\n│ │ │ └── core.pyi # Type stubs for the Rust extension module (update when PyO3 APIs change)\n│ │ ├── connectors/ # External system connectors (localfs, postgres, qdrant, lancedb, google_drive)\n│ │ ├── connectorkits/ # Connector building utilities\n│ │ ├── resources/ # Abstractions: file.py (FileLike), chunk.py (Chunk), schema.py\n│ │ └── ops/ # Operations: text.py (RecursiveSplitter), sentence_transformers.py\n│ └── tests/ # Python tests\n│\n├── examples/ # Example applications\n├── docs/ # Documentation\n└── dev/ # Development utilities\n```\n\n## Key Concepts\n\n### Mental model: declarative data pipelines\n\nCocoIndex uses a **declarative** programming model — you specify *what* your output should look like (target states), not *how* to incrementally update it. The engine handles change detection and applies minimal updates automatically.\n\nThink of it like:\n\n* **React**: declare UI as function of state → React re-renders what changed\n* **Spreadsheets**: declare formulas → cells recompute when inputs change\n* **CocoIndex**: declare target states as function of source → engine syncs what changed\n\n### Core concepts\n\n**App** — The top-level runnable unit. Bundles a main function with its arguments. When you call `app.update()`, the main function runs as the root processing component.\n\n**Processing Component** — The unit of execution that owns a set of target states. Created by `mount()` or `use_mount()` at a specific component path. When a component finishes, its target states sync atomically to external systems.\n\n**Component Path** — Stable identifier for a processing component across runs. Created via `coco.component_subpath(\"process\", filename)`. CocoIndex uses component paths to:\n\n* Match components to their previous runs for change detection\n* Determine ownership of target states (if a path disappears, its target states are cleaned up)\n\n**Target State** — What you want to exist in an external system (a file, a database row, a table). You *declare* target states; CocoIndex keeps them in sync — creating, updating, or removing as needed.\n\n**Target** — The API object used to declare target states (e.g., `DirTarget`, `TableTarget`). Targets can be nested: a container target state (directory/table) provides a Target for declaring child target states (files/rows).\n\n**Function** — A Python function decorated with `@coco.fn`. Use `memo=True` to enable memoization (skip execution when inputs and code are unchanged).\n\n**Context** — React-style provider mechanism for sharing resources. Define keys with `ContextKey[T]`, provide values in lifespan via `builder.provide()`, use in functions via `coco.use_context(key)`.\n\n### Key APIs\n\n```python\n# Mounting processing components (subpath auto-derived from fn.__name__)\nawait coco.mount(fn, *args, **kw) # child runs independently\nresult = await coco.use_mount(fn, *args, **kw) # returns value directly\n\n# Explicit subpath (for multi-part paths or multiple mounts of same function)\nawait coco.mount(coco.component_subpath(\"process\", filename), fn, *args, **kw)\nresult = await coco.use_mount(coco.component_subpath(\"name\"), fn, *args, **kw)\n\n# Component subpath composition\nsubpath = coco.component_subpath(\"process\", filename) # multiple parts\nsubpath = coco.component_subpath(\"a\") / \"b\" / \"c\" # chaining with /\n\n# Using component_subpath as context manager (applies to all nested mount calls)\nwith coco.component_subpath(\"process\"):\n for f in files:\n await coco.mount(coco.component_subpath(str(f.relative_path)), process_file, f, target)\n\n# Declaring target states (typically via Target methods)\ndir_target.declare_file(filename=name, content=data)\ntable_target.declare_row(row=MyRow(...))\n\n# Using context values\ndb = coco.use_context(PG_DB) # retrieve value provided in lifespan\n\n# Explicit context management (for ThreadPoolExecutor)\nctx = coco.get_component_context()\nwith ctx.attach():\n # coco APIs work correctly in this thread\n coco.mount(...)\n```\n\n**Mount handles:**\n\n* `mount()` → `ComponentMountHandle`: call `await handle.ready()` to wait until target states are synced\n* `use_mount()` → returns the result value directly (awaitable)\n\n### How syncing works\n\nWhen a processing component finishes, CocoIndex compares its declared target states with those from the previous run at the same component path:\n\n* New target states → create (insert row, create file)\n* Changed target states → update\n* Missing target states → delete\n\nChanges are applied atomically per component. If a source item is deleted (path no longer mounted), all its target states are cleaned up automatically.\n\n### Example\n\n```python\nimport pathlib\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, target: localfs.DirTarget) -> None:\n html = _markdown_it.render(await file.read_text())\n outname = \"__\".join(file.file_path.path.parts) + \".html\"\n target.declare_file(filename=outname, content=html)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n target = await coco.use_mount(localfs.declare_dir_target, outdir)\n\n files = localfs.walk_dir(\n sourcedir,\n recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n )\n await coco.mount_each(process_file, files.items(), target)\n\n\napp = coco.App(\n coco.AppConfig(name=\"FilesTransform\"),\n app_main,\n sourcedir=pathlib.Path(\"./docs\"),\n outdir=pathlib.Path(\"./out\"),\n)\napp.update_blocking(report_to_stdout=True)\n```\n\n## Code Conventions\n\n### Internal vs External Modules\n\nWe distinguish between **internal modules** (under packages with `_` prefix, e.g. `_internal.*` or `connectors.*._source`) and **external modules** (which users can directly import).\n\n**External modules** (user-facing, e.g. `cocoindex/ops/sentence_transformers.py`):\n\n* Be strict about not leaking implementation details\n* Use `__all__` to explicitly list public exports\n* Prefix ALL non-public symbols with `_`, including:\n * Standard library imports: `import threading as _threading`, `import typing as _typing`\n * Third-party imports: `import numpy as _np`, `from numpy.typing import NDArray as _NDArray`\n * Internal package imports: `from cocoindex.resources import schema as _schema`\n* Exception: `TYPE_CHECKING` imports for type hints don't need prefixing\n\n**Internal modules** (e.g. `cocoindex/_internal/component_ctx.py`, `**/_target.py`):\n\n* Less strict since users shouldn't import these directly\n* Standard library and internal imports don't need underscore prefix\n* Only prefix symbols that are truly private to the module itself (e.g. `_context_var` for a module-private ContextVar)\n\n### Minimize API surface until deemed necessary\n\nWhen adding a new public API (function, class, kwarg, configuration option), prefer the smallest surface that solves the concrete need in front of you. Do not pre-expose tunables, hooks, or alternatives \"in case someone needs them.\" Adding a kwarg later is straightforward and backwards-compatible; removing or renaming one is disruptive. If a knob is currently a hardcoded constant inside the implementation, leave it there until a real use case demands it — at which point promoting it to a kwarg is mechanical. This applies equally to optional parameters, callback hooks, parser/transformer plug-points, and configuration keys.\n\n### General principles (also covered by `/review-changes`)\n\n- **Top-level imports.** Defer to in-function only for a real circular dependency or a heavy import that isn't always needed.\n- **Specific types over `Any`.** Use concrete types — including concrete types from third-party libraries. `Any` only when the type is truly generic and no downstream code needs to downcast it.\n- **Validate and exchange early.** When a value enters as a weaker form (`str`, `Any`, raw identifier), convert to the strong type at the earliest point. Don't propagate the weak form. Concrete example in this codebase: connector handlers receive a `ContextKey` string as part of a target-state key. The parent handler resolves the key once (when constructing the child handler) and passes the typed connection directly — the child stores `_pool: asyncpg.Pool`, not `_db_key: str`.\n- **`NamedTuple`/small dataclass for multi-value returns.** Access fields by name (`result.can_reuse`) at call sites.\n- **Exceptions for exceptional situations only.** Reserve exceptions (Python) and errors (Rust) for truly unexpected failures — not for end-of-iteration, \"not found\", or other expected state transitions. Use explicit return values (sentinel, enum variant, `None`) for those.\n- **Single source of truth, delete dead code, honest names.** When the same value/logic appears in multiple places, consolidate. When changes make code unreachable, delete it (and its tests, and any dead config knobs). Function and field names should describe what the implementation does today.\n\n- Rust code follows [dev/rust-guidelines.md](dev/rust-guidelines.md).\n\n### Testing Guidelines\n\nWe prefer end-to-end tests on user-facing APIs, over unit tests on smaller internal functions. With this said, there're cases where unit tests are necessary, e.g. for internal logic with various situations and edge cases, in which case it's usually easier to cover various scenarios with unit tests.\n\n#### Test Environment Setup\n\nUse `common.create_test_env(__file__)` to create a CocoIndex `Environment` for tests. It derives a unique `db_path` from the test file path and picks up the current event loop automatically.\n\n* **Sync tests** (module-level creation): Call at module level — the Environment creates a background loop for async callbacks.\n\n ```python\n coco_env = common.create_test_env(__file__)\n ```\n\n* **Async tests with async resources** (e.g., asyncpg pools): Call inside an async fixture so the Environment binds to the test's running event loop (same loop the pool is on). Use the `suffix` parameter when each test needs its own Environment:\n\n ```python\n @pytest_asyncio.fixture\n async def pg_env(request: pytest.FixtureRequest) -> Any:\n pool = await asyncpg.create_pool(dsn)\n coco_env = common.create_test_env(__file__, suffix=request.node.name)\n coco_env.context_provider.provide(DB_KEY, pool)\n yield pool, coco_env\n await pool.close()\n ```\n\n The `suffix` ensures each test gets a unique `db_path`, avoiding \"environment already open\" errors.\n\n#### Testcontainers for Database Tests\n\nUse `testcontainers[postgres]` (in the `build-test` dependency group) to spin up real database instances automatically — no manual setup or environment variables needed. Use a module-scoped sync fixture for the container and a function-scoped async fixture for per-test resources:\n\n```python\n_PG_IMAGE = \"postgres:16-alpine\"\n\npytestmark = [pytest.mark.requires_docker(_PG_IMAGE)]\n\n@pytest.fixture(scope=\"module\")\ndef pg_dsn() -> Any:\n with PostgresContainer(_PG_IMAGE) as pg:\n dsn = pg.get_connection_url().replace(\"postgresql+psycopg2://\", \"postgresql://\")\n yield dsn\n\n@pytest_asyncio.fixture\nasync def pool(pg_dsn: str) -> Any:\n p = await asyncpg.create_pool(pg_dsn)\n yield p\n await p.close()\n```\n\nList every image a test's containers use in its `requires_docker(...)` marker. `python/tests/conftest.py` pulls missing images after collection, before the first test runs; otherwise testcontainers pulls inside the fixture, and a cold-cache download counts against the first test's `pytest-timeout`.\n\n### LMDB write paths\n\n- All LMDB writes must go through `Storage::run_txn` (uses the single-writer batcher).\n- Do not open a heed write txn directly or wrap the env in a separate mutex/semaphore — bypassing the batcher loses fsync coalescing and regresses concurrent-submit throughput by 10-100×.\n- LMDB has no savepoints. If a sub-operation needs to \"abort\" without failing its caller, handle it at the body level (e.g. return a sentinel result without writing). A body that returns `Err` fails only its own caller: the batcher drops its writes by aborting the txn and re-running the bodies ahead of it, so every body must be safe to re-run.\n- An LMDB write txn must begin and end (commit or abort) on the same OS thread: the writer lock is thread-owned and LMDB ignores a failed release, so a txn that migrates between runtime workers blocks every later writer for good. heed marks `RwTxn` as `Send` regardless, so the compiler won't catch it. `Storage::run_txn` runs each batch on one blocking thread for this reason; never hold a write txn across an `.await` on the multi-thread runtime.\n\n### Sync vs Async\n\nThe Rust core (`rust/core`, `rust/utils`) uses **async-first** design with Tokio. The `rust/py` crate bridges Rust async to Python, offering both sync and async APIs:\n\n* Rust core exposes async functions\n* `rust/py` provides sync wrappers that use `block_on()` to call async Rust from sync Python\n* Python's `cocoindex` API is **async-first**: mount APIs (`mount`, `use_mount`, `mount_each`, `map`) are all `async def`; `App.update()`/`App.drop()` are async; sync entry points (`App.update_blocking()`, `App.drop_blocking()`, `start_blocking()`, `stop_blocking()`) are available for CLI and blocking contexts\n\nWhen adding new functionality that involves I/O or concurrency:\n\n* Implement async in Rust\n* Bridge to Python via `rust/py`, providing both sync and async variants if needed\n\n## Versioning\n\nThe current codebase is for CocoIndex v1, which is a fundamental redesign from CocoIndex v0. CocoIndex v1 lives on the `main` branch; the legacy v0 code is preserved on the `v0` branch.\n\n## Docs diagrams\n\nAll diagrams embedded in docs pages are built as inline Astro components under `docs/src/components/diagrams/`, not exported SVG files. See [docs/src/components/diagrams/README.md](docs/src/components/diagrams/README.md) for the palette, primitives, shared CSS classes, animation conventions, and step-by-step authoring guidelines. Follow that guide for any new or updated diagram.\n\n## Agent Tooling\n\n### Semantic code search over this repo (`cocoindex-code`)\n\nTo find where a concept lives, prefer semantic search over blind `grep`. This repo keeps a local\n`cocoindex-code` index; query it from any agent's shell with the `ccc` CLI:\n\n```bash\nccc search \"where is X handled\" # semantic search; add --refresh to reindex first\n```\n\nIf `ccc search` reports no index, set it up once:\n\n```bash\ncommand -v ccc >/dev/null || uv tool install 'cocoindex-code[full]' # per machine; local embeddings, no API key\nccc init -f && ccc index # repo-scoped index (first build ~1-2 min)\n```\n\nEverything is local — SQLite index, local embeddings, no keys — and a background daemon keeps it\nfresh. On a brand-new machine `ccc init` asks once for an embedding provider; choose the local\ndefault. The `-f` matters: if a parent directory is also a `ccc` project, plain `ccc init` refuses\nand search silently uses the parent's index, so run `ccc status` to confirm the root is this repo.\nWe register no MCP server on purpose — an `ccc mcp` mode exists, but the CLI costs no context until\ninvoked and behaves the same across agents. Source:\n[github.com/cocoindex-io/cocoindex-code](https://github.com/cocoindex-io/cocoindex-code).\n\n## Agent Playbooks\n\nReusable task-specific agent guidance lives under `dev/agent-skills/`. These playbooks use a `SKILL.md` format so Claude Code can load them directly from `.claude/skills/`, but the content is intended to be readable by any coding agent.\n\n- For docs diagrams, read `dev/agent-skills/cocoindex-diagrams/SKILL.md`.\n- For new target connectors, read `dev/agent-skills/target-connector/SKILL.md`.\n- For upgrading example package versions, read `dev/agent-skills/upgrade-examples/SKILL.md`.\n\nAgent Skills-compatible clients, including Codex, discover the checked-in symlinks under `.agents/skills/`. The public CocoIndex skill is exposed there as `cocoindex`, and contributor playbooks are exposed there from `dev/agent-skills/`.\n\nClaude Code compatibility remains under `.claude/`: hooks/settings live there, and `.claude/skills/` mirrors the contributor playbooks with symlinks. Do not duplicate skill content under `.agents/skills/` or `.claude/skills/`; edit the source directories (`skills/cocoindex/` for the public skill, `dev/agent-skills/` for contributor playbooks) and keep dot-directory entries as symlinks.\n\nPortable agent checks live under `dev/agent-checks/`. Claude Code hooks in `.claude/hooks/` are thin adapters that call these scripts when relevant files changed.\n","CLAUDE.md":"# CLAUDE.md\n\n@AGENTS.md\n","skills/cocoindex/SKILL.md":"---\nname: cocoindex\ndescription: This skill should be used when building data processing pipelines with CocoIndex, a Python library for incremental data transformation. Use when the task involves processing files/data into databases, creating vector embeddings, building knowledge graphs, ETL workflows, or any data pipeline requiring automatic change detection and incremental updates. CocoIndex is Python-native (supports any Python types), has no DSL, and uses version 1.0.0 or later.\n---\n\n# CocoIndex\n\nCocoIndex is a Python library for building incremental data processing pipelines with declarative target states. Think spreadsheets or React for data pipelines: declare what the output should look like based on current input, and CocoIndex automatically handles incremental updates, change detection, and syncing to external systems.\n\n## Overview\n\nCocoIndex enables building data pipelines that:\n\n- **Automatically handle incremental updates**: Only reprocess changed data\n- **Use declarative target states**: Declare what should exist, not how to update\n- **Support any Python types**: No custom DSL -- use dataclasses, Pydantic, NamedTuple\n- **Provide function memoization**: Skip expensive operations when inputs/code unchanged\n- **Sync to multiple targets**: PostgreSQL, SQLite, LanceDB, Qdrant, SurrealDB, Apache Doris, file systems, Kafka\n\n**Key principle**: `TargetState = Transform(SourceState)`\n\n## When to Use This Skill\n\nUse this skill when building pipelines that involve:\n\n- **Document processing**: PDF/Markdown conversion, text extraction, chunking\n- **Vector embeddings**: Embedding documents/code for semantic search\n- **Database transformations**: ETL from source DB to target DB\n- **Knowledge graphs**: Extract entities and relationships from data\n- **LLM-based extraction**: Structured data extraction using LLMs\n- **File-based pipelines**: Transform files from one format to another\n- **Incremental indexing**: Keep search indexes up-to-date with source changes\n- **Streaming pipelines**: Kafka-based real-time data processing\n\n## Quick Start: Creating a New Project\n\n### Initialize Project\n\n```bash\ncocoindex init my-project\ncd my-project\n```\n\nThis creates: `main.py`, `pyproject.toml`, `README.md`. The generated `main.py` sets the database location in its lifespan via `builder.settings.db_path = pathlib.Path(\"./cocoindex.db\")`.\n\n### Add Dependencies\n\n```toml\n# For vector embeddings with PostgreSQL\ndependencies = [\"cocoindex>=1.0.0\", \"sentence-transformers\", \"asyncpg\"]\n\n# For LLM extraction\ndependencies = [\"cocoindex>=1.0.0\", \"litellm\", \"instructor\", \"pydantic>=2.0\"]\n```\n\nSee [references/setup_project.md](references/setup_project.md) for complete examples.\n\n### Run the Pipeline\n\n```bash\nuv run cocoindex update main.py # installs dependencies, then runs\n```\n\n## Core Concepts\n\n### 1. Apps\n\nAn **App** is the top-level executable that binds a main function with parameters:\n\n```python\nimport cocoindex as coco\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n ...\n\napp = coco.App(\n coco.AppConfig(name=\"MyApp\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n)\n```\n\n### 2. Functions (`@coco.fn`)\n\nThe `@coco.fn` decorator marks functions as CocoIndex processing functions. Add `memo=True` to skip re-execution when inputs/code are unchanged:\n\n```python\n@coco.fn(memo=True)\nasync def expensive_operation(data: str) -> Result:\n # LLM call, embedding generation, heavy computation\n return await expensive_transform(data)\n```\n\n**Key parameters:**\n- `memo=True` -- Enable memoization (skip if inputs/code unchanged)\n- `version=1` -- Explicit version bump to force re-execution\n- `batching=True` -- Auto-batch concurrent calls (async only)\n- `runner=coco.GPU` -- Serialize GPU-bound execution\n\n### 3. Processing Components\n\nA **processing component** groups an item's processing with its target states.\n\n**Mount components** with `mount_each()` (preferred for lists) or `mount()`:\n\n```python\n# One component per item (preferred for lists)\nawait coco.mount_each(process_file, files.items(), target_table)\n\n# Single component (subpath auto-derived from fn.__name__)\nawait coco.mount(setup_fn, arg1)\n\n# Dependent component (blocks until result returned)\nresult = await coco.use_mount(init_fn)\n\n# Explicit subpath (when you need a specific path, e.g. in loops)\nawait coco.mount(coco.component_subpath(\"item\", item_id), process_item, item)\n```\n\n**Key points:**\n- All mount APIs are `async`\n- `mount()`, `use_mount()`, and `mount_each()` auto-derive subpath from `fn.__name__`; optional explicit subpath as first arg\n- Use `use_mount()` when you need the return value\n- Use stable component paths for proper memoization and cleanup\n\n### 4. Target States\n\n**Declare** what should exist -- CocoIndex handles creation/update/deletion:\n\n```python\n# Database row target\ntable.declare_row(row=MyRecord(id=1, name=\"example\"))\n\n# File target\nlocalfs.declare_file(outdir / \"output.txt\", content, create_parent_dirs=True)\n\n# Kafka message target\ntopic_target.declare_target_state(key=\"msg-1\", value=json.dumps(data))\n```\n\n### 5. Context for Shared Resources\n\nUse `ContextKey` to share expensive resources (DB connections, models) across components:\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n# In processing functions:\nembedder = coco.use_context(EMBEDDER)\n```\n\nThe `@coco.lifespan` decorator registers the function to the default CocoIndex environment, which is shared among all apps by default. `ContextKey` also serves as the stable identity for sources/targets -- the `key` string must remain stable across runs.\n\n### 6. ID Generation\n\nGenerate stable, unique identifiers that persist across incremental updates:\n\n```python\nfrom cocoindex.resources.id import generate_id, IdGenerator\n\n# Deterministic: same dep -> same ID\nchunk_id = await generate_id(chunk.text)\n\n# Always distinct: each call -> new ID, even with same dep\nid_gen = IdGenerator()\nfor chunk in chunks:\n chunk_id = await id_gen.next_id(chunk.text)\n```\n\n### 7. Catch-Up vs Live Mode\n\nBy default, `app.update()` runs in **catch-up mode**: it scans all sources, processes what changed since the last run (memoized components are skipped), syncs target states, and returns. Each call still has to scan sources to discover changes.\n\n**Live mode** keeps the app running after catch-up and lets components stream changes continuously from their sources (e.g., file watcher, Kafka consumer), applying them with very low latency.\n\n```python\n# Enable live mode\napp.update_blocking(live=True)\n# Or: cocoindex update main.py -L\n```\n\nTwo things are needed: (1) enable live mode on the app, and (2) use a source that supports live updates.\n\n- **`LiveMapView`** sources (e.g., `localfs.walk_dir(..., live=True)`) scan current state first, then watch for changes. They also work in catch-up mode -- write the pipeline once, choose mode at run time.\n- **`LiveMapFeed`** sources (e.g., `kafka.topic_as_map()`) only stream changes with no initial snapshot. `mount_each()` auto-detects these and creates a live component internally.\n\n```python\n# LocalFS with live watching\nfiles = localfs.walk_dir(sourcedir, live=True, ...)\nawait coco.mount_each(process_file, files.items(), target)\n\n# Kafka -- inherently live\nitems = kafka.topic_as_map(consumer, [\"my-topic\"])\nawait coco.mount_each(process_message, items, target)\n```\n\n## Common Pipeline Patterns\n\n### Pattern 1: File Transformation\n\n```python\nimport pathlib\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, outdir: pathlib.Path) -> None:\n content = await file.read_text()\n transformed = transform_content(content)\n outname = file.file_path.path.stem + \".out\"\n localfs.declare_file(outdir / outname, transformed, create_parent_dirs=True)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n files = localfs.walk_dir(\n sourcedir,\n recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n )\n await coco.mount_each(process_file, files.items(), outdir)\n\napp = coco.App(\n coco.AppConfig(name=\"Transform\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n outdir=pathlib.Path(\"./out\"),\n)\n```\n\n### Pattern 2: Vector Embedding Pipeline\n\n```python\nimport pathlib\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator, Annotated\n\nimport asyncpg\nfrom numpy.typing import NDArray\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs, postgres\nfrom cocoindex.ops.text import RecursiveSplitter\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\nfrom cocoindex.resources.chunk import Chunk\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom cocoindex.resources.id import IdGenerator\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n_splitter = RecursiveSplitter()\n\n@dataclass\nclass DocEmbedding:\n id: int\n filename: str\n text: str\n embedding: Annotated[NDArray, EMBEDDER] # Dimensions inferred from ContextKey\n chunk_start: int\n chunk_end: int\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n@coco.fn\nasync def process_chunk(\n chunk: Chunk, filename: pathlib.PurePath,\n id_gen: IdGenerator, table: postgres.TableTarget[DocEmbedding],\n) -> None:\n table.declare_row(row=DocEmbedding(\n id=await id_gen.next_id(chunk.text),\n filename=str(filename),\n text=chunk.text,\n embedding=await coco.use_context(EMBEDDER).embed(chunk.text),\n chunk_start=chunk.start.char_offset,\n chunk_end=chunk.end.char_offset,\n ))\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, table: postgres.TableTarget[DocEmbedding]) -> None:\n text = await file.read_text()\n chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500)\n id_gen = IdGenerator()\n await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n target_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"embeddings\",\n table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=[\"id\"]),\n )\n target_table.declare_vector_index(column=\"embedding\")\n\n files = localfs.walk_dir(sourcedir, recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]))\n await coco.mount_each(process_file, files.items(), target_table)\n\napp = coco.App(coco.AppConfig(name=\"Embedding\"), app_main, sourcedir=pathlib.Path(\"./data\"))\n```\n\n### Pattern 3: LLM-Based Extraction\n\n```python\nimport instructor\nfrom pydantic import BaseModel\nfrom litellm import acompletion\n\n_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)\n\nclass ExtractionResult(BaseModel):\n title: str\n topics: list[str]\n\n@coco.fn(memo=True) # Memo avoids re-calling LLM\nasync def extract_and_store(content: str, message_id: int, table) -> None:\n result = await _instructor_client.chat.completions.create(\n model=\"gpt-4\",\n response_model=ExtractionResult,\n messages=[{\"role\": \"user\", \"content\": f\"Extract topics: {content}\"}],\n )\n table.declare_row(row=Message(id=message_id, title=result.title, content=content))\n```\n\n## Connectors and Operations\n\nCocoIndex provides connectors for reading from and writing to external systems:\n\n| Connector | Source | Target | Vectors | Use Case |\n|-----------|--------|--------|---------|----------|\n| **PostgreSQL** | Y | Y | pgvector | Production SQL + vectors |\n| **SQLite** | - | Y | sqlite-vec | Local SQL + vectors |\n| **LanceDB** | - | Y | Y | Cloud-native vector DB |\n| **Qdrant** | - | Y | Y | Specialized vector DB |\n| **SurrealDB** | - | Y | Y | Graph + document DB |\n| **Apache Doris** | - | Y | Y | Analytical DB + vectors |\n| **LocalFS** | Y | Y | N/A | File-based pipelines |\n| **Amazon S3** | Y | - | N/A | Cloud object storage |\n| **Kafka** | Y | Y | N/A | Streaming pipelines |\n| **Google Drive** | Y | - | N/A | Cloud file source |\n\nFor detailed connector documentation, see [references/connectors.md](references/connectors.md).\n\n## Text and Embedding Operations\n\n### Text Splitting\n\n```python\nfrom cocoindex.ops.text import RecursiveSplitter, detect_code_language\n\nsplitter = RecursiveSplitter()\nlanguage = detect_code_language(filename=\"example.py\")\nchunks = splitter.split(text, chunk_size=1000, chunk_overlap=200, language=language)\n```\n\n### Embeddings\n\n```python\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\n\nembedder = SentenceTransformerEmbedder(\"sentence-transformers/all-MiniLM-L6-v2\")\nembedding = await embedder.embed(text) # Returns NDArray\n```\n\n## CLI Commands\n\n```bash\ncocoindex init my-project # Create new project\ncocoindex update main.py # Run app\ncocoindex update main.py:my_app # Run specific app\ncocoindex update main.py -L # Run in live mode (continuous)\ncocoindex update main.py --full-reprocess # Reprocess everything\ncocoindex drop main.py [-f] # Drop and reset all state\ncocoindex ls [main.py] # List apps\ncocoindex show main.py [--tree] # Show component paths\n```\n\n## Best Practices\n\n### 1. Use `@coco.fn` on All Processing Functions\n\nEvery function that participates in the pipeline (declares target states, calls mount APIs, etc.) must be decorated with `@coco.fn`.\n\n### 2. Add Memoization for Expensive Operations\n\n```python\n@coco.fn(memo=True) # Skip re-execution when inputs/code unchanged\nasync def process_chunk(chunk, table):\n embedding = await embedder.embed(chunk.text) # Expensive!\n table.declare_row(...)\n```\n\n### 3. Use Stable Component Paths\n\n```python\n# Good: Stable identifiers\ncoco.component_subpath(\"file\", str(file.file_path.path))\ncoco.component_subpath(\"record\", record.id)\n\n# Bad: Unstable identifiers\ncoco.component_subpath(\"file\", file) # Object reference\ncoco.component_subpath(\"idx\", idx) # Index changes\n```\n\n### 4. Use Context for Shared Resources\n\n```python\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n yield\n```\n\n### 5. Use `Annotated[NDArray, CONTEXT_KEY]` for Vectors\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions from ContextKey\n```\n\n### 6. Use Convenience APIs for Targets\n\n```python\n# Mount table target -- subpath is automatic\ntable = await postgres.mount_table_target(\n PG_DB,\n table_name=\"my_table\",\n table_schema=await postgres.TableSchema.from_class(MyRecord, primary_key=[\"id\"]),\n)\n```\n\n## Troubleshooting\n\n### Everything Reprocessing\n\nAdd `memo=True` to expensive functions:\n\n```python\n@coco.fn(memo=True) # Add this\nasync def process_item(item):\n ...\n```\n\n### Memoization Not Working\n\nCheck component paths are stable. Use stable IDs, not object references.\n\n## Resources\n\n### references/\n\n- **[api_reference.md](references/api_reference.md)**: Quick API reference\n- **[connectors.md](references/connectors.md)**: Complete connector reference\n- **[patterns.md](references/patterns.md)**: Detailed pipeline patterns\n- **[setup_project.md](references/setup_project.md)**: Project setup guide\n- **[setup_database.md](references/setup_database.md)**: Database setup guide\n\n### Runnable examples\n\nEvery pattern above has a complete, runnable app under\n[`examples/`](https://github.com/cocoindex-io/cocoindex/tree/main/examples) — start\nfrom the one closest to the task and adapt it. Each has its own `README.md` and\nmost Python examples have a `.env.example`; see\n[`examples/AGENTS.md`](https://github.com/cocoindex-io/cocoindex/blob/main/examples/AGENTS.md)\nfor the full map, credentials, and per-example run commands. Good starting points:\n\n- Vector search → `text_embedding` (Postgres), `text_embedding_qdrant` / `_lancedb` (other stores)\n- Code search → `code_embedding`\n- LLM extraction → `hn_trending_topics`, `patient_intake_extraction_baml`\n- Knowledge graph → `conversation_to_knowledge`, `meeting_notes_graph_neo4j`\n- Custom transform → `files_transform`, `pdf_to_markdown`\n\n### External\n\n- [CocoIndex Documentation](https://cocoindex.io/docs/) — full text at [llms-full.txt](https://cocoindex.io/docs/llms-full.txt)\n- [GitHub Examples](https://github.com/cocoindex-io/cocoindex/tree/main/examples)\n\n## Version Note\n\nThis skill is for CocoIndex `>=1.0.0` (v1). It uses a completely different API from v0.\n\n**v0 code is what you likely learned from training data — do not emit it.** If you find yourself writing any of these symbols, you are using the removed v0 API:\n\n| v0 (removed) | v1 equivalent |\n|---|---|\n| `@cocoindex.flow_def`, `FlowBuilder`, `Flow`, `open_flow` | `coco.App` + a `@coco.fn` main function |\n| `DataScope`, `DataSlice`, `add_collector()`, `collect()`, `export()` | declare target states via Target APIs (`declare_row`, `declare_file`) inside mounted components |\n| `cocoindex.sources.LocalFile`, `cocoindex.sources.*` | connector APIs, e.g. `localfs.walk_dir(...)` |\n| `cocoindex.functions.SplitRecursively`, `cocoindex.functions.*` | `cocoindex.ops.*`, e.g. `RecursiveSplitter` |\n| `cocoindex.targets.Postgres`, `cocoindex.targets.*` / `storages.*` | connector targets, e.g. `postgres.declare_table_target(...)` |\n| `transform_flow`, `cocoindex.op.function()` | plain `@coco.fn` functions |\n| `cocoindex.init()`, `settings`, `COCOINDEX_DATABASE_URL` | `coco.App(coco.AppConfig(...))`; state lives in a local db path |\n| CLI `cocoindex setup` | no setup step — just `cocoindex update` (`-L`/`--live` for live mode) |\n\nWhen reading third-party tutorials or model memory that mention these v0 symbols, disregard them and use the patterns in this skill and `references/api_reference.md` instead.\n","examples/AGENTS.md":"# AGENTS.md — CocoIndex examples\n\nGuidance for AI coding agents (Claude Code, Cursor, etc.) working in this `examples/`\ndirectory. Most top-level Python subfolders are self-contained, runnable\nCocoIndex **v1** apps; `rust/` contains Rust ports with per-example READMEs.\n\n## Before you write CocoIndex code: install the skill\n\nCocoIndex v1 is a fundamental redesign from v0. Without context, LLMs tend to\nhallucinate the v0 flow-builder DSL and deprecated decorators. Install the\nofficial skill first — it teaches the correct v1 API. Quickest portable path\nfor Codex and other Agent Skills-compatible clients is the hosted skill:\n\n```sh\nmkdir -p .agents/skills/cocoindex/references\ncurl -fsSL https://cocoindex.io/docs/skill.md -o .agents/skills/cocoindex/SKILL.md\nfor f in api_reference connectors patterns setup_database setup_project; do\n curl -fsSL https://cocoindex.io/docs/references/$f.md -o .agents/skills/cocoindex/references/$f.md\ndone\n```\n\nOr clone the repo and copy the folder:\n\n```sh\ngit clone --depth=1 https://github.com/cocoindex-io/cocoindex.git /tmp/cocoindex-skill\nmkdir -p .agents/skills && cp -r /tmp/cocoindex-skill/skills/cocoindex .agents/skills/\n```\n\nFor Claude Code's native project path, use `.claude/skills/cocoindex` instead\nof `.agents/skills/cocoindex`; the skill format is the same.\n\nThe skill itself lives at [`skills/cocoindex/`](../skills/cocoindex) (SKILL.md +\n`references/`). For Cursor, copy `SKILL.md` into `.cursor/rules/`. Full machine-readable\ndocs: (index) and\n (everything, including example\nwalkthroughs). Regular docs pages also have raw Markdown twins by replacing the\ntrailing slash with `.md` (e.g.\n`https://cocoindex.io/docs/programming_guide/core_concepts.md`).\n\n## The v1 mental model\n\n`target_state = transform(source_state)`. You declare what the target should look\nlike; the Rust engine keeps it in sync, reprocessing only what changed (state is\ntracked in a local LMDB store — **no database is required for the engine itself**,\nonly when an example writes to one). Key APIs: `@coco.fn`, `mount` / `use_mount` /\n`mount_each`, `ContextKey`, target-state declarations. See the skill for details.\n\n## Running examples\n\nMost Python examples are standalone projects with their own `pyproject.toml`:\n\n```sh\ncd \ncp .env.example .env # if present — fill in the blanks (see below)\npip install -e . # or: uv pip install -e .\ncocoindex update main # catch-up: scan sources, sync, exit\ncocoindex update -L main # live mode: catch up, then watch for changes (where supported)\n```\n\nUse the example's README as the source of truth. Known exceptions:\n\n- `multi_codebase_summarization`, `audio_to_text`,\n `patient_intake_extraction_baml`, `patient_intake_extraction_dspy`:\n `cocoindex update main.py`\n- `conversation_to_knowledge`: `cocoindex update conv_knowledge.app`\n- `image_search`, `image_search_colpali`: start the API with\n `python -m uvicorn api:app --reload --host 0.0.0.0 --port 8000`; it runs the\n CocoIndex app in live mode, then start the frontend from `frontend/`.\n- `csv_to_kafka`, `kafka_to_lancedb`: `cocoindex update -L main.py`\n- `rust/`: follow that example's README. Many use\n `cargo run -- index` for indexing and `cargo run -- query \"...\"` for search;\n a few take custom paths or service-specific subcommands.\n\nSome examples expose a query/CLI demo via `python main.py \"\"`; check the\nexample's `README.md`. Examples that need extra services or a code-gen step\n(e.g. `baml generate`) say so in their README.\n\n## Environment / credentials\n\nWhen an example needs credentials or service configuration, required env vars\nare templated in that example's **`.env.example`** — `cp` it to `.env` and fill\nin the blanks; both `python main.py` and the `cocoindex` CLI load `.env`\nautomatically. Common ones:\n\n- `POSTGRES_URL` — for Postgres/pgvector targets. Local instance:\n `docker compose -f ../../dev/postgres.yaml up -d` from inside an example\n directory.\n- `OPENAI_API_KEY` / `GEMINI_API_KEY` — for examples that call an LLM.\n- Service-specific (`QDRANT_URL`, `LANCEDB_URI`, `NEO4J_*`, `KAFKA_*`,\n `GOOGLE_SERVICE_ACCOUNT_CREDENTIAL`, …) — see that example's `.env.example`.\n\nExamples with no `.env.example` (e.g. `files_transform`, `pdf_to_markdown`) run\nfully locally with no credentials.\n\n**Never commit secrets.** The `.env` files tracked in this repo hold only\nnon-secret defaults (`COCOINDEX_DB`, local service URLs); keep API keys and\ncredentials in your local `.env` edits and out of commits.\n\n## The examples\n\nA walkthrough URL means there's a step-by-step guide at\n`https://cocoindex.io/docs/examples//`; otherwise start from the example's README.\n\n### Vector indexes (embed → store → search by meaning)\n- `text_embedding` — Markdown → pgvector; the simplest end-to-end index. *(walkthrough: text-embedding)*\n- `code_embedding` — repo → Tree-sitter chunks → pgvector; query code in English. *(walkthrough: index-codebase)*\n- `text_embedding_qdrant` / `text_embedding_lancedb` / `text_embedding_turbopuffer` — same flow, different vector store.\n- `code_embedding_lancedb` — code chunks → LanceDB.\n- `pdf_embedding` — PDFs → markdown → chunks → pgvector.\n- `paper_metadata` — extract title/authors/abstract from PDFs → Postgres + embeddings.\n- `amazon_s3_embedding` / `gdrive_text_embedding` / `oci_object_storage_embedding` — same flow, remote source (S3 / Google Drive / OCI).\n- `postgres_source` — read from an existing Postgres table as the source.\n- `entire_session_search` — semantic search over AI coding sessions captured by Entire.\n- `sec_edgar_analytics` — multi-format SEC filings → Apache Doris with a vector **and** a full-text index; hybrid (semantic + keyword) RRF search. *(walkthrough: sec-edgar-analytics)*\n\n### Multimodal\n- `image_search` — CLIP embeddings + Qdrant, queried via FastAPI + React.\n- `image_search_colpali` — ColPali multi-vector model + Qdrant MaxSim.\n- `multi_format_indexing` — PDFs + images as page screenshots → ColPali → Qdrant; no OCR, no chunking. *(walkthrough: multi-format-indexing)*\n- `face_recognition` — detect faces (dlib) → 128-d embeddings → Qdrant face search. *(walkthrough: face-recognition)*\n- `audio_to_text` — transcribe audio with LiteLLM → Postgres.\n- `slides_to_speech` — slides → vision-LLM notes → Pocket TTS narration → LanceDB semantic search. *(walkthrough: slides-to-speech)*\n\n### Structured extraction (LLM / BAML / DSPy)\n- `multi_codebase_summarization` — LLM per-file summaries across many repos. *(walkthrough: multi-codebase-summarization)*\n- `hn_trending_topics` — scrape HackerNews → LLM topic extraction → Postgres.\n- `manuals_llm_extraction` — PDF manuals → Markdown (docling) → typed module records → Postgres. *(walkthrough: manuals-llm-extraction)*\n- `patient_intake_extraction_baml` / `patient_intake_extraction_dspy` — structured PDF extraction with BAML / DSPy (Gemini vision).\n\n### Knowledge graphs\n- `conversation_to_knowledge` — YouTube podcasts → SurrealDB knowledge graph. *(walkthrough: podcast-to-knowledge-graph)*\n- `docs_to_knowledge_graph` — Markdown docs → Neo4j concept graph of LLM-extracted triples. *(walkthrough: docs-to-knowledge-graph)*\n- `product_recommendation` — product catalog → LLM taxonomy extraction → Neo4j recommendation graph. *(walkthrough: product-recommendation)*\n- `meeting_notes_graph_neo4j` / `meeting_notes_graph_falkordb` — Google Drive meeting notes → Neo4j / FalkorDB graph.\n- `meeting_notes_graph_surrealdb` — local Markdown meeting notes → SurrealDB graph.\n\n### Custom sources / targets / streaming\n- `pdf_to_markdown` — incremental PDF → Markdown with docling (local, no services). *(walkthrough: pdf-to-markdown)*\n- `files_transform` — watch Markdown files → HTML, live mode (local, no services).\n- `csv_to_kafka` — watch CSVs → publish rows to Kafka.\n- `kafka_to_lancedb` — consume Kafka → route to LanceDB tables.\n\n### Rust\n- `rust/` — Rust ports of many of the above, using the CocoIndex Rust API.\n\n## Conventions for edits\n\n- Keep each Python example self-contained: its own `pyproject.toml` and\n `README.md`; add `.env.example` when credentials or configurable services are\n required. When you add an example, also add a line to `EXAMPLE_CATALOG` in the\n docs repo (`docs/src/data/examples.ts`) so it appears in `/docs/llms.txt`.\n- Match the surrounding code's low comment density.\n- Don't commit generated artifacts (`cocoindex.db`, `__pycache__`, build output) —\n they're already git-ignored.\n","dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md)","dev/agent-skills/cocoindex-diagrams/references/pitfalls.md":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/pitfalls.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/pitfalls.md)","dev/agent-skills/cocoindex-diagrams/references/workflow.md":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/workflow.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/workflow.md)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\nThis file provides guidance to coding agents working with code in this repository.\nClaude Code compatibility is kept through `CLAUDE.md`, which imports this file.\n\n## Build and Test Commands\n\nThis project uses [uv](https://docs.astral.sh/uv/) for Python project management.\n\n### Building\n\n```bash\nuv run maturin develop # Build Rust code and install Python package (required after Rust changes)\n```\n\n### Testing\n\n```bash\ncargo test # Run Rust tests\nuv run mypy # Type check Python code\nuv run pytest python/ # Run Python tests (use after both Rust and Python changes)\n```\n\n### Code Formatting and Linting\n\nPython linting and formatting go through `prek`, which runs the `ruff-check` and\n`ruff-format` hooks from `.pre-commit-config.yaml` with the exact ruff version\nand file set CI uses. A `ruff` from `uv run`, `uvx`, or Homebrew is usually a\ndifferent version than the hook pins, so run the hooks rather than `ruff` itself.\n\n```bash\nuv tool install prek # one-time; some hooks also need `protoc` (brew install protobuf)\nprek run ruff-check ruff-format --all-files # Lint (applying safe fixes), then format Python code (same as CI)\ncargo fmt # Format Rust code\n./dev/run_cargo_clippy.sh # Lint Rust code (same as CI)\n```\n\n### Pre-submission Checks\n\nCI enforces the full pre-commit hook suite from `.pre-commit-config.yaml`, which\ncovers more than the commands above — notably regenerating the CLI docs\n(`docs/src/content/docs/cli.mdx`) whenever `cli.py` changes. Run it locally\nbefore pushing:\n\n```bash\nprek run --all-files\n```\n\n### Pull Requests and CI\n\nOpen a pull request as a draft and mark it ready once the checks above pass:\ndrafts run no CI. When `main` has a merge queue, a ready pull request runs only\nthe fast checks, and the queue runs the full suite on the commit that lands.\nFor a full run on a branch before then, use `gh workflow run CI.yml --ref `.\nThe CI tiers, the merge queue and stacked pull requests are described in\n[CI.md](https://github.com/cocoindex-io/.github/blob/main/CI.md).\n\n### Workflow Summary\n\n| Change Type | Commands to Run |\n|-------------|-----------------|\n| Rust code only | `uv run maturin develop && cargo test` |\n| Python code only | `uv run mypy && uv run pytest python/` |\n| Both Rust and Python | Run all commands from both categories above |\n| Python lint and format | `prek run ruff-check ruff-format --all-files` |\n| Before pushing | `prek run --all-files` |\n\nAfter changing a core engine API (`rust/core`), run `cargo check --workspace --all-targets` to catch call sites in every workspace crate — both `rust/py` and the Rust SDK under `rust/sdk/` call core directly, and the SDK is easy to miss when grepping. The `examples/rust/*` and `benchmarks/*/rust` crates are standalone workspaces that consume the SDK; the `prek` suite's `cargo check (rust examples + benchmarks)` hook covers those.\n\n## Code Structure\n\n```\ncocoindex/\n├── rust/ # Rust crates (workspace)\n│ ├── core/ # Core engine crate\n│ │ └── src/\n│ │ ├── engine/ # Core engine\n│ │ ├── state/ # States of the core engine\n│ │ └── inspect/ # Database inspection utilities\n│ ├── py/ # Python bindings (PyO3); calls core engine APIs\n│ ├── py_utils/ # Python-Rust utility helpers (error, convert, future)\n│ ├── utils/ # General utilities: error, batching, fingerprint, etc.\n│ ├── code_ast/ # Tree-sitter foundation: language registry (links all grammars), CodeSource, positions\n│ ├── ops_text/ # Text processing operations (splitter, language detection)\n│ ├── code_match/ # Structural code pattern matching over tree-sitter ASTs\n│ └── sdk/\n│ ├── cocoindex/ # Rust SDK (`cocoindex` crate); calls core engine APIs directly, like py/\n│ └── cocoindex_macros/ # Proc macros for the Rust SDK (`#[function]`)\n│\n├── python/\n│ ├── cocoindex/ # Python package\n│ │ ├── __init__.py # Package entry point\n│ │ ├── cli.py # CLI commands\n│ │ ├── _internal/ # Internal implementation for the core engine\n│ │ │ ├── api.py # Public API: mount, use_mount, mount_each, map, mount_target, App, fn, start/stop\n│ │ │ ├── app.py # App base implementation\n│ │ │ ├── context_keys.py # ContextKey and ContextProvider\n│ │ │ ├── environment.py # Environment and lifespan handling\n│ │ │ ├── function.py # @coco.fn decorator implementation\n│ │ │ ├── component_ctx.py # ComponentContext and component_subpath\n│ │ │ ├── target_state.py # Target state implementation\n│ │ │ └── core.pyi # Type stubs for the Rust extension module (update when PyO3 APIs change)\n│ │ ├── connectors/ # External system connectors (localfs, postgres, qdrant, lancedb, google_drive)\n│ │ ├── connectorkits/ # Connector building utilities\n│ │ ├── resources/ # Abstractions: file.py (FileLike), chunk.py (Chunk), schema.py\n│ │ └── ops/ # Operations: text.py (RecursiveSplitter), sentence_transformers.py\n│ └── tests/ # Python tests\n│\n├── examples/ # Example applications\n├── docs/ # Documentation\n└── dev/ # Development utilities\n```\n\n## Key Concepts\n\n### Mental model: declarative data pipelines\n\nCocoIndex uses a **declarative** programming model — you specify *what* your output should look like (target states), not *how* to incrementally update it. The engine handles change detection and applies minimal updates automatically.\n\nThink of it like:\n\n* **React**: declare UI as function of state → React re-renders what changed\n* **Spreadsheets**: declare formulas → cells recompute when inputs change\n* **CocoIndex**: declare target states as function of source → engine syncs what changed\n\n### Core concepts\n\n**App** — The top-level runnable unit. Bundles a main function with its arguments. When you call `app.update()`, the main function runs as the root processing component.\n\n**Processing Component** — The unit of execution that owns a set of target states. Created by `mount()` or `use_mount()` at a specific component path. When a component finishes, its target states sync atomically to external systems.\n\n**Component Path** — Stable identifier for a processing component across runs. Created via `coco.component_subpath(\"process\", filename)`. CocoIndex uses component paths to:\n\n* Match components to their previous runs for change detection\n* Determine ownership of target states (if a path disappears, its target states are cleaned up)\n\n**Target State** — What you want to exist in an external system (a file, a database row, a table). You *declare* target states; CocoIndex keeps them in sync — creating, updating, or removing as needed.\n\n**Target** — The API object used to declare target states (e.g., `DirTarget`, `TableTarget`). Targets can be nested: a container target state (directory/table) provides a Target for declaring child target states (files/rows).\n\n**Function** — A Python function decorated with `@coco.fn`. Use `memo=True` to enable memoization (skip execution when inputs and code are unchanged).\n\n**Context** — React-style provider mechanism for sharing resources. Define keys with `ContextKey[T]`, provide values in lifespan via `builder.provide()`, use in functions via `coco.use_context(key)`.\n\n### Key APIs\n\n```python\n# Mounting processing components (subpath auto-derived from fn.__name__)\nawait coco.mount(fn, *args, **kw) # child runs independently\nresult = await coco.use_mount(fn, *args, **kw) # returns value directly\n\n# Explicit subpath (for multi-part paths or multiple mounts of same function)\nawait coco.mount(coco.component_subpath(\"process\", filename), fn, *args, **kw)\nresult = await coco.use_mount(coco.component_subpath(\"name\"), fn, *args, **kw)\n\n# Component subpath composition\nsubpath = coco.component_subpath(\"process\", filename) # multiple parts\nsubpath = coco.component_subpath(\"a\") / \"b\" / \"c\" # chaining with /\n\n# Using component_subpath as context manager (applies to all nested mount calls)\nwith coco.component_subpath(\"process\"):\n for f in files:\n await coco.mount(coco.component_subpath(str(f.relative_path)), process_file, f, target)\n\n# Declaring target states (typically via Target methods)\ndir_target.declare_file(filename=name, content=data)\ntable_target.declare_row(row=MyRow(...))\n\n# Using context values\ndb = coco.use_context(PG_DB) # retrieve value provided in lifespan\n\n# Explicit context management (for ThreadPoolExecutor)\nctx = coco.get_component_context()\nwith ctx.attach():\n # coco APIs work correctly in this thread\n coco.mount(...)\n```\n\n**Mount handles:**\n\n* `mount()` → `ComponentMountHandle`: call `await handle.ready()` to wait until target states are synced\n* `use_mount()` → returns the result value directly (awaitable)\n\n### How syncing works\n\nWhen a processing component finishes, CocoIndex compares its declared target states with those from the previous run at the same component path:\n\n* New target states → create (insert row, create file)\n* Changed target states → update\n* Missing target states → delete\n\nChanges are applied atomically per component. If a source item is deleted (path no longer mounted), all its target states are cleaned up automatically.\n\n### Example\n\n```python\nimport pathlib\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, target: localfs.DirTarget) -> None:\n html = _markdown_it.render(await file.read_text())\n outname = \"__\".join(file.file_path.path.parts) + \".html\"\n target.declare_file(filename=outname, content=html)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n target = await coco.use_mount(localfs.declare_dir_target, outdir)\n\n files = localfs.walk_dir(\n sourcedir,\n recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n )\n await coco.mount_each(process_file, files.items(), target)\n\n\napp = coco.App(\n coco.AppConfig(name=\"FilesTransform\"),\n app_main,\n sourcedir=pathlib.Path(\"./docs\"),\n outdir=pathlib.Path(\"./out\"),\n)\napp.update_blocking(report_to_stdout=True)\n```\n\n## Code Conventions\n\n### Internal vs External Modules\n\nWe distinguish between **internal modules** (under packages with `_` prefix, e.g. `_internal.*` or `connectors.*._source`) and **external modules** (which users can directly import).\n\n**External modules** (user-facing, e.g. `cocoindex/ops/sentence_transformers.py`):\n\n* Be strict about not leaking implementation details\n* Use `__all__` to explicitly list public exports\n* Prefix ALL non-public symbols with `_`, including:\n * Standard library imports: `import threading as _threading`, `import typing as _typing`\n * Third-party imports: `import numpy as _np`, `from numpy.typing import NDArray as _NDArray`\n * Internal package imports: `from cocoindex.resources import schema as _schema`\n* Exception: `TYPE_CHECKING` imports for type hints don't need prefixing\n\n**Internal modules** (e.g. `cocoindex/_internal/component_ctx.py`, `**/_target.py`):\n\n* Less strict since users shouldn't import these directly\n* Standard library and internal imports don't need underscore prefix\n* Only prefix symbols that are truly private to the module itself (e.g. `_context_var` for a module-private ContextVar)\n\n### Minimize API surface until deemed necessary\n\nWhen adding a new public API (function, class, kwarg, configuration option), prefer the smallest surface that solves the concrete need in front of you. Do not pre-expose tunables, hooks, or alternatives \"in case someone needs them.\" Adding a kwarg later is straightforward and backwards-compatible; removing or renaming one is disruptive. If a knob is currently a hardcoded constant inside the implementation, leave it there until a real use case demands it — at which point promoting it to a kwarg is mechanical. This applies equally to optional parameters, callback hooks, parser/transformer plug-points, and configuration keys.\n\n### General principles (also covered by `/review-changes`)\n\n- **Top-level imports.** Defer to in-function only for a real circular dependency or a heavy import that isn't always needed.\n- **Specific types over `Any`.** Use concrete types — including concrete types from third-party libraries. `Any` only when the type is truly generic and no downstream code needs to downcast it.\n- **Validate and exchange early.** When a value enters as a weaker form (`str`, `Any`, raw identifier), convert to the strong type at the earliest point. Don't propagate the weak form. Concrete example in this codebase: connector handlers receive a `ContextKey` string as part of a target-state key. The parent handler resolves the key once (when constructing the child handler) and passes the typed connection directly — the child stores `_pool: asyncpg.Pool`, not `_db_key: str`.\n- **`NamedTuple`/small dataclass for multi-value returns.** Access fields by name (`result.can_reuse`) at call sites.\n- **Exceptions for exceptional situations only.** Reserve exceptions (Python) and errors (Rust) for truly unexpected failures — not for end-of-iteration, \"not found\", or other expected state transitions. Use explicit return values (sentinel, enum variant, `None`) for those.\n- **Single source of truth, delete dead code, honest names.** When the same value/logic appears in multiple places, consolidate. When changes make code unreachable, delete it (and its tests, and any dead config knobs). Function and field names should describe what the implementation does today.\n\n- Rust code follows [dev/rust-guidelines.md](dev/rust-guidelines.md).\n\n### Testing Guidelines\n\nWe prefer end-to-end tests on user-facing APIs, over unit tests on smaller internal functions. With this said, there're cases where unit tests are necessary, e.g. for internal logic with various situations and edge cases, in which case it's usually easier to cover various scenarios with unit tests.\n\n#### Test Environment Setup\n\nUse `common.create_test_env(__file__)` to create a CocoIndex `Environment` for tests. It derives a unique `db_path` from the test file path and picks up the current event loop automatically.\n\n* **Sync tests** (module-level creation): Call at module level — the Environment creates a background loop for async callbacks.\n\n ```python\n coco_env = common.create_test_env(__file__)\n ```\n\n* **Async tests with async resources** (e.g., asyncpg pools): Call inside an async fixture so the Environment binds to the test's running event loop (same loop the pool is on). Use the `suffix` parameter when each test needs its own Environment:\n\n ```python\n @pytest_asyncio.fixture\n async def pg_env(request: pytest.FixtureRequest) -> Any:\n pool = await asyncpg.create_pool(dsn)\n coco_env = common.create_test_env(__file__, suffix=request.node.name)\n coco_env.context_provider.provide(DB_KEY, pool)\n yield pool, coco_env\n await pool.close()\n ```\n\n The `suffix` ensures each test gets a unique `db_path`, avoiding \"environment already open\" errors.\n\n#### Testcontainers for Database Tests\n\nUse `testcontainers[postgres]` (in the `build-test` dependency group) to spin up real database instances automatically — no manual setup or environment variables needed. Use a module-scoped sync fixture for the container and a function-scoped async fixture for per-test resources:\n\n```python\n_PG_IMAGE = \"postgres:16-alpine\"\n\npytestmark = [pytest.mark.requires_docker(_PG_IMAGE)]\n\n@pytest.fixture(scope=\"module\")\ndef pg_dsn() -> Any:\n with PostgresContainer(_PG_IMAGE) as pg:\n dsn = pg.get_connection_url().replace(\"postgresql+psycopg2://\", \"postgresql://\")\n yield dsn\n\n@pytest_asyncio.fixture\nasync def pool(pg_dsn: str) -> Any:\n p = await asyncpg.create_pool(pg_dsn)\n yield p\n await p.close()\n```\n\nList every image a test's containers use in its `requires_docker(...)` marker. `python/tests/conftest.py` pulls missing images after collection, before the first test runs; otherwise testcontainers pulls inside the fixture, and a cold-cache download counts against the first test's `pytest-timeout`.\n\n### LMDB write paths\n\n- All LMDB writes must go through `Storage::run_txn` (uses the single-writer batcher).\n- Do not open a heed write txn directly or wrap the env in a separate mutex/semaphore — bypassing the batcher loses fsync coalescing and regresses concurrent-submit throughput by 10-100×.\n- LMDB has no savepoints. If a sub-operation needs to \"abort\" without failing its caller, handle it at the body level (e.g. return a sentinel result without writing). A body that returns `Err` fails only its own caller: the batcher drops its writes by aborting the txn and re-running the bodies ahead of it, so every body must be safe to re-run.\n- An LMDB write txn must begin and end (commit or abort) on the same OS thread: the writer lock is thread-owned and LMDB ignores a failed release, so a txn that migrates between runtime workers blocks every later writer for good. heed marks `RwTxn` as `Send` regardless, so the compiler won't catch it. `Storage::run_txn` runs each batch on one blocking thread for this reason; never hold a write txn across an `.await` on the multi-thread runtime.\n\n### Sync vs Async\n\nThe Rust core (`rust/core`, `rust/utils`) uses **async-first** design with Tokio. The `rust/py` crate bridges Rust async to Python, offering both sync and async APIs:\n\n* Rust core exposes async functions\n* `rust/py` provides sync wrappers that use `block_on()` to call async Rust from sync Python\n* Python's `cocoindex` API is **async-first**: mount APIs (`mount`, `use_mount`, `mount_each`, `map`) are all `async def`; `App.update()`/`App.drop()` are async; sync entry points (`App.update_blocking()`, `App.drop_blocking()`, `start_blocking()`, `stop_blocking()`) are available for CLI and blocking contexts\n\nWhen adding new functionality that involves I/O or concurrency:\n\n* Implement async in Rust\n* Bridge to Python via `rust/py`, providing both sync and async variants if needed\n\n## Versioning\n\nThe current codebase is for CocoIndex v1, which is a fundamental redesign from CocoIndex v0. CocoIndex v1 lives on the `main` branch; the legacy v0 code is preserved on the `v0` branch.\n\n## Docs diagrams\n\nAll diagrams embedded in docs pages are built as inline Astro components under `docs/src/components/diagrams/`, not exported SVG files. See [docs/src/components/diagrams/README.md](docs/src/components/diagrams/README.md) for the palette, primitives, shared CSS classes, animation conventions, and step-by-step authoring guidelines. Follow that guide for any new or updated diagram.\n\n## Agent Tooling\n\n### Semantic code search over this repo (`cocoindex-code`)\n\nTo find where a concept lives, prefer semantic search over blind `grep`. This repo keeps a local\n`cocoindex-code` index; query it from any agent's shell with the `ccc` CLI:\n\n```bash\nccc search \"where is X handled\" # semantic search; add --refresh to reindex first\n```\n\nIf `ccc search` reports no index, set it up once:\n\n```bash\ncommand -v ccc >/dev/null || uv tool install 'cocoindex-code[full]' # per machine; local embeddings, no API key\nccc init -f && ccc index # repo-scoped index (first build ~1-2 min)\n```\n\nEverything is local — SQLite index, local embeddings, no keys — and a background daemon keeps it\nfresh. On a brand-new machine `ccc init` asks once for an embedding provider; choose the local\ndefault. The `-f` matters: if a parent directory is also a `ccc` project, plain `ccc init` refuses\nand search silently uses the parent's index, so run `ccc status` to confirm the root is this repo.\nWe register no MCP server on purpose — an `ccc mcp` mode exists, but the CLI costs no context until\ninvoked and behaves the same across agents. Source:\n[github.com/cocoindex-io/cocoindex-code](https://github.com/cocoindex-io/cocoindex-code).\n\n## Agent Playbooks\n\nReusable task-specific agent guidance lives under `dev/agent-skills/`. These playbooks use a `SKILL.md` format so Claude Code can load them directly from `.claude/skills/`, but the content is intended to be readable by any coding agent.\n\n- For docs diagrams, read `dev/agent-skills/cocoindex-diagrams/SKILL.md`.\n- For new target connectors, read `dev/agent-skills/target-connector/SKILL.md`.\n- For upgrading example package versions, read `dev/agent-skills/upgrade-examples/SKILL.md`.\n\nAgent Skills-compatible clients, including Codex, discover the checked-in symlinks under `.agents/skills/`. The public CocoIndex skill is exposed there as `cocoindex`, and contributor playbooks are exposed there from `dev/agent-skills/`.\n\nClaude Code compatibility remains under `.claude/`: hooks/settings live there, and `.claude/skills/` mirrors the contributor playbooks with symlinks. Do not duplicate skill content under `.agents/skills/` or `.claude/skills/`; edit the source directories (`skills/cocoindex/` for the public skill, `dev/agent-skills/` for contributor playbooks) and keep dot-directory entries as symlinks.\n\nPortable agent checks live under `dev/agent-checks/`. Claude Code hooks in `.claude/hooks/` are thin adapters that call these scripts when relevant files changed.\n","isInternal":false,"tokens":4990,"sizeBytes":22038},{"name":"AGENTS.md","path":"examples/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/examples/AGENTS.md","title":"examples - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md — CocoIndex examples\n\nGuidance for AI coding agents (Claude Code, Cursor, etc.) working in this `examples/`\ndirectory. Most top-level Python subfolders are self-contained, runnable\nCocoIndex **v1** apps; `rust/` contains Rust ports with per-example READMEs.\n\n## Before you write CocoIndex code: install the skill\n\nCocoIndex v1 is a fundamental redesign from v0. Without context, LLMs tend to\nhallucinate the v0 flow-builder DSL and deprecated decorators. Install the\nofficial skill first — it teaches the correct v1 API. Quickest portable path\nfor Codex and other Agent Skills-compatible clients is the hosted skill:\n\n```sh\nmkdir -p .agents/skills/cocoindex/references\ncurl -fsSL https://cocoindex.io/docs/skill.md -o .agents/skills/cocoindex/SKILL.md\nfor f in api_reference connectors patterns setup_database setup_project; do\n curl -fsSL https://cocoindex.io/docs/references/$f.md -o .agents/skills/cocoindex/references/$f.md\ndone\n```\n\nOr clone the repo and copy the folder:\n\n```sh\ngit clone --depth=1 https://github.com/cocoindex-io/cocoindex.git /tmp/cocoindex-skill\nmkdir -p .agents/skills && cp -r /tmp/cocoindex-skill/skills/cocoindex .agents/skills/\n```\n\nFor Claude Code's native project path, use `.claude/skills/cocoindex` instead\nof `.agents/skills/cocoindex`; the skill format is the same.\n\nThe skill itself lives at [`skills/cocoindex/`](../skills/cocoindex) (SKILL.md +\n`references/`). For Cursor, copy `SKILL.md` into `.cursor/rules/`. Full machine-readable\ndocs: (index) and\n (everything, including example\nwalkthroughs). Regular docs pages also have raw Markdown twins by replacing the\ntrailing slash with `.md` (e.g.\n`https://cocoindex.io/docs/programming_guide/core_concepts.md`).\n\n## The v1 mental model\n\n`target_state = transform(source_state)`. You declare what the target should look\nlike; the Rust engine keeps it in sync, reprocessing only what changed (state is\ntracked in a local LMDB store — **no database is required for the engine itself**,\nonly when an example writes to one). Key APIs: `@coco.fn`, `mount` / `use_mount` /\n`mount_each`, `ContextKey`, target-state declarations. See the skill for details.\n\n## Running examples\n\nMost Python examples are standalone projects with their own `pyproject.toml`:\n\n```sh\ncd \ncp .env.example .env # if present — fill in the blanks (see below)\npip install -e . # or: uv pip install -e .\ncocoindex update main # catch-up: scan sources, sync, exit\ncocoindex update -L main # live mode: catch up, then watch for changes (where supported)\n```\n\nUse the example's README as the source of truth. Known exceptions:\n\n- `multi_codebase_summarization`, `audio_to_text`,\n `patient_intake_extraction_baml`, `patient_intake_extraction_dspy`:\n `cocoindex update main.py`\n- `conversation_to_knowledge`: `cocoindex update conv_knowledge.app`\n- `image_search`, `image_search_colpali`: start the API with\n `python -m uvicorn api:app --reload --host 0.0.0.0 --port 8000`; it runs the\n CocoIndex app in live mode, then start the frontend from `frontend/`.\n- `csv_to_kafka`, `kafka_to_lancedb`: `cocoindex update -L main.py`\n- `rust/`: follow that example's README. Many use\n `cargo run -- index` for indexing and `cargo run -- query \"...\"` for search;\n a few take custom paths or service-specific subcommands.\n\nSome examples expose a query/CLI demo via `python main.py \"\"`; check the\nexample's `README.md`. Examples that need extra services or a code-gen step\n(e.g. `baml generate`) say so in their README.\n\n## Environment / credentials\n\nWhen an example needs credentials or service configuration, required env vars\nare templated in that example's **`.env.example`** — `cp` it to `.env` and fill\nin the blanks; both `python main.py` and the `cocoindex` CLI load `.env`\nautomatically. Common ones:\n\n- `POSTGRES_URL` — for Postgres/pgvector targets. Local instance:\n `docker compose -f ../../dev/postgres.yaml up -d` from inside an example\n directory.\n- `OPENAI_API_KEY` / `GEMINI_API_KEY` — for examples that call an LLM.\n- Service-specific (`QDRANT_URL`, `LANCEDB_URI`, `NEO4J_*`, `KAFKA_*`,\n `GOOGLE_SERVICE_ACCOUNT_CREDENTIAL`, …) — see that example's `.env.example`.\n\nExamples with no `.env.example` (e.g. `files_transform`, `pdf_to_markdown`) run\nfully locally with no credentials.\n\n**Never commit secrets.** The `.env` files tracked in this repo hold only\nnon-secret defaults (`COCOINDEX_DB`, local service URLs); keep API keys and\ncredentials in your local `.env` edits and out of commits.\n\n## The examples\n\nA walkthrough URL means there's a step-by-step guide at\n`https://cocoindex.io/docs/examples//`; otherwise start from the example's README.\n\n### Vector indexes (embed → store → search by meaning)\n- `text_embedding` — Markdown → pgvector; the simplest end-to-end index. *(walkthrough: text-embedding)*\n- `code_embedding` — repo → Tree-sitter chunks → pgvector; query code in English. *(walkthrough: index-codebase)*\n- `text_embedding_qdrant` / `text_embedding_lancedb` / `text_embedding_turbopuffer` — same flow, different vector store.\n- `code_embedding_lancedb` — code chunks → LanceDB.\n- `pdf_embedding` — PDFs → markdown → chunks → pgvector.\n- `paper_metadata` — extract title/authors/abstract from PDFs → Postgres + embeddings.\n- `amazon_s3_embedding` / `gdrive_text_embedding` / `oci_object_storage_embedding` — same flow, remote source (S3 / Google Drive / OCI).\n- `postgres_source` — read from an existing Postgres table as the source.\n- `entire_session_search` — semantic search over AI coding sessions captured by Entire.\n- `sec_edgar_analytics` — multi-format SEC filings → Apache Doris with a vector **and** a full-text index; hybrid (semantic + keyword) RRF search. *(walkthrough: sec-edgar-analytics)*\n\n### Multimodal\n- `image_search` — CLIP embeddings + Qdrant, queried via FastAPI + React.\n- `image_search_colpali` — ColPali multi-vector model + Qdrant MaxSim.\n- `multi_format_indexing` — PDFs + images as page screenshots → ColPali → Qdrant; no OCR, no chunking. *(walkthrough: multi-format-indexing)*\n- `face_recognition` — detect faces (dlib) → 128-d embeddings → Qdrant face search. *(walkthrough: face-recognition)*\n- `audio_to_text` — transcribe audio with LiteLLM → Postgres.\n- `slides_to_speech` — slides → vision-LLM notes → Pocket TTS narration → LanceDB semantic search. *(walkthrough: slides-to-speech)*\n\n### Structured extraction (LLM / BAML / DSPy)\n- `multi_codebase_summarization` — LLM per-file summaries across many repos. *(walkthrough: multi-codebase-summarization)*\n- `hn_trending_topics` — scrape HackerNews → LLM topic extraction → Postgres.\n- `manuals_llm_extraction` — PDF manuals → Markdown (docling) → typed module records → Postgres. *(walkthrough: manuals-llm-extraction)*\n- `patient_intake_extraction_baml` / `patient_intake_extraction_dspy` — structured PDF extraction with BAML / DSPy (Gemini vision).\n\n### Knowledge graphs\n- `conversation_to_knowledge` — YouTube podcasts → SurrealDB knowledge graph. *(walkthrough: podcast-to-knowledge-graph)*\n- `docs_to_knowledge_graph` — Markdown docs → Neo4j concept graph of LLM-extracted triples. *(walkthrough: docs-to-knowledge-graph)*\n- `product_recommendation` — product catalog → LLM taxonomy extraction → Neo4j recommendation graph. *(walkthrough: product-recommendation)*\n- `meeting_notes_graph_neo4j` / `meeting_notes_graph_falkordb` — Google Drive meeting notes → Neo4j / FalkorDB graph.\n- `meeting_notes_graph_surrealdb` — local Markdown meeting notes → SurrealDB graph.\n\n### Custom sources / targets / streaming\n- `pdf_to_markdown` — incremental PDF → Markdown with docling (local, no services). *(walkthrough: pdf-to-markdown)*\n- `files_transform` — watch Markdown files → HTML, live mode (local, no services).\n- `csv_to_kafka` — watch CSVs → publish rows to Kafka.\n- `kafka_to_lancedb` — consume Kafka → route to LanceDB tables.\n\n### Rust\n- `rust/` — Rust ports of many of the above, using the CocoIndex Rust API.\n\n## Conventions for edits\n\n- Keep each Python example self-contained: its own `pyproject.toml` and\n `README.md`; add `.env.example` when credentials or configurable services are\n required. When you add an example, also add a line to `EXAMPLE_CATALOG` in the\n docs repo (`docs/src/data/examples.ts`) so it appears in `/docs/llms.txt`.\n- Match the surrounding code's low comment density.\n- Don't commit generated artifacts (`cocoindex.db`, `__pycache__`, build output) —\n they're already git-ignored.\n","isInternal":false,"tokens":2232,"sizeBytes":8697},{"name":"layout-patterns.md","path":"dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md)","tokens":300,"sizeBytes":252},{"name":"pitfalls.md","path":"dev/agent-skills/cocoindex-diagrams/references/pitfalls.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/pitfalls.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/pitfalls.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/pitfalls.md)","tokens":300,"sizeBytes":238},{"name":"workflow.md","path":"dev/agent-skills/cocoindex-diagrams/references/workflow.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/workflow.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/workflow.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/workflow.md)","tokens":300,"sizeBytes":238},{"name":"api_reference.md","path":"skills/cocoindex/references/api_reference.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/skills/cocoindex/references/api_reference.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# CocoIndex API Reference\n\nQuick reference for the most commonly used CocoIndex APIs.\n\n## Import Convention\n\n```python\nimport cocoindex as coco\n```\n\nAll APIs live directly on the `coco` module. There is no separate `cocoindex.asyncio` module.\n\n---\n\n## `@coco.fn` Decorator\n\nMark a function as a CocoIndex processing function.\n\n```python\n@coco.fn\nasync def my_function(arg1: str) -> None: ...\n\n@coco.fn(memo=True, version=1)\nasync def expensive_fn(data: str) -> Result: ...\n\n# Force async interface for a sync function (useful for batching)\n@coco.fn.as_async(memo=True, batching=True)\ndef batch_embed(texts: list[str]) -> list[NDArray]: ...\n```\n\n**Parameters:**\n- `memo: bool = False` -- Enable memoization (skip if inputs/code unchanged)\n- `version: int | None = None` -- Explicit version bump to force re-execution\n- `logic_tracking: \"full\" | \"self\" | None = \"full\"` -- How code changes are tracked\n- `batching: bool = False` -- Auto-batch concurrent calls (async only)\n- `max_batch_size: int | None = None` -- Max batch size\n- `runner: Runner | None = None` -- e.g. `coco.GPU` for serialized GPU execution\n\n---\n\n## Mount APIs (all async)\n\nAll mount APIs accept an optional `ComponentSubpath` as their first argument. When omitted, the subpath is auto-derived from `Symbol(fn.__name__)`. Provide an explicit subpath when mounting the same function multiple times, using multi-part paths, or needing a specific path name.\n\n### `coco.mount()`\n\nMount a processing component in the background.\n\n```python\n# Subpath auto-derived from fn.__name__\nhandle = await coco.mount(processor_fn, *args, **kwargs)\n\n# Explicit subpath\nhandle = await coco.mount(coco.component_subpath(\"name\"), processor_fn, *args, **kwargs)\n\nawait handle.ready() # Optional: wait until component finishes\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `processor_fn` -- Function (or LiveComponent class) to run.\n- `*args, **kwargs` -- Arguments passed to the function.\n\n**Returns:** `ComponentMountHandle`\n\n### `coco.use_mount()`\n\nMount a dependent component and return its result. Parent depends on the child.\n\n```python\n# Subpath auto-derived from fn.__name__\nresult = await coco.use_mount(init_fn, *args, **kwargs)\n\n# Explicit subpath\nresult = await coco.use_mount(coco.component_subpath(\"setup\"), init_fn, *args, **kwargs)\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `processor_fn` -- Function to run.\n- `*args, **kwargs` -- Arguments passed to the function.\n\n**Returns:** The return value of `processor_fn`.\n\n### `coco.mount_each()`\n\nMount one component per item in a keyed iterable. Preferred for processing lists.\n\n```python\n# Subpath auto-derived from fn.__name__\nawait coco.mount_each(process_file, files.items(), *extra_args)\n\n# Explicit subpath\nawait coco.mount_each(coco.component_subpath(\"process\"), process_file, files.items(), table)\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `fn` -- Function to run per item. Item value is passed as first argument.\n- `items` -- Keyed iterable of `(StableKey, T)` pairs, or a `LiveMapFeed` for live mode.\n- `*args, **kwargs` -- Additional arguments passed to `fn` after the item.\n\n**Returns:** `ComponentMountHandle`\n\n### `coco.mount_target()`\n\nMount a target state, ensuring the container is applied before returning the child provider.\n\n```python\nprovider = await coco.mount_target(target_state)\n```\n\nPrefer connector convenience methods (`postgres.mount_table_target()`, etc.) which call this internally.\n\n### `coco.map()`\n\nRun a function concurrently on each item. No processing components are created -- pure concurrent execution within the current component.\n\n```python\nresults = await coco.map(process_chunk, chunks, *extra_args)\n```\n\n**Parameters:**\n- `fn` -- Async function to apply. Item is passed as first argument.\n- `items` -- Iterable or async iterable.\n- `*args, **kwargs` -- Additional arguments passed to `fn` after the item.\n\n**Returns:** `list[T]`\n\n### `coco.component_subpath()`\n\nCreate a stable component path for mounting.\n\n```python\ncoco.component_subpath(\"setup\")\ncoco.component_subpath(\"file\", str(file_path))\ncoco.component_subpath(\"record\", record.id)\n\n# Chaining with /\nsubpath = coco.component_subpath(\"a\") / \"b\" / \"c\"\n\n# Context manager form (applies to all nested mount calls)\nwith coco.component_subpath(\"process\"):\n for f in files:\n await coco.mount(coco.component_subpath(str(f.path)), process_file, f)\n```\n\n**StableKey types:** `str | int | bool | bytes | uuid.UUID | Symbol | tuple[StableKey, ...]`\n\n---\n\n## Context System\n\n### `coco.ContextKey[T]`\n\nType-safe key for sharing resources. The `key` string is the stable identity across runs.\n\n```python\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n```\n\n- `detect_change=True` -- Opt in to auto-invalidate dependent memos when value changes (models, configs)\n- `detect_change=False` (default) -- For resources not affecting computation (DB connections, loggers)\n\n### `builder.provide()`\n\nRegister a resource in context (used in lifespan).\n\n```python\nbuilder.provide(PG_DB, pool)\nbuilder.provide_with(KEY, context_manager) # Sync CM\nawait builder.provide_async_with(KEY, async_cm) # Async CM\n```\n\n### `coco.use_context()`\n\nRetrieve a resource from context inside a processing function.\n\n```python\npool = coco.use_context(PG_DB)\nembedder = coco.use_context(EMBEDDER)\n```\n\n---\n\n## Lifespan\n\n### `@coco.lifespan`\n\nDefine environment setup/teardown. Registered to the default environment.\n\n```python\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(MODEL))\n yield\n```\n\nCan also be sync:\n\n```python\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n builder.settings.db_path = pathlib.Path(\"./my.db\")\n yield\n```\n\n---\n\n## App\n\n### `coco.App`\n\n```python\napp = coco.App(\n coco.AppConfig(name=\"MyApp\"),\n main_fn,\n **params,\n)\n\n# Async\nawait app.update()\nhandle = app.update(live=True) # Live mode\nawait app.drop()\n\n# Sync (blocking)\napp.update_blocking(report_to_stdout=True)\napp.drop_blocking()\n```\n\n### `coco.AppConfig`\n\n```python\ncoco.AppConfig(\n name=\"MyApp\", # Required\n environment=env, # Optional: custom Environment\n max_inflight_components=1024, # Optional: concurrency limit\n)\n```\n\n### Start/Stop (for programmatic usage)\n\n```python\n# Context manager (preferred)\nasync with coco.runtime():\n await app.update()\n\n# Or manually\nawait coco.start()\ntry:\n await app.update()\nfinally:\n await coco.stop()\n\n# Sync variants\nwith coco.runtime():\n app.update_blocking()\n```\n\n---\n\n## Exception Handlers\n\n### Global (in lifespan)\n\n```python\nbuilder.set_exception_handler(my_handler)\n```\n\n### Scoped (in processing functions)\n\n```python\nasync with coco.exception_handler(my_handler):\n await coco.mount_each(process_file, files.items(), table)\n```\n\n### Handler Signature\n\n```python\nasync def my_handler(exc: BaseException, ctx: coco.ExceptionContext) -> None:\n logger.error(f\"Error in {ctx.stable_path}: {exc}\")\n```\n\n`ExceptionContext` provides: `env_name`, `stable_path`, `processor_name`, `mount_kind`, `parent_stable_path`, `is_background`, `source`, `original_exception`.\n\n---\n\n## Live Mode\n\nRun the pipeline continuously, streaming changes from sources:\n\n```python\n# CLI\ncocoindex update main.py -L\n\n# Programmatic\nhandle = app.update(live=True)\nasync for snapshot in handle.watch():\n print(snapshot.stats)\n```\n\nSources that support live mode:\n- `localfs.walk_dir(..., live=True)` -- File watching\n- `kafka.topic_as_map(consumer, topics)` -- Kafka topic consumption\n\n---\n\n## CLI Commands\n\n```bash\ncocoindex init [PROJECT_NAME] # Create new project\ncocoindex update APP_TARGET # Run app once\ncocoindex update APP_TARGET -L # Run in live mode\ncocoindex update APP_TARGET --full-reprocess # Force reprocess all\ncocoindex update APP_TARGET --reset # Reset state before running\ncocoindex drop APP_TARGET [-f] # Drop app and all state\ncocoindex ls [APP_TARGET] [--db PATH] # List apps\ncocoindex show APP_TARGET [--tree] # Show component paths\n```\n\n**APP_TARGET format:** `main.py`, `main.py:app_name`, `my_module:app_name`\n\n---\n\n## Text Operations\n\n**Import:** `from cocoindex.ops.text import ...`\n\n### `detect_code_language()`\n\n```python\nlanguage = detect_code_language(filename=\"example.py\") # -> \"python\"\n```\n\n### `RecursiveSplitter`\n\n```python\nsplitter = RecursiveSplitter()\nchunks = splitter.split(\n text,\n chunk_size=1000,\n chunk_overlap=200,\n min_chunk_size=300,\n language=\"python\", # Syntax-aware splitting\n)\n# Each chunk: Chunk(text=..., start=TextPosition(...), end=TextPosition(...))\n```\n\n### `SeparatorSplitter`\n\n```python\nsplitter = SeparatorSplitter(separators_regex=[r\"\\n\\n\", r\"\\n\"])\nchunks = splitter.split(text)\n```\n\n---\n\n## Embedding Operations\n\n**Import:** `from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder`\n\n```python\nembedder = SentenceTransformerEmbedder(\"sentence-transformers/all-MiniLM-L6-v2\")\nembedding = await embedder.embed(text) # -> NDArray (float32)\n```\n\n**As VectorSchemaProvider:**\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions\n```\n\n---\n\n## File Resources\n\n**Import:** `from cocoindex.resources.file import ...`\n\n### `FileLike` (async file object)\n\n```python\ntext = await file.read_text()\ndata = await file.read() # bytes, lazy/cached\nfp = await file.content_fingerprint()\n```\n\n### `PatternFilePathMatcher`\n\n```python\nmatcher = PatternFilePathMatcher(\n included_patterns=[\"**/*.py\", \"**/*.md\"],\n excluded_patterns=[\".*/**\", \"__pycache__/**\"],\n)\n```\n\n---\n\n## ID Generation\n\n**Import:** `from cocoindex.resources.id import ...`\n\n```python\n# Deterministic: same dep -> same ID\nchunk_id = await generate_id(chunk.text)\nchunk_uuid = generate_uuid(chunk.text)\n\n# Distinct per call (even with same dep)\nid_gen = IdGenerator()\nchunk_id = await id_gen.next_id(chunk.text)\n\nuuid_gen = UuidGenerator()\nchunk_uuid = uuid_gen.next_uuid(chunk.text)\n```\n\n---\n\n## Vector Schema\n\n**Import:** `from cocoindex.resources.schema import VectorSchema`\n\n```python\n# Via ContextKey (preferred -- auto-infer from embedder)\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER]\n\n# Explicit dimensions\nschema = VectorSchema(dtype=np.dtype(np.float32), size=384)\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, schema]\n```\n\n---\n\n## See Also\n\n- [Connectors Reference](./connectors.md) -- Database and system connectors\n- [Patterns Reference](./patterns.md) -- Common pipeline patterns\n- [Setup Project](./setup_project.md) -- Project setup guide\n- [Setup Database](./setup_database.md) -- Database setup guide\n","isInternal":false,"tokens":2766,"sizeBytes":11274},{"name":"connectors.md","path":"skills/cocoindex/references/connectors.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/skills/cocoindex/references/connectors.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# CocoIndex Connectors Reference\n\nComprehensive reference for all CocoIndex connectors.\n\n## Common Patterns\n\n### Target Setup\n\nAll target connectors follow this pattern:\n\n1. **Provide** connection via `ContextKey` in lifespan\n2. **Mount** target using `mount_*_target(ContextKey, ...)` convenience method\n3. **Declare** rows/points/files on the returned target object\n\nTarget connectors take a `ContextKey` (not the connection object directly) as their first argument. This enables CocoIndex to track the identity of the target across runs.\n\n### Vector Support\n\nMost connectors support vector embeddings via `VectorSchemaProvider`:\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions from ContextKey\n```\n\n---\n\n## PostgreSQL\n\n**Import**: `from cocoindex.connectors import postgres`\n\n**Capabilities**: Source + Target | **Vector**: pgvector\n\n### Connection Setup\n\n```python\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n yield\n```\n\n### As Target\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Annotated\nfrom numpy.typing import NDArray\n\n@dataclass\nclass Embedding:\n id: int\n text: str\n vector: Annotated[NDArray, EMBEDDER]\n\n# Mount table target\ntarget_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"embeddings\",\n table_schema=await postgres.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n pg_schema_name=\"my_schema\", # Optional, default \"public\"\n)\n\n# Optional: declare vector index\ntarget_table.declare_vector_index(column=\"vector\")\n\n# Declare rows\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### As Source\n\n```python\n@dataclass\nclass Record:\n id: int\n name: str\n\nsource = postgres.PgTableSource(\n coco.use_context(PG_DB),\n table_name=\"my_table\",\n row_type=Record,\n)\n\n# Iterate (async)\nfetcher = source.fetch_rows()\nasync for record in fetcher:\n ...\n\n# Keyed iteration for mount_each\nawait coco.mount_each(process_record, source.fetch_rows().items(key=lambda r: r.id), table)\n```\n\n### Type Mapping\n\n| Python | PostgreSQL |\n|--------|-----------|\n| `bool` | `boolean` |\n| `int` | `bigint` |\n| `float` | `double precision` |\n| `str` | `text` |\n| `bytes` | `bytea` |\n| `UUID` | `uuid` |\n| `datetime` | `timestamp with time zone` |\n| `list`, `dict` | `jsonb` |\n| `NDArray` + vector schema | `vector(n)` or `halfvec(n)` |\n\n### SQL Command Attachments\n\n```python\ntarget_table.declare_sql_command_attachment(\n name=\"my_index\",\n setup_sql=\"CREATE INDEX ...\",\n teardown_sql=\"DROP INDEX ...\", # Optional\n)\n```\n\n---\n\n## SQLite\n\n**Import**: `from cocoindex.connectors import sqlite`\n\n**Capabilities**: Target only | **Vector**: sqlite-vec\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import sqlite\n\nSQLITE_DB = coco.ContextKey[sqlite.ManagedConnection](\"sqlite_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n conn = sqlite.connect(\"./data.db\", load_vec=\"auto\")\n builder.provide(SQLITE_DB, conn)\n yield\n conn.close()\n```\n\n### As Target\n\n```python\ntarget_table = await sqlite.mount_table_target(\n SQLITE_DB,\n table_name=\"embeddings\",\n table_schema=await sqlite.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n)\n\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### Type Mapping\n\n| Python | SQLite |\n|--------|--------|\n| `bool` | `INTEGER` (0/1) |\n| `int` | `INTEGER` |\n| `float` | `REAL` |\n| `str` | `TEXT` |\n| `bytes` | `BLOB` |\n| `datetime` | `TEXT` (ISO format) |\n| `list`, `dict` | `TEXT` (JSON) |\n| `NDArray` + vector schema | `BLOB` (sqlite-vec) |\n\n---\n\n## LanceDB\n\n**Import**: `from cocoindex.connectors import lancedb`\n\n**Capabilities**: Target only | **Vector**: Native | **Storage**: Local or cloud (S3, GCS)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import lancedb\n\nLANCE_DB = coco.ContextKey[lancedb.LanceAsyncConnection](\"lance_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n conn = await lancedb.connect_async(\"./lancedb_data\") # or \"s3://bucket/path\"\n builder.provide(LANCE_DB, conn)\n yield\n```\n\n### As Target\n\n```python\ntarget_table = await lancedb.mount_table_target(\n LANCE_DB,\n table_name=\"embeddings\",\n table_schema=await lancedb.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n)\n\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### Type Mapping\n\n| Python | PyArrow |\n|--------|---------|\n| `bool` | `bool` |\n| `int` | `int64` |\n| `float` | `float64` |\n| `str` | `string` |\n| `bytes` | `binary` |\n| `list`, `dict` | `string` (JSON) |\n| `NDArray` + vector schema | `fixed_size_list` |\n\n---\n\n## Qdrant\n\n**Import**: `from cocoindex.connectors import qdrant`\n\n**Capabilities**: Target only | **Vector**: Native | **Model**: Point-oriented\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import qdrant\n\nfrom qdrant_client import QdrantClient\nQDRANT_DB = coco.ContextKey[QdrantClient](\"qdrant_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n client = qdrant.create_client(\"http://localhost:6333\")\n builder.provide(QDRANT_DB, client)\n yield\n```\n\n### As Target (Single Vector)\n\n```python\ncollection = await qdrant.mount_collection_target(\n QDRANT_DB,\n collection_name=\"embeddings\",\n schema=await qdrant.CollectionSchema.create(\n vectors=qdrant.QdrantVectorDef(schema=EMBEDDER, distance=\"cosine\"),\n ),\n)\n\ncollection.declare_point(point=qdrant.PointStruct(\n id=\"point-1\",\n vector=embedding_array.tolist(),\n payload={\"text\": \"hello\"},\n))\n```\n\n### As Target (Named Vectors)\n\n```python\nschema = await qdrant.CollectionSchema.create(\n vectors={\n \"text\": qdrant.QdrantVectorDef(schema=text_embedder_key, distance=\"cosine\"),\n \"image\": qdrant.QdrantVectorDef(schema=image_embedder_key, distance=\"cosine\"),\n },\n)\ncollection = await qdrant.mount_collection_target(QDRANT_DB, \"multimodal\", schema=schema)\n\ncollection.declare_point(point=qdrant.PointStruct(\n id=\"point-1\",\n vector={\"text\": text_vec.tolist(), \"image\": image_vec.tolist()},\n payload={\"title\": \"example\"},\n))\n```\n\n### Distance Metrics\n\n- `\"cosine\"` -- Cosine similarity (default)\n- `\"dot\"` -- Dot product\n- `\"euclid\"` -- Euclidean distance (L2)\n\n---\n\n## SurrealDB\n\n**Import**: `from cocoindex.connectors import surrealdb`\n\n**Capabilities**: Target only | **Vector**: Native | **Model**: Graph + document\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import surrealdb\n\nSURREAL_DB = coco.ContextKey[surrealdb.ConnectionFactory](\"surreal_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n factory = surrealdb.ConnectionFactory(\n url=\"ws://localhost:8000/rpc\",\n namespace=\"my_ns\",\n database=\"my_db\",\n credentials={\"username\": \"root\", \"password\": \"root\"},\n )\n builder.provide(SURREAL_DB, factory)\n yield\n```\n\n### As Target (Table)\n\n```python\ntarget_table = await surrealdb.mount_table_target(\n SURREAL_DB,\n table_name=\"records\",\n table_schema=await surrealdb.TableSchema.from_class(MyRecord),\n)\n\ntarget_table.declare_row(row=MyRecord(id=\"rec1\", name=\"example\"))\n```\n\n### As Target (Relation)\n\n```python\nrelation_target = await surrealdb.mount_relation_target(\n SURREAL_DB,\n table_name=\"likes\",\n from_table=users_table,\n to_table=items_table,\n table_schema=await surrealdb.TableSchema.from_class(MyRelation),\n)\nrelation_target.declare_relation(from_id=\"user:1\", to_id=\"item:1\", record=MyRelation(...))\n```\n\n---\n\n## Neo4j\n\n**Import**: `from cocoindex.connectors import neo4j`\n\n**Capabilities**: Target only | **Vector**: Native (vector index) | **Model**: Knowledge graph (nodes + relationships)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import neo4j\n\nNEO4J_DB = coco.ContextKey[neo4j.ConnectionFactory](\"neo4j_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n neo4j_db = neo4j.ConnectionFactory(\n uri=\"bolt://localhost:7687\",\n auth=(\"neo4j\", \"cocoindex\"),\n database=\"neo4j\", # Optional, default \"neo4j\"\n )\n builder.provide(NEO4J_DB, neo4j_db)\n yield\n await neo4j_db.close() # closes the driver the factory opened on this loop\n```\n\n### As Target (Nodes + Relationships)\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass Person:\n name: str\n\n@dataclass\nclass Task:\n description: str\n\n@dataclass\nclass AssignedRel:\n priority: int\n\n# Mount node tables (each label is a table)\nperson_table = await neo4j.mount_table_target(\n NEO4J_DB,\n \"Person\",\n await neo4j.TableSchema.from_class(Person, primary_key=\"name\"),\n primary_key=\"name\",\n)\ntask_table = await neo4j.mount_table_target(\n NEO4J_DB,\n \"Task\",\n await neo4j.TableSchema.from_class(Task, primary_key=\"description\"),\n primary_key=\"description\",\n)\n\n# Mount a relationship target between two node tables\nassigned_rel = await neo4j.mount_relation_target(\n NEO4J_DB, \"ASSIGNED_TO\", person_table, task_table,\n await neo4j.TableSchema.from_class(AssignedRel),\n)\n\n# Declare nodes\nperson_table.declare_record(row=Person(name=\"Alice\")) # alias: declare_row\ntask_table.declare_record(row=Task(description=\"ship v1\"))\n\n# Declare edges (record optional; PK auto-derived from endpoints when omitted)\nassigned_rel.declare_relation(\n from_id=\"Alice\", to_id=\"ship v1\", record=AssignedRel(priority=1),\n)\n```\n\n### Vector Index\n\n```python\nperson_table.declare_vector_index(\n field=\"embedding\",\n dimension=384,\n metric=\"cosine\", # or \"euclidean\"\n)\n```\n\n### Type Override\n\n```python\nfrom typing import Annotated\n\n@dataclass\nclass Row:\n id: str\n score: Annotated[float, neo4j.Neo4jType(\"decimal\", encoder=str)]\n```\n\n---\n\n## FalkorDB\n\n**Import**: `from cocoindex.connectors import falkordb`\n\n**Capabilities**: Target only | **Vector**: Native (vector index) | **Model**: Knowledge graph (nodes + relationships)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import falkordb\n\nFALKOR_DB = coco.ContextKey[falkordb.ConnectionFactory](\"falkor_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n falkor_db = falkordb.ConnectionFactory(\n uri=\"falkor://localhost:6379\",\n graph=\"knowledge_graph\", # Optional, default \"default\"\n )\n builder.provide(FALKOR_DB, falkor_db)\n yield\n await falkor_db.close() # closes the client the factory opened on this loop\n```\n\n### As Target (Nodes + Relationships)\n\n```python\nperson_table = await falkordb.mount_table_target(\n FALKOR_DB,\n \"Person\",\n await falkordb.TableSchema.from_class(Person, primary_key=\"name\"),\n primary_key=\"name\",\n)\ntask_table = await falkordb.mount_table_target(\n FALKOR_DB,\n \"Task\",\n await falkordb.TableSchema.from_class(Task, primary_key=\"description\"),\n primary_key=\"description\",\n)\n\n# Mount relationship target (no schema -> PK auto-derived from endpoints)\nassigned_rel = await falkordb.mount_relation_target(\n FALKOR_DB, \"ASSIGNED_TO\", person_table, task_table,\n)\n\n# Declare nodes\nperson_table.declare_record(row=Person(name=\"Alice\")) # alias: declare_row\ntask_table.declare_record(row=Task(description=\"ship v1\"))\n\n# Declare edges\nassigned_rel.declare_relation(from_id=\"Alice\", to_id=\"ship v1\")\n```\n\n### Vector Index\n\n```python\nperson_table.declare_vector_index(\n field=\"embedding\",\n dimension=384,\n metric=\"cosine\", # or \"euclidean\", \"ip\"\n)\n```\n\n### Type Override\n\n```python\nfrom typing import Annotated\n\n@dataclass\nclass Row:\n id: str\n score: Annotated[float, falkordb.FalkorType(\"decimal\", encoder=str)]\n```\n\n---\n\n## LocalFS\n\n**Import**: `from cocoindex.connectors import localfs`\n\n**Capabilities**: Source + Target\n\n### As Source\n\n```python\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import PatternFilePathMatcher\n\nfiles = localfs.walk_dir(\n pathlib.Path(\"./data\"),\n recursive=True,\n path_matcher=PatternFilePathMatcher(\n included_patterns=[\"**/*.py\", \"**/*.md\"],\n excluded_patterns=[\".*/**\", \"__pycache__/**\"],\n ),\n live=True, # Enable file watching for live mode\n)\n\n# Process each file\nawait coco.mount_each(process_file, files.items(), target_table)\n```\n\n### As Target (Single File)\n\n```python\nlocalfs.declare_file(\n outdir / \"output.txt\",\n \"file content\",\n create_parent_dirs=True,\n)\n```\n\n### As Target (Directory)\n\n```python\ndir_target = await localfs.mount_dir_target(pathlib.Path(\"./output\"))\ndir_target.declare_file(\"file1.txt\", \"content 1\")\ndir_target.declare_file(\"subdir/file2.txt\", \"content 2\")\n```\n\n### Stable File Paths\n\nUse `localfs.FilePath` for stable memoization when the base directory might move:\n\n```python\nfiles = localfs.walk_dir(localfs.FilePath(path=\"./data\"), ...)\n```\n\n---\n\n## Amazon S3\n\n**Import**: `from cocoindex.connectors import amazon_s3`\n\n**Capabilities**: Source only\n\n### Connection Setup\n\n```python\nimport aiobotocore.session\nfrom aiobotocore.client import AioBaseClient\n\nS3_CLIENT = coco.ContextKey[AioBaseClient](\"s3_client\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n session = aiobotocore.session.get_session()\n async with session.create_client(\"s3\") as s3_client:\n builder.provide(S3_CLIENT, s3_client)\n yield\n```\n\n### Reading Objects\n\n```python\nfrom cocoindex.connectors import amazon_s3\n\nclient = coco.use_context(S3_CLIENT)\n\n# List and iterate objects\nwalker = amazon_s3.list_objects(\n client, \"my-bucket\",\n prefix=\"data/\",\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.json\"]),\n max_file_size=10_000_000,\n)\n\nawait coco.mount_each(process_file, walker.items(), target)\n\n# Single object\nfile = await amazon_s3.get_object(client, \"s3://my-bucket/path/to/file.json\")\ncontent = await amazon_s3.read(client, \"s3://my-bucket/path/to/file.json\")\n```\n\n---\n\n## Kafka\n\n**Import**: `from cocoindex.connectors import kafka`\n\n**Capabilities**: Source + Target\n\n### As Source (Live)\n\n```python\nfrom confluent_kafka.aio import AIOConsumer\nfrom cocoindex.connectors import kafka\n\nconsumer = AIOConsumer({\n \"bootstrap.servers\": \"localhost:9092\",\n \"group.id\": \"my-group\",\n \"enable.auto.commit\": \"false\",\n \"auto.offset.reset\": \"earliest\",\n})\n\nitems = kafka.topic_as_map(consumer, [\"my-topic\"])\nawait coco.mount_each(process_message, items, target_table)\n```\n\n`topic_as_map()` returns a `LiveMapFeed`, which `mount_each()` auto-detects for live mode.\n\n### As Target\n\n```python\nfrom confluent_kafka.aio import AIOProducer\n\nKAFKA_PRODUCER = coco.ContextKey[AIOProducer](\"kafka_producer\")\n\ntopic_target = await kafka.mount_kafka_topic_target(KAFKA_PRODUCER, \"my-topic\")\ntopic_target.declare_target_state(key=\"msg-key\", value=json.dumps(data))\n```\n\n---\n\n## Apache Doris\n\n**Import**: `from cocoindex.connectors import doris`\n\n**Capabilities**: Target only | **Vector**: Native | **Features**: Vector indexes, inverted indexes, Stream Load\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import doris\n\nDORIS_DB = coco.ContextKey[doris.ManagedConnection](\"doris_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n config = doris.DorisConnectionConfig(\n fe_host=\"localhost\",\n database=\"my_db\",\n fe_http_port=8080,\n query_port=9030,\n username=\"root\",\n password=\"\",\n )\n conn = doris.connect(config)\n builder.provide(DORIS_DB, conn)\n yield\n```\n\n### As Target\n\n```python\ntarget_table = await doris.mount_table_target(\n DORIS_DB,\n table_name=\"embeddings\",\n table_schema=await doris.TableSchema.from_class(MyRecord, primary_key=[\"id\"]),\n vector_indexes=[doris.VectorIndexDef(field_name=\"embedding\", metric_type=\"cosine_distance\")],\n)\n\ntarget_table.declare_row(row=MyRecord(id=1, text=\"hello\", embedding=vec))\n```\n\n---\n\n## Google Drive\n\n**Import**: `from cocoindex.connectors import google_drive`\n\n**Capabilities**: Source only\n\n```python\nfrom cocoindex.connectors import google_drive\n\nsource = google_drive.GoogleDriveSource(\n service_account_credential_path=\"credentials.json\",\n root_folder_ids=[\"folder-id\"],\n)\nawait coco.mount_each(process_file, source.items(), target)\n```\n\n---\n\n## Connector Comparison\n\n| Connector | Source | Target | Vectors | Best For |\n|-----------|--------|--------|---------|----------|\n| **PostgreSQL** | Y | Y | pgvector | Production SQL + vectors |\n| **SQLite** | - | Y | sqlite-vec | Local SQL + vectors |\n| **LanceDB** | - | Y | Native | Cloud-native vector DB |\n| **Qdrant** | - | Y | Native | Specialized vector search |\n| **SurrealDB** | - | Y | Native | Graph + document DB |\n| **Neo4j** | - | Y | Native | Knowledge graphs |\n| **FalkorDB** | - | Y | Native | Knowledge graphs (Redis) |\n| **Apache Doris** | - | Y | Native | Analytical DB + vectors |\n| **LocalFS** | Y | Y | N/A | File-based pipelines |\n| **Amazon S3** | Y | - | N/A | Cloud object storage |\n| **Kafka** | Y | Y | N/A | Streaming pipelines |\n| **Google Drive** | Y | - | N/A | Cloud file source |\n\n---\n\n## See Also\n\n- [API Reference](./api_reference.md)\n- [Patterns](./patterns.md)\n- [Setup Database](./setup_database.md)\n","isInternal":false,"tokens":4540,"sizeBytes":17627},{"name":"patterns.md","path":"skills/cocoindex/references/patterns.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/skills/cocoindex/references/patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# CocoIndex Common Patterns\n\nCommon patterns and workflows for building CocoIndex pipelines.\n\n## Core Pattern: TargetState = Transform(SourceState)\n\nAll CocoIndex pipelines follow this declarative pattern:\n\n1. Read source state\n2. Transform\n3. Declare target state\n\nCocoIndex handles the incremental sync automatically.\n\n---\n\n## Pattern 1: File Transformation Pipeline\n\n**Use case**: Transform files from one format to another (e.g., Markdown -> HTML, PDF -> Markdown)\n\n```python\nimport pathlib\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom markdown_it import MarkdownIt\n\n_markdown_it = MarkdownIt(\"gfm-like\")\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, outdir: pathlib.Path) -> None:\n html = _markdown_it.render(await file.read_text())\n outname = \"__\".join(file.file_path.path.parts) + \".html\"\n localfs.declare_file(outdir / outname, html, create_parent_dirs=True)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n files = localfs.walk_dir(\n sourcedir,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n live=True, # Enable live file watching\n )\n await coco.mount_each(process_file, files.items(), outdir)\n\napp = coco.App(\n coco.AppConfig(name=\"FilesTransform\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n outdir=pathlib.Path(\"./output_html\"),\n)\n```\n\n**Key points:**\n- `memo=True` -- Skip reprocessing unchanged files\n- `mount_each()` -- One component per file; keys from `files.items()` become component subpaths\n- `live=True` on `walk_dir()` -- Enables file watching for live mode\n- Auto-cleanup -- Deleting source file automatically removes output file\n\n---\n\n## Pattern 2: Vector Embedding Pipeline\n\n**Use case**: Chunk and embed documents for semantic search\n\n```python\nimport pathlib\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator, Annotated\n\nimport asyncpg\nfrom numpy.typing import NDArray\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs, postgres\nfrom cocoindex.ops.text import RecursiveSplitter\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\nfrom cocoindex.resources.chunk import Chunk\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom cocoindex.resources.id import IdGenerator\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n_splitter = RecursiveSplitter()\n\n@dataclass\nclass DocEmbedding:\n id: int\n filename: str\n chunk_start: int\n chunk_end: int\n text: str\n embedding: Annotated[NDArray, EMBEDDER]\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n@coco.fn\nasync def process_chunk(\n chunk: Chunk, filename: pathlib.PurePath,\n id_gen: IdGenerator, table: postgres.TableTarget[DocEmbedding],\n) -> None:\n table.declare_row(row=DocEmbedding(\n id=await id_gen.next_id(chunk.text),\n filename=str(filename),\n chunk_start=chunk.start.char_offset,\n chunk_end=chunk.end.char_offset,\n text=chunk.text,\n embedding=await coco.use_context(EMBEDDER).embed(chunk.text),\n ))\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, table: postgres.TableTarget[DocEmbedding]) -> None:\n text = await file.read_text()\n chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language=\"markdown\")\n id_gen = IdGenerator()\n await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n target_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"doc_embeddings\",\n table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=[\"id\"]),\n )\n target_table.declare_vector_index(column=\"embedding\")\n\n files = localfs.walk_dir(sourcedir, recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]))\n await coco.mount_each(process_file, files.items(), target_table)\n\napp = coco.App(coco.AppConfig(name=\"TextEmbedding\"), app_main,\n sourcedir=pathlib.Path(\"./markdown_files\"))\n```\n\n**Key points:**\n- `mount_table_target(PG_DB, ...)` -- Takes `ContextKey` as first argument\n- `Annotated[NDArray, EMBEDDER]` -- Vector annotation uses ContextKey for auto dimension inference\n- `map()` -- Concurrent execution within a component (no child components created)\n- `IdGenerator` -- Generates stable unique IDs for chunks across incremental updates\n- `memo=True` on `process_file` -- Skips unchanged files entirely\n\n---\n\n## Pattern 3: Database Source -> Transform -> Database Target\n\n**Use case**: Transform data from one database to another\n\n```python\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator\n\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nSOURCE_DB_URL = \"postgres://localhost/source_db\"\nTARGET_DB_URL = \"postgres://localhost/target_db\"\n\nSOURCE_DB = coco.ContextKey[asyncpg.Pool](\"source_db\")\nTARGET_DB = coco.ContextKey[asyncpg.Pool](\"target_db\")\n\n@dataclass\nclass SourceRecord:\n id: int\n name: str\n value: float\n\n@dataclass\nclass TargetRecord:\n id: int\n name: str\n value: float\n processed: bool\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with (\n await asyncpg.create_pool(SOURCE_DB_URL) as source_pool,\n await asyncpg.create_pool(TARGET_DB_URL) as target_pool,\n ):\n builder.provide(SOURCE_DB, source_pool)\n builder.provide(TARGET_DB, target_pool)\n yield\n\n@coco.fn(memo=True)\nasync def process_record(record: SourceRecord, target_table: postgres.TableTarget[TargetRecord]) -> None:\n target_table.declare_row(row=TargetRecord(\n id=record.id, name=record.name.upper(),\n value=record.value * 2, processed=True,\n ))\n\n@coco.fn\nasync def app_main() -> None:\n target_table = await postgres.mount_table_target(\n TARGET_DB,\n table_name=\"target_records\",\n table_schema=await postgres.TableSchema.from_class(TargetRecord, primary_key=[\"id\"]),\n )\n\n source = postgres.PgTableSource(\n coco.use_context(SOURCE_DB),\n table_name=\"source_records\",\n row_type=SourceRecord,\n )\n await coco.mount_each(\n coco.component_subpath(\"record\"),\n process_record,\n source.fetch_rows().items(key=lambda r: r.id),\n target_table,\n )\n\napp = coco.App(coco.AppConfig(name=\"DatabaseTransform\"), app_main)\n```\n\n---\n\n## Pattern 4: LLM-Based Extraction Pipeline\n\n**Use case**: Extract structured data using LLMs\n\n```python\nimport instructor\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator\nfrom pydantic import BaseModel\nfrom litellm import acompletion\n\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\n\n_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)\n\nclass ExtractedTopic(BaseModel):\n name: str\n description: str\n\nclass ExtractionResult(BaseModel):\n title: str\n topics: list[ExtractedTopic]\n\n@dataclass\nclass Message:\n id: int\n title: str\n content: str\n\n@dataclass\nclass Topic:\n message_id: int\n name: str\n description: str\n\n@coco.fn(memo=True)\nasync def extract_and_store(\n content: str, message_id: int,\n messages_table: postgres.TableTarget[Message],\n topics_table: postgres.TableTarget[Topic],\n) -> None:\n result = await _instructor_client.chat.completions.create(\n model=\"gpt-4\",\n response_model=ExtractionResult,\n messages=[{\"role\": \"user\", \"content\": f\"Extract topics:\\n\\n{content}\"}],\n )\n messages_table.declare_row(row=Message(id=message_id, title=result.title, content=content))\n for topic in result.topics:\n topics_table.declare_row(row=Topic(\n message_id=message_id, name=topic.name, description=topic.description,\n ))\n\n@coco.fn\nasync def app_main(input_texts: list[str]) -> None:\n messages_table = await postgres.mount_table_target(\n PG_DB, table_name=\"messages\",\n table_schema=await postgres.TableSchema.from_class(Message, primary_key=[\"id\"]),\n )\n topics_table = await postgres.mount_table_target(\n PG_DB, table_name=\"topics\",\n table_schema=await postgres.TableSchema.from_class(Topic, primary_key=[\"message_id\", \"name\"]),\n )\n for idx, text in enumerate(input_texts):\n await coco.mount(\n coco.component_subpath(\"text\", idx),\n extract_and_store, text, idx, messages_table, topics_table,\n )\n\napp = coco.App(coco.AppConfig(name=\"LLMExtraction\"), app_main, input_texts=[\"text1...\", \"text2...\"])\n```\n\n**Key points:**\n- `memo=True` on extraction -- Avoids re-calling LLM for unchanged inputs\n- Multiple target tables -- Declare rows in multiple tables from single component\n- Pydantic models -- For structured LLM outputs\n\n---\n\n## Pattern 5: Kafka Streaming Pipeline\n\n**Use case**: Process Kafka messages and write to a database\n\n```python\nimport json\nfrom collections.abc import AsyncIterator\nfrom dataclasses import dataclass\n\nfrom confluent_kafka import Message\nfrom confluent_kafka.aio import AIOConsumer\n\nimport cocoindex as coco\nfrom cocoindex.connectors import kafka, lancedb\n\nLANCE_DB = coco.ContextKey[lancedb.LanceAsyncConnection](\"lance_db\")\n\n@dataclass\nclass Product:\n sku: str\n name: str\n category: str\n price: float\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n conn = await lancedb.connect_async(\"./lancedb_data\")\n builder.provide(LANCE_DB, conn)\n yield\n\n@coco.fn\nasync def process_message(msg: Message, table: lancedb.TableTarget[Product]) -> None:\n value = msg.value()\n if value is None:\n return\n row = json.loads(value.decode() if isinstance(value, bytes) else value)\n table.declare_row(row=Product(**{**row, \"price\": float(row[\"price\"])}))\n\n@coco.fn\nasync def app_main() -> None:\n products_table = await lancedb.mount_table_target(\n LANCE_DB, table_name=\"products\",\n table_schema=await lancedb.TableSchema.from_class(Product, primary_key=[\"sku\"]),\n )\n\n consumer = AIOConsumer({\n \"bootstrap.servers\": \"localhost:9092\",\n \"group.id\": \"my-group\",\n \"enable.auto.commit\": \"false\",\n \"auto.offset.reset\": \"earliest\",\n })\n items = kafka.topic_as_map(consumer, [\"products-topic\"])\n await coco.mount_each(process_message, items, products_table)\n\napp = coco.App(coco.AppConfig(name=\"KafkaToLanceDB\"), app_main)\n```\n\n**Key points:**\n- `kafka.topic_as_map()` returns a `LiveMapFeed` -- `mount_each()` auto-detects for live mode\n- Pipeline runs continuously in live mode, processing new messages as they arrive\n\n---\n\n## Pattern 6: Context Management for Shared Resources\n\n**Use case**: Share expensive resources across components\n\n```python\nimport cocoindex as coco\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\n\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\nCONFIG = coco.ContextKey[dict](\"config\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n builder.provide(CONFIG, {\"chunk_size\": 1000, \"overlap\": 200})\n yield\n\n@coco.fn\nasync def process_item(text: str) -> None:\n embedder = coco.use_context(EMBEDDER)\n config = coco.use_context(CONFIG)\n embedding = await embedder.embed(text)\n ...\n```\n\n**Key points:**\n- `detect_change=True` -- Opt in to invalidate memos when value changes (use for models/config affecting output)\n- `detect_change=False` (default) -- For resources not affecting computation (DB connections, loggers)\n- `ContextKey` name is stable identity -- avoid renaming across runs\n\n---\n\n## Common Anti-Patterns to Avoid\n\n### Missing `@coco.fn` Decorator\n\n```python\n# BAD: Missing decorator\nasync def process_file(file, table):\n table.declare_row(...)\n\n# GOOD:\n@coco.fn\nasync def process_file(file, table):\n table.declare_row(...)\n```\n\n### Reprocessing Everything\n\n```python\n# BAD: No memoization\n@coco.fn # Missing memo=True\nasync def process_file(file, table):\n embedding = await embedder.embed(await file.read_text()) # Expensive!\n\n# GOOD:\n@coco.fn(memo=True)\nasync def process_file(file, table):\n ...\n```\n\n### Unstable Component Paths\n\n```python\n# BAD: Using object references or indices\nfor file in files:\n await coco.mount(coco.component_subpath(file), ...) # Object ref\nfor idx, item in enumerate(items):\n await coco.mount(coco.component_subpath(idx), ...) # Index changes\n\n# GOOD: Use stable identifiers\nawait coco.mount_each(process_file, files.items(), table) # Keys from items()\n```\n\n### Loading Resources Per Component\n\n```python\n# BAD: Loading model in every component\n@coco.fn\nasync def process(text):\n model = SentenceTransformer(\"model\") # Loaded repeatedly!\n\n# GOOD: Load once in lifespan, use via context\nembedder = coco.use_context(EMBEDDER)\n```\n\n### Mixing Target State with Side Effects\n\n```python\n# BAD: Side effects not detected\n@coco.fn\nasync def process(data):\n requests.post(\"https://api.example.com\", json=data) # Not detected!\n\n# GOOD: Only declare target states\ntable.declare_row(row=result)\n```\n\n---\n\n## See Also\n\n- [Connectors Reference](./connectors.md)\n- [API Reference](./api_reference.md)\n- [Setup Project](./setup_project.md)\n","isInternal":false,"tokens":3311,"sizeBytes":13957},{"name":"setup_database.md","path":"skills/cocoindex/references/setup_database.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/skills/cocoindex/references/setup_database.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Database Setup Guide\n\nSetup instructions for databases used with CocoIndex.\n\n## PostgreSQL with pgvector\n\n### Using Docker (Recommended)\n\n```bash\ncat > docker-compose.yml < docker-compose.yml <=1.0.0\",\n \"sentence-transformers\",\n \"asyncpg\",\n]\n```\n\n### PostgreSQL Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"asyncpg\",\n]\n```\n\n### SQLite Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"sqlite-vec\",\n]\n```\n\n### LanceDB Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"lancedb\",\n]\n```\n\n### Qdrant Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"qdrant-client\",\n]\n```\n\n### Kafka Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"confluent-kafka\",\n]\n```\n\n### LLM-Based Extraction\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"litellm\",\n \"instructor\",\n \"pydantic>=2.0\",\n \"asyncpg\",\n]\n```\n\n---\n\n## Environment Configuration\n\n### `.env` File\n\nThe `cocoindex` CLI automatically loads `.env` from the current directory (via `find_dotenv`).\n\n```bash\n# CocoIndex internal database (optional fallback).\n# Only used if the lifespan does not set builder.settings.db_path.\n# The `cocoindex init` template sets db_path in the lifespan instead, so this is not needed there.\nCOCOINDEX_DB=./cocoindex.db\n\n# PostgreSQL (if using)\nPOSTGRES_URL=postgres://user:pass@localhost/db\n\n# Qdrant (if using)\nQDRANT_URL=http://localhost:6333\n\n# API keys (if using LLM extraction)\nOPENAI_API_KEY=your-openai-api-key\nANTHROPIC_API_KEY=your-anthropic-api-key\n```\n\n### Manual Settings (in lifespan)\n\n```python\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n builder.settings.db_path = pathlib.Path(\"./custom.db\")\n yield\n```\n\n---\n\n## Running Your Pipeline\n\n```bash\nuv sync # Install dependencies\nuv run cocoindex update main.py # Run pipeline\nuv run cocoindex update main.py -L # Run in live mode\nuv run cocoindex show main.py # Show component paths\nuv run cocoindex drop main.py -f # Reset everything\n```\n\nIf `pyproject.toml` has `[tool.uv.sources]` or `[[tool.uv.index]]` tables, install with uv. pip ignores those tables, so it would fetch the packages they pin from PyPI instead.\n\n---\n\n## Common Issues\n\n### Import Errors\n\n```bash\nuv sync\n```\n\n### Database Connection Errors\n\nVerify database is running and `.env` has correct URLs. See [setup_database.md](./setup_database.md).\n\n---\n\n## See Also\n\n- [Database Setup](./setup_database.md)\n- [Patterns](./patterns.md)\n- [API Reference](./api_reference.md)\n","isInternal":false,"tokens":808,"sizeBytes":2990},{"name":"SKILL.md","path":"skills/cocoindex/SKILL.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/skills/cocoindex/SKILL.md","title":"cocoindex","category":"anthropic-skill","format":"markdown","content":"---\nname: cocoindex\ndescription: This skill should be used when building data processing pipelines with CocoIndex, a Python library for incremental data transformation. Use when the task involves processing files/data into databases, creating vector embeddings, building knowledge graphs, ETL workflows, or any data pipeline requiring automatic change detection and incremental updates. CocoIndex is Python-native (supports any Python types), has no DSL, and uses version 1.0.0 or later.\n---\n\n# CocoIndex\n\nCocoIndex is a Python library for building incremental data processing pipelines with declarative target states. Think spreadsheets or React for data pipelines: declare what the output should look like based on current input, and CocoIndex automatically handles incremental updates, change detection, and syncing to external systems.\n\n## Overview\n\nCocoIndex enables building data pipelines that:\n\n- **Automatically handle incremental updates**: Only reprocess changed data\n- **Use declarative target states**: Declare what should exist, not how to update\n- **Support any Python types**: No custom DSL -- use dataclasses, Pydantic, NamedTuple\n- **Provide function memoization**: Skip expensive operations when inputs/code unchanged\n- **Sync to multiple targets**: PostgreSQL, SQLite, LanceDB, Qdrant, SurrealDB, Apache Doris, file systems, Kafka\n\n**Key principle**: `TargetState = Transform(SourceState)`\n\n## When to Use This Skill\n\nUse this skill when building pipelines that involve:\n\n- **Document processing**: PDF/Markdown conversion, text extraction, chunking\n- **Vector embeddings**: Embedding documents/code for semantic search\n- **Database transformations**: ETL from source DB to target DB\n- **Knowledge graphs**: Extract entities and relationships from data\n- **LLM-based extraction**: Structured data extraction using LLMs\n- **File-based pipelines**: Transform files from one format to another\n- **Incremental indexing**: Keep search indexes up-to-date with source changes\n- **Streaming pipelines**: Kafka-based real-time data processing\n\n## Quick Start: Creating a New Project\n\n### Initialize Project\n\n```bash\ncocoindex init my-project\ncd my-project\n```\n\nThis creates: `main.py`, `pyproject.toml`, `README.md`. The generated `main.py` sets the database location in its lifespan via `builder.settings.db_path = pathlib.Path(\"./cocoindex.db\")`.\n\n### Add Dependencies\n\n```toml\n# For vector embeddings with PostgreSQL\ndependencies = [\"cocoindex>=1.0.0\", \"sentence-transformers\", \"asyncpg\"]\n\n# For LLM extraction\ndependencies = [\"cocoindex>=1.0.0\", \"litellm\", \"instructor\", \"pydantic>=2.0\"]\n```\n\nSee [references/setup_project.md](references/setup_project.md) for complete examples.\n\n### Run the Pipeline\n\n```bash\nuv run cocoindex update main.py # installs dependencies, then runs\n```\n\n## Core Concepts\n\n### 1. Apps\n\nAn **App** is the top-level executable that binds a main function with parameters:\n\n```python\nimport cocoindex as coco\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n ...\n\napp = coco.App(\n coco.AppConfig(name=\"MyApp\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n)\n```\n\n### 2. Functions (`@coco.fn`)\n\nThe `@coco.fn` decorator marks functions as CocoIndex processing functions. Add `memo=True` to skip re-execution when inputs/code are unchanged:\n\n```python\n@coco.fn(memo=True)\nasync def expensive_operation(data: str) -> Result:\n # LLM call, embedding generation, heavy computation\n return await expensive_transform(data)\n```\n\n**Key parameters:**\n- `memo=True` -- Enable memoization (skip if inputs/code unchanged)\n- `version=1` -- Explicit version bump to force re-execution\n- `batching=True` -- Auto-batch concurrent calls (async only)\n- `runner=coco.GPU` -- Serialize GPU-bound execution\n\n### 3. Processing Components\n\nA **processing component** groups an item's processing with its target states.\n\n**Mount components** with `mount_each()` (preferred for lists) or `mount()`:\n\n```python\n# One component per item (preferred for lists)\nawait coco.mount_each(process_file, files.items(), target_table)\n\n# Single component (subpath auto-derived from fn.__name__)\nawait coco.mount(setup_fn, arg1)\n\n# Dependent component (blocks until result returned)\nresult = await coco.use_mount(init_fn)\n\n# Explicit subpath (when you need a specific path, e.g. in loops)\nawait coco.mount(coco.component_subpath(\"item\", item_id), process_item, item)\n```\n\n**Key points:**\n- All mount APIs are `async`\n- `mount()`, `use_mount()`, and `mount_each()` auto-derive subpath from `fn.__name__`; optional explicit subpath as first arg\n- Use `use_mount()` when you need the return value\n- Use stable component paths for proper memoization and cleanup\n\n### 4. Target States\n\n**Declare** what should exist -- CocoIndex handles creation/update/deletion:\n\n```python\n# Database row target\ntable.declare_row(row=MyRecord(id=1, name=\"example\"))\n\n# File target\nlocalfs.declare_file(outdir / \"output.txt\", content, create_parent_dirs=True)\n\n# Kafka message target\ntopic_target.declare_target_state(key=\"msg-1\", value=json.dumps(data))\n```\n\n### 5. Context for Shared Resources\n\nUse `ContextKey` to share expensive resources (DB connections, models) across components:\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n# In processing functions:\nembedder = coco.use_context(EMBEDDER)\n```\n\nThe `@coco.lifespan` decorator registers the function to the default CocoIndex environment, which is shared among all apps by default. `ContextKey` also serves as the stable identity for sources/targets -- the `key` string must remain stable across runs.\n\n### 6. ID Generation\n\nGenerate stable, unique identifiers that persist across incremental updates:\n\n```python\nfrom cocoindex.resources.id import generate_id, IdGenerator\n\n# Deterministic: same dep -> same ID\nchunk_id = await generate_id(chunk.text)\n\n# Always distinct: each call -> new ID, even with same dep\nid_gen = IdGenerator()\nfor chunk in chunks:\n chunk_id = await id_gen.next_id(chunk.text)\n```\n\n### 7. Catch-Up vs Live Mode\n\nBy default, `app.update()` runs in **catch-up mode**: it scans all sources, processes what changed since the last run (memoized components are skipped), syncs target states, and returns. Each call still has to scan sources to discover changes.\n\n**Live mode** keeps the app running after catch-up and lets components stream changes continuously from their sources (e.g., file watcher, Kafka consumer), applying them with very low latency.\n\n```python\n# Enable live mode\napp.update_blocking(live=True)\n# Or: cocoindex update main.py -L\n```\n\nTwo things are needed: (1) enable live mode on the app, and (2) use a source that supports live updates.\n\n- **`LiveMapView`** sources (e.g., `localfs.walk_dir(..., live=True)`) scan current state first, then watch for changes. They also work in catch-up mode -- write the pipeline once, choose mode at run time.\n- **`LiveMapFeed`** sources (e.g., `kafka.topic_as_map()`) only stream changes with no initial snapshot. `mount_each()` auto-detects these and creates a live component internally.\n\n```python\n# LocalFS with live watching\nfiles = localfs.walk_dir(sourcedir, live=True, ...)\nawait coco.mount_each(process_file, files.items(), target)\n\n# Kafka -- inherently live\nitems = kafka.topic_as_map(consumer, [\"my-topic\"])\nawait coco.mount_each(process_message, items, target)\n```\n\n## Common Pipeline Patterns\n\n### Pattern 1: File Transformation\n\n```python\nimport pathlib\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, outdir: pathlib.Path) -> None:\n content = await file.read_text()\n transformed = transform_content(content)\n outname = file.file_path.path.stem + \".out\"\n localfs.declare_file(outdir / outname, transformed, create_parent_dirs=True)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n files = localfs.walk_dir(\n sourcedir,\n recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n )\n await coco.mount_each(process_file, files.items(), outdir)\n\napp = coco.App(\n coco.AppConfig(name=\"Transform\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n outdir=pathlib.Path(\"./out\"),\n)\n```\n\n### Pattern 2: Vector Embedding Pipeline\n\n```python\nimport pathlib\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator, Annotated\n\nimport asyncpg\nfrom numpy.typing import NDArray\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs, postgres\nfrom cocoindex.ops.text import RecursiveSplitter\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\nfrom cocoindex.resources.chunk import Chunk\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom cocoindex.resources.id import IdGenerator\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n_splitter = RecursiveSplitter()\n\n@dataclass\nclass DocEmbedding:\n id: int\n filename: str\n text: str\n embedding: Annotated[NDArray, EMBEDDER] # Dimensions inferred from ContextKey\n chunk_start: int\n chunk_end: int\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n@coco.fn\nasync def process_chunk(\n chunk: Chunk, filename: pathlib.PurePath,\n id_gen: IdGenerator, table: postgres.TableTarget[DocEmbedding],\n) -> None:\n table.declare_row(row=DocEmbedding(\n id=await id_gen.next_id(chunk.text),\n filename=str(filename),\n text=chunk.text,\n embedding=await coco.use_context(EMBEDDER).embed(chunk.text),\n chunk_start=chunk.start.char_offset,\n chunk_end=chunk.end.char_offset,\n ))\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, table: postgres.TableTarget[DocEmbedding]) -> None:\n text = await file.read_text()\n chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500)\n id_gen = IdGenerator()\n await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n target_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"embeddings\",\n table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=[\"id\"]),\n )\n target_table.declare_vector_index(column=\"embedding\")\n\n files = localfs.walk_dir(sourcedir, recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]))\n await coco.mount_each(process_file, files.items(), target_table)\n\napp = coco.App(coco.AppConfig(name=\"Embedding\"), app_main, sourcedir=pathlib.Path(\"./data\"))\n```\n\n### Pattern 3: LLM-Based Extraction\n\n```python\nimport instructor\nfrom pydantic import BaseModel\nfrom litellm import acompletion\n\n_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)\n\nclass ExtractionResult(BaseModel):\n title: str\n topics: list[str]\n\n@coco.fn(memo=True) # Memo avoids re-calling LLM\nasync def extract_and_store(content: str, message_id: int, table) -> None:\n result = await _instructor_client.chat.completions.create(\n model=\"gpt-4\",\n response_model=ExtractionResult,\n messages=[{\"role\": \"user\", \"content\": f\"Extract topics: {content}\"}],\n )\n table.declare_row(row=Message(id=message_id, title=result.title, content=content))\n```\n\n## Connectors and Operations\n\nCocoIndex provides connectors for reading from and writing to external systems:\n\n| Connector | Source | Target | Vectors | Use Case |\n|-----------|--------|--------|---------|----------|\n| **PostgreSQL** | Y | Y | pgvector | Production SQL + vectors |\n| **SQLite** | - | Y | sqlite-vec | Local SQL + vectors |\n| **LanceDB** | - | Y | Y | Cloud-native vector DB |\n| **Qdrant** | - | Y | Y | Specialized vector DB |\n| **SurrealDB** | - | Y | Y | Graph + document DB |\n| **Apache Doris** | - | Y | Y | Analytical DB + vectors |\n| **LocalFS** | Y | Y | N/A | File-based pipelines |\n| **Amazon S3** | Y | - | N/A | Cloud object storage |\n| **Kafka** | Y | Y | N/A | Streaming pipelines |\n| **Google Drive** | Y | - | N/A | Cloud file source |\n\nFor detailed connector documentation, see [references/connectors.md](references/connectors.md).\n\n## Text and Embedding Operations\n\n### Text Splitting\n\n```python\nfrom cocoindex.ops.text import RecursiveSplitter, detect_code_language\n\nsplitter = RecursiveSplitter()\nlanguage = detect_code_language(filename=\"example.py\")\nchunks = splitter.split(text, chunk_size=1000, chunk_overlap=200, language=language)\n```\n\n### Embeddings\n\n```python\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\n\nembedder = SentenceTransformerEmbedder(\"sentence-transformers/all-MiniLM-L6-v2\")\nembedding = await embedder.embed(text) # Returns NDArray\n```\n\n## CLI Commands\n\n```bash\ncocoindex init my-project # Create new project\ncocoindex update main.py # Run app\ncocoindex update main.py:my_app # Run specific app\ncocoindex update main.py -L # Run in live mode (continuous)\ncocoindex update main.py --full-reprocess # Reprocess everything\ncocoindex drop main.py [-f] # Drop and reset all state\ncocoindex ls [main.py] # List apps\ncocoindex show main.py [--tree] # Show component paths\n```\n\n## Best Practices\n\n### 1. Use `@coco.fn` on All Processing Functions\n\nEvery function that participates in the pipeline (declares target states, calls mount APIs, etc.) must be decorated with `@coco.fn`.\n\n### 2. Add Memoization for Expensive Operations\n\n```python\n@coco.fn(memo=True) # Skip re-execution when inputs/code unchanged\nasync def process_chunk(chunk, table):\n embedding = await embedder.embed(chunk.text) # Expensive!\n table.declare_row(...)\n```\n\n### 3. Use Stable Component Paths\n\n```python\n# Good: Stable identifiers\ncoco.component_subpath(\"file\", str(file.file_path.path))\ncoco.component_subpath(\"record\", record.id)\n\n# Bad: Unstable identifiers\ncoco.component_subpath(\"file\", file) # Object reference\ncoco.component_subpath(\"idx\", idx) # Index changes\n```\n\n### 4. Use Context for Shared Resources\n\n```python\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n yield\n```\n\n### 5. Use `Annotated[NDArray, CONTEXT_KEY]` for Vectors\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions from ContextKey\n```\n\n### 6. Use Convenience APIs for Targets\n\n```python\n# Mount table target -- subpath is automatic\ntable = await postgres.mount_table_target(\n PG_DB,\n table_name=\"my_table\",\n table_schema=await postgres.TableSchema.from_class(MyRecord, primary_key=[\"id\"]),\n)\n```\n\n## Troubleshooting\n\n### Everything Reprocessing\n\nAdd `memo=True` to expensive functions:\n\n```python\n@coco.fn(memo=True) # Add this\nasync def process_item(item):\n ...\n```\n\n### Memoization Not Working\n\nCheck component paths are stable. Use stable IDs, not object references.\n\n## Resources\n\n### references/\n\n- **[api_reference.md](references/api_reference.md)**: Quick API reference\n- **[connectors.md](references/connectors.md)**: Complete connector reference\n- **[patterns.md](references/patterns.md)**: Detailed pipeline patterns\n- **[setup_project.md](references/setup_project.md)**: Project setup guide\n- **[setup_database.md](references/setup_database.md)**: Database setup guide\n\n### Runnable examples\n\nEvery pattern above has a complete, runnable app under\n[`examples/`](https://github.com/cocoindex-io/cocoindex/tree/main/examples) — start\nfrom the one closest to the task and adapt it. Each has its own `README.md` and\nmost Python examples have a `.env.example`; see\n[`examples/AGENTS.md`](https://github.com/cocoindex-io/cocoindex/blob/main/examples/AGENTS.md)\nfor the full map, credentials, and per-example run commands. Good starting points:\n\n- Vector search → `text_embedding` (Postgres), `text_embedding_qdrant` / `_lancedb` (other stores)\n- Code search → `code_embedding`\n- LLM extraction → `hn_trending_topics`, `patient_intake_extraction_baml`\n- Knowledge graph → `conversation_to_knowledge`, `meeting_notes_graph_neo4j`\n- Custom transform → `files_transform`, `pdf_to_markdown`\n\n### External\n\n- [CocoIndex Documentation](https://cocoindex.io/docs/) — full text at [llms-full.txt](https://cocoindex.io/docs/llms-full.txt)\n- [GitHub Examples](https://github.com/cocoindex-io/cocoindex/tree/main/examples)\n\n## Version Note\n\nThis skill is for CocoIndex `>=1.0.0` (v1). It uses a completely different API from v0.\n\n**v0 code is what you likely learned from training data — do not emit it.** If you find yourself writing any of these symbols, you are using the removed v0 API:\n\n| v0 (removed) | v1 equivalent |\n|---|---|\n| `@cocoindex.flow_def`, `FlowBuilder`, `Flow`, `open_flow` | `coco.App` + a `@coco.fn` main function |\n| `DataScope`, `DataSlice`, `add_collector()`, `collect()`, `export()` | declare target states via Target APIs (`declare_row`, `declare_file`) inside mounted components |\n| `cocoindex.sources.LocalFile`, `cocoindex.sources.*` | connector APIs, e.g. `localfs.walk_dir(...)` |\n| `cocoindex.functions.SplitRecursively`, `cocoindex.functions.*` | `cocoindex.ops.*`, e.g. `RecursiveSplitter` |\n| `cocoindex.targets.Postgres`, `cocoindex.targets.*` / `storages.*` | connector targets, e.g. `postgres.declare_table_target(...)` |\n| `transform_flow`, `cocoindex.op.function()` | plain `@coco.fn` functions |\n| `cocoindex.init()`, `settings`, `COCOINDEX_DATABASE_URL` | `coco.App(coco.AppConfig(...))`; state lives in a local db path |\n| CLI `cocoindex setup` | no setup step — just `cocoindex update` (`-L`/`--live` for live mode) |\n\nWhen reading third-party tutorials or model memory that mention these v0 symbols, disregard them and use the patterns in this skill and `references/api_reference.md` instead.\n","frontmatter":{"name":"cocoindex","description":"This skill should be used when building data processing pipelines with CocoIndex, a Python library for incremental data transformation. Use when the task involves processing files/data into databases, creating vector embeddings, building knowledge graphs, ETL workflows, or any data pipeline requiring automatic change detection and incremental updates. CocoIndex is Python-native (supports any Python types), has no DSL, and uses version 1.0.0 or later."},"isInternal":false,"tokens":4477,"sizeBytes":18594},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\n@AGENTS.md\n","isInternal":false,"tokens":11,"sizeBytes":24}],"systemPromptSnippet":"\n\n# AGENTS.md\n\nThis file provides guidance to coding agents working with code in this repository.\nClaude Code compatibility is kept through `CLAUDE.md`, which imports this file.\n\n## Build and Test Commands\n\nThis project uses [uv](https://docs.astral.sh/uv/) for Python project management.\n\n### Building\n\n```bash\nuv run maturin develop # Build Rust code and install Python package (required after Rust changes)\n```\n\n### Testing\n\n```bash\ncargo test # Run Rust tests\nuv run mypy # Type check Python code\nuv run pytest python/ # Run Python tests (use after both Rust and Python changes)\n```\n\n### Code Formatting and Linting\n\nPython linting and formatting go through `prek`, which runs the `ruff-check` and\n`ruff-format` hooks from `.pre-commit-config.yaml` with the exact ruff version\nand file set CI uses. A `ruff` from `uv run`, `uvx`, or Homebrew is usually a\ndifferent version than the hook pins, so run the hooks rather than `ruff` itself.\n\n```bash\nuv tool install prek # one-time; some hooks also need `protoc` (brew install protobuf)\nprek run ruff-check ruff-format --all-files # Lint (applying safe fixes), then format Python code (same as CI)\ncargo fmt # Format Rust code\n./dev/run_cargo_clippy.sh # Lint Rust code (same as CI)\n```\n\n### Pre-submission Checks\n\nCI enforces the full pre-commit hook suite from `.pre-commit-config.yaml`, which\ncovers more than the commands above — notably regenerating the CLI docs\n(`docs/src/content/docs/cli.mdx`) whenever `cli.py` changes. Run it locally\nbefore pushing:\n\n```bash\nprek run --all-files\n```\n\n### Pull Requests and CI\n\nOpen a pull request as a draft and mark it ready once the checks above pass:\ndrafts run no CI. When `main` has a merge queue, a ready pull request runs only\nthe fast checks, and the queue runs the full suite on the commit that lands.\nFor a full run on a branch before then, use `gh workflow run CI.yml --ref `.\nThe CI tiers, the merge queue and stacked pull requests are described in\n[CI.md](https://github.com/cocoindex-io/.github/blob/main/CI.md).\n\n### Workflow Summary\n\n| Change Type | Commands to Run |\n|-------------|-----------------|\n| Rust code only | `uv run maturin develop && cargo test` |\n| Python code only | `uv run mypy && uv run pytest python/` |\n| Both Rust and Python | Run all commands from both categories above |\n| Python lint and format | `prek run ruff-check ruff-format --all-files` |\n| Before pushing | `prek run --all-files` |\n\nAfter changing a core engine API (`rust/core`), run `cargo check --workspace --all-targets` to catch call sites in every workspace crate — both `rust/py` and the Rust SDK under `rust/sdk/` call core directly, and the SDK is easy to miss when grepping. The `examples/rust/*` and `benchmarks/*/rust` crates are standalone workspaces that consume the SDK; the `prek` suite's `cargo check (rust examples + benchmarks)` hook covers those.\n\n## Code Structure\n\n```\ncocoindex/\n├── rust/ # Rust crates (workspace)\n│ ├── core/ # Core engine crate\n│ │ └── src/\n│ │ ├── engine/ # Core engine\n│ │ ├── state/ # States of the core engine\n│ │ └── inspect/ # Database inspection utilities\n│ ├── py/ # Python bindings (PyO3); calls core engine APIs\n│ ├── py_utils/ # Python-Rust utility helpers (error, convert, future)\n│ ├── utils/ # General utilities: error, batching, fingerprint, etc.\n│ ├── code_ast/ # Tree-sitter foundation: language registry (links all grammars), CodeSource, positions\n│ ├── ops_text/ # Text processing operations (splitter, language detection)\n│ ├── code_match/ # Structural code pattern matching over tree-sitter ASTs\n│ └── sdk/\n│ ├── cocoindex/ # Rust SDK (`cocoindex` crate); calls core engine APIs directly, like py/\n│ └── cocoindex_macros/ # Proc macros for the Rust SDK (`#[function]`)\n│\n├── python/\n│ ├── cocoindex/ # Python package\n│ │ ├── __init__.py # Package entry point\n│ │ ├── cli.py # CLI commands\n│ │ ├── _internal/ # Internal implementation for the core engine\n│ │ │ ├── api.py # Public API: mount, use_mount, mount_each, map, mount_target, App, fn, start/stop\n│ │ │ ├── app.py # App base implementation\n│ │ │ ├── context_keys.py # ContextKey and ContextProvider\n│ │ │ ├── environment.py # Environment and lifespan handling\n│ │ │ ├── function.py # @coco.fn decorator implementation\n│ │ │ ├── component_ctx.py # ComponentContext and component_subpath\n│ │ │ ├── target_state.py # Target state implementation\n│ │ │ └── core.pyi # Type stubs for the Rust extension module (update when PyO3 APIs change)\n│ │ ├── connectors/ # External system connectors (localfs, postgres, qdrant, lancedb, google_drive)\n│ │ ├── connectorkits/ # Connector building utilities\n│ │ ├── resources/ # Abstractions: file.py (FileLike), chunk.py (Chunk), schema.py\n│ │ └── ops/ # Operations: text.py (RecursiveSplitter), sentence_transformers.py\n│ └── tests/ # Python tests\n│\n├── examples/ # Example applications\n├── docs/ # Documentation\n└── dev/ # Development utilities\n```\n\n## Key Concepts\n\n### Mental model: declarative data pipelines\n\nCocoIndex uses a **declarative** programming model — you specify *what* your output should look like (target states), not *how* to incrementally update it. The engine handles change detection and applies minimal updates automatically.\n\nThink of it like:\n\n* **React**: declare UI as function of state → React re-renders what changed\n* **Spreadsheets**: declare formulas → cells recompute when inputs change\n* **CocoIndex**: declare target states as function of source → engine syncs what changed\n\n### Core concepts\n\n**App** — The top-level runnable unit. Bundles a main function with its arguments. When you call `app.update()`, the main function runs as the root processing component.\n\n**Processing Component** — The unit of execution that owns a set of target states. Created by `mount()` or `use_mount()` at a specific component path. When a component finishes, its target states sync atomically to external systems.\n\n**Component Path** — Stable identifier for a processing component across runs. Created via `coco.component_subpath(\"process\", filename)`. CocoIndex uses component paths to:\n\n* Match components to their previous runs for change detection\n* Determine ownership of target states (if a path disappears, its target states are cleaned up)\n\n**Target State** — What you want to exist in an external system (a file, a database row, a table). You *declare* target states; CocoIndex keeps them in sync — creating, updating, or removing as needed.\n\n**Target** — The API object used to declare target states (e.g., `DirTarget`, `TableTarget`). Targets can be nested: a container target state (directory/table) provides a Target for declaring child target states (files/rows).\n\n**Function** — A Python function decorated with `@coco.fn`. Use `memo=True` to enable memoization (skip execution when inputs and code are unchanged).\n\n**Context** — React-style provider mechanism for sharing resources. Define keys with `ContextKey[T]`, provide values in lifespan via `builder.provide()`, use in functions via `coco.use_context(key)`.\n\n### Key APIs\n\n```python\n# Mounting processing components (subpath auto-derived from fn.__name__)\nawait coco.mount(fn, *args, **kw) # child runs independently\nresult = await coco.use_mount(fn, *args, **kw) # returns value directly\n\n# Explicit subpath (for multi-part paths or multiple mounts of same function)\nawait coco.mount(coco.component_subpath(\"process\", filename), fn, *args, **kw)\nresult = await coco.use_mount(coco.component_subpath(\"name\"), fn, *args, **kw)\n\n# Component subpath composition\nsubpath = coco.component_subpath(\"process\", filename) # multiple parts\nsubpath = coco.component_subpath(\"a\") / \"b\" / \"c\" # chaining with /\n\n# Using component_subpath as context manager (applies to all nested mount calls)\nwith coco.component_subpath(\"process\"):\n for f in files:\n await coco.mount(coco.component_subpath(str(f.relative_path)), process_file, f, target)\n\n# Declaring target states (typically via Target methods)\ndir_target.declare_file(filename=name, content=data)\ntable_target.declare_row(row=MyRow(...))\n\n# Using context values\ndb = coco.use_context(PG_DB) # retrieve value provided in lifespan\n\n# Explicit context management (for ThreadPoolExecutor)\nctx = coco.get_component_context()\nwith ctx.attach():\n # coco APIs work correctly in this thread\n coco.mount(...)\n```\n\n**Mount handles:**\n\n* `mount()` → `ComponentMountHandle`: call `await handle.ready()` to wait until target states are synced\n* `use_mount()` → returns the result value directly (awaitable)\n\n### How syncing works\n\nWhen a processing component finishes, CocoIndex compares its declared target states with those from the previous run at the same component path:\n\n* New target states → create (insert row, create file)\n* Changed target states → update\n* Missing target states → delete\n\nChanges are applied atomically per component. If a source item is deleted (path no longer mounted), all its target states are cleaned up automatically.\n\n### Example\n\n```python\nimport pathlib\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, target: localfs.DirTarget) -> None:\n html = _markdown_it.render(await file.read_text())\n outname = \"__\".join(file.file_path.path.parts) + \".html\"\n target.declare_file(filename=outname, content=html)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n target = await coco.use_mount(localfs.declare_dir_target, outdir)\n\n files = localfs.walk_dir(\n sourcedir,\n recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n )\n await coco.mount_each(process_file, files.items(), target)\n\n\napp = coco.App(\n coco.AppConfig(name=\"FilesTransform\"),\n app_main,\n sourcedir=pathlib.Path(\"./docs\"),\n outdir=pathlib.Path(\"./out\"),\n)\napp.update_blocking(report_to_stdout=True)\n```\n\n## Code Conventions\n\n### Internal vs External Modules\n\nWe distinguish between **internal modules** (under packages with `_` prefix, e.g. `_internal.*` or `connectors.*._source`) and **external modules** (which users can directly import).\n\n**External modules** (user-facing, e.g. `cocoindex/ops/sentence_transformers.py`):\n\n* Be strict about not leaking implementation details\n* Use `__all__` to explicitly list public exports\n* Prefix ALL non-public symbols with `_`, including:\n * Standard library imports: `import threading as _threading`, `import typing as _typing`\n * Third-party imports: `import numpy as _np`, `from numpy.typing import NDArray as _NDArray`\n * Internal package imports: `from cocoindex.resources import schema as _schema`\n* Exception: `TYPE_CHECKING` imports for type hints don't need prefixing\n\n**Internal modules** (e.g. `cocoindex/_internal/component_ctx.py`, `**/_target.py`):\n\n* Less strict since users shouldn't import these directly\n* Standard library and internal imports don't need underscore prefix\n* Only prefix symbols that are truly private to the module itself (e.g. `_context_var` for a module-private ContextVar)\n\n### Minimize API surface until deemed necessary\n\nWhen adding a new public API (function, class, kwarg, configuration option), prefer the smallest surface that solves the concrete need in front of you. Do not pre-expose tunables, hooks, or alternatives \"in case someone needs them.\" Adding a kwarg later is straightforward and backwards-compatible; removing or renaming one is disruptive. If a knob is currently a hardcoded constant inside the implementation, leave it there until a real use case demands it — at which point promoting it to a kwarg is mechanical. This applies equally to optional parameters, callback hooks, parser/transformer plug-points, and configuration keys.\n\n### General principles (also covered by `/review-changes`)\n\n- **Top-level imports.** Defer to in-function only for a real circular dependency or a heavy import that isn't always needed.\n- **Specific types over `Any`.** Use concrete types — including concrete types from third-party libraries. `Any` only when the type is truly generic and no downstream code needs to downcast it.\n- **Validate and exchange early.** When a value enters as a weaker form (`str`, `Any`, raw identifier), convert to the strong type at the earliest point. Don't propagate the weak form. Concrete example in this codebase: connector handlers receive a `ContextKey` string as part of a target-state key. The parent handler resolves the key once (when constructing the child handler) and passes the typed connection directly — the child stores `_pool: asyncpg.Pool`, not `_db_key: str`.\n- **`NamedTuple`/small dataclass for multi-value returns.** Access fields by name (`result.can_reuse`) at call sites.\n- **Exceptions for exceptional situations only.** Reserve exceptions (Python) and errors (Rust) for truly unexpected failures — not for end-of-iteration, \"not found\", or other expected state transitions. Use explicit return values (sentinel, enum variant, `None`) for those.\n- **Single source of truth, delete dead code, honest names.** When the same value/logic appears in multiple places, consolidate. When changes make code unreachable, delete it (and its tests, and any dead config knobs). Function and field names should describe what the implementation does today.\n\n- Rust code follows [dev/rust-guidelines.md](dev/rust-guidelines.md).\n\n### Testing Guidelines\n\nWe prefer end-to-end tests on user-facing APIs, over unit tests on smaller internal functions. With this said, there're cases where unit tests are necessary, e.g. for internal logic with various situations and edge cases, in which case it's usually easier to cover various scenarios with unit tests.\n\n#### Test Environment Setup\n\nUse `common.create_test_env(__file__)` to create a CocoIndex `Environment` for tests. It derives a unique `db_path` from the test file path and picks up the current event loop automatically.\n\n* **Sync tests** (module-level creation): Call at module level — the Environment creates a background loop for async callbacks.\n\n ```python\n coco_env = common.create_test_env(__file__)\n ```\n\n* **Async tests with async resources** (e.g., asyncpg pools): Call inside an async fixture so the Environment binds to the test's running event loop (same loop the pool is on). Use the `suffix` parameter when each test needs its own Environment:\n\n ```python\n @pytest_asyncio.fixture\n async def pg_env(request: pytest.FixtureRequest) -> Any:\n pool = await asyncpg.create_pool(dsn)\n coco_env = common.create_test_env(__file__, suffix=request.node.name)\n coco_env.context_provider.provide(DB_KEY, pool)\n yield pool, coco_env\n await pool.close()\n ```\n\n The `suffix` ensures each test gets a unique `db_path`, avoiding \"environment already open\" errors.\n\n#### Testcontainers for Database Tests\n\nUse `testcontainers[postgres]` (in the `build-test` dependency group) to spin up real database instances automatically — no manual setup or environment variables needed. Use a module-scoped sync fixture for the container and a function-scoped async fixture for per-test resources:\n\n```python\n_PG_IMAGE = \"postgres:16-alpine\"\n\npytestmark = [pytest.mark.requires_docker(_PG_IMAGE)]\n\n@pytest.fixture(scope=\"module\")\ndef pg_dsn() -> Any:\n with PostgresContainer(_PG_IMAGE) as pg:\n dsn = pg.get_connection_url().replace(\"postgresql+psycopg2://\", \"postgresql://\")\n yield dsn\n\n@pytest_asyncio.fixture\nasync def pool(pg_dsn: str) -> Any:\n p = await asyncpg.create_pool(pg_dsn)\n yield p\n await p.close()\n```\n\nList every image a test's containers use in its `requires_docker(...)` marker. `python/tests/conftest.py` pulls missing images after collection, before the first test runs; otherwise testcontainers pulls inside the fixture, and a cold-cache download counts against the first test's `pytest-timeout`.\n\n### LMDB write paths\n\n- All LMDB writes must go through `Storage::run_txn` (uses the single-writer batcher).\n- Do not open a heed write txn directly or wrap the env in a separate mutex/semaphore — bypassing the batcher loses fsync coalescing and regresses concurrent-submit throughput by 10-100×.\n- LMDB has no savepoints. If a sub-operation needs to \"abort\" without failing its caller, handle it at the body level (e.g. return a sentinel result without writing). A body that returns `Err` fails only its own caller: the batcher drops its writes by aborting the txn and re-running the bodies ahead of it, so every body must be safe to re-run.\n- An LMDB write txn must begin and end (commit or abort) on the same OS thread: the writer lock is thread-owned and LMDB ignores a failed release, so a txn that migrates between runtime workers blocks every later writer for good. heed marks `RwTxn` as `Send` regardless, so the compiler won't catch it. `Storage::run_txn` runs each batch on one blocking thread for this reason; never hold a write txn across an `.await` on the multi-thread runtime.\n\n### Sync vs Async\n\nThe Rust core (`rust/core`, `rust/utils`) uses **async-first** design with Tokio. The `rust/py` crate bridges Rust async to Python, offering both sync and async APIs:\n\n* Rust core exposes async functions\n* `rust/py` provides sync wrappers that use `block_on()` to call async Rust from sync Python\n* Python's `cocoindex` API is **async-first**: mount APIs (`mount`, `use_mount`, `mount_each`, `map`) are all `async def`; `App.update()`/`App.drop()` are async; sync entry points (`App.update_blocking()`, `App.drop_blocking()`, `start_blocking()`, `stop_blocking()`) are available for CLI and blocking contexts\n\nWhen adding new functionality that involves I/O or concurrency:\n\n* Implement async in Rust\n* Bridge to Python via `rust/py`, providing both sync and async variants if needed\n\n## Versioning\n\nThe current codebase is for CocoIndex v1, which is a fundamental redesign from CocoIndex v0. CocoIndex v1 lives on the `main` branch; the legacy v0 code is preserved on the `v0` branch.\n\n## Docs diagrams\n\nAll diagrams embedded in docs pages are built as inline Astro components under `docs/src/components/diagrams/`, not exported SVG files. See [docs/src/components/diagrams/README.md](docs/src/components/diagrams/README.md) for the palette, primitives, shared CSS classes, animation conventions, and step-by-step authoring guidelines. Follow that guide for any new or updated diagram.\n\n## Agent Tooling\n\n### Semantic code search over this repo (`cocoindex-code`)\n\nTo find where a concept lives, prefer semantic search over blind `grep`. This repo keeps a local\n`cocoindex-code` index; query it from any agent's shell with the `ccc` CLI:\n\n```bash\nccc search \"where is X handled\" # semantic search; add --refresh to reindex first\n```\n\nIf `ccc search` reports no index, set it up once:\n\n```bash\ncommand -v ccc >/dev/null || uv tool install 'cocoindex-code[full]' # per machine; local embeddings, no API key\nccc init -f && ccc index # repo-scoped index (first build ~1-2 min)\n```\n\nEverything is local — SQLite index, local embeddings, no keys — and a background daemon keeps it\nfresh. On a brand-new machine `ccc init` asks once for an embedding provider; choose the local\ndefault. The `-f` matters: if a parent directory is also a `ccc` project, plain `ccc init` refuses\nand search silently uses the parent's index, so run `ccc status` to confirm the root is this repo.\nWe register no MCP server on purpose — an `ccc mcp` mode exists, but the CLI costs no context until\ninvoked and behaves the same across agents. Source:\n[github.com/cocoindex-io/cocoindex-code](https://github.com/cocoindex-io/cocoindex-code).\n\n## Agent Playbooks\n\nReusable task-specific agent guidance lives under `dev/agent-skills/`. These playbooks use a `SKILL.md` format so Claude Code can load them directly from `.claude/skills/`, but the content is intended to be readable by any coding agent.\n\n- For docs diagrams, read `dev/agent-skills/cocoindex-diagrams/SKILL.md`.\n- For new target connectors, read `dev/agent-skills/target-connector/SKILL.md`.\n- For upgrading example package versions, read `dev/agent-skills/upgrade-examples/SKILL.md`.\n\nAgent Skills-compatible clients, including Codex, discover the checked-in symlinks under `.agents/skills/`. The public CocoIndex skill is exposed there as `cocoindex`, and contributor playbooks are exposed there from `dev/agent-skills/`.\n\nClaude Code compatibility remains under `.claude/`: hooks/settings live there, and `.claude/skills/` mirrors the contributor playbooks with symlinks. Do not duplicate skill content under `.agents/skills/` or `.claude/skills/`; edit the source directories (`skills/cocoindex/` for the public skill, `dev/agent-skills/` for contributor playbooks) and keep dot-directory entries as symlinks.\n\nPortable agent checks live under `dev/agent-checks/`. Claude Code hooks in `.claude/hooks/` are thin adapters that call these scripts when relevant files changed.\n\n\n# AGENTS.md — CocoIndex examples\n\nGuidance for AI coding agents (Claude Code, Cursor, etc.) working in this `examples/`\ndirectory. Most top-level Python subfolders are self-contained, runnable\nCocoIndex **v1** apps; `rust/` contains Rust ports with per-example READMEs.\n\n## Before you write CocoIndex code: install the skill\n\nCocoIndex v1 is a fundamental redesign from v0. Without context, LLMs tend to\nhallucinate the v0 flow-builder DSL and deprecated decorators. Install the\nofficial skill first — it teaches the correct v1 API. Quickest portable path\nfor Codex and other Agent Skills-compatible clients is the hosted skill:\n\n```sh\nmkdir -p .agents/skills/cocoindex/references\ncurl -fsSL https://cocoindex.io/docs/skill.md -o .agents/skills/cocoindex/SKILL.md\nfor f in api_reference connectors patterns setup_database setup_project; do\n curl -fsSL https://cocoindex.io/docs/references/$f.md -o .agents/skills/cocoindex/references/$f.md\ndone\n```\n\nOr clone the repo and copy the folder:\n\n```sh\ngit clone --depth=1 https://github.com/cocoindex-io/cocoindex.git /tmp/cocoindex-skill\nmkdir -p .agents/skills && cp -r /tmp/cocoindex-skill/skills/cocoindex .agents/skills/\n```\n\nFor Claude Code's native project path, use `.claude/skills/cocoindex` instead\nof `.agents/skills/cocoindex`; the skill format is the same.\n\nThe skill itself lives at [`skills/cocoindex/`](../skills/cocoindex) (SKILL.md +\n`references/`). For Cursor, copy `SKILL.md` into `.cursor/rules/`. Full machine-readable\ndocs: (index) and\n (everything, including example\nwalkthroughs). Regular docs pages also have raw Markdown twins by replacing the\ntrailing slash with `.md` (e.g.\n`https://cocoindex.io/docs/programming_guide/core_concepts.md`).\n\n## The v1 mental model\n\n`target_state = transform(source_state)`. You declare what the target should look\nlike; the Rust engine keeps it in sync, reprocessing only what changed (state is\ntracked in a local LMDB store — **no database is required for the engine itself**,\nonly when an example writes to one). Key APIs: `@coco.fn`, `mount` / `use_mount` /\n`mount_each`, `ContextKey`, target-state declarations. See the skill for details.\n\n## Running examples\n\nMost Python examples are standalone projects with their own `pyproject.toml`:\n\n```sh\ncd \ncp .env.example .env # if present — fill in the blanks (see below)\npip install -e . # or: uv pip install -e .\ncocoindex update main # catch-up: scan sources, sync, exit\ncocoindex update -L main # live mode: catch up, then watch for changes (where supported)\n```\n\nUse the example's README as the source of truth. Known exceptions:\n\n- `multi_codebase_summarization`, `audio_to_text`,\n `patient_intake_extraction_baml`, `patient_intake_extraction_dspy`:\n `cocoindex update main.py`\n- `conversation_to_knowledge`: `cocoindex update conv_knowledge.app`\n- `image_search`, `image_search_colpali`: start the API with\n `python -m uvicorn api:app --reload --host 0.0.0.0 --port 8000`; it runs the\n CocoIndex app in live mode, then start the frontend from `frontend/`.\n- `csv_to_kafka`, `kafka_to_lancedb`: `cocoindex update -L main.py`\n- `rust/`: follow that example's README. Many use\n `cargo run -- index` for indexing and `cargo run -- query \"...\"` for search;\n a few take custom paths or service-specific subcommands.\n\nSome examples expose a query/CLI demo via `python main.py \"\"`; check the\nexample's `README.md`. Examples that need extra services or a code-gen step\n(e.g. `baml generate`) say so in their README.\n\n## Environment / credentials\n\nWhen an example needs credentials or service configuration, required env vars\nare templated in that example's **`.env.example`** — `cp` it to `.env` and fill\nin the blanks; both `python main.py` and the `cocoindex` CLI load `.env`\nautomatically. Common ones:\n\n- `POSTGRES_URL` — for Postgres/pgvector targets. Local instance:\n `docker compose -f ../../dev/postgres.yaml up -d` from inside an example\n directory.\n- `OPENAI_API_KEY` / `GEMINI_API_KEY` — for examples that call an LLM.\n- Service-specific (`QDRANT_URL`, `LANCEDB_URI`, `NEO4J_*`, `KAFKA_*`,\n `GOOGLE_SERVICE_ACCOUNT_CREDENTIAL`, …) — see that example's `.env.example`.\n\nExamples with no `.env.example` (e.g. `files_transform`, `pdf_to_markdown`) run\nfully locally with no credentials.\n\n**Never commit secrets.** The `.env` files tracked in this repo hold only\nnon-secret defaults (`COCOINDEX_DB`, local service URLs); keep API keys and\ncredentials in your local `.env` edits and out of commits.\n\n## The examples\n\nA walkthrough URL means there's a step-by-step guide at\n`https://cocoindex.io/docs/examples//`; otherwise start from the example's README.\n\n### Vector indexes (embed → store → search by meaning)\n- `text_embedding` — Markdown → pgvector; the simplest end-to-end index. *(walkthrough: text-embedding)*\n- `code_embedding` — repo → Tree-sitter chunks → pgvector; query code in English. *(walkthrough: index-codebase)*\n- `text_embedding_qdrant` / `text_embedding_lancedb` / `text_embedding_turbopuffer` — same flow, different vector store.\n- `code_embedding_lancedb` — code chunks → LanceDB.\n- `pdf_embedding` — PDFs → markdown → chunks → pgvector.\n- `paper_metadata` — extract title/authors/abstract from PDFs → Postgres + embeddings.\n- `amazon_s3_embedding` / `gdrive_text_embedding` / `oci_object_storage_embedding` — same flow, remote source (S3 / Google Drive / OCI).\n- `postgres_source` — read from an existing Postgres table as the source.\n- `entire_session_search` — semantic search over AI coding sessions captured by Entire.\n- `sec_edgar_analytics` — multi-format SEC filings → Apache Doris with a vector **and** a full-text index; hybrid (semantic + keyword) RRF search. *(walkthrough: sec-edgar-analytics)*\n\n### Multimodal\n- `image_search` — CLIP embeddings + Qdrant, queried via FastAPI + React.\n- `image_search_colpali` — ColPali multi-vector model + Qdrant MaxSim.\n- `multi_format_indexing` — PDFs + images as page screenshots → ColPali → Qdrant; no OCR, no chunking. *(walkthrough: multi-format-indexing)*\n- `face_recognition` — detect faces (dlib) → 128-d embeddings → Qdrant face search. *(walkthrough: face-recognition)*\n- `audio_to_text` — transcribe audio with LiteLLM → Postgres.\n- `slides_to_speech` — slides → vision-LLM notes → Pocket TTS narration → LanceDB semantic search. *(walkthrough: slides-to-speech)*\n\n### Structured extraction (LLM / BAML / DSPy)\n- `multi_codebase_summarization` — LLM per-file summaries across many repos. *(walkthrough: multi-codebase-summarization)*\n- `hn_trending_topics` — scrape HackerNews → LLM topic extraction → Postgres.\n- `manuals_llm_extraction` — PDF manuals → Markdown (docling) → typed module records → Postgres. *(walkthrough: manuals-llm-extraction)*\n- `patient_intake_extraction_baml` / `patient_intake_extraction_dspy` — structured PDF extraction with BAML / DSPy (Gemini vision).\n\n### Knowledge graphs\n- `conversation_to_knowledge` — YouTube podcasts → SurrealDB knowledge graph. *(walkthrough: podcast-to-knowledge-graph)*\n- `docs_to_knowledge_graph` — Markdown docs → Neo4j concept graph of LLM-extracted triples. *(walkthrough: docs-to-knowledge-graph)*\n- `product_recommendation` — product catalog → LLM taxonomy extraction → Neo4j recommendation graph. *(walkthrough: product-recommendation)*\n- `meeting_notes_graph_neo4j` / `meeting_notes_graph_falkordb` — Google Drive meeting notes → Neo4j / FalkorDB graph.\n- `meeting_notes_graph_surrealdb` — local Markdown meeting notes → SurrealDB graph.\n\n### Custom sources / targets / streaming\n- `pdf_to_markdown` — incremental PDF → Markdown with docling (local, no services). *(walkthrough: pdf-to-markdown)*\n- `files_transform` — watch Markdown files → HTML, live mode (local, no services).\n- `csv_to_kafka` — watch CSVs → publish rows to Kafka.\n- `kafka_to_lancedb` — consume Kafka → route to LanceDB tables.\n\n### Rust\n- `rust/` — Rust ports of many of the above, using the CocoIndex Rust API.\n\n## Conventions for edits\n\n- Keep each Python example self-contained: its own `pyproject.toml` and\n `README.md`; add `.env.example` when credentials or configurable services are\n required. When you add an example, also add a line to `EXAMPLE_CATALOG` in the\n docs repo (`docs/src/data/examples.ts`) so it appears in `/docs/llms.txt`.\n- Match the surrounding code's low comment density.\n- Don't commit generated artifacts (`cocoindex.db`, `__pycache__`, build output) —\n they're already git-ignored.\n\n\n# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/layout-patterns.md)\n\n\n# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/pitfalls.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/pitfalls.md)\n\n\n# References Skill\n\nPath: `dev/agent-skills/cocoindex-diagrams/references/workflow.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/cocoindex-io/cocoindex/HEAD/dev/agent-skills/cocoindex-diagrams/references/workflow.md)\n\n\n# CocoIndex API Reference\n\nQuick reference for the most commonly used CocoIndex APIs.\n\n## Import Convention\n\n```python\nimport cocoindex as coco\n```\n\nAll APIs live directly on the `coco` module. There is no separate `cocoindex.asyncio` module.\n\n---\n\n## `@coco.fn` Decorator\n\nMark a function as a CocoIndex processing function.\n\n```python\n@coco.fn\nasync def my_function(arg1: str) -> None: ...\n\n@coco.fn(memo=True, version=1)\nasync def expensive_fn(data: str) -> Result: ...\n\n# Force async interface for a sync function (useful for batching)\n@coco.fn.as_async(memo=True, batching=True)\ndef batch_embed(texts: list[str]) -> list[NDArray]: ...\n```\n\n**Parameters:**\n- `memo: bool = False` -- Enable memoization (skip if inputs/code unchanged)\n- `version: int | None = None` -- Explicit version bump to force re-execution\n- `logic_tracking: \"full\" | \"self\" | None = \"full\"` -- How code changes are tracked\n- `batching: bool = False` -- Auto-batch concurrent calls (async only)\n- `max_batch_size: int | None = None` -- Max batch size\n- `runner: Runner | None = None` -- e.g. `coco.GPU` for serialized GPU execution\n\n---\n\n## Mount APIs (all async)\n\nAll mount APIs accept an optional `ComponentSubpath` as their first argument. When omitted, the subpath is auto-derived from `Symbol(fn.__name__)`. Provide an explicit subpath when mounting the same function multiple times, using multi-part paths, or needing a specific path name.\n\n### `coco.mount()`\n\nMount a processing component in the background.\n\n```python\n# Subpath auto-derived from fn.__name__\nhandle = await coco.mount(processor_fn, *args, **kwargs)\n\n# Explicit subpath\nhandle = await coco.mount(coco.component_subpath(\"name\"), processor_fn, *args, **kwargs)\n\nawait handle.ready() # Optional: wait until component finishes\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `processor_fn` -- Function (or LiveComponent class) to run.\n- `*args, **kwargs` -- Arguments passed to the function.\n\n**Returns:** `ComponentMountHandle`\n\n### `coco.use_mount()`\n\nMount a dependent component and return its result. Parent depends on the child.\n\n```python\n# Subpath auto-derived from fn.__name__\nresult = await coco.use_mount(init_fn, *args, **kwargs)\n\n# Explicit subpath\nresult = await coco.use_mount(coco.component_subpath(\"setup\"), init_fn, *args, **kwargs)\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `processor_fn` -- Function to run.\n- `*args, **kwargs` -- Arguments passed to the function.\n\n**Returns:** The return value of `processor_fn`.\n\n### `coco.mount_each()`\n\nMount one component per item in a keyed iterable. Preferred for processing lists.\n\n```python\n# Subpath auto-derived from fn.__name__\nawait coco.mount_each(process_file, files.items(), *extra_args)\n\n# Explicit subpath\nawait coco.mount_each(coco.component_subpath(\"process\"), process_file, files.items(), table)\n```\n\n**Parameters:**\n- `subpath` (optional) -- Component subpath. Auto-derived from `fn.__name__` when omitted.\n- `fn` -- Function to run per item. Item value is passed as first argument.\n- `items` -- Keyed iterable of `(StableKey, T)` pairs, or a `LiveMapFeed` for live mode.\n- `*args, **kwargs` -- Additional arguments passed to `fn` after the item.\n\n**Returns:** `ComponentMountHandle`\n\n### `coco.mount_target()`\n\nMount a target state, ensuring the container is applied before returning the child provider.\n\n```python\nprovider = await coco.mount_target(target_state)\n```\n\nPrefer connector convenience methods (`postgres.mount_table_target()`, etc.) which call this internally.\n\n### `coco.map()`\n\nRun a function concurrently on each item. No processing components are created -- pure concurrent execution within the current component.\n\n```python\nresults = await coco.map(process_chunk, chunks, *extra_args)\n```\n\n**Parameters:**\n- `fn` -- Async function to apply. Item is passed as first argument.\n- `items` -- Iterable or async iterable.\n- `*args, **kwargs` -- Additional arguments passed to `fn` after the item.\n\n**Returns:** `list[T]`\n\n### `coco.component_subpath()`\n\nCreate a stable component path for mounting.\n\n```python\ncoco.component_subpath(\"setup\")\ncoco.component_subpath(\"file\", str(file_path))\ncoco.component_subpath(\"record\", record.id)\n\n# Chaining with /\nsubpath = coco.component_subpath(\"a\") / \"b\" / \"c\"\n\n# Context manager form (applies to all nested mount calls)\nwith coco.component_subpath(\"process\"):\n for f in files:\n await coco.mount(coco.component_subpath(str(f.path)), process_file, f)\n```\n\n**StableKey types:** `str | int | bool | bytes | uuid.UUID | Symbol | tuple[StableKey, ...]`\n\n---\n\n## Context System\n\n### `coco.ContextKey[T]`\n\nType-safe key for sharing resources. The `key` string is the stable identity across runs.\n\n```python\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n```\n\n- `detect_change=True` -- Opt in to auto-invalidate dependent memos when value changes (models, configs)\n- `detect_change=False` (default) -- For resources not affecting computation (DB connections, loggers)\n\n### `builder.provide()`\n\nRegister a resource in context (used in lifespan).\n\n```python\nbuilder.provide(PG_DB, pool)\nbuilder.provide_with(KEY, context_manager) # Sync CM\nawait builder.provide_async_with(KEY, async_cm) # Async CM\n```\n\n### `coco.use_context()`\n\nRetrieve a resource from context inside a processing function.\n\n```python\npool = coco.use_context(PG_DB)\nembedder = coco.use_context(EMBEDDER)\n```\n\n---\n\n## Lifespan\n\n### `@coco.lifespan`\n\nDefine environment setup/teardown. Registered to the default environment.\n\n```python\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(MODEL))\n yield\n```\n\nCan also be sync:\n\n```python\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n builder.settings.db_path = pathlib.Path(\"./my.db\")\n yield\n```\n\n---\n\n## App\n\n### `coco.App`\n\n```python\napp = coco.App(\n coco.AppConfig(name=\"MyApp\"),\n main_fn,\n **params,\n)\n\n# Async\nawait app.update()\nhandle = app.update(live=True) # Live mode\nawait app.drop()\n\n# Sync (blocking)\napp.update_blocking(report_to_stdout=True)\napp.drop_blocking()\n```\n\n### `coco.AppConfig`\n\n```python\ncoco.AppConfig(\n name=\"MyApp\", # Required\n environment=env, # Optional: custom Environment\n max_inflight_components=1024, # Optional: concurrency limit\n)\n```\n\n### Start/Stop (for programmatic usage)\n\n```python\n# Context manager (preferred)\nasync with coco.runtime():\n await app.update()\n\n# Or manually\nawait coco.start()\ntry:\n await app.update()\nfinally:\n await coco.stop()\n\n# Sync variants\nwith coco.runtime():\n app.update_blocking()\n```\n\n---\n\n## Exception Handlers\n\n### Global (in lifespan)\n\n```python\nbuilder.set_exception_handler(my_handler)\n```\n\n### Scoped (in processing functions)\n\n```python\nasync with coco.exception_handler(my_handler):\n await coco.mount_each(process_file, files.items(), table)\n```\n\n### Handler Signature\n\n```python\nasync def my_handler(exc: BaseException, ctx: coco.ExceptionContext) -> None:\n logger.error(f\"Error in {ctx.stable_path}: {exc}\")\n```\n\n`ExceptionContext` provides: `env_name`, `stable_path`, `processor_name`, `mount_kind`, `parent_stable_path`, `is_background`, `source`, `original_exception`.\n\n---\n\n## Live Mode\n\nRun the pipeline continuously, streaming changes from sources:\n\n```python\n# CLI\ncocoindex update main.py -L\n\n# Programmatic\nhandle = app.update(live=True)\nasync for snapshot in handle.watch():\n print(snapshot.stats)\n```\n\nSources that support live mode:\n- `localfs.walk_dir(..., live=True)` -- File watching\n- `kafka.topic_as_map(consumer, topics)` -- Kafka topic consumption\n\n---\n\n## CLI Commands\n\n```bash\ncocoindex init [PROJECT_NAME] # Create new project\ncocoindex update APP_TARGET # Run app once\ncocoindex update APP_TARGET -L # Run in live mode\ncocoindex update APP_TARGET --full-reprocess # Force reprocess all\ncocoindex update APP_TARGET --reset # Reset state before running\ncocoindex drop APP_TARGET [-f] # Drop app and all state\ncocoindex ls [APP_TARGET] [--db PATH] # List apps\ncocoindex show APP_TARGET [--tree] # Show component paths\n```\n\n**APP_TARGET format:** `main.py`, `main.py:app_name`, `my_module:app_name`\n\n---\n\n## Text Operations\n\n**Import:** `from cocoindex.ops.text import ...`\n\n### `detect_code_language()`\n\n```python\nlanguage = detect_code_language(filename=\"example.py\") # -> \"python\"\n```\n\n### `RecursiveSplitter`\n\n```python\nsplitter = RecursiveSplitter()\nchunks = splitter.split(\n text,\n chunk_size=1000,\n chunk_overlap=200,\n min_chunk_size=300,\n language=\"python\", # Syntax-aware splitting\n)\n# Each chunk: Chunk(text=..., start=TextPosition(...), end=TextPosition(...))\n```\n\n### `SeparatorSplitter`\n\n```python\nsplitter = SeparatorSplitter(separators_regex=[r\"\\n\\n\", r\"\\n\"])\nchunks = splitter.split(text)\n```\n\n---\n\n## Embedding Operations\n\n**Import:** `from cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder`\n\n```python\nembedder = SentenceTransformerEmbedder(\"sentence-transformers/all-MiniLM-L6-v2\")\nembedding = await embedder.embed(text) # -> NDArray (float32)\n```\n\n**As VectorSchemaProvider:**\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions\n```\n\n---\n\n## File Resources\n\n**Import:** `from cocoindex.resources.file import ...`\n\n### `FileLike` (async file object)\n\n```python\ntext = await file.read_text()\ndata = await file.read() # bytes, lazy/cached\nfp = await file.content_fingerprint()\n```\n\n### `PatternFilePathMatcher`\n\n```python\nmatcher = PatternFilePathMatcher(\n included_patterns=[\"**/*.py\", \"**/*.md\"],\n excluded_patterns=[\".*/**\", \"__pycache__/**\"],\n)\n```\n\n---\n\n## ID Generation\n\n**Import:** `from cocoindex.resources.id import ...`\n\n```python\n# Deterministic: same dep -> same ID\nchunk_id = await generate_id(chunk.text)\nchunk_uuid = generate_uuid(chunk.text)\n\n# Distinct per call (even with same dep)\nid_gen = IdGenerator()\nchunk_id = await id_gen.next_id(chunk.text)\n\nuuid_gen = UuidGenerator()\nchunk_uuid = uuid_gen.next_uuid(chunk.text)\n```\n\n---\n\n## Vector Schema\n\n**Import:** `from cocoindex.resources.schema import VectorSchema`\n\n```python\n# Via ContextKey (preferred -- auto-infer from embedder)\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER]\n\n# Explicit dimensions\nschema = VectorSchema(dtype=np.dtype(np.float32), size=384)\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, schema]\n```\n\n---\n\n## See Also\n\n- [Connectors Reference](./connectors.md) -- Database and system connectors\n- [Patterns Reference](./patterns.md) -- Common pipeline patterns\n- [Setup Project](./setup_project.md) -- Project setup guide\n- [Setup Database](./setup_database.md) -- Database setup guide\n\n\n# CocoIndex Connectors Reference\n\nComprehensive reference for all CocoIndex connectors.\n\n## Common Patterns\n\n### Target Setup\n\nAll target connectors follow this pattern:\n\n1. **Provide** connection via `ContextKey` in lifespan\n2. **Mount** target using `mount_*_target(ContextKey, ...)` convenience method\n3. **Declare** rows/points/files on the returned target object\n\nTarget connectors take a `ContextKey` (not the connection object directly) as their first argument. This enables CocoIndex to track the identity of the target across runs.\n\n### Vector Support\n\nMost connectors support vector embeddings via `VectorSchemaProvider`:\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions from ContextKey\n```\n\n---\n\n## PostgreSQL\n\n**Import**: `from cocoindex.connectors import postgres`\n\n**Capabilities**: Source + Target | **Vector**: pgvector\n\n### Connection Setup\n\n```python\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n yield\n```\n\n### As Target\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Annotated\nfrom numpy.typing import NDArray\n\n@dataclass\nclass Embedding:\n id: int\n text: str\n vector: Annotated[NDArray, EMBEDDER]\n\n# Mount table target\ntarget_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"embeddings\",\n table_schema=await postgres.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n pg_schema_name=\"my_schema\", # Optional, default \"public\"\n)\n\n# Optional: declare vector index\ntarget_table.declare_vector_index(column=\"vector\")\n\n# Declare rows\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### As Source\n\n```python\n@dataclass\nclass Record:\n id: int\n name: str\n\nsource = postgres.PgTableSource(\n coco.use_context(PG_DB),\n table_name=\"my_table\",\n row_type=Record,\n)\n\n# Iterate (async)\nfetcher = source.fetch_rows()\nasync for record in fetcher:\n ...\n\n# Keyed iteration for mount_each\nawait coco.mount_each(process_record, source.fetch_rows().items(key=lambda r: r.id), table)\n```\n\n### Type Mapping\n\n| Python | PostgreSQL |\n|--------|-----------|\n| `bool` | `boolean` |\n| `int` | `bigint` |\n| `float` | `double precision` |\n| `str` | `text` |\n| `bytes` | `bytea` |\n| `UUID` | `uuid` |\n| `datetime` | `timestamp with time zone` |\n| `list`, `dict` | `jsonb` |\n| `NDArray` + vector schema | `vector(n)` or `halfvec(n)` |\n\n### SQL Command Attachments\n\n```python\ntarget_table.declare_sql_command_attachment(\n name=\"my_index\",\n setup_sql=\"CREATE INDEX ...\",\n teardown_sql=\"DROP INDEX ...\", # Optional\n)\n```\n\n---\n\n## SQLite\n\n**Import**: `from cocoindex.connectors import sqlite`\n\n**Capabilities**: Target only | **Vector**: sqlite-vec\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import sqlite\n\nSQLITE_DB = coco.ContextKey[sqlite.ManagedConnection](\"sqlite_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n conn = sqlite.connect(\"./data.db\", load_vec=\"auto\")\n builder.provide(SQLITE_DB, conn)\n yield\n conn.close()\n```\n\n### As Target\n\n```python\ntarget_table = await sqlite.mount_table_target(\n SQLITE_DB,\n table_name=\"embeddings\",\n table_schema=await sqlite.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n)\n\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### Type Mapping\n\n| Python | SQLite |\n|--------|--------|\n| `bool` | `INTEGER` (0/1) |\n| `int` | `INTEGER` |\n| `float` | `REAL` |\n| `str` | `TEXT` |\n| `bytes` | `BLOB` |\n| `datetime` | `TEXT` (ISO format) |\n| `list`, `dict` | `TEXT` (JSON) |\n| `NDArray` + vector schema | `BLOB` (sqlite-vec) |\n\n---\n\n## LanceDB\n\n**Import**: `from cocoindex.connectors import lancedb`\n\n**Capabilities**: Target only | **Vector**: Native | **Storage**: Local or cloud (S3, GCS)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import lancedb\n\nLANCE_DB = coco.ContextKey[lancedb.LanceAsyncConnection](\"lance_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n conn = await lancedb.connect_async(\"./lancedb_data\") # or \"s3://bucket/path\"\n builder.provide(LANCE_DB, conn)\n yield\n```\n\n### As Target\n\n```python\ntarget_table = await lancedb.mount_table_target(\n LANCE_DB,\n table_name=\"embeddings\",\n table_schema=await lancedb.TableSchema.from_class(Embedding, primary_key=[\"id\"]),\n)\n\ntarget_table.declare_row(row=Embedding(id=1, text=\"hello\", vector=vec))\n```\n\n### Type Mapping\n\n| Python | PyArrow |\n|--------|---------|\n| `bool` | `bool` |\n| `int` | `int64` |\n| `float` | `float64` |\n| `str` | `string` |\n| `bytes` | `binary` |\n| `list`, `dict` | `string` (JSON) |\n| `NDArray` + vector schema | `fixed_size_list` |\n\n---\n\n## Qdrant\n\n**Import**: `from cocoindex.connectors import qdrant`\n\n**Capabilities**: Target only | **Vector**: Native | **Model**: Point-oriented\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import qdrant\n\nfrom qdrant_client import QdrantClient\nQDRANT_DB = coco.ContextKey[QdrantClient](\"qdrant_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n client = qdrant.create_client(\"http://localhost:6333\")\n builder.provide(QDRANT_DB, client)\n yield\n```\n\n### As Target (Single Vector)\n\n```python\ncollection = await qdrant.mount_collection_target(\n QDRANT_DB,\n collection_name=\"embeddings\",\n schema=await qdrant.CollectionSchema.create(\n vectors=qdrant.QdrantVectorDef(schema=EMBEDDER, distance=\"cosine\"),\n ),\n)\n\ncollection.declare_point(point=qdrant.PointStruct(\n id=\"point-1\",\n vector=embedding_array.tolist(),\n payload={\"text\": \"hello\"},\n))\n```\n\n### As Target (Named Vectors)\n\n```python\nschema = await qdrant.CollectionSchema.create(\n vectors={\n \"text\": qdrant.QdrantVectorDef(schema=text_embedder_key, distance=\"cosine\"),\n \"image\": qdrant.QdrantVectorDef(schema=image_embedder_key, distance=\"cosine\"),\n },\n)\ncollection = await qdrant.mount_collection_target(QDRANT_DB, \"multimodal\", schema=schema)\n\ncollection.declare_point(point=qdrant.PointStruct(\n id=\"point-1\",\n vector={\"text\": text_vec.tolist(), \"image\": image_vec.tolist()},\n payload={\"title\": \"example\"},\n))\n```\n\n### Distance Metrics\n\n- `\"cosine\"` -- Cosine similarity (default)\n- `\"dot\"` -- Dot product\n- `\"euclid\"` -- Euclidean distance (L2)\n\n---\n\n## SurrealDB\n\n**Import**: `from cocoindex.connectors import surrealdb`\n\n**Capabilities**: Target only | **Vector**: Native | **Model**: Graph + document\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import surrealdb\n\nSURREAL_DB = coco.ContextKey[surrealdb.ConnectionFactory](\"surreal_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n factory = surrealdb.ConnectionFactory(\n url=\"ws://localhost:8000/rpc\",\n namespace=\"my_ns\",\n database=\"my_db\",\n credentials={\"username\": \"root\", \"password\": \"root\"},\n )\n builder.provide(SURREAL_DB, factory)\n yield\n```\n\n### As Target (Table)\n\n```python\ntarget_table = await surrealdb.mount_table_target(\n SURREAL_DB,\n table_name=\"records\",\n table_schema=await surrealdb.TableSchema.from_class(MyRecord),\n)\n\ntarget_table.declare_row(row=MyRecord(id=\"rec1\", name=\"example\"))\n```\n\n### As Target (Relation)\n\n```python\nrelation_target = await surrealdb.mount_relation_target(\n SURREAL_DB,\n table_name=\"likes\",\n from_table=users_table,\n to_table=items_table,\n table_schema=await surrealdb.TableSchema.from_class(MyRelation),\n)\nrelation_target.declare_relation(from_id=\"user:1\", to_id=\"item:1\", record=MyRelation(...))\n```\n\n---\n\n## Neo4j\n\n**Import**: `from cocoindex.connectors import neo4j`\n\n**Capabilities**: Target only | **Vector**: Native (vector index) | **Model**: Knowledge graph (nodes + relationships)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import neo4j\n\nNEO4J_DB = coco.ContextKey[neo4j.ConnectionFactory](\"neo4j_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n neo4j_db = neo4j.ConnectionFactory(\n uri=\"bolt://localhost:7687\",\n auth=(\"neo4j\", \"cocoindex\"),\n database=\"neo4j\", # Optional, default \"neo4j\"\n )\n builder.provide(NEO4J_DB, neo4j_db)\n yield\n await neo4j_db.close() # closes the driver the factory opened on this loop\n```\n\n### As Target (Nodes + Relationships)\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass Person:\n name: str\n\n@dataclass\nclass Task:\n description: str\n\n@dataclass\nclass AssignedRel:\n priority: int\n\n# Mount node tables (each label is a table)\nperson_table = await neo4j.mount_table_target(\n NEO4J_DB,\n \"Person\",\n await neo4j.TableSchema.from_class(Person, primary_key=\"name\"),\n primary_key=\"name\",\n)\ntask_table = await neo4j.mount_table_target(\n NEO4J_DB,\n \"Task\",\n await neo4j.TableSchema.from_class(Task, primary_key=\"description\"),\n primary_key=\"description\",\n)\n\n# Mount a relationship target between two node tables\nassigned_rel = await neo4j.mount_relation_target(\n NEO4J_DB, \"ASSIGNED_TO\", person_table, task_table,\n await neo4j.TableSchema.from_class(AssignedRel),\n)\n\n# Declare nodes\nperson_table.declare_record(row=Person(name=\"Alice\")) # alias: declare_row\ntask_table.declare_record(row=Task(description=\"ship v1\"))\n\n# Declare edges (record optional; PK auto-derived from endpoints when omitted)\nassigned_rel.declare_relation(\n from_id=\"Alice\", to_id=\"ship v1\", record=AssignedRel(priority=1),\n)\n```\n\n### Vector Index\n\n```python\nperson_table.declare_vector_index(\n field=\"embedding\",\n dimension=384,\n metric=\"cosine\", # or \"euclidean\"\n)\n```\n\n### Type Override\n\n```python\nfrom typing import Annotated\n\n@dataclass\nclass Row:\n id: str\n score: Annotated[float, neo4j.Neo4jType(\"decimal\", encoder=str)]\n```\n\n---\n\n## FalkorDB\n\n**Import**: `from cocoindex.connectors import falkordb`\n\n**Capabilities**: Target only | **Vector**: Native (vector index) | **Model**: Knowledge graph (nodes + relationships)\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import falkordb\n\nFALKOR_DB = coco.ContextKey[falkordb.ConnectionFactory](\"falkor_db\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n falkor_db = falkordb.ConnectionFactory(\n uri=\"falkor://localhost:6379\",\n graph=\"knowledge_graph\", # Optional, default \"default\"\n )\n builder.provide(FALKOR_DB, falkor_db)\n yield\n await falkor_db.close() # closes the client the factory opened on this loop\n```\n\n### As Target (Nodes + Relationships)\n\n```python\nperson_table = await falkordb.mount_table_target(\n FALKOR_DB,\n \"Person\",\n await falkordb.TableSchema.from_class(Person, primary_key=\"name\"),\n primary_key=\"name\",\n)\ntask_table = await falkordb.mount_table_target(\n FALKOR_DB,\n \"Task\",\n await falkordb.TableSchema.from_class(Task, primary_key=\"description\"),\n primary_key=\"description\",\n)\n\n# Mount relationship target (no schema -> PK auto-derived from endpoints)\nassigned_rel = await falkordb.mount_relation_target(\n FALKOR_DB, \"ASSIGNED_TO\", person_table, task_table,\n)\n\n# Declare nodes\nperson_table.declare_record(row=Person(name=\"Alice\")) # alias: declare_row\ntask_table.declare_record(row=Task(description=\"ship v1\"))\n\n# Declare edges\nassigned_rel.declare_relation(from_id=\"Alice\", to_id=\"ship v1\")\n```\n\n### Vector Index\n\n```python\nperson_table.declare_vector_index(\n field=\"embedding\",\n dimension=384,\n metric=\"cosine\", # or \"euclidean\", \"ip\"\n)\n```\n\n### Type Override\n\n```python\nfrom typing import Annotated\n\n@dataclass\nclass Row:\n id: str\n score: Annotated[float, falkordb.FalkorType(\"decimal\", encoder=str)]\n```\n\n---\n\n## LocalFS\n\n**Import**: `from cocoindex.connectors import localfs`\n\n**Capabilities**: Source + Target\n\n### As Source\n\n```python\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import PatternFilePathMatcher\n\nfiles = localfs.walk_dir(\n pathlib.Path(\"./data\"),\n recursive=True,\n path_matcher=PatternFilePathMatcher(\n included_patterns=[\"**/*.py\", \"**/*.md\"],\n excluded_patterns=[\".*/**\", \"__pycache__/**\"],\n ),\n live=True, # Enable file watching for live mode\n)\n\n# Process each file\nawait coco.mount_each(process_file, files.items(), target_table)\n```\n\n### As Target (Single File)\n\n```python\nlocalfs.declare_file(\n outdir / \"output.txt\",\n \"file content\",\n create_parent_dirs=True,\n)\n```\n\n### As Target (Directory)\n\n```python\ndir_target = await localfs.mount_dir_target(pathlib.Path(\"./output\"))\ndir_target.declare_file(\"file1.txt\", \"content 1\")\ndir_target.declare_file(\"subdir/file2.txt\", \"content 2\")\n```\n\n### Stable File Paths\n\nUse `localfs.FilePath` for stable memoization when the base directory might move:\n\n```python\nfiles = localfs.walk_dir(localfs.FilePath(path=\"./data\"), ...)\n```\n\n---\n\n## Amazon S3\n\n**Import**: `from cocoindex.connectors import amazon_s3`\n\n**Capabilities**: Source only\n\n### Connection Setup\n\n```python\nimport aiobotocore.session\nfrom aiobotocore.client import AioBaseClient\n\nS3_CLIENT = coco.ContextKey[AioBaseClient](\"s3_client\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n session = aiobotocore.session.get_session()\n async with session.create_client(\"s3\") as s3_client:\n builder.provide(S3_CLIENT, s3_client)\n yield\n```\n\n### Reading Objects\n\n```python\nfrom cocoindex.connectors import amazon_s3\n\nclient = coco.use_context(S3_CLIENT)\n\n# List and iterate objects\nwalker = amazon_s3.list_objects(\n client, \"my-bucket\",\n prefix=\"data/\",\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.json\"]),\n max_file_size=10_000_000,\n)\n\nawait coco.mount_each(process_file, walker.items(), target)\n\n# Single object\nfile = await amazon_s3.get_object(client, \"s3://my-bucket/path/to/file.json\")\ncontent = await amazon_s3.read(client, \"s3://my-bucket/path/to/file.json\")\n```\n\n---\n\n## Kafka\n\n**Import**: `from cocoindex.connectors import kafka`\n\n**Capabilities**: Source + Target\n\n### As Source (Live)\n\n```python\nfrom confluent_kafka.aio import AIOConsumer\nfrom cocoindex.connectors import kafka\n\nconsumer = AIOConsumer({\n \"bootstrap.servers\": \"localhost:9092\",\n \"group.id\": \"my-group\",\n \"enable.auto.commit\": \"false\",\n \"auto.offset.reset\": \"earliest\",\n})\n\nitems = kafka.topic_as_map(consumer, [\"my-topic\"])\nawait coco.mount_each(process_message, items, target_table)\n```\n\n`topic_as_map()` returns a `LiveMapFeed`, which `mount_each()` auto-detects for live mode.\n\n### As Target\n\n```python\nfrom confluent_kafka.aio import AIOProducer\n\nKAFKA_PRODUCER = coco.ContextKey[AIOProducer](\"kafka_producer\")\n\ntopic_target = await kafka.mount_kafka_topic_target(KAFKA_PRODUCER, \"my-topic\")\ntopic_target.declare_target_state(key=\"msg-key\", value=json.dumps(data))\n```\n\n---\n\n## Apache Doris\n\n**Import**: `from cocoindex.connectors import doris`\n\n**Capabilities**: Target only | **Vector**: Native | **Features**: Vector indexes, inverted indexes, Stream Load\n\n### Connection Setup\n\n```python\nfrom cocoindex.connectors import doris\n\nDORIS_DB = coco.ContextKey[doris.ManagedConnection](\"doris_db\")\n\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n config = doris.DorisConnectionConfig(\n fe_host=\"localhost\",\n database=\"my_db\",\n fe_http_port=8080,\n query_port=9030,\n username=\"root\",\n password=\"\",\n )\n conn = doris.connect(config)\n builder.provide(DORIS_DB, conn)\n yield\n```\n\n### As Target\n\n```python\ntarget_table = await doris.mount_table_target(\n DORIS_DB,\n table_name=\"embeddings\",\n table_schema=await doris.TableSchema.from_class(MyRecord, primary_key=[\"id\"]),\n vector_indexes=[doris.VectorIndexDef(field_name=\"embedding\", metric_type=\"cosine_distance\")],\n)\n\ntarget_table.declare_row(row=MyRecord(id=1, text=\"hello\", embedding=vec))\n```\n\n---\n\n## Google Drive\n\n**Import**: `from cocoindex.connectors import google_drive`\n\n**Capabilities**: Source only\n\n```python\nfrom cocoindex.connectors import google_drive\n\nsource = google_drive.GoogleDriveSource(\n service_account_credential_path=\"credentials.json\",\n root_folder_ids=[\"folder-id\"],\n)\nawait coco.mount_each(process_file, source.items(), target)\n```\n\n---\n\n## Connector Comparison\n\n| Connector | Source | Target | Vectors | Best For |\n|-----------|--------|--------|---------|----------|\n| **PostgreSQL** | Y | Y | pgvector | Production SQL + vectors |\n| **SQLite** | - | Y | sqlite-vec | Local SQL + vectors |\n| **LanceDB** | - | Y | Native | Cloud-native vector DB |\n| **Qdrant** | - | Y | Native | Specialized vector search |\n| **SurrealDB** | - | Y | Native | Graph + document DB |\n| **Neo4j** | - | Y | Native | Knowledge graphs |\n| **FalkorDB** | - | Y | Native | Knowledge graphs (Redis) |\n| **Apache Doris** | - | Y | Native | Analytical DB + vectors |\n| **LocalFS** | Y | Y | N/A | File-based pipelines |\n| **Amazon S3** | Y | - | N/A | Cloud object storage |\n| **Kafka** | Y | Y | N/A | Streaming pipelines |\n| **Google Drive** | Y | - | N/A | Cloud file source |\n\n---\n\n## See Also\n\n- [API Reference](./api_reference.md)\n- [Patterns](./patterns.md)\n- [Setup Database](./setup_database.md)\n\n\n# CocoIndex Common Patterns\n\nCommon patterns and workflows for building CocoIndex pipelines.\n\n## Core Pattern: TargetState = Transform(SourceState)\n\nAll CocoIndex pipelines follow this declarative pattern:\n\n1. Read source state\n2. Transform\n3. Declare target state\n\nCocoIndex handles the incremental sync automatically.\n\n---\n\n## Pattern 1: File Transformation Pipeline\n\n**Use case**: Transform files from one format to another (e.g., Markdown -> HTML, PDF -> Markdown)\n\n```python\nimport pathlib\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom markdown_it import MarkdownIt\n\n_markdown_it = MarkdownIt(\"gfm-like\")\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, outdir: pathlib.Path) -> None:\n html = _markdown_it.render(await file.read_text())\n outname = \"__\".join(file.file_path.path.parts) + \".html\"\n localfs.declare_file(outdir / outname, html, create_parent_dirs=True)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n files = localfs.walk_dir(\n sourcedir,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n live=True, # Enable live file watching\n )\n await coco.mount_each(process_file, files.items(), outdir)\n\napp = coco.App(\n coco.AppConfig(name=\"FilesTransform\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n outdir=pathlib.Path(\"./output_html\"),\n)\n```\n\n**Key points:**\n- `memo=True` -- Skip reprocessing unchanged files\n- `mount_each()` -- One component per file; keys from `files.items()` become component subpaths\n- `live=True` on `walk_dir()` -- Enables file watching for live mode\n- Auto-cleanup -- Deleting source file automatically removes output file\n\n---\n\n## Pattern 2: Vector Embedding Pipeline\n\n**Use case**: Chunk and embed documents for semantic search\n\n```python\nimport pathlib\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator, Annotated\n\nimport asyncpg\nfrom numpy.typing import NDArray\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs, postgres\nfrom cocoindex.ops.text import RecursiveSplitter\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\nfrom cocoindex.resources.chunk import Chunk\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom cocoindex.resources.id import IdGenerator\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n_splitter = RecursiveSplitter()\n\n@dataclass\nclass DocEmbedding:\n id: int\n filename: str\n chunk_start: int\n chunk_end: int\n text: str\n embedding: Annotated[NDArray, EMBEDDER]\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n@coco.fn\nasync def process_chunk(\n chunk: Chunk, filename: pathlib.PurePath,\n id_gen: IdGenerator, table: postgres.TableTarget[DocEmbedding],\n) -> None:\n table.declare_row(row=DocEmbedding(\n id=await id_gen.next_id(chunk.text),\n filename=str(filename),\n chunk_start=chunk.start.char_offset,\n chunk_end=chunk.end.char_offset,\n text=chunk.text,\n embedding=await coco.use_context(EMBEDDER).embed(chunk.text),\n ))\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, table: postgres.TableTarget[DocEmbedding]) -> None:\n text = await file.read_text()\n chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500, language=\"markdown\")\n id_gen = IdGenerator()\n await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n target_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"doc_embeddings\",\n table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=[\"id\"]),\n )\n target_table.declare_vector_index(column=\"embedding\")\n\n files = localfs.walk_dir(sourcedir, recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]))\n await coco.mount_each(process_file, files.items(), target_table)\n\napp = coco.App(coco.AppConfig(name=\"TextEmbedding\"), app_main,\n sourcedir=pathlib.Path(\"./markdown_files\"))\n```\n\n**Key points:**\n- `mount_table_target(PG_DB, ...)` -- Takes `ContextKey` as first argument\n- `Annotated[NDArray, EMBEDDER]` -- Vector annotation uses ContextKey for auto dimension inference\n- `map()` -- Concurrent execution within a component (no child components created)\n- `IdGenerator` -- Generates stable unique IDs for chunks across incremental updates\n- `memo=True` on `process_file` -- Skips unchanged files entirely\n\n---\n\n## Pattern 3: Database Source -> Transform -> Database Target\n\n**Use case**: Transform data from one database to another\n\n```python\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator\n\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nSOURCE_DB_URL = \"postgres://localhost/source_db\"\nTARGET_DB_URL = \"postgres://localhost/target_db\"\n\nSOURCE_DB = coco.ContextKey[asyncpg.Pool](\"source_db\")\nTARGET_DB = coco.ContextKey[asyncpg.Pool](\"target_db\")\n\n@dataclass\nclass SourceRecord:\n id: int\n name: str\n value: float\n\n@dataclass\nclass TargetRecord:\n id: int\n name: str\n value: float\n processed: bool\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with (\n await asyncpg.create_pool(SOURCE_DB_URL) as source_pool,\n await asyncpg.create_pool(TARGET_DB_URL) as target_pool,\n ):\n builder.provide(SOURCE_DB, source_pool)\n builder.provide(TARGET_DB, target_pool)\n yield\n\n@coco.fn(memo=True)\nasync def process_record(record: SourceRecord, target_table: postgres.TableTarget[TargetRecord]) -> None:\n target_table.declare_row(row=TargetRecord(\n id=record.id, name=record.name.upper(),\n value=record.value * 2, processed=True,\n ))\n\n@coco.fn\nasync def app_main() -> None:\n target_table = await postgres.mount_table_target(\n TARGET_DB,\n table_name=\"target_records\",\n table_schema=await postgres.TableSchema.from_class(TargetRecord, primary_key=[\"id\"]),\n )\n\n source = postgres.PgTableSource(\n coco.use_context(SOURCE_DB),\n table_name=\"source_records\",\n row_type=SourceRecord,\n )\n await coco.mount_each(\n coco.component_subpath(\"record\"),\n process_record,\n source.fetch_rows().items(key=lambda r: r.id),\n target_table,\n )\n\napp = coco.App(coco.AppConfig(name=\"DatabaseTransform\"), app_main)\n```\n\n---\n\n## Pattern 4: LLM-Based Extraction Pipeline\n\n**Use case**: Extract structured data using LLMs\n\n```python\nimport instructor\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator\nfrom pydantic import BaseModel\nfrom litellm import acompletion\n\nimport asyncpg\nimport cocoindex as coco\nfrom cocoindex.connectors import postgres\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\n\n_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)\n\nclass ExtractedTopic(BaseModel):\n name: str\n description: str\n\nclass ExtractionResult(BaseModel):\n title: str\n topics: list[ExtractedTopic]\n\n@dataclass\nclass Message:\n id: int\n title: str\n content: str\n\n@dataclass\nclass Topic:\n message_id: int\n name: str\n description: str\n\n@coco.fn(memo=True)\nasync def extract_and_store(\n content: str, message_id: int,\n messages_table: postgres.TableTarget[Message],\n topics_table: postgres.TableTarget[Topic],\n) -> None:\n result = await _instructor_client.chat.completions.create(\n model=\"gpt-4\",\n response_model=ExtractionResult,\n messages=[{\"role\": \"user\", \"content\": f\"Extract topics:\\n\\n{content}\"}],\n )\n messages_table.declare_row(row=Message(id=message_id, title=result.title, content=content))\n for topic in result.topics:\n topics_table.declare_row(row=Topic(\n message_id=message_id, name=topic.name, description=topic.description,\n ))\n\n@coco.fn\nasync def app_main(input_texts: list[str]) -> None:\n messages_table = await postgres.mount_table_target(\n PG_DB, table_name=\"messages\",\n table_schema=await postgres.TableSchema.from_class(Message, primary_key=[\"id\"]),\n )\n topics_table = await postgres.mount_table_target(\n PG_DB, table_name=\"topics\",\n table_schema=await postgres.TableSchema.from_class(Topic, primary_key=[\"message_id\", \"name\"]),\n )\n for idx, text in enumerate(input_texts):\n await coco.mount(\n coco.component_subpath(\"text\", idx),\n extract_and_store, text, idx, messages_table, topics_table,\n )\n\napp = coco.App(coco.AppConfig(name=\"LLMExtraction\"), app_main, input_texts=[\"text1...\", \"text2...\"])\n```\n\n**Key points:**\n- `memo=True` on extraction -- Avoids re-calling LLM for unchanged inputs\n- Multiple target tables -- Declare rows in multiple tables from single component\n- Pydantic models -- For structured LLM outputs\n\n---\n\n## Pattern 5: Kafka Streaming Pipeline\n\n**Use case**: Process Kafka messages and write to a database\n\n```python\nimport json\nfrom collections.abc import AsyncIterator\nfrom dataclasses import dataclass\n\nfrom confluent_kafka import Message\nfrom confluent_kafka.aio import AIOConsumer\n\nimport cocoindex as coco\nfrom cocoindex.connectors import kafka, lancedb\n\nLANCE_DB = coco.ContextKey[lancedb.LanceAsyncConnection](\"lance_db\")\n\n@dataclass\nclass Product:\n sku: str\n name: str\n category: str\n price: float\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n conn = await lancedb.connect_async(\"./lancedb_data\")\n builder.provide(LANCE_DB, conn)\n yield\n\n@coco.fn\nasync def process_message(msg: Message, table: lancedb.TableTarget[Product]) -> None:\n value = msg.value()\n if value is None:\n return\n row = json.loads(value.decode() if isinstance(value, bytes) else value)\n table.declare_row(row=Product(**{**row, \"price\": float(row[\"price\"])}))\n\n@coco.fn\nasync def app_main() -> None:\n products_table = await lancedb.mount_table_target(\n LANCE_DB, table_name=\"products\",\n table_schema=await lancedb.TableSchema.from_class(Product, primary_key=[\"sku\"]),\n )\n\n consumer = AIOConsumer({\n \"bootstrap.servers\": \"localhost:9092\",\n \"group.id\": \"my-group\",\n \"enable.auto.commit\": \"false\",\n \"auto.offset.reset\": \"earliest\",\n })\n items = kafka.topic_as_map(consumer, [\"products-topic\"])\n await coco.mount_each(process_message, items, products_table)\n\napp = coco.App(coco.AppConfig(name=\"KafkaToLanceDB\"), app_main)\n```\n\n**Key points:**\n- `kafka.topic_as_map()` returns a `LiveMapFeed` -- `mount_each()` auto-detects for live mode\n- Pipeline runs continuously in live mode, processing new messages as they arrive\n\n---\n\n## Pattern 6: Context Management for Shared Resources\n\n**Use case**: Share expensive resources across components\n\n```python\nimport cocoindex as coco\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\n\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\nCONFIG = coco.ContextKey[dict](\"config\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n builder.provide(CONFIG, {\"chunk_size\": 1000, \"overlap\": 200})\n yield\n\n@coco.fn\nasync def process_item(text: str) -> None:\n embedder = coco.use_context(EMBEDDER)\n config = coco.use_context(CONFIG)\n embedding = await embedder.embed(text)\n ...\n```\n\n**Key points:**\n- `detect_change=True` -- Opt in to invalidate memos when value changes (use for models/config affecting output)\n- `detect_change=False` (default) -- For resources not affecting computation (DB connections, loggers)\n- `ContextKey` name is stable identity -- avoid renaming across runs\n\n---\n\n## Common Anti-Patterns to Avoid\n\n### Missing `@coco.fn` Decorator\n\n```python\n# BAD: Missing decorator\nasync def process_file(file, table):\n table.declare_row(...)\n\n# GOOD:\n@coco.fn\nasync def process_file(file, table):\n table.declare_row(...)\n```\n\n### Reprocessing Everything\n\n```python\n# BAD: No memoization\n@coco.fn # Missing memo=True\nasync def process_file(file, table):\n embedding = await embedder.embed(await file.read_text()) # Expensive!\n\n# GOOD:\n@coco.fn(memo=True)\nasync def process_file(file, table):\n ...\n```\n\n### Unstable Component Paths\n\n```python\n# BAD: Using object references or indices\nfor file in files:\n await coco.mount(coco.component_subpath(file), ...) # Object ref\nfor idx, item in enumerate(items):\n await coco.mount(coco.component_subpath(idx), ...) # Index changes\n\n# GOOD: Use stable identifiers\nawait coco.mount_each(process_file, files.items(), table) # Keys from items()\n```\n\n### Loading Resources Per Component\n\n```python\n# BAD: Loading model in every component\n@coco.fn\nasync def process(text):\n model = SentenceTransformer(\"model\") # Loaded repeatedly!\n\n# GOOD: Load once in lifespan, use via context\nembedder = coco.use_context(EMBEDDER)\n```\n\n### Mixing Target State with Side Effects\n\n```python\n# BAD: Side effects not detected\n@coco.fn\nasync def process(data):\n requests.post(\"https://api.example.com\", json=data) # Not detected!\n\n# GOOD: Only declare target states\ntable.declare_row(row=result)\n```\n\n---\n\n## See Also\n\n- [Connectors Reference](./connectors.md)\n- [API Reference](./api_reference.md)\n- [Setup Project](./setup_project.md)\n\n\n# Database Setup Guide\n\nSetup instructions for databases used with CocoIndex.\n\n## PostgreSQL with pgvector\n\n### Using Docker (Recommended)\n\n```bash\ncat > docker-compose.yml < docker-compose.yml <\n# Project Setup Guide\n\nSetting up CocoIndex projects for different use cases.\n\n## Creating a New Project\n\n```bash\ncocoindex init my-project\ncd my-project\n```\n\nThis creates: `main.py`, `pyproject.toml`, `README.md`. The generated `main.py` sets the internal database location in its lifespan via `builder.settings.db_path = pathlib.Path(\"./cocoindex.db\")`.\n\n```bash\nuv run cocoindex update main.py # installs dependencies, then runs\n```\n\n## Dependencies by Use Case\n\n### Vector Embedding Pipeline\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"sentence-transformers\",\n \"asyncpg\",\n]\n```\n\n### PostgreSQL Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"asyncpg\",\n]\n```\n\n### SQLite Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"sqlite-vec\",\n]\n```\n\n### LanceDB Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"lancedb\",\n]\n```\n\n### Qdrant Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"qdrant-client\",\n]\n```\n\n### Kafka Integration\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"confluent-kafka\",\n]\n```\n\n### LLM-Based Extraction\n\n```toml\n[project]\ndependencies = [\n \"cocoindex>=1.0.0\",\n \"litellm\",\n \"instructor\",\n \"pydantic>=2.0\",\n \"asyncpg\",\n]\n```\n\n---\n\n## Environment Configuration\n\n### `.env` File\n\nThe `cocoindex` CLI automatically loads `.env` from the current directory (via `find_dotenv`).\n\n```bash\n# CocoIndex internal database (optional fallback).\n# Only used if the lifespan does not set builder.settings.db_path.\n# The `cocoindex init` template sets db_path in the lifespan instead, so this is not needed there.\nCOCOINDEX_DB=./cocoindex.db\n\n# PostgreSQL (if using)\nPOSTGRES_URL=postgres://user:pass@localhost/db\n\n# Qdrant (if using)\nQDRANT_URL=http://localhost:6333\n\n# API keys (if using LLM extraction)\nOPENAI_API_KEY=your-openai-api-key\nANTHROPIC_API_KEY=your-anthropic-api-key\n```\n\n### Manual Settings (in lifespan)\n\n```python\n@coco.lifespan\ndef coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:\n builder.settings.db_path = pathlib.Path(\"./custom.db\")\n yield\n```\n\n---\n\n## Running Your Pipeline\n\n```bash\nuv sync # Install dependencies\nuv run cocoindex update main.py # Run pipeline\nuv run cocoindex update main.py -L # Run in live mode\nuv run cocoindex show main.py # Show component paths\nuv run cocoindex drop main.py -f # Reset everything\n```\n\nIf `pyproject.toml` has `[tool.uv.sources]` or `[[tool.uv.index]]` tables, install with uv. pip ignores those tables, so it would fetch the packages they pin from PyPI instead.\n\n---\n\n## Common Issues\n\n### Import Errors\n\n```bash\nuv sync\n```\n\n### Database Connection Errors\n\nVerify database is running and `.env` has correct URLs. See [setup_database.md](./setup_database.md).\n\n---\n\n## See Also\n\n- [Database Setup](./setup_database.md)\n- [Patterns](./patterns.md)\n- [API Reference](./api_reference.md)\n\n\n---\nname: cocoindex\ndescription: This skill should be used when building data processing pipelines with CocoIndex, a Python library for incremental data transformation. Use when the task involves processing files/data into databases, creating vector embeddings, building knowledge graphs, ETL workflows, or any data pipeline requiring automatic change detection and incremental updates. CocoIndex is Python-native (supports any Python types), has no DSL, and uses version 1.0.0 or later.\n---\n\n# CocoIndex\n\nCocoIndex is a Python library for building incremental data processing pipelines with declarative target states. Think spreadsheets or React for data pipelines: declare what the output should look like based on current input, and CocoIndex automatically handles incremental updates, change detection, and syncing to external systems.\n\n## Overview\n\nCocoIndex enables building data pipelines that:\n\n- **Automatically handle incremental updates**: Only reprocess changed data\n- **Use declarative target states**: Declare what should exist, not how to update\n- **Support any Python types**: No custom DSL -- use dataclasses, Pydantic, NamedTuple\n- **Provide function memoization**: Skip expensive operations when inputs/code unchanged\n- **Sync to multiple targets**: PostgreSQL, SQLite, LanceDB, Qdrant, SurrealDB, Apache Doris, file systems, Kafka\n\n**Key principle**: `TargetState = Transform(SourceState)`\n\n## When to Use This Skill\n\nUse this skill when building pipelines that involve:\n\n- **Document processing**: PDF/Markdown conversion, text extraction, chunking\n- **Vector embeddings**: Embedding documents/code for semantic search\n- **Database transformations**: ETL from source DB to target DB\n- **Knowledge graphs**: Extract entities and relationships from data\n- **LLM-based extraction**: Structured data extraction using LLMs\n- **File-based pipelines**: Transform files from one format to another\n- **Incremental indexing**: Keep search indexes up-to-date with source changes\n- **Streaming pipelines**: Kafka-based real-time data processing\n\n## Quick Start: Creating a New Project\n\n### Initialize Project\n\n```bash\ncocoindex init my-project\ncd my-project\n```\n\nThis creates: `main.py`, `pyproject.toml`, `README.md`. The generated `main.py` sets the database location in its lifespan via `builder.settings.db_path = pathlib.Path(\"./cocoindex.db\")`.\n\n### Add Dependencies\n\n```toml\n# For vector embeddings with PostgreSQL\ndependencies = [\"cocoindex>=1.0.0\", \"sentence-transformers\", \"asyncpg\"]\n\n# For LLM extraction\ndependencies = [\"cocoindex>=1.0.0\", \"litellm\", \"instructor\", \"pydantic>=2.0\"]\n```\n\nSee [references/setup_project.md](references/setup_project.md) for complete examples.\n\n### Run the Pipeline\n\n```bash\nuv run cocoindex update main.py # installs dependencies, then runs\n```\n\n## Core Concepts\n\n### 1. Apps\n\nAn **App** is the top-level executable that binds a main function with parameters:\n\n```python\nimport cocoindex as coco\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n ...\n\napp = coco.App(\n coco.AppConfig(name=\"MyApp\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n)\n```\n\n### 2. Functions (`@coco.fn`)\n\nThe `@coco.fn` decorator marks functions as CocoIndex processing functions. Add `memo=True` to skip re-execution when inputs/code are unchanged:\n\n```python\n@coco.fn(memo=True)\nasync def expensive_operation(data: str) -> Result:\n # LLM call, embedding generation, heavy computation\n return await expensive_transform(data)\n```\n\n**Key parameters:**\n- `memo=True` -- Enable memoization (skip if inputs/code unchanged)\n- `version=1` -- Explicit version bump to force re-execution\n- `batching=True` -- Auto-batch concurrent calls (async only)\n- `runner=coco.GPU` -- Serialize GPU-bound execution\n\n### 3. Processing Components\n\nA **processing component** groups an item's processing with its target states.\n\n**Mount components** with `mount_each()` (preferred for lists) or `mount()`:\n\n```python\n# One component per item (preferred for lists)\nawait coco.mount_each(process_file, files.items(), target_table)\n\n# Single component (subpath auto-derived from fn.__name__)\nawait coco.mount(setup_fn, arg1)\n\n# Dependent component (blocks until result returned)\nresult = await coco.use_mount(init_fn)\n\n# Explicit subpath (when you need a specific path, e.g. in loops)\nawait coco.mount(coco.component_subpath(\"item\", item_id), process_item, item)\n```\n\n**Key points:**\n- All mount APIs are `async`\n- `mount()`, `use_mount()`, and `mount_each()` auto-derive subpath from `fn.__name__`; optional explicit subpath as first arg\n- Use `use_mount()` when you need the return value\n- Use stable component paths for proper memoization and cleanup\n\n### 4. Target States\n\n**Declare** what should exist -- CocoIndex handles creation/update/deletion:\n\n```python\n# Database row target\ntable.declare_row(row=MyRecord(id=1, name=\"example\"))\n\n# File target\nlocalfs.declare_file(outdir / \"output.txt\", content, create_parent_dirs=True)\n\n# Kafka message target\ntopic_target.declare_target_state(key=\"msg-1\", value=json.dumps(data))\n```\n\n### 5. Context for Shared Resources\n\nUse `ContextKey` to share expensive resources (DB connections, models) across components:\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n# In processing functions:\nembedder = coco.use_context(EMBEDDER)\n```\n\nThe `@coco.lifespan` decorator registers the function to the default CocoIndex environment, which is shared among all apps by default. `ContextKey` also serves as the stable identity for sources/targets -- the `key` string must remain stable across runs.\n\n### 6. ID Generation\n\nGenerate stable, unique identifiers that persist across incremental updates:\n\n```python\nfrom cocoindex.resources.id import generate_id, IdGenerator\n\n# Deterministic: same dep -> same ID\nchunk_id = await generate_id(chunk.text)\n\n# Always distinct: each call -> new ID, even with same dep\nid_gen = IdGenerator()\nfor chunk in chunks:\n chunk_id = await id_gen.next_id(chunk.text)\n```\n\n### 7. Catch-Up vs Live Mode\n\nBy default, `app.update()` runs in **catch-up mode**: it scans all sources, processes what changed since the last run (memoized components are skipped), syncs target states, and returns. Each call still has to scan sources to discover changes.\n\n**Live mode** keeps the app running after catch-up and lets components stream changes continuously from their sources (e.g., file watcher, Kafka consumer), applying them with very low latency.\n\n```python\n# Enable live mode\napp.update_blocking(live=True)\n# Or: cocoindex update main.py -L\n```\n\nTwo things are needed: (1) enable live mode on the app, and (2) use a source that supports live updates.\n\n- **`LiveMapView`** sources (e.g., `localfs.walk_dir(..., live=True)`) scan current state first, then watch for changes. They also work in catch-up mode -- write the pipeline once, choose mode at run time.\n- **`LiveMapFeed`** sources (e.g., `kafka.topic_as_map()`) only stream changes with no initial snapshot. `mount_each()` auto-detects these and creates a live component internally.\n\n```python\n# LocalFS with live watching\nfiles = localfs.walk_dir(sourcedir, live=True, ...)\nawait coco.mount_each(process_file, files.items(), target)\n\n# Kafka -- inherently live\nitems = kafka.topic_as_map(consumer, [\"my-topic\"])\nawait coco.mount_each(process_message, items, target)\n```\n\n## Common Pipeline Patterns\n\n### Pattern 1: File Transformation\n\n```python\nimport pathlib\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, outdir: pathlib.Path) -> None:\n content = await file.read_text()\n transformed = transform_content(content)\n outname = file.file_path.path.stem + \".out\"\n localfs.declare_file(outdir / outname, transformed, create_parent_dirs=True)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path, outdir: pathlib.Path) -> None:\n files = localfs.walk_dir(\n sourcedir,\n recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]),\n )\n await coco.mount_each(process_file, files.items(), outdir)\n\napp = coco.App(\n coco.AppConfig(name=\"Transform\"),\n app_main,\n sourcedir=pathlib.Path(\"./data\"),\n outdir=pathlib.Path(\"./out\"),\n)\n```\n\n### Pattern 2: Vector Embedding Pipeline\n\n```python\nimport pathlib\nfrom dataclasses import dataclass\nfrom typing import AsyncIterator, Annotated\n\nimport asyncpg\nfrom numpy.typing import NDArray\n\nimport cocoindex as coco\nfrom cocoindex.connectors import localfs, postgres\nfrom cocoindex.ops.text import RecursiveSplitter\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\nfrom cocoindex.resources.chunk import Chunk\nfrom cocoindex.resources.file import FileLike, PatternFilePathMatcher\nfrom cocoindex.resources.id import IdGenerator\n\nDATABASE_URL = \"postgres://cocoindex:cocoindex@localhost/cocoindex\"\nPG_DB = coco.ContextKey[asyncpg.Pool](\"pg_db\")\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n_splitter = RecursiveSplitter()\n\n@dataclass\nclass DocEmbedding:\n id: int\n filename: str\n text: str\n embedding: Annotated[NDArray, EMBEDDER] # Dimensions inferred from ContextKey\n chunk_start: int\n chunk_end: int\n\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n builder.provide(EMBEDDER, SentenceTransformerEmbedder(\"all-MiniLM-L6-v2\"))\n yield\n\n@coco.fn\nasync def process_chunk(\n chunk: Chunk, filename: pathlib.PurePath,\n id_gen: IdGenerator, table: postgres.TableTarget[DocEmbedding],\n) -> None:\n table.declare_row(row=DocEmbedding(\n id=await id_gen.next_id(chunk.text),\n filename=str(filename),\n text=chunk.text,\n embedding=await coco.use_context(EMBEDDER).embed(chunk.text),\n chunk_start=chunk.start.char_offset,\n chunk_end=chunk.end.char_offset,\n ))\n\n@coco.fn(memo=True)\nasync def process_file(file: FileLike, table: postgres.TableTarget[DocEmbedding]) -> None:\n text = await file.read_text()\n chunks = _splitter.split(text, chunk_size=2000, chunk_overlap=500)\n id_gen = IdGenerator()\n await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)\n\n@coco.fn\nasync def app_main(sourcedir: pathlib.Path) -> None:\n target_table = await postgres.mount_table_target(\n PG_DB,\n table_name=\"embeddings\",\n table_schema=await postgres.TableSchema.from_class(DocEmbedding, primary_key=[\"id\"]),\n )\n target_table.declare_vector_index(column=\"embedding\")\n\n files = localfs.walk_dir(sourcedir, recursive=True,\n path_matcher=PatternFilePathMatcher(included_patterns=[\"**/*.md\"]))\n await coco.mount_each(process_file, files.items(), target_table)\n\napp = coco.App(coco.AppConfig(name=\"Embedding\"), app_main, sourcedir=pathlib.Path(\"./data\"))\n```\n\n### Pattern 3: LLM-Based Extraction\n\n```python\nimport instructor\nfrom pydantic import BaseModel\nfrom litellm import acompletion\n\n_instructor_client = instructor.from_litellm(acompletion, mode=instructor.Mode.JSON)\n\nclass ExtractionResult(BaseModel):\n title: str\n topics: list[str]\n\n@coco.fn(memo=True) # Memo avoids re-calling LLM\nasync def extract_and_store(content: str, message_id: int, table) -> None:\n result = await _instructor_client.chat.completions.create(\n model=\"gpt-4\",\n response_model=ExtractionResult,\n messages=[{\"role\": \"user\", \"content\": f\"Extract topics: {content}\"}],\n )\n table.declare_row(row=Message(id=message_id, title=result.title, content=content))\n```\n\n## Connectors and Operations\n\nCocoIndex provides connectors for reading from and writing to external systems:\n\n| Connector | Source | Target | Vectors | Use Case |\n|-----------|--------|--------|---------|----------|\n| **PostgreSQL** | Y | Y | pgvector | Production SQL + vectors |\n| **SQLite** | - | Y | sqlite-vec | Local SQL + vectors |\n| **LanceDB** | - | Y | Y | Cloud-native vector DB |\n| **Qdrant** | - | Y | Y | Specialized vector DB |\n| **SurrealDB** | - | Y | Y | Graph + document DB |\n| **Apache Doris** | - | Y | Y | Analytical DB + vectors |\n| **LocalFS** | Y | Y | N/A | File-based pipelines |\n| **Amazon S3** | Y | - | N/A | Cloud object storage |\n| **Kafka** | Y | Y | N/A | Streaming pipelines |\n| **Google Drive** | Y | - | N/A | Cloud file source |\n\nFor detailed connector documentation, see [references/connectors.md](references/connectors.md).\n\n## Text and Embedding Operations\n\n### Text Splitting\n\n```python\nfrom cocoindex.ops.text import RecursiveSplitter, detect_code_language\n\nsplitter = RecursiveSplitter()\nlanguage = detect_code_language(filename=\"example.py\")\nchunks = splitter.split(text, chunk_size=1000, chunk_overlap=200, language=language)\n```\n\n### Embeddings\n\n```python\nfrom cocoindex.ops.sentence_transformers import SentenceTransformerEmbedder\n\nembedder = SentenceTransformerEmbedder(\"sentence-transformers/all-MiniLM-L6-v2\")\nembedding = await embedder.embed(text) # Returns NDArray\n```\n\n## CLI Commands\n\n```bash\ncocoindex init my-project # Create new project\ncocoindex update main.py # Run app\ncocoindex update main.py:my_app # Run specific app\ncocoindex update main.py -L # Run in live mode (continuous)\ncocoindex update main.py --full-reprocess # Reprocess everything\ncocoindex drop main.py [-f] # Drop and reset all state\ncocoindex ls [main.py] # List apps\ncocoindex show main.py [--tree] # Show component paths\n```\n\n## Best Practices\n\n### 1. Use `@coco.fn` on All Processing Functions\n\nEvery function that participates in the pipeline (declares target states, calls mount APIs, etc.) must be decorated with `@coco.fn`.\n\n### 2. Add Memoization for Expensive Operations\n\n```python\n@coco.fn(memo=True) # Skip re-execution when inputs/code unchanged\nasync def process_chunk(chunk, table):\n embedding = await embedder.embed(chunk.text) # Expensive!\n table.declare_row(...)\n```\n\n### 3. Use Stable Component Paths\n\n```python\n# Good: Stable identifiers\ncoco.component_subpath(\"file\", str(file.file_path.path))\ncoco.component_subpath(\"record\", record.id)\n\n# Bad: Unstable identifiers\ncoco.component_subpath(\"file\", file) # Object reference\ncoco.component_subpath(\"idx\", idx) # Index changes\n```\n\n### 4. Use Context for Shared Resources\n\n```python\n@coco.lifespan\nasync def coco_lifespan(builder: coco.EnvironmentBuilder) -> AsyncIterator[None]:\n async with await asyncpg.create_pool(DATABASE_URL) as pool:\n builder.provide(PG_DB, pool)\n yield\n```\n\n### 5. Use `Annotated[NDArray, CONTEXT_KEY]` for Vectors\n\n```python\nEMBEDDER = coco.ContextKey[SentenceTransformerEmbedder](\"embedder\")\n\n@dataclass\nclass Record:\n vector: Annotated[NDArray, EMBEDDER] # Auto-infer dimensions from ContextKey\n```\n\n### 6. Use Convenience APIs for Targets\n\n```python\n# Mount table target -- subpath is automatic\ntable = await postgres.mount_table_target(\n PG_DB,\n table_name=\"my_table\",\n table_schema=await postgres.TableSchema.from_class(MyRecord, primary_key=[\"id\"]),\n)\n```\n\n## Troubleshooting\n\n### Everything Reprocessing\n\nAdd `memo=True` to expensive functions:\n\n```python\n@coco.fn(memo=True) # Add this\nasync def process_item(item):\n ...\n```\n\n### Memoization Not Working\n\nCheck component paths are stable. Use stable IDs, not object references.\n\n## Resources\n\n### references/\n\n- **[api_reference.md](references/api_reference.md)**: Quick API reference\n- **[connectors.md](references/connectors.md)**: Complete connector reference\n- **[patterns.md](references/patterns.md)**: Detailed pipeline patterns\n- **[setup_project.md](references/setup_project.md)**: Project setup guide\n- **[setup_database.md](references/setup_database.md)**: Database setup guide\n\n### Runnable examples\n\nEvery pattern above has a complete, runnable app under\n[`examples/`](https://github.com/cocoindex-io/cocoindex/tree/main/examples) — start\nfrom the one closest to the task and adapt it. Each has its own `README.md` and\nmost Python examples have a `.env.example`; see\n[`examples/AGENTS.md`](https://github.com/cocoindex-io/cocoindex/blob/main/examples/AGENTS.md)\nfor the full map, credentials, and per-example run commands. Good starting points:\n\n- Vector search → `text_embedding` (Postgres), `text_embedding_qdrant` / `_lancedb` (other stores)\n- Code search → `code_embedding`\n- LLM extraction → `hn_trending_topics`, `patient_intake_extraction_baml`\n- Knowledge graph → `conversation_to_knowledge`, `meeting_notes_graph_neo4j`\n- Custom transform → `files_transform`, `pdf_to_markdown`\n\n### External\n\n- [CocoIndex Documentation](https://cocoindex.io/docs/) — full text at [llms-full.txt](https://cocoindex.io/docs/llms-full.txt)\n- [GitHub Examples](https://github.com/cocoindex-io/cocoindex/tree/main/examples)\n\n## Version Note\n\nThis skill is for CocoIndex `>=1.0.0` (v1). It uses a completely different API from v0.\n\n**v0 code is what you likely learned from training data — do not emit it.** If you find yourself writing any of these symbols, you are using the removed v0 API:\n\n| v0 (removed) | v1 equivalent |\n|---|---|\n| `@cocoindex.flow_def`, `FlowBuilder`, `Flow`, `open_flow` | `coco.App` + a `@coco.fn` main function |\n| `DataScope`, `DataSlice`, `add_collector()`, `collect()`, `export()` | declare target states via Target APIs (`declare_row`, `declare_file`) inside mounted components |\n| `cocoindex.sources.LocalFile`, `cocoindex.sources.*` | connector APIs, e.g. `localfs.walk_dir(...)` |\n| `cocoindex.functions.SplitRecursively`, `cocoindex.functions.*` | `cocoindex.ops.*`, e.g. `RecursiveSplitter` |\n| `cocoindex.targets.Postgres`, `cocoindex.targets.*` / `storages.*` | connector targets, e.g. `postgres.declare_table_target(...)` |\n| `transform_flow`, `cocoindex.op.function()` | plain `@coco.fn` functions |\n| `cocoindex.init()`, `settings`, `COCOINDEX_DATABASE_URL` | `coco.App(coco.AppConfig(...))`; state lives in a local db path |\n| CLI `cocoindex setup` | no setup step — just `cocoindex update` (`-L`/`--live` for live mode) |\n\nWhen reading third-party tutorials or model memory that mention these v0 symbols, disregard them and use the patterns in this skill and `references/api_reference.md` instead.\n\n\n# CLAUDE.md\n\n@AGENTS.md\n"}