# Technical Documentation: potpie-ai/potpie > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/potpie-ai/potpie) ### Overview (`README.md`)

Potpie AI logo

Docs Tests PyPI Discord Apache 2.0

[Potpie](https://potpie.ai) turns your codebase and software development lifecycle into a living context graph for AI agents. It indexes code, structure, decisions, source history, team knowledge and engineering workflows, so agents can answer questions, plan changes, debug failures, and write code with project-specific context. ## Install and setup Potpie ### Step 1: install the CLI through [PyPI](https://pypi.org/project/potpie/) with `uv` or `pip`: ```bash uv tool install potpie ``` or: ```bash python3 -m pip install --user potpie ``` > [!NOTE] > `uv tool install potpie` is recommended for CLI installs because global > mutation of Python packages is generally not recommended. ### Step 2: Run the Potpie setup wizard ```bash potpie setup ``` The setup wizard provisions local config, storage, the daemon, a default pot, and agent skills. It also lets you choose integrations and the coding harness Potpie should configure. ### Step 3: Open your configured harness Potpie is already integrated into your selected harness. You can start using Potpie with the repo of your choice. Open your previously selected harness and ask it to use Potpie for the repo. > [!NOTE] > You don't need to run a separate manual ingest command. The CLI registers > sources and the configured agent can ingest or update project context when the > task requires it. You can view your context graph in the web UI: ```bash potpie ui ``` This will open a graph explorer in your browser. ## Basic CLI user checklist The main CLI commands are: | Command | Purpose | | --- | --- | | `potpie setup` | Run first-time local setup for config, daemon, default pot, and agent skills. | | `potpie login` | Sign in to Potpie for account-backed and managed features. | | `potpie github login` / `potpie linear login` | Connect source integrations you want agents to use. | | `potpie status` | Show context readiness for the active pot, including daemon, graph, and skill checks. | | `potpie auth status` | Show configured integration auth status. | | `potpie auth status --verify` | Verify integration credentials with lightweight API checks. | | `potpie doctor` | Run local diagnostics for daemon, backend capabilities, and skill drift. | | `potpie source add repo .` | Register the current repo as a source for the resolved pot. | | `potpie pot list` / `potpie pot use ` | List pots and choose the active workspace. | | `potpie resolve ""` | Pull the context an agent should read before doing a task. | | `potpie search ""` | Look up a specific file, workflow, bug, decision, or convention. | | `potpie record --type --summary ""` | Write a durable project learning. | | `potpie graph ...` | Use lower-level graph reads, quality checks, proposals, and commits. | | `potpie ui` | Open the local graph explorer served by the daemon. | | `potpie skills install --agent ` | Install or refresh Potpie guidance for an agent harness. | Examples: ```bash potpie setup --repo . --agent claude potpie github login potpie auth status potpie status potpie source add repo . potpie resolve "what should I know before working in this repository?" potpie search "authentication flow" potpie record --type decision --summary "Prefer the Potpie CLI for graph work" ``` You can find an exhaustive list with more examples in our [docs](https://docs.potpie.ai). ## Integrations and Coding Harnesses Potpie supports a variety of integrations and coding harnesses, with more coming. If your team needs a new integration or harness, please [raise a ticket](https://github.com/potpie-ai/potpie/issues/new/choose). ### Integrations | Tool | Description | | --- | --- | | ![GitHub][github-badge] | Index repositories, pull requests, issues, reviews and source history. | | ![Linear][linear-badge] | Index teams, issues, projects and documents. | | ![Jira][jira-badge] | Index projects, issues, status and changelog context. | | ![Confluence][confluence-badge] | Index spaces, pages, runbooks and decisions. | ### Coding harnesses | Tool | Description | | --- | --- | | ![Claude Code][claude-badge] | Install Potpie instructions and skills for Claude Code. | | ![OpenAI Codex][codex-badge] | Install Potpie instructions and skills for OpenAI Codex. | | ![Cursor][cursor-badge] | Install Potpie instructions and skills for Cursor. | | ![OpenCode][opencode-badge] | Install Potpie skills for OpenCode. | [github-badge]: https://img.shields.io/badge/GitHub-111827?style=flat-square&logo=github&logoColor=white [linear-badge]: https://img.shields.io/badge/Linear-111827?style=flat-square&logo=linear&logoColor=white [jira-badge]: https://img.shields.io/badge/Jira-111827?style=flat-square&logo=jira&logoColor=white [confluence-badge]: https://img.shields.io/badge/Confluence-111827?style=flat-square&logo=confluence&logoColor=white [claude-badge]: https://img.shields.io/badge/Claude%20Code-111827?style=flat-square&logo=anthropic&logoColor=white [codex-badge]: https://img.shields.io/badge/OpenAI%20Codex-111827?style=flat-square&logo=openai&logoColor=white [cursor-badge]: https://img.shields.io/badge/Cursor-111827?style=flat-square&logo=cursor&logoColor=white [opencode-badge]: https://img.shields.io/badge/OpenCode-111827?style=flat-square&logo=opencode&logoColor=white ## Architecture Potpie's current architecture is CLI-first. CLI is designed to be used by both humans and agents. Read the deeper architecture notes in [`docs/context-graph/architecture.md`](https://github.com/potpie-ai/potpie/blob/main/docs/context-graph/architecture.md). ## License This project is licensed under the Apache 2.0 License. See [LICENSE](https://github.com/potpie-ai/potpie/blob/main/LICENSE). ## Community & Support - [GitHub Issues](https://github.com/potpie-ai/potpie/issues): bugs and repository-scoped requests - [Discord](https://discord.gg/ryk5CMD5v6): community discussion and support - [Docs](https://docs.potpie.ai): setup, product guides, and integration details New contributions are always welcome. Read the [Contributing Guide](https://github.com/potpie-ai/potpie/blob/main/.github/CONTRIBUTING.md) to set up your environment, understand the workflow, and open a pull request. Contributors --- ### Overview (`potpie/cli/README.md`) # Potpie CLI (`potpie`) The command-line entrypoint for the context graph. Context operations route through the typed engine client; product operations use explicit Potpie-owned services from one local runtime composition. - **Entrypoint:** `potpie/cli/main.py` (registered as the `potpie` console script in `pyproject.toml` → `[project.scripts]`). - **Command groups:** `potpie/cli/commands/` — one module per `cli-flow.md` section (`bootstrap`, `query`, `pots`/`source`, `host`, `daemon`, `ledger`, `graph`, `timeline`, `backend`, `skills`, `cloud`). - **Cross-cutting contract:** `commands/_common.py` owns `--json` output, the exit-code map (0 ok / 1 validation / 2 unavailable / 3 degraded / 4 auth), the structured error shape (`code`/`message`/`detail`/`recommended_next_action`), origin routing (local or managed host) and active-pot resolution. An unbuilt capability surfaces as the structured not-implemented contract (`CapabilityNotImplemented`), never a traceback. ## Agent compatibility commands The CLI exposes four compatibility commands with distinct response contracts: - `potpie resolve` and `potpie search` return an `AgentEnvelope` with no server-side synthesis. - `potpie record` returns a record receipt containing `status`, `record_id`, and `mutations_applied`. - `potpie status` returns readiness information for the selected pot and scope, including the recommended recipe. | Command | Use | |------------|-----| | `potpie resolve` | Primary bounded-context wrap for a task. | | `potpie search` | Narrow follow-up lookup. | | `potpie record` | Record a durable learning (decision, fix, preference, …). | | `potpie status` | Cheap pot/scope readiness + recommended recipe. | Integration credential status is intentionally outside this four-tool surface: use `potpie auth status [--verify]` for local provider auth state. ## Managed hosts The same commands run against a hosted context graph. The local daemon needs no configuration; a managed host needs its base URL and an API key, stored in `/cli_hosts.json` (`0600`) by `potpie host set`: ```bash printf '%s\n' "$KEY" | potpie host set https://your-host.example --token - potpie host use managed # make it the active host potpie pot list --managed potpie pot use managed: --also-default-for-current-repo potpie host use local # back to this machine ``` - The key never travels on argv: `--token -` reads it from stdin, and a literal `--token ` is refused. `POTPIE_MANAGED_URL` / `POTPIE_MANAGED_TOKEN` override the stored host for one run. - One command's host is the prefix of a qualified ref (`managed:`, `local:`), else the global `--host local|managed`, else the active host. A bare ref resolves on that host only; when it is not there, the error names the other host's match. - Engine commands on a managed host go through `RemoteEngineClient` (`potpie.runtime.remote`); pot, source, grant, `whoami` and `status` commands go through the host's `PotDirectory` (`potpie.pots.directory`). `daemon`, `skills`, `setup`, integration `auth` and harness configuration always stay on this machine. - `potpie ui --pot managed:` opens the explorer served by the local daemon with `?host=managed`; the daemon holds the key, the browser never sees it. - A host older than this client answers `not_implemented` (exit `2`) for what it lacks; everything else keeps working. - Python callers use `potpie.runtime.connect()`: `HostConnection.engine(selector)`, `.pots` (synchronous, any thread) and `.extension(name, request, result_type)`; synchronous code runs them with `.run(awaitable)`. The full contract is in [`docs/context-graph/cli-flow.md`](../../docs/context-graph/cli-flow.md#hosts-commandshostpy). ## Authoritative reference The full command catalog, flags, profiles (local vs managed), and the output contract live in **[`docs/context-graph/cli-flow.md`](../../docs/context-graph/cli-flow.md)**. The end-state architecture (services, ports, composition roots) is in **[`docs/context-graph/architecture.md`](../../docs/context-graph/architecture.md)**. Run `potpie --help` (or `python -m potpie.cli.main --help`) to list the live commands. ## Local install (this repo) Repo-local development installs the CLI with: ```bash make cli-install make cli-status ``` That path builds the graph-explorer UI, stops any old daemon, and installs the editable package. Published-package users should use `uv tool install potpie` or `pip install potpie` instead. ## Agent harness install `potpie skills install [] --agent claude` materializes the packaged skill bundle into an agent harness through the explicit root skill service. The default scope is global, so skills are installed once into the selected harness's user-level skills directory. The shipped templates live under `potpie/cli/templates/`, with one source for each kind of file: - `agent_bundle/.agents/skills/` — every skill. All harnesses install their skills from here, remapped to the harness's own skills directory. - `routing/POTPIE.md` — the compact instruction block merged into a harness's `AGENTS.md` or `CLAUDE.md`, globally and per repository. | Harness | Global path | |---------|-------------| | Cursor | `~/.cursor/skills//SKILL.md` | | Claude Code | `~/.claude/skills//SKILL.md` | | OpenCode | `~/.config/opencode/skills//SKILL.md` | | Codex | `$HOME/.agents/skills//SKILL.md` | `POTPIE_HARNESS_HOME` moves these roots away from the real home directory (the test suite pins it); `CONTEXT_ENGINE_HOME` deliberately does not, because a harness keeps reading its own home whatever Potpie's state directory is. For harnesses with documented file-backed global instructions, a bundle install or update (no skill id) also refreshes the compact Potpie managed block in `~/.claude/CLAUDE.md` and `~/.codex/AGENTS.md`; naming one skill id installs only that skill. Existing user-authored content is preserved; Potpie only appends or updates the `` / `` managed section, and `skills remove --all` takes that section back out. Remove one global skill with `potpie skills remove --agent claude`, or delete every globally installed Potpie skill for a harness with `potpie skills remove --all --agent claude`. Use `--scope project --path .` for repo-local cleanup. A skill removed by id is remembered as disabled, so a later bundle install skips it until it is installed again by id. `potpie --json skills status --agent ` lists `installed`, `missing`, `outdated`, `drifted` (installed, but its files no longer match the bundle — a subset of `outdated` with the same repair) and `disabled`. `--path` is resolved against the caller's working directory and must already exist. Use `--scope project --path .` for repo-local installs. The bundle teaches feature / debugging / review / operations / docs / onboarding workflows over the CLI graph surface. Agents see an advisory `skills` block in `potpie status` with missing/outdated skills and the exact install command. Repo-local `AGENTS.md` and `CLAUDE.md` files are merged the same way as global instruction files, so setup does not replace existing agent instructions. --- ### Overview (`potpie/integrations/README.md`) # potpie-integrations External integrations for Potpie: OAuth providers, `project_sources`, Linear sync, and HTTP routers. Layout mirrors `context-engine` (hexagonal): - `integrations/domain/` — registry, provider definitions, shared schemas - `integrations/application/` — services, provider bootstrap - `integrations/adapters/outbound/` — persistence models, OAuth clients, Linear GraphQL, crypto - `integrations/adapters/inbound/http/` — FastAPI routers mounted by the main app The installable Python package root is `integrations` (not top-level `domain`/`application`) so it does not collide with the `context-engine` editable install. --- ### Overview (`potpie/parsing/README.md`) # Parsing Extract code graphs from repositories. ## Rust ```rust use parsing_rs::{extract_graph, GraphPayload, NodePayload, RelationshipPayload}; let graph: GraphPayload = extract_graph("/path/to/repo"); // Nodes: FILE, CLASS, INTERFACE, FUNCTION for node in graph.nodes { println!("{}: {} @ {}:{}", node.node_type, node.name, node.file, node.line); } // Relationships: CONTAINS, REFERENCES for rel in graph.relationships { println!("{} → {}", rel.source_id, rel.target_id); } ``` ## Python ```python from parsing.py_graph import create_graph import networkx as nx G = create_graph("/path/to/repo") # Returns NetworkX MultiDiGraph ``` ## Node Types - `FILE` - Source file - `CLASS` / `INTERFACE` - Type definitions - `FUNCTION` - Methods and functions ## Relationship Types - `CONTAINS` - FILE → CLASS/FUNCTION - `REFERENCES` - FUNCTION → FUNCTION/CLASS (calls, usage) ## Supported Languages Python, Rust, JavaScript, TypeScript, Go, Java, C, C++, Ruby, PHP, C#, Elixir, OCaml, Elisp, Elm, QL ## Build ```bash cd app/src/parsing && maturin develop ``` ## Note This crate creates **graphs only**. Embeddings and search are handled by the parent `feat/colbert` module (uses Qdrant + Neo4j). ## FFF Workspace Search (Lexical) The FFF search surface indexes a checked-out workspace folder on demand and provides in-memory path/content search. ```rust use parsing_rs::{build_workspace_index, search_files}; let index = build_workspace_index("/path/to/workspace").expect("workspace should be indexed"); let files = index.search_files("auth", 5); assert!(!files.is_empty()); ``` ```python import parsing_rs index = parsing_rs.build_workspace_index("/path/to/workspace") print(index.file_count(), index.content_file_count()) print([(r.path, r.score) for r in index.search_files("auth", 5)]) ``` This is lexical, in-memory search only. It intentionally does not do semantic matching. Pre-sandbox `list_files` behavior (`potpie/parsing/src/git.rs`) is unchanged. --- ### Overview (`spec/modules/README.md`) --- id: SPEC-MODULES-INDEX title: Context Runtime Module Index kind: index revision: 1 maturity: draft owners: - team:potpie depends_on: [] change_ref: null --- # Context Runtime Module Index The module contracts divide root Potpie capability ownership, domain behavior, Potpie lease and resource management, process hosting, and presentation into separate boundaries. ## Read Order 1. [Potpie Capabilities](potpie-capabilities.md) 2. [Context Engine](context-engine.md) 3. [Potpie Resource Manager](potpie-resource-manager.md) 4. [Daemon](daemon.md) 5. [CLI](cli.md) Read [the system contract](../system.md) first for the controller path, hosted domain path, trust boundary, error taxonomy, and destructive-intent flow. ## Ownership Summary | Module | Owns | Excludes | |---|---|---| | Potpie capabilities | Root configuration, pots, skills, setup, identity lifecycle, cross-capability composition | Context Engine domain semantics, daemon protocol, CLI presentation, generic source-layer buckets | | Context Engine | Thin domain façade, one bound identity, focused domain modules, typed engine outcomes | Product selection, authorization policy, provisioning, daemon, presentation | | Resource Manager | Selection, authorization, host resources, explicit dependency composition, authorized context leases | Engine-operation dispatch, generic results, service lookup, daemon transport, presentation | | Daemon subsystem | External controller, one foreground runtime, discovery, authenticated readiness, typed handlers | Domain semantics, terminal behavior | | CLI | User intent, controller/client selection, human and machine presentation, streams and exits | Resource composition, daemon internals, domain semantics | ## Execution Direction ```text CLI -> typed daemon handler -> handler requests lease from Resource Manager -> Resource Manager issues lease -> handler invokes Context Engine ``` The Resource Manager is in the acquisition path but not the domain-operation dispatch path. Daemon context-domain handlers depend directly on the Context Engine façade for invocation and on the Resource Manager for lease acquisition. Context Engine does not import or depend on any Potpie product boundary. The Potpie capability contract makes the root ownership structure explicit without changing this execution direction. ## Lifecycle Direction ```text CLI -> external daemon controller -> foreground runtime CLI -> typed daemon client -> live readiness and runtime operations ``` Controller process observation never substitutes for the authenticated readiness handshake. ## Deferred Detail The exact public engine API, lease representation, Resource Manager cache policy, controller API, daemon implementation source and protocol, concurrency matrix, destructive-intent representation, machine JSON envelope, and compatibility sequence are recorded in [the question log](../questions/open.md). ## Conformance These are accepted target contracts. No module implementation or verification claim exists until a conformance record pins an accepted contract and implementation ref. --- ### Overview (`potpie/context-engine/src/potpie_context_engine/benchmarks/README.md`) # Context Engine Benchmarks A live, no-mock benchmark harness for the context engine. Every scenario: 1. Creates an **ephemeral pot** on a configured engine. 2. (If declared) loads the **canonical Acme universe seed** — services, team, ADRs, runbooks — through the same `/events/reconcile` path scenarios use. 3. Replays a sequence of **real recorded webhook payloads** (GitHub PRs, Linear issues, Slack messages, repo docs, alerts, deploys) plus any **distractor events** declared by the scenario, interleaved on the ingestion timeline. 4. Snapshots the resulting graph and asserts **ingestion / ontology quality** (right entities, right edges, no soft-downgrades, and — new — that the graph stays *clean* of distractor entities the scenario explicitly forbids). 5. Issues a `context_resolve` call and asserts **retrieval correctness** (cited the right events, used the right include keys, did not cite distractor events, and — for TIME scenarios — stayed inside the declared window). 6. Hands the response to an **Opus LLM judge** that scores synthesis quality against a per-scenario rubric, with optional per-dimension attribution for composite scenarios. 7. Drops the pot. The full design lives in `docs/context-graph/bench-plan.md`. This file documents how to *run* the bench; the plan documents what it *measures*. ## Why no mocks The previous benchmark mocked the reconciliation agent and used canned `MockIntelligenceProvider` responses, which is exactly what the headline 81 % score was measuring. The redesign rejects that: reconciliation is the engine's product, so the bench drives it for real. ## Use-case taxonomy (v3, 2026-05-20) Four **knowledge dimensions** + a composite modifier: | Code | What it tests | Default weights (ing / ret / syn) | |---|---|---| | `PREF` | Project preferences — conventions, ADRs, framework choices | 20 / 40 / 40 | | `INFRA` | Project infra / architecture — topology, owners, environments | 30 / 40 / 30 | | `TIME` | Timeline — recent changes, chronology, change-to-symptom links | 40 / 30 / 30 | | `BUG` | Bug repo — prior incidents, fixes, recurrence detection | 25 / 35 / 40 | | `COMBO` | Composite scenarios that exercise ≥ 2 of the above | 30 / 35 / 35 | `COMBO` scenarios declare `dimensions: [PREF, INFRA]` (etc.); the report breaks their score out per dimension. ## Layout ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Multi-source ingestion The engine ships production-grade readers for **GitHub** and **Linear**, and a minimal **Notion** reader. The four other connector kinds the bench uses — **Slack**, **Repo Docs**, **Alerting**, **Deploy** — are wired up as *passive stub connectors* (see `potpie/context-engine/adapters/outbound/connectors/_bench_stubs.py`). They emit a minimal `ReconciliationPlan` per envelope (one canonical entity, no edges) and let the reconciliation agent do the rest. The contract for swapping in a production reader later is the same `SourceConnectorPort` they implement. ## Two ways to run the engine The bench can drive the engine two ways: 1. **HTTP (default).** Talks to a running engine over `POTPIE_BENCH_API_URL` — which is the app's own API port (`:8001`). Needs gunicorn on 8001 + a Celery worker on the `context-graph-etl` queue. 2. **In-process (`--local` / `POTPIE_BENCH_INPROCESS=1`).** Builds the engine container in-process and reconciles **inline** against the shared Postgres/Neo4j — **no HTTP server on :8001 and no Celery worker required**. This keeps the bench non-blocking for the rest of the app (the app keeps 8001) and is fully self-contained. Ingestion is single-threaded and batched (submit-all → reconcile-once → read-back), which also sidesteps the threads-pool *"Event loop is closed"* reconciliation race. Implemented in `core/local_engine.py` (`InProcessEngineClient`). ```bash # In-process: needs only Postgres + Neo4j up (e.g. `make infra-up`) + OPENAI_API_KEY. python -m benchmarks run --local --tier quick --ingest-timeout 600 python -m benchmarks run --local --scenario infra_topology_basic --skip-judge ``` ## Configuration For the **HTTP** path the bench talks to a real engine. Required environment: | Variable | Purpose | |---|---| | `POTPIE_BENCH_INPROCESS` | `1` to run the engine in-process (no :8001 server, no worker). Same as `run --local`. | | `POTPIE_BENCH_API_URL` | (HTTP path) Engine base URL (e.g. `http://127.0.0.1:8001`). Falls back to `POTPIE_API_URL`. | | `POTPIE_BENCH_API_KEY` | (HTTP path) API key for an account that can create pots. Falls back to `POTPIE_API_KEY`. | | `OPENAI_API_KEY` | Used by the LLM judge (default model `gpt-5.4`). | | `POTPIE_BENCH_JUDGE_MODEL` | Optional. Override the judge model. Default `gpt-5.4`. | | `POTPIE_BENCH_REPO` | Optional. `owner/repo` to attach to each ephemeral pot. Default `acme/sandbox`. | | `CONTEXT_ENGINE_RECONCILIATION_MODEL` | Optional. Reconciliation-agent model on the engine side. Default `openai-responses:gpt-5.4-mini`. | ## Running ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Operational commands explained - **`smoke`** — runs every scenario through the bench-side pipeline (fixture resolution, evaluator math, reporting) against synthetic snapshots. No engine required. Sub-second. The right gate on every PR — catches harness regressions before the long run. - **`probe`** — pre-flight against the live engine. Confirms `/context/status` returns the expected connector kinds, then submits one minimal envelope per kind and waits ~15 s for it to leave `queued`. If reconciliation is dead this command finishes in seconds with a clear diagnostic instead of 11 minutes of timeouts mid-run. - **`run --concurrency N`** — pots are isolated by construction, so scenarios run in parallel safely. See "Concurrency & progress" below for how the two engine modes parallelize differently. - **`run-light`** — curated 5-scenario subset (one per dimension: PREF / INFRA / TIME / BUG / COMBO), 5-way parallel, **invariant** judging on. The fastest end-to-end signal we have. Use it as the smoke test before a full `run`. ## Schema-independent (invariant) grading The default judge grades the agent's answer against a per-scenario rubric whose criteria name specific fixture ids (OPS-218, ADR-021), and the deterministic axes grade against the engine's graph shape and the include vocabulary it currently happens to expose. Both move when the ontology / read trunk changes, even when the agent's answer is just as good. `run --invariant` (and the default `run-light`) replaces that with a single judge call that grades the answer against **only the scenario's input signal events** — no rubric, no fixture ids, no include vocab. Four 0..100 sub-scores: | Score | Question | |---|---| | `faithfulness` | Every concrete claim in the answer is supported by the events. Hallucinated identifiers / people / fixes collapse this score. | | `coverage` | The answer surfaces the facts a careful reader of the input events would consider essential to the question. | | `clarity` | The answer is structured so a working engineer can understand what happened in seconds. | | `usefulness` | The answer concretely helps the user do whatever the question implies (debug, decide, follow a convention, plan a change). | Aggregate: weighted mean (default 30/30/20/20 — faithfulness + coverage carry 60 %, so cosmetic gains can't paper over hallucinations or omissions). Pass score: 65. In invariant mode the ingestion + retrieval axes still run end-to-end (the engine has to actually ingest the events and answer the query) but they don't gate pass/fail and they don't enter the aggregate. They stay in the JSON report as diagnostics so you can still see e.g. "the engine is now ignoring this include key" alongside the answer-level score. The judge is configurable via `POTPIE_BENCH_INVARIANT_JUDGE_MODEL` (falls back to `POTPIE_BENCH_JUDGE_MODEL`, default `gpt-5.4`). ## Concurrency & progress **Progress logging.** A run prints a start banner (mode / concurrency / tier / judge), a `[i/N]` line per scenario with phase breadcrumbs (events ingested → reconcile time → graph snapshot size → retrieval → synthesis), a live `passed k/done` tally, and an end summary (total time, per-scenario pass/fail, slowest scenarios). `benchmarks.*` logs show by default; `-v` / `--verbose` additionally surfaces the chatty engine + HTTP loggers. **Concurrency model** (`--concurrency N`): | Engine mode | How `--concurrency N` parallelizes | Why | |---|---|---| | **HTTP** (default) | N **threads** fan scenarios out; each thread submits + polls over HTTP while reconciliation runs in the engine's Celery worker. Within a scenario, seeds + distractors submit via a 6-worker pool; signals stay ordered. | HTTP calls are thread-safe; the worker does the heavy LLM work. Throughput is bounded by the worker's pool (run it `--pool=solo` to dodge the agent's threads race). | | **In-process** (`--local`) | N **processes** (one scenario per subprocess). Within a scenario, ingestion is single-threaded (submit-all → reconcile-once → read-back). | The reconciliation agent does a per-batch `asyncio.run`; sharing a process across scenarios (threads) hits an "Event loop is closed" race. Separate processes give each scenario its own loop / agent / DB session and genuinely parallelize the LLM-bound reconciliation. | **Cost note.** The dominant per-scenario cost is the reconciliation agent — one LLM run over the scenario's events. The 12-event Acme universe is re-reconciled per scenario (e.g. ~4–5 min for a ~19-event scenario), and `hard`/`adversarial` scenarios add heavy distractors. So: prefer `--concurrency` (≈ linear speedup, bounded by CPU/LLM rate limits), keep the judge off (`--skip-judge`) for cheap dry runs, and dial distractor `count:` down if a run is too slow. (A future `--reuse-pot` that seeds the universe once is the biggest remaining win — see bench-plan §10.) ## Authoring a new scenario 1. Pick a use case (`PREF`, `INFRA`, `TIME`, `BUG`, or `COMBO`). 2. Capture or reuse the **raw event envelopes** the scenario depends on. See `fixtures/README.md`. 3. Write a `.yaml` under `use_cases//scenarios/`. Schema (everything past `judge:` is unchanged from v1; everything above is v3): ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Opt-in sub-axis gates Coverage and precision are reported as separate numbers on every scenario, but they do not block pass/fail by default. Scenarios that *care* about a specific recall or purity floor can declare it: ```yaml post_ingest_assertions: coverage_floor: 80 # ingestion axis fails if <80% of assertions match precision_floor: 90 # ingestion axis fails if forbidden entities slip in # ... structural assertions ... retrieval_assertions: coverage_floor: 100 # every must_cite event must actually be cited precision_floor: 80 # forbidden citations cost real points # ... must_cite_event_id etc. ... ``` Without these fields, sub-axes are diagnostic-only and never gate. Every scenario is graded on three axes (ingestion / retrieval / synthesis) plus two sub-axes (coverage / precision). The aggregate scenario score is the weighted mean of the primary axes; weights come from the use case unless `axis_weights:` overrides. ## Distractor generation Hand-authoring 25:1 distractor ratios is busywork. Use the generator: ```bash python -m benchmarks.tools.gen_distractors \ --template fixtures/raw_events/github/pr_merged__998__inventory_unrelated.json \ --count 25 \ --id-prefix noise_inventory_pr \ --out fixtures/raw_events/noise/inventory/ ``` Then reference the output directory in the scenario via a glob: ```yaml distractor_events: - { event: "noise/inventory/noise_inventory_pr__*.json", at: "-21d..-7d", count: 25 } ``` ## Tiers - **quick** — runs on every PR. Target 30 scenarios total. Each scenario should complete in < 90 s (excluding the LLM judge, which is opt-in via `--skip-judge` for cheap dry runs). - **extended** — runs nightly or before a release. No hard cap; expect minutes-to-hours. Target 60 scenarios. ## Reports Reports are JSON-first (`reports/.json`) with a `--format markdown` view for humans. Four aggregation panels appear in every report: - **By use case** — count, aggregate, ingestion / retrieval / synthesis, coverage, precision. - **By dimension** — composite scenarios decomposed across PREF / INFRA / TIME / BUG. - **By difficulty** — easy / medium / hard / adversarial curve. - **By source mix** — single / dual / full / adversarial curve. A baseline diff (`--baseline reports/main.json`, planned) highlights regressions per panel cell so a small synthesis dip on `hard` doesn't get hidden by a stable easy-tier retrieval score. --- ### Index (`docs/index.md`) --- title: Potpie description: Turn your codebase and software development lifecycle into a living context graph for AI agents. --- ## Overview [Potpie](https://potpie.ai) turns your codebase and software development lifecycle into a living context graph for AI agents. It indexes code, structure, decisions, source history, team knowledge and engineering workflows, so agents can answer questions, plan changes, debug failures, and write code with project-specific context. ## Start here - [Installation](./getting-started/installation.md) - [First steps](./getting-started/first-steps.md) - [Integrations and coding harnesses](./guides/integrations.md) - [CLI reference](./reference/cli.md) - [Context Graph](./context-graph/index.md) ## Architecture Potpie's current architecture is CLI-first. CLI is designed to be used by both humans and agents. Read the deeper architecture notes in [Context graph architecture](./context-graph/architecture.md). ## License This project is licensed under the Apache 2.0 License. See [LICENSE](https://github.com/potpie-ai/potpie/blob/main/LICENSE). ## Community and support - [GitHub Issues](https://github.com/potpie-ai/potpie/issues): bugs and repository-scoped requests - [Discord](https://discord.gg/ryk5CMD5v6): community discussion and support - [Docs](https://docs.potpie.ai): setup, product guides, and integration details New contributions are always welcome. Read the [Contributing Guide](https://github.com/potpie-ai/potpie/blob/main/.github/CONTRIBUTING.md) to set up your environment, understand the workflow, and open a pull request. --- ### Context Graph Index (`docs/context-graph/index.md`) --- title: Context Graph description: "Durable project memory for AI agents: claims, pots, and the graph workbench." --- ## Overview The Context Graph is Potpie's durable, shared **project memory for AI agents** — a compact store of sourced **claims** (decisions, ownership, infra topology, prior bugs/fixes, conventions, features) so an agent doesn't rebuild context from raw code, PRs, tickets, and chat on every task. Humans and agents talk to the **same `potpie` CLI**. The `potpie graph …` workbench is **shipped today** (data-plane contract `v1.5`, ontology `2026-06-graph`) alongside the legacy `resolve`/`search`/`record` compatibility wrappers — there is no separate "future V2." The same Pot Management, Graph, and Skill Manager service modules run inside either a local daemon or a managed backend API; **state stays local by default**. ```mermaid flowchart TB cg_human["Humans"] cg_agent["Agents
(Claude Code, Codex, Cursor…)"] cg_cli["potpie CLI
(graph workbench + V1 wrappers)"] cg_human --> cg_cli cg_agent --> cg_cli subgraph cg_local["Local profile — shipped, default"] direction TB cg_daemon["local daemon
(detached by default)"] cg_services["services:
Pot Mgmt · Graph · Skill Manager"] cg_backend["GraphBackend
(default falkordb_lite, embedded)"] cg_daemon --> cg_services --> cg_backend end subgraph cg_managed["Managed backend — roadmap"] direction TB cg_api["managed API
(same service modules)"] cg_hosted[("hosted stores")] cg_api --> cg_hosted end cg_ledger["Event Ledger
(source-event service: webhooks · polling · cursors)"] cg_cli --> cg_daemon cg_cli -. "managed routing
(CapabilityNotImplemented)" .-> cg_api cg_services -. "pull events (external clients are stubs)" .-> cg_ledger ``` ## Target OSS default **Published package** (PyPI): ```bash uv tool install potpie # recommended # or: pip install potpie potpie setup # provisions config, local stores, the default pot, the daemon, and skills potpie status ``` The base package is the complete local product (CLI, daemon, local graph backend). Semantic search with sentence-transformers is the optional `embeddings` extra (`uv tool install 'potpie[embeddings]'`); without it, setup falls back to the bundled hashing embedder. **Repo-local development** (this checkout): use `make cli-install` — it builds the graph-explorer UI, stops any old daemon, and installs the editable CLI. Do not use raw `uv tool install --editable ./potpie/context-engine` for day-to-day reinstalls. ```bash make cli-install make cli-status # confirm PATH / uv-tool install potpie setup potpie status ``` `setup` also registers your repo as a source. It does not scan the working tree — harness-led ingestion writes the graph. The OSS/CLI default backend is **`falkordb_lite`** — an embedded FalkorDB over a local file, with **no Docker, server, Neo4j, or cloud key required**; override it with `--backend` or `CONTEXT_ENGINE_BACKEND` (precedence: `CONTEXT_ENGINE_BACKEND` > legacy `GRAPH_DB_BACKEND` > `falkordb_lite`). Full flags live in [`cli-flow.md`](./cli-flow.md). > **Roadmap (not yet wired):** Managed-backend routing is designed but not > functional — `pot use --managed`, `pot list --managed`, and the whole `cloud` > group raise `CapabilityNotImplemented`. The **external** Event Ledger clients > (`ledger pull/query` against real providers) are TODO stubs. The live "ledger" > today is the internal Postgres event store described in > [`ingestion-nudge.md`](./ingestion-nudge.md). ## Start here | Doc | What it answers | |---|---| | [`vision.md`](./vision.md) | What the Context Graph is and why; claims-not-payloads; harness-owned intelligence; the three product boundaries (local OSS / managed [roadmap] / Event Ledger [roadmap]); pots-as-tenancy; anti-goals. | | [`architecture.md`](./architecture.md) | Hexagonal layers; the two composition roots (local agent spine vs ingestion server); the daemon model; the `GraphBackend` port + 6 capabilities + the backend coverage table; per-pot scoping, the resource store for document payloads, and backend selection. | | [`ontology.md`](./ontology.md) | The three declarative catalogs (25 entity types / 27 predicates + `RELATED_TO` / record types); contract constants (versions, 7 truth classes, 10 mutation ops, 6 source authorities); 8 subgraphs / 10 views; identity keys and the environment qualifier. | | [`querying.md`](./querying.md) | Reading: the CLI Graph Surface Lite and compatibility commands; the single read trunk and 10 readers; the `AgentEnvelope` (ranked evidence, no server-side answers); ranking; the 3-axis model (Retrieve / Filter / Traverse — all shipped). | | [`writing.md`](./writing.md) | Writing: the flat semantic-mutation DSL (10 ops); validation + runtime risk; the canonical write door `graph propose` → `graph commit --verify` (with `graph mutate` and `record` as the legacy wrappers); revision-based concurrency and receipts; inbox; quality. | | [`ingestion-nudge.md`](./ingestion-nudge.md) | How raw episodes/events enter; the internal Postgres event store vs the external Event Ledger seam; connectors (github/notion only); windowed reconciliation (off by default); the zero-token nudge trigger model. | | [`skills.md`](./skills.md) | Harness-owned intelligence; the bundled CLI skills (potpie-graph teaches propose/commit); how an agent acts on nudges; the separate server-side reconciliation skill surface (not the same thing). | | [`cli-flow.md`](./cli-flow.md) | The full `potpie` command reference, grouped, with flags, exit-code contract, snapshots, and the canonical journey. | | [`observability.md`](./observability.md) | What logs, traces, metrics, and readiness report; span names. | | [`bench-plan.md`](./bench-plan.md) | How graph quality is validated across backends (invariant judge, `run-light`). | ## Vocabulary | Term | Meaning | |---|---| | **Pot** | Unit of isolation/tenancy. Every query, source, inbox item, claim, semantic mutation, and graph operation is scoped to one pot; the pot id **is** the storage `group_id`. A pot is local or managed; the active pot determines routing. Cross-pot federation is an anti-goal. | | **Daemon** | Local background process (`potpie/daemon/lifecycle.py`) for lifecycle, IPC, health, and logs — **not** the business layer. Default host mode is detached (`daemon`); it also serves the read-only `potpie ui` explorer. | | **Services** | Root capability services own pots, sources, setup, skills, and local identity; Graph Service owns graph reads, semantic mutations, and the workbench. Runtime composition assembles them for the local daemon. | | **GraphBackend** | Swappable capability bundle of 6 ports — canonical `mutation` + `claim_query`, plus rebuildable projections `semantic`, `inspection`, `analytics`, `snapshot` — with `capabilities()`/`provision()`. Default profile `falkordb_lite`. | | **Skill Manager** | CLI-managed skill catalog/installer for agent harnesses. Skills teach agents how to use the CLI; they are not graph facts or new tools. | | **Event Ledger** | Separate managed-or-self-hostable source-event service (webhooks, polling, replay cursors). Graph consumers *pull* and track their own cursor/apply state; the ledger is **not** the graph source of truth. (External clients are stubs — roadmap.) | | **Claim** | A compact, sourced fact — the graph stores claims + source refs, **never** full payloads (diffs, doc bodies, transcripts). Each claim carries a truth class that feeds the ranker. | | **Retrieval card** | The single text a claim is embedded and searched as (agent-authored `description` leads). One builder is shared by the embed-on-write and read paths, so retrieval quality tracks the description the agent writes. | The active package lives under [`potpie/context-engine/`](https://github.com/potpie-ai/potpie/tree/main/potpie/context-engine). Where any older note disagrees with these docs, these docs (code-verified on `main`) win. --- ### Async Migration Plan (`docs/async-migration-plan.md`) --- title: Async migration plan description: Step-by-step plan to replace blocking libraries with native async equivalents. --- ## Context The previous PRs used `asyncio.to_thread` to wrap blocking calls. The code review flagged: - Thread-safety bugs (DB sessions passed to worker threads) - Unbounded queue in custom stream generator - Daemon thread-per-stream with no cap - `StreamingResponse` with sync generator already runs in Starlette's thread pool (~40 tokens) — the custom thread+queue was unnecessary overhead - Thread pool saturation risk under load **New direction:** Replace blocking libraries with native async equivalents. Use `to_thread` only as a temporary bridge for code that cannot be converted yet. Measure every step. --- ## Step 0: Revert thread-based changes **Branch:** `fix/wait-for-task-start-thread` (5 commits) Revert these changes that introduced `asyncio.to_thread` wrappers and the custom stream generator: | File | What to revert | |------|----------------| | `conversation_routing.py` | Remove `redis_stream_generator_async`, revert `start_celery_task_and_stream` to sync, remove `asyncio.to_thread` around `ensure_unique_run_id` and `wait_for_task_start` | | `conversations_router.py` | Remove `asyncio.to_thread` wrappers around `session_service`, `redis_manager`, `ensure_unique_run_id`; restore `local_mode` / request / UA detection in `create_conversation` | | `conversation_service.py` | Remove `asyncio.to_thread` around `history_manager.*`, `session_service.*`, `redis_manager.*`, `celery_app.control.revoke` | | `redis_streaming.py` | Revert `stop_event` parameter, revert xread block from 1000ms back to 5000ms. Keep `require_running` in `wait_for_task_start` (that's a correctness fix, not a threading change) | | `api/router.py` | Remove `asyncio.to_thread` around `ensure_unique_run_id` | | `tunnel_router.py` | Remove `asyncio.to_thread` wrappers | | `tunnel_service.py` | Revert scan_iter and TTL changes only if they were part of the thread PR. Keep if they were independent fixes | | `socket_service.py` | Remove `close()` method if it was added as part of the thread PR | | `main.py` | Remove shutdown handler if it was part of the thread PR | | `github_service.py` | Remove `asyncio.to_thread` around `redis.get`/`redis.setex` in `get_project_structure_async` | **Keep from the thread PR:** - `wait_for_task_start(require_running=True)` — correctness fix, not a threading change - Timeout additions (timeout=30 on requests calls) — safety fix **Keep from `fix/sync-timeouts-only`:** - All timeout additions (auth_service, github_service, parse_webhook_helper) - `parsing_controller._query_parsing_status_sync` with `asyncio.to_thread` — acceptable bridge until async DB migration --- ## Step 1: AsyncRedisStreamManager for FastAPI **What:** Create `redis.asyncio` client alongside the existing sync client. Add async methods for all operations called from FastAPI. **Files to change:** - `redis_streaming.py` — add `AsyncRedisStreamManager` class using `redis.asyncio` - `async def get_task_status()` - `async def set_task_status()` - `async def set_task_id()` - `async def get_task_id()` - `async def publish_event()` - `async def set_cancellation()` - `async def get_stream_snapshot()` - `async def clear_session()` - `async def wait_for_task_start()` — use `asyncio.sleep` instead of `time.sleep` - `conversation_routing.py` — use `AsyncRedisStreamManager` in `start_celery_task_and_stream` (make it `async def`), `start_celery_task_and_wait` - `ensure_unique_run_id` → `async_ensure_unique_run_id` using async Redis - Leave sync `redis_stream_generator` as-is (Starlette thread pool handles it) - `session_service.py` — add `AsyncSessionService` using `AsyncRedisStreamManager`. Use `await async_redis.scan(...)` instead of `keys()` - `conversations_router.py` — switch to async versions - `api/router.py` — switch to async versions - `conversation_service.py` — `stop_generation` uses async Redis methods **Keep unchanged:** - Sync `RedisStreamManager` — still used by Celery tasks (`agent_tasks.py`, `parsing_tasks.py`) - Sync `redis_stream_generator` — Starlette runs it in thread pool **Test script: `scripts/benchmark_redis_async.py`** ``` Measures: 1. Time to complete N concurrent POST /message requests (streaming) 2. Time to complete N concurrent GET /active-session requests 3. Time to complete N concurrent GET /task-status requests 4. Compare: before (sync Redis on event loop) vs after (native async Redis) 5. Report: p50, p95, p99 latency; requests/sec; event loop blocked time (via middleware) ``` **PR:** `feat/async-redis-stream-manager` --- ## Step 2: AsyncSession for hot-path DB services **What:** Migrate services called from FastAPI to use `AsyncSession`. ### 2a: UsageService **Why first:** Called on every `create_conversation` and `post_message` (check_usage_limit). Highest frequency. - `usage_service.py` — replace `SessionLocal()` + `session.query()` with `AsyncSessionLocal` + `await async_session.execute(select(...))` - Inject `AsyncSession` from FastAPI dependency **Test script: `scripts/benchmark_usage_check.py`** ``` Measures: 1. N concurrent POST /conversations (each triggers check_usage_limit) 2. Latency distribution before/after ``` **PR:** `feat/async-usage-service` ### 2b: ChatHistoryService (dual-path) ✅ **Why:** Used from both FastAPI (store_message, stop_generation) and Celery (agent_tasks). Cannot just swap to async. - Create `AsyncChatHistoryService` with `AsyncSession`: - `async def get_session_history()` - `add_message_chunk()` (in-memory buffer only) - `async def flush_message_buffer()` - `async def save_partial_ai_message()` - `ConversationService.create()` — accepts optional `async_db`; when provided, builds `AsyncChatHistoryService` and uses it on FastAPI path - FastAPI path uses async variant; Celery path uses existing sync `ChatHistoryService` **Test script: `scripts/benchmark_chat_flow.py`** ``` Measures: 1. N concurrent POST /message (streaming) — end-to-end including DB writes 2. N concurrent POST /stop — stop_generation with DB save 3. Latency distribution ``` **PR:** `feat/async-chat-history-service` ### 2c: ShareChatService, AccessService, UserService ✅ - **AsyncShareChatService**: New class in `access_service.py` using `AsyncSession`; `share_chat`, `get_shared_emails`, `remove_access` (select/update/commit). - **conversations_router**: Share/access endpoints use `get_async_db` and `AsyncShareChatService(async_db)`. - **AsyncUserService**: New class in `user_service.py`; async `get_user_by_uid`, `get_user_id_by_email`, `get_user_by_email`, `get_user_ids_by_emails`, `create_user`, `update_last_login`. Sync `UserService` retained for Celery/main. - **api/router**: `get_api_key_user` uses `AsyncUserService(async_db).get_user_by_uid`. Auth routes (2d) remain on sync UserService for now. **Test script: `scripts/benchmark_share_access.py`** ``` Measures: 1. N concurrent POST /share 2. N concurrent GET /shared-emails 3. N concurrent DELETE /access ``` **PR:** `feat/async-share-access-user-services` ### 2d: Auth routes (signup, sso_login, provider endpoints) ✅ - auth_router.py: inject `get_async_db`, use `AsyncUserService(async_db)` for user lookups (get_user_by_uid, get_user_by_email, update_last_login) in signup, sso_login, get_my_account - `UnifiedAuthService` remains sync for this PR; full async migration can follow **Test script: `scripts/benchmark_auth.py`** ``` Measures: 1. N concurrent POST /signup 2. N concurrent POST /sso/login 3. N concurrent GET /providers/me 4. Latency distribution ``` **PR:** `feat/async-auth-db` ### 2e: GithubService DB queries ✅ - `get_combined_user_repos(user_id, async_session=...)` and `get_repos_for_user(user_id, async_session=...)` accept optional `AsyncSession`; when provided use `select()` + `await session.execute()` - `code_provider_controller.get_user_repos` and github_router pass `async_db` into `get_combined_user_repos` **PR:** `feat/async-github-db` (included in feat/async-share-access-user-services) --- ## Step 3: Async Redis for non-streaming services ✅ **What:** Add `redis.asyncio` to services that do simple Redis get/set from FastAPI. | Service | Change | |---------|--------| | `tunnel_service.py` | ✅ Async client + `get_workspace_tunnel_record_async`, `set_workspace_tunnel_record_async`, `list_user_tunnels_async`. Sync `list_user_tunnels` uses `scan_iter` not `keys`. Workspace record `set` uses TTL (`WORKSPACE_TUNNEL_RECORD_TTL`). | | `tunnel_router.py` | ✅ Both routes call `await tunnel_service.get_workspace_tunnel_record_async(workspace_id)`. | | `github_service.py` | ✅ Lazy shared `_get_async_redis_cache()`; `get_project_structure_async` uses async Redis for cache get/setex when available. | | `branch_cache.py` | ✅ Optional `_async_redis_client`; `get_branches_async` for FastAPI path. Controller `get_branch_list` uses `await branch_cache.get_branches_async`. | **Test script: `scripts/benchmark_tunnel_github.py`** ``` Measures: 1. N concurrent GET /tunnel/workspace/{id} 2. N concurrent GET /tunnel/workspace/{id}/socket-status 3. N concurrent GET /github/user-repos (triggers branch cache / repo listing) 4. Latency distribution (p50, p95, p99); req/s Usage: WORKSPACE_ID=<16-hex> AUTH_HEADER="Bearer " CONCURRENT=5 ROUNDS=2 uv run python scripts/benchmark_tunnel_github.py ``` **PR:** `feat/async-redis-tunnel-github` --- ## Step 4: Async HTTP clients ✅ **What:** Replace `requests` with `httpx` where called from FastAPI; offload sync-only SDKs to threads. | Location | Change | |----------|--------| | `auth_service.py` | Sync login uses `httpx.Client`; added `login_async` with `httpx.AsyncClient` (timeout 10s/30s). Route calls `await auth_handler.login_async`. | | `auth_router.py` | `send_slack_message` uses `httpx.AsyncClient().post` with timeout. | | `parse_webhook_helper.py` | `send_slack_notification` uses `httpx.AsyncClient().post` with timeout. | | `linear_client.py` | Sync `execute_query` uses `httpx.Client` (timeout 10s/30s); added `execute_query_async`, `get_issue_async`, `update_issue_async`, `comment_create_async`. Linear tools use async methods. | | `email_helper.py` | `resend.Emails.send` wrapped in `await asyncio.to_thread(resend.Emails.send, params)` in both `send_email` and `send_parsing_failure_alert`. | | `posthog_helper.py` | `send_event`: when event loop is running, uses `loop.run_in_executor(None, _capture_sync)` (fire-and-forget); otherwise runs `_capture_sync` inline. | **Test script: `scripts/benchmark_login.py`** (optional; not added in this PR) **PR:** Combined with Step 3 in `feat/async-redis-tunnel-github` --- ## Step 5: SearchService async DB - `search_service.py` — migrate `search_codebase` to `AsyncSession` - `knowledge_graph_router.py` — use async session for project verification query **PR:** `feat/async-search-service` --- ## Test infrastructure ### Shared benchmark harness: `scripts/benchmark_harness.py` ```python """ Usage: python scripts/benchmark_harness.py --target