{"owner":"vectorize-io","repo":"hindsight","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","CLAUDE.md"],"files":{"AGENTS.md":"# AGENTS.md\n\nSee [CLAUDE.md](./CLAUDE.md) for project documentation and coding conventions.\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nHindsight is an agent memory system that provides long-term memory for AI agents using biomimetic data structures. Memories are organized as:\n- **World facts**: General knowledge (\"The sky is blue\")\n- **Experience facts**: Personal experiences (\"I visited Paris in 2023\")\n- **Mental models**: Consolidated knowledge synthesized from facts (\"User prefers functional programming patterns\")\n\n## Development Commands\n\n### Local Development (API + UI)\n```bash\n# Start both API server and control plane UI\n./scripts/dev/start.sh\n```\n\n### API Server (Python/FastAPI)\n```bash\n# Start API server only (loads .env automatically)\n./scripts/dev/start-api.sh\n\n# Run all tests (parallelized with pytest-xdist)\ncd hindsight-api-slim && uv run pytest tests/\n\n# Run specific test file\ncd hindsight-api-slim && uv run pytest tests/test_http_api_integration.py -v\n\n# Run single test function\ncd hindsight-api-slim && uv run pytest tests/test_retain.py::test_retain_simple -v\n\n# Lint and format\ncd hindsight-api-slim && uv run ruff check .\ncd hindsight-api-slim && uv run ruff format .\n\n# Type checking (uses ty - extremely fast type checker from Astral)\ncd hindsight-api-slim && uv run ty check hindsight_api/\n```\n\n### Control Plane (Next.js)\n```bash\n./scripts/dev/start-control-plane.sh\n# Or manually:\ncd hindsight-control-plane && npm run dev\n```\n\n### Documentation Site (Docusaurus)\n```bash\n./scripts/dev/start-docs.sh\n```\n\n\n### Generating Clients/OpenAPI\n```bash\n# Regenerate OpenAPI spec after API changes (REQUIRED after changing endpoints)\n./scripts/generate-openapi.sh\n\n# Regenerate all client SDKs (Python, TypeScript, Rust)\n./scripts/generate-clients.sh\n```\n\n### Benchmarks\n```bash\n# Accuracy benchmarks\n./scripts/benchmarks/run-longmemeval.sh\n./scripts/benchmarks/run-locomo.sh\n\n# Performance benchmarks\n./scripts/benchmarks/run-perf-test.sh                      # System perf (mock LLM + pg0)\n./scripts/benchmarks/run-perf-test.sh --scale tiny          # Quick smoke test\n./scripts/benchmarks/run-consolidation.sh\n\n# Results viewer\n./scripts/benchmarks/start-visualizer.sh  # View results at localhost:8001\n```\n\n## Architecture\n\n### Monorepo Structure\n- **hindsight-api-slim/**: Core FastAPI server with memory engine (Python, uv)\n- **hindsight-control-plane/**: Admin UI (Next.js, npm)\n- **hindsight-cli/**: CLI tool (Rust, cargo, uses progenitor for API client)\n- **hindsight-clients/**: Generated SDK clients (Python, TypeScript, Rust)\n- **hindsight-docs/**: Docusaurus documentation site\n- **hindsight-integrations/**: Framework integrations (LiteLLM, CrewAI, LangGraph, Pydantic AI, AG2, Claude Code, etc.)\n- **hindsight-dev/**: Development tools and benchmarks\n\n### Core Engine (hindsight-api-slim/hindsight_api/engine/)\n- `memory_engine.py`: Main orchestrator for retain/recall/reflect operations\n- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, VertexAI, Groq, MiniMax, Ollama, LM Studio, LiteLLM, Claude Code\n- `embeddings.py`: Embedding generation (local sentence-transformers or TEI)\n- `cross_encoder.py`: Reranking (local or TEI)\n- `entity_resolver.py`: Entity extraction and normalization\n- `query_analyzer.py`: Query intent analysis\n\n**retain/**: Memory ingestion pipeline\n- `orchestrator.py`: Coordinates the retain flow\n- `fact_extraction.py`: LLM-based fact extraction from content\n- `link_utils.py`: Entity link creation and management\n\n**search/**: Multi-strategy retrieval\n- `retrieval.py`: Main retrieval orchestrator\n- `graph_retrieval.py`: Graph retrieval abstract base class\n- `link_expansion_retrieval.py`: Link expansion graph retrieval\n- `fusion.py`: Reciprocal rank fusion for combining results\n- `reranking.py`: Cross-encoder reranking\n\n### API Layer (hindsight-api-slim/hindsight_api/api/)\n- `http.py`: FastAPI HTTP routers for all REST endpoints\n- `mcp.py`: Model Context Protocol server implementation\n\nMain operations:\n- **Retain**: Store memories, extracts facts/entities/relationships\n- **Recall**: Retrieve memories via 4 parallel strategies (semantic, BM25, graph, temporal) + reranking\n- **Reflect**: Disposition-aware reasoning using memories and mental models.\n\n### Database\nPostgreSQL with pgvector. Schema managed via Alembic migrations in `hindsight-api-slim/hindsight_api/alembic/`. Migrations run automatically on API startup.\n\nKey tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`\n\n### Adding Database Migrations\n\nHindsight runs the same Alembic tree against PostgreSQL and Oracle 23ai. Each\nmigration file dispatches through `run_for_dialect`, which calls either\n`_pg_upgrade` or `_oracle_upgrade` based on the live connection. A pytest lint\n(`tests/test_migration_shape.py`) fails CI if a migration omits the dispatcher.\n\n1. **Create a new migration file** in `hindsight-api-slim/hindsight_api/alembic/versions/`:\n   - File name format: `<revision_id>_<description>.py` (e.g., `f1a2b3c4d5e6_add_new_index.py`)\n   - Use a unique hex revision ID (12 chars)\n   - Set `down_revision` to the previous migration's revision ID\n\n2. **Migration template** (the `script.py.mako` template scaffolds this; fill in the bodies):\n   ```python\n   \"\"\"Description of the migration\n\n   Revision ID: f1a2b3c4d5e6\n   Revises: <previous_revision_id>\n   Create Date: YYYY-MM-DD\n   \"\"\"\n   from collections.abc import Sequence\n   from alembic import context, op\n\n   from hindsight_api.alembic._dialect import run_for_dialect\n\n   revision: str = \"f1a2b3c4d5e6\"\n   down_revision: str | Sequence[str] | None = \"<previous_revision_id>\"\n   branch_labels: str | Sequence[str] | None = None\n   depends_on: str | Sequence[str] | None = None\n\n\n   def _pg_schema_prefix() -> str:\n       \"\"\"Schema-qualifier for raw SQL on PG (multi-tenant search_path).\"\"\"\n       schema = context.config.get_main_option(\"target_schema\")\n       return f'\"{schema}\".' if schema else \"\"\n\n\n   def _pg_upgrade() -> None:\n       schema = _pg_schema_prefix()\n       op.execute(f\"CREATE INDEX ... ON {schema}table_name(...)\")\n\n\n   def _pg_downgrade() -> None:\n       schema = _pg_schema_prefix()\n       op.execute(f\"DROP INDEX IF EXISTS {schema}index_name\")\n\n\n   def _oracle_upgrade() -> None:\n       # Oracle 23ai equivalent. Use op.get_bind().exec_driver_sql for forms\n       # that Alembic core does not model (vector/text indexes, partitions).\n       op.execute(\"CREATE INDEX ... ON table_name(...)\")\n\n\n   def _oracle_downgrade() -> None:\n       op.execute(\"DROP INDEX IF EXISTS index_name\")\n\n\n   def upgrade() -> None:\n       run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)\n\n\n   def downgrade() -> None:\n       run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)\n   ```\n\n   **Dialect-only migrations.** If a change genuinely doesn't apply to one\n   dialect (e.g. enabling `pg_trgm` is PG-only), omit the unused slot:\n   ```python\n   def upgrade() -> None:\n       run_for_dialect(pg=_pg_upgrade)  # oracle slot intentionally absent → no-op\n   ```\n   Make the asymmetry deliberate. Don't leave an Oracle slot empty just because\n   you didn't think about it — copy-pasting a PG migration without the Oracle\n   half is exactly how schemas drift.\n\n3. **Run migrations locally**:\n   ```bash\n   # Set database URL and run migrations for the base schema plus all tenants\n   uv run hindsight-admin run-db-migration\n\n   # Run on a specific tenant schema\n   uv run hindsight-admin run-db-migration --schema tenant_xyz\n   ```\n\n## Key Conventions\n\n### Code Quality\n\n**Before writing code, read `.claude/skills/code-review/SKILL.md`** for the full coding standards (Python style, type safety, TypeScript style, general principles).\n\n**Always run the lint script after making Python or TypeScript/Node changes:**\n```bash\n./scripts/hooks/lint.sh\n```\n\nDead-code detection runs in CI (the `check-unused-code` job) at two levels:\n- **Blocking:** unused imports (ruff `F401`) and variables (`F841`) — `lint.sh` auto-removes\n  them and `verify-generated-files` fails on any leftover diff; and **knip** for orphaned\n  control-plane files / unused (or unlisted) `package.json` dependencies.\n- **Advisory:** whole unused Python functions (vulture) and unused control-plane *exports*\n  (the shadcn/ui surface is kept on purpose) — surfaced, not gated.\n\nRun both locally with:\n```bash\n./scripts/hooks/check-unused.sh\n```\n\n**After completing any implementation work, run `/code-review`** to verify your changes against project standards (missing tests, dead code, type safety, etc.). Fix any \"must fix\" issues before considering the task done.\n\n**MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all \"must fix\" issues are resolved.\n\n### Testing\n\nMost tests are deterministic (MockLLM, pure functions) — assert directly.\n\n**Tests that verify LLM behaviour use a real LLM + an LLM-as-judge.** When the thing under test is *how the model interprets a prompt* (classification, attribution, dimension preservation, instruction-following), MockLLM can't simulate it and exact string/enum asserts flake across providers and runs. Use this pattern instead:\n\n1. Mark the test module `pytestmark = pytest.mark.hs_llm_core` (single-provider; CI runs it in the core-LLM job). Use `hs_llm_mat` only for provider-matrix acceptance tests.\n2. Call the real pipeline (`LLMConfig.from_env()`, `_get_raw_config()`), e.g. `extract_facts_from_text(...)`.\n3. Assert with the judge, not string matching:\n   ```python\n   from tests.llm_judge import assert_meets_criteria\n   facts_summary = \"\\n\".join(f\"- [{f.fact_type}] {f.fact}\" for f in facts)\n   await assert_meets_criteria(\n       response=facts_summary,\n       criteria=\"The first-person user statements are classified 'world' and attributed to the user, not the agent.\",\n       context=\"What the input said and who was speaking.\",\n   )\n   ```\n\nRules of thumb:\n- **Judge anything non-deterministic** — including `fact_type` classification and speaker attribution. Do NOT hard-assert `fact_type == \"...\"`; pass a `[fact_type] fact` summary to the judge instead. Structural facts that ARE deterministic (counts, presence of a field, that a substring was injected into a prompt) stay as direct asserts in fast unit tests.\n- **Split the test surface**: cover the deterministic mechanics (prompt assembly, suppression logic) with fast non-LLM unit tests, and the model-following behaviour with one `hs_llm_core` judge test. (Example pair: `test_narrator_resolution.py` + `test_narrator_context_override.py`.)\n- The judge model is independent of the test provider (defaults to Gemini); never judge with the same call you're testing.\n\n### Memory Banks\n- Each bank is an isolated memory store (like a \"brain\" for one user/agent)\n- Banks have dispositions (skepticism, literalism, empathy traits 1-5) affecting reflect\n- Banks can have background context\n- Bank isolation is strict - no cross-bank data leakage\n\n### API Design\n- All endpoints operate on a single bank per request\n- Multi-bank queries are client responsibility to orchestrate\n- Disposition traits only affect reflect, not recall\n\n### Control Plane API Routes\n\nWhen adding or modifying parameters in the dataplane API (hindsight-api), you must also update the control plane routes that proxy to it:\n\n1. **API Routes** (`hindsight-control-plane/src/app/api/`):\n   - `recall/route.ts` - proxies to `/v1/default/banks/{bank_id}/memories/recall`\n   - `reflect/route.ts` - proxies to `/v1/default/banks/{bank_id}/reflect`\n   - `memories/retain/route.ts` - proxies to `/v1/default/banks/{bank_id}/memories/retain`\n   - Other routes follow the same pattern\n\n2. **Client types** (`hindsight-control-plane/src/lib/api.ts`):\n   - Update the TypeScript type definitions for `recall()`, `reflect()`, `retain()` etc.\n\n3. **Checklist when adding new API parameters**:\n   - Add parameter extraction in the route handler (destructure from `body`)\n   - Pass the parameter to the SDK call\n   - Update the client type definition in `lib/api.ts`\n   - Update any UI components that need to use the new parameter\n\n### Harness Attribution (which coding agent wrote a document)\n\n`hindsight-integrations/hindsight-coding-agents/` stamps the coding agent on every\ndocument it retains, so the control plane can show its logo instead of another\n`key=value` chip:\n\n- `metadata.harness = \"<id>\"` — the authoritative field\n- tag `harness:<id>` — the same value, so the documents list can filter on it\n\nThe ids are defined by that integration's HookSpecs\n(`src/harness/hook-lifecycle.ts`) plus the persistent-plugin entrypoints\nregistered in `src/harness/registry.ts`, whose id is their\n`createPluginEntry(...)` argument — currently `antigravity-cli`, `claude-code`,\n`cline-cli`, `codex`, `copilot-cli`, `cursor-cli`, `devin-cli`, `grok-build`,\n`kilo`, `opencode`.\n\nThe control plane resolves the value in\n`hindsight-control-plane/src/lib/harness-logo.ts` (metadata wins over the tag) and\nrenders it with `components/ui/harness-logo.tsx` in the documents table and the\ndocument detail dialog. **Adding a harness to the integration means adding it to\nthat registry in the same change**: copy its icon from\n`hindsight-docs/static/img/icons/` (or take it from the agent's own brand assets\nwhen the docs site carries none) into\n`hindsight-control-plane/public/img/harness/` and add one entry. Don't register\nids nothing writes — a test asserts the registry matches the emitted set, plus an\nexplicit list of retired ids kept so already-retained documents keep their logo.\nAn unregistered harness is not an error: it renders no logo and still shows as\nordinary metadata.\n\n### Adding New Integrations\n\nEvery new integration in `hindsight-integrations/` must satisfy all of the following before it can be merged:\n\n1. **Tests are required** — tests must simulate or exercise the external system (mock the framework's interfaces and verify the integration actually calls Hindsight correctly). Pure unit tests of helper functions are not sufficient.\n2. **CI job** — add a test job in `.github/workflows/test.yml` following the existing pattern (e.g., `test-crewai-integration`). The job must build, install deps, and run `uv run pytest tests -v`. Also add the integration to `detect-changes` outputs so it only runs when its files change.\n3. **Release process** — add the integration name to the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh` so it can be released via the standard release workflow.\n4. **Follow project code standards** — Python style, type safety, no raw dicts for structured data, no multi-item tuple returns (see `.claude/skills/code-review/SKILL.md`).\n\nIf any of these are missing, the integration is incomplete and must not be pushed or merged.\n\n### Changelogs\n\nNever add \"Unreleased\" entries to changelogs (e.g. `hindsight-docs/src/pages/changelog/**`). Changelog entries are written by the release script (`./scripts/release-integration.sh`) when a version is actually cut. If a bug fix or feature needs documenting before release, describe it in the PR/commit — the release tooling will surface it in the published changelog section.\n\n### Adding New API Configuration Flags\n\nConfiguration follows a hierarchical system: **Global (env vars) → Tenant (via extension) → Bank (database)**.\n\nFields must be categorized as either **hierarchical** (can be overridden per-tenant/bank) or **static** (server-level only).\n\n#### Adding a New Configuration Field\n\n1. **config.py** (`hindsight-api-slim/hindsight_api/config.py`):\n   - Add `ENV_*` constant for the environment variable name (e.g., `ENV_MY_SETTING = \"HINDSIGHT_API_MY_SETTING\"`)\n   - Add `DEFAULT_*` constant for the default value\n   - Add field to `HindsightConfig` dataclass with type annotation\n   - **Mark as configurable** by adding to `_CONFIGURABLE_FIELDS` set if the field should be overridable per-tenant/bank via API\n   - Add initialization in `from_env()` method\n\n   ```python\n   # Configurable field (can be overridden per-tenant/bank via API)\n   _CONFIGURABLE_FIELDS = {\n       ...,\n       \"my_setting\",  # Add here for configurable\n   }\n\n   # Static field - just don't add to _CONFIGURABLE_FIELDS\n   ```\n\n2. **main.py** (`hindsight-api-slim/hindsight_api/main.py`):\n   - No change is needed for ordinary environment-backed config fields. The CLI starts from `_get_raw_config()`,\n     so new `HindsightConfig` fields are carried through automatically.\n   - If the new field should be overridable by a CLI flag, add the argparse option in `_parse_cli_args()` and include\n     that field in the `dataclasses.replace(config, ...)` call near the \"CLI override\" comment.\n\n3. **Use hierarchical config in MemoryEngine**:\n   ```python\n   # Config is resolved automatically per bank via ConfigResolver\n   config_dict = await self._config_resolver.get_bank_config(bank_id, context)\n   value = config_dict[\"my_setting\"]\n   ```\n\n4. **Use static config** (non-hierarchical):\n   ```python\n   from ...config import get_config\n   config = get_config()\n   value = config.my_static_field\n   ```\n\n5. **Documentation** (`hindsight-docs/docs/developer/configuration.md`):\n   - Add to appropriate section table with Variable, Description, Default\n   - Mark if it's hierarchical (can be overridden per-bank)\n\n6. **Env template** (`.env.example`):\n   - Add the variable to the appropriate section, commented if optional, with a\n     short inline comment describing it (mirror the documentation entry).\n   - This file is the single source of truth for the env template:\n     `scripts/dev/setup.sh` copies it to `.env`, and `hindsight-embed` ships a\n     bundled copy (`hindsight-embed/hindsight_embed/env.example`) that seeds\n     embed/profile configs. After editing `.env.example`, re-copy it to the\n     embed package (`cp .env.example hindsight-embed/hindsight_embed/env.example`)\n     or the `test_bundled_template_matches_repo_root` sync test will fail.\n\n#### Hierarchical vs Static Guidelines\n\n**Hierarchical** (per-bank overridable):\n- LLM settings (provider, model, API key, base URL)\n- Operation-specific settings (retain mode, chunk size, etc.)\n- Feature flags that vary by customer/bank\n\n**Static** (server-level only):\n- Infrastructure settings (database URL, port, host)\n- Global limits (max concurrent operations)\n- System-wide feature flags\n\n## Environment Setup\n\n```bash\ncp .env.example .env\n# Edit .env with the LLM provider/model and credentials for your setup\n\n# Python deps\nuv sync --directory hindsight-api-slim/\n\n# Node deps (uses npm workspaces)\nnpm install\n```\n\nCommon LLM settings:\n- `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, minimax, ollama, lmstudio\n- `HINDSIGHT_API_LLM_API_KEY`: API key for providers that require one\n- `HINDSIGHT_API_LLM_MODEL`: Model name (defaults are provider-specific)\n\nOptional (uses local models by default):\n- `HINDSIGHT_API_EMBEDDINGS_PROVIDER`: local (default) or tei\n- `HINDSIGHT_API_RERANKER_PROVIDER`: local (default) or tei\n- `HINDSIGHT_API_DATABASE_URL`: External PostgreSQL (uses embedded pg0 by default)\n- `HINDSIGHT_API_ENABLE_BANK_CONFIG_API`: Enable per-bank config API (default: true)\n"}}