{"owner":"HKUDS","repo":"LightRAG","hasSkills":true,"totalSkillsCount":8,"totalTokensCount":23470,"categories":["anthropic-skill","generic","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["lightrag/tools/README_CLEAN_LLM_QUERY_CACHE.md","prompts/UserCustomizePrompts.md","lightrag/tools/README_KG_INTEGRITY_REPAIR.md","lightrag/tools/README_MIGRATE_LLM_CACHE.md","lightrag/tools/README_MIGRATE_GRAPH_STORAGE.md","AGENTS.md","lightrag/tools/README_REBUILD_VDB.md","lightrag/tools/README_SOURCE_CONFLICT_REPAIR.md"],"skills":{"lightrag/tools/README_CLEAN_LLM_QUERY_CACHE.md":"# LLM Query Cache Cleanup Tool - User Guide\n\n## Overview\n\nThis tool cleans up LightRAG's LLM query cache from KV storage implementations. It specifically targets query caches generated during RAG query operations (modes: `mix`, `hybrid`, `local`, `global`), including both query and keywords caches.\n\n## Supported Storage Types\n\n1. **JsonKVStorage** - File-based JSON storage\n2. **RedisKVStorage** - Redis database storage\n3. **PGKVStorage** - PostgreSQL database storage\n4. **MongoKVStorage** - MongoDB database storage\n5. **OpenSearchKVStorage** - OpenSearch index storage\n\n## Cache Types\n\nThe tool cleans up the following query cache types:\n\n### Query Cache Modes (4 types)\n- `mix:*` - Mixed mode query caches\n- `hybrid:*` - Hybrid mode query caches\n- `local:*` - Local mode query caches\n- `global:*` - Global mode query caches\n\n### Cache Content Types (2 types)\n- `*:query:*` - Query result caches\n- `*:keywords:*` - Keywords extraction caches\n\n### Cache Key Format\n```\n<mode>:<cache_type>:<hash>\n```\n\nExamples:\n- `mix:query:5ce04d25e957c290216cee5bfe6344fa`\n- `mix:keywords:fee77b98244a0b047ce95e21060de60e`\n- `global:query:abc123def456...`\n- `local:keywords:789xyz...`\n\n**Important Note**: This tool does NOT clean extraction caches (`default:extract:*` and `default:summary:*`). Use the migration tool or manual deletion for those caches.\n\n## Prerequisites\n\n- The tool reads storage configuration from environment variables\n- Ensure the target storage is properly configured and accessible\n- Backup important data before running cleanup operations\n\n## Usage\n\n### Basic Usage\n\nRun from the LightRAG project root directory:\n\n```bash\npython -m lightrag.tools.clean_llm_query_cache\n# or\npython lightrag/tools/clean_llm_query_cache.py\n```\n\n### Interactive Workflow\n\nThe tool guides you through the following steps:\n\n#### 1. Select Storage Type\n```\n============================================================\nLLM Query Cache Cleanup Tool - LightRAG\n============================================================\n\n=== Storage Setup ===\n\nSupported KV Storage Types:\n[1] JsonKVStorage\n[2] RedisKVStorage\n[3] PGKVStorage\n[4] MongoKVStorage\n[5] OpenSearchKVStorage\n\nSelect storage type (1-5) (Press Enter to exit): 1\n```\n\n**Note**: You can press Enter or type `0` at any prompt to exit gracefully.\n\n#### 2. Storage Validation\nThe tool will:\n- Check required environment variables\n- Auto-detect workspace configuration\n- Initialize and connect to storage\n- Verify connection status\n\n```\nChecking configuration...\n✓ All required environment variables are set\n\nInitializing storage...\n- Storage Type: JsonKVStorage\n- Workspace: space1\n- Connection Status: ✓ Success\n```\n\n#### 3. View Cache Statistics\n\nThe tool displays a detailed breakdown of query caches by mode and type:\n\n```\nCounting query cache records...\n\n📊 Query Cache Statistics (Before Cleanup):\n┌────────────┬────────────┬────────────┬────────────┐\n│ Mode       │ Query      │ Keywords   │ Total      │\n├────────────┼────────────┼────────────┼────────────┤\n│ mix        │      1,234 │        567 │      1,801 │\n│ hybrid     │        890 │        423 │      1,313 │\n│ local      │      2,345 │      1,123 │      3,468 │\n│ global     │        678 │        345 │      1,023 │\n├────────────┼────────────┼────────────┼────────────┤\n│ Total      │      5,147 │      2,458 │      7,605 │\n└────────────┴────────────┴────────────┴────────────┘\n```\n\n#### 4. Select Cleanup Scope\n\nChoose what type of caches to delete:\n\n```\n=== Cleanup Options ===\n[1] Delete all query caches (both query and keywords)\n[2] Delete query caches only (keep keywords)\n[3] Delete keywords caches only (keep query)\n[0] Cancel\n\nSelect cleanup option (0-3): 1\n```\n\n**Cleanup Types:**\n- **Option 1 (all)**: Deletes both query and keywords caches across all modes\n- **Option 2 (query)**: Deletes only query caches, preserves keywords caches\n- **Option 3 (keywords)**: Deletes only keywords caches, preserves query caches\n\n#### 5. Confirm Deletion\n\nReview the cleanup plan and confirm:\n\n```\n============================================================\nCleanup Confirmation\n============================================================\nStorage: JsonKVStorage (workspace: space1)\nCleanup Type: all\nRecords to Delete: 7,605 / 7,605\n\n⚠️  WARNING: This will delete ALL query caches across all modes!\n\nContinue with deletion? (y/n): y\n```\n\n#### 6. Execute Cleanup\n\nThe tool performs batch deletion with real-time progress:\n\n**JsonKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing 1,000 records at a time from JsonKVStorage\n\nBatch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓\nBatch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓\n...\nBatch 8/8: ████████████████████ 7,605/7,605 (100.0%) ✓\n\nPersisting changes to storage...\n✓ Changes persisted successfully\n```\n\n**RedisKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing Redis keys in batches of 1,000\n\nBatch 1: Deleted 1,000 keys (Total: 1,000) ✓\nBatch 2: Deleted 1,000 keys (Total: 2,000) ✓\n...\n```\n\n**PostgreSQL Example:**\n```\n=== Starting Cleanup ===\n💡 Executing PostgreSQL DELETE query\n\n✓ Deleted 7,605 records in 0.45s\n```\n\n**MongoDB Example:**\n```\n=== Starting Cleanup ===\n💡 Executing MongoDB deleteMany operations\n\nPattern 1/8: Deleted 1,234 records ✓\nPattern 2/8: Deleted 567 records ✓\n...\nTotal deleted: 7,605 records\n```\n\n**OpenSearchKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing 1,000 records at a time from OpenSearchKVStorage\n\nBatch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓\nBatch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓\n...\n```\n\n#### 7. Review Cleanup Report\n\nThe tool provides a comprehensive final report:\n\n**Successful Cleanup:**\n```\n============================================================\nCleanup Complete - Final Report\n============================================================\n\n📊 Statistics:\n  Total records to delete:  7,605\n  Total batches:            8\n  Successful batches:       8\n  Failed batches:           0\n  Successfully deleted:     7,605\n  Failed to delete:         0\n  Success rate:             100.00%\n\n📈 Before/After Comparison:\n  Total caches before:      7,605\n  Total caches after:       0\n  Net reduction:            7,605\n\n============================================================\n✓ SUCCESS: All records cleaned up successfully!\n============================================================\n\n📊 Query Cache Statistics (After Cleanup):\n┌────────────┬────────────┬────────────┬────────────┐\n│ Mode       │ Query      │ Keywords   │ Total      │\n├────────────┼────────────┼────────────┼────────────┤\n│ mix        │          0 │          0 │          0 │\n│ hybrid     │          0 │          0 │          0 │\n│ local      │          0 │          0 │          0 │\n│ global     │          0 │          0 │          0 │\n├────────────┼────────────┼────────────┼────────────┤\n│ Total      │          0 │          0 │          0 │\n└────────────┴────────────┴────────────┴────────────┘\n```\n\n**Cleanup with Errors:**\n```\n============================================================\nCleanup Complete - Final Report\n============================================================\n\n📊 Statistics:\n  Total records to delete:  7,605\n  Total batches:            8\n  Successful batches:       7\n  Failed batches:           1\n  Successfully deleted:     6,605\n  Failed to delete:         1,000\n  Success rate:             86.85%\n\n📈 Before/After Comparison:\n  Total caches before:      7,605\n  Total caches after:       1,000\n  Net reduction:            6,605\n\n⚠️  Errors encountered: 1\n\nError Details:\n------------------------------------------------------------\n\nError Summary:\n  - ConnectionError: 1 occurrence(s)\n\nFirst 5 errors:\n\n  1. Batch 3\n     Type: ConnectionError\n     Message: Connection timeout after 30s\n     Records lost: 1,000\n\n============================================================\n⚠️  WARNING: Cleanup completed with errors!\n   Please review the error details above.\n============================================================\n```\n\n## Technical Details\n\n### Workspace Handling\n\nThe tool retrieves workspace in the following priority order:\n\n1. **Storage-specific workspace environment variables**\n   - PGKVStorage: `POSTGRES_WORKSPACE`\n   - MongoKVStorage: `MONGODB_WORKSPACE`\n   - RedisKVStorage: `REDIS_WORKSPACE`\n   - OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`\n\n2. **Generic workspace environment variable**\n   - `WORKSPACE`\n\n3. **Default value**\n   - Empty string (uses storage's default workspace)\n\n### Batch Deletion\n\n- Default batch size: 1000 records/batch\n- Prevents memory overflow and connection timeouts\n- Each batch is processed independently\n- Failed batches are logged but don't stop cleanup\n\n### Storage-Specific Deletion Strategies\n\n#### JsonKVStorage\n- Collects all matching keys first (snapshot approach)\n- Deletes in batches with lock protection\n- Fast in-memory operations\n\n#### RedisKVStorage\n- Uses SCAN with pattern matching\n- Pipeline DELETE for batch operations\n- Cursor-based iteration for large datasets\n\n#### PostgreSQL\n- Single DELETE query with OR conditions\n- Efficient server-side bulk deletion\n- Uses LIKE patterns for mode/type matching\n\n#### MongoDB\n- Multiple deleteMany operations (one per pattern)\n- Regex-based document matching\n- Returns exact deletion counts\n\n### Pattern Matching Implementation\n\n**JsonKVStorage:**\n```python\n# Direct key prefix matching\nif key.startswith(\"mix:query:\") or key.startswith(\"mix:keywords:\")\n```\n\n**RedisKVStorage:**\n```python\n# SCAN with namespace-prefixed patterns\npattern = f\"{namespace}:mix:query:*\"\ncursor, keys = await redis.scan(cursor, match=pattern)\n```\n\n**PostgreSQL:**\n```python\n# SQL LIKE conditions\nWHERE id LIKE 'mix:query:%' OR id LIKE 'mix:keywords:%'\n```\n\n**MongoDB:**\n```python\n# Regex queries on _id field\n{\"_id\": {\"$regex\": \"^mix:query:\"}}\n```\n\n**OpenSearchKVStorage:**\n```python\n# Scan raw hits, then match cache key prefixes in Python\nif hit[\"_id\"].startswith(\"mix:query:\"):\n```\n\n## Error Handling & Resilience\n\nThe tool implements comprehensive error tracking:\n\n### Batch-Level Error Tracking\n- Each batch is independently error-checked\n- Failed batches are logged with full details\n- Successful batches commit even if later batches fail\n- Real-time progress shows ✓ (success) or ✗ (failed)\n\n### Error Reporting\nAfter cleanup completes, a detailed report includes:\n- **Statistics**: Total records, success/failure counts, success rate\n- **Before/After Comparison**: Net reduction in cache count\n- **Error Summary**: Grouped by error type with occurrence counts\n- **Error Details**: Batch number, error type, message, and records lost\n- **Recommendations**: Clear indication of success or need for review\n\n### Verification\n- Post-cleanup count verification\n- Before/after statistics comparison\n- Identifies partial cleanup scenarios\n\n## Important Notes\n\n1. **Irreversible Operation**\n   - Deleted caches cannot be recovered\n   - Always backup important data before cleanup\n   - Test on non-production data first\n\n2. **Performance Impact**\n   - Query performance may degrade temporarily after cleanup\n   - Caches will rebuild on subsequent queries\n   - Consider cleanup during off-peak hours\n\n3. **Selective Cleanup**\n   - Choose cleanup scope carefully\n   - Keywords caches may be valuable for future queries\n   - Query caches rebuild faster than keywords caches\n\n4. **Workspace Isolation**\n   - Cleanup only affects the selected workspace\n   - Other workspaces remain untouched\n   - Verify workspace before confirming cleanup\n\n5. **Interrupt and Resume**\n   - Cleanup can be interrupted at any time (Ctrl+C)\n   - Already deleted records cannot be recovered\n   - No automatic resume - must run tool again\n\n## Storage Configuration\n\nThe tool supports multiple configuration methods with the following priority:\n\n1. **Environment variables** (highest priority)\n2. **Default values** (lowest priority)\n\n### Environment Variable Configuration\n\nConfigure storage settings in your `.env` file:\n\n#### Workspace Configuration (Optional)\n\n```bash\n# Generic workspace (shared by all storages)\nWORKSPACE=space1\n\n# Or configure independent workspace for specific storage\nPOSTGRES_WORKSPACE=pg_space\nMONGODB_WORKSPACE=mongo_space\nREDIS_WORKSPACE=redis_space\n```\n\n**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string\n\n#### JsonKVStorage\n\n```bash\nWORKING_DIR=./rag_storage\n```\n\n#### RedisKVStorage\n\n```bash\nREDIS_URI=redis://localhost:6379\n```\n\n#### PGKVStorage\n\n```bash\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_DATABASE=your_database\n```\n\n#### MongoKVStorage\n\n```bash\nMONGO_URI=mongodb://root:root@localhost:27017/\nMONGO_DATABASE=LightRAG\n```\n\n#### OpenSearchKVStorage\n\n```bash\nOPENSEARCH_HOSTS=localhost:9200\nOPENSEARCH_WORKSPACE=search_space\n```\n\nIf environment variables are not provided, the tool falls back to built-in defaults where available.\n\n## Troubleshooting\n\n### Missing Environment Variables\n```\n⚠️  Warning: Missing environment variables: POSTGRES_USER, POSTGRES_PASSWORD\n```\n**Solution**: Add missing variables to your `.env` file\n\n### Connection Failed\n```\n✗ Initialization failed: Connection refused\n```\n**Solutions**:\n- Check if database service is running\n- Verify connection parameters (host, port, credentials)\n- Check firewall settings\n- Ensure network connectivity for remote databases\n\n### No Caches Found\n```\n⚠️  No query caches found in storage\n```\n**Possible Reasons**:\n- No queries have been run yet\n- Caches were already cleaned\n- Wrong workspace selected\n- Different storage type was used for queries\n\n### Partial Cleanup\n```\n⚠️  WARNING: Cleanup completed with errors!\n```\n**Solutions**:\n- Check error details in the report\n- Verify storage connection stability\n- Re-run tool to clean remaining caches\n- Check storage capacity and permissions\n\n## Use Cases\n\n### Use Case 1: Clean All Query Caches\n\n**Scenario**: Free up storage space by removing all query caches\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 1 (all) -> Confirm (y)\n```\n\n**Result**: All query and keywords caches deleted, maximum storage freed\n\n### Use Case 2: Refresh Query Caches Only\n\n**Scenario**: Force query cache rebuild while keeping keywords\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 2 (query only) -> Confirm (y)\n```\n\n**Result**: Query caches deleted, keywords preserved for faster rebuild\n\n### Use Case 3: Clean Stale Keywords\n\n**Scenario**: Remove outdated keywords while keeping recent query results\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 3 (keywords only) -> Confirm (y)\n```\n\n**Result**: Keywords deleted, query caches preserved\n\n### Use Case 4: Workspace-Specific Cleanup\n\n**Scenario**: Clean caches for a specific workspace\n\n```bash\n# Configure workspace\nexport WORKSPACE=development\n\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Cleanup option -> Confirm (y)\n```\n\n**Result**: Only development workspace caches cleaned\n\n## Best Practices\n\n1. **Backup Before Cleanup**\n   - Always backup your storage before major cleanup\n   - Test cleanup on non-production data first\n   - Document cleanup decisions\n\n2. **Monitor Performance**\n   - Watch storage metrics during cleanup\n   - Monitor query performance after cleanup\n   - Allow time for cache rebuild\n\n3. **Scheduled Cleanup**\n   - Clean caches periodically (weekly/monthly)\n   - Automate cleanup for development environments\n   - Keep production cleanup manual for safety\n\n4. **Selective Deletion**\n   - Consider cleanup scope based on needs\n   - Keywords caches are harder to rebuild\n   - Query caches rebuild automatically\n\n5. **Storage Capacity**\n   - Monitor storage usage trends\n   - Clean caches before reaching capacity limits\n   - Archive old data if needed\n\n## Comparison with Migration Tool\n\n| Feature | Cleanup Tool | Migration Tool |\n|---------|-------------|----------------|\n| **Purpose** | Delete query caches | Migrate extraction caches |\n| **Cache Types** | mix/hybrid/local/global | default:extract/summary |\n| **Modes** | query, keywords | extract, summary |\n| **Operation** | Deletion | Copy between storages |\n| **Reversible** | No | Yes (source unchanged) |\n| **Use Case** | Free storage, refresh caches | Change storage backend |\n\n## Limitations\n\n1. **Single Storage Operation**\n   - Can only clean one storage type at a time\n   - To clean multiple storages, run tool multiple times\n\n2. **No Dry Run Mode**\n   - Deletion is immediate after confirmation\n   - No preview-only mode available\n   - Test on non-production first\n\n3. **No Selective Mode Cleanup**\n   - Cannot clean only specific modes (e.g., only `mix`)\n   - Cleanup applies to all modes for selected cache type\n   - All-or-nothing per cache type\n\n4. **No Scheduled Cleanup**\n   - Manual execution required\n   - No built-in scheduling\n   - Use cron/scheduler if automation needed\n\n5. **Verification Limitations**\n   - Post-cleanup verification may fail in error scenarios\n   - Manual verification recommended for critical operations\n\n## Future Enhancements\n\nPotential improvements for future versions:\n\n- Selective mode cleanup (e.g., clean only `mix` mode)\n- Age-based cleanup (delete caches older than X days)\n- Size-based cleanup (delete largest caches first)\n- Dry run mode for safe preview\n- Automated scheduling support\n- Cache statistics export\n- Incremental cleanup with pause/resume\n\n## Support\n\nFor issues, questions, or feature requests:\n- Check the error details in the cleanup report\n- Review storage configuration\n- Verify workspace settings\n- Test with a small dataset first\n- Report bugs through project issue tracker\n","prompts/UserCustomizePrompts.md":"# Folder for User-Defined Prompts\n\nThis folder is designated for storing customized prompt templates created by users.\n","lightrag/tools/README_KG_INTEGRITY_REPAIR.md":"# KG Integrity Audit / Recovery-Anchor Repair\n\nOffline companion to the issue #3400 recoverable-mutation work.\n\nSince #3400, ingestion prewrites per-document **recovery anchors**\n(`full_entities` / `full_relations`) before mutating the graph, so purge,\nretry, and scan rollback can always discover a document's contributions.\nInstallations with data ingested **before** that change (or written through\ndirect KG paths like `ainsert_custom_kg`, which sit outside the\ndocument-level guarantee) may hold graph contributions with no anchor — such\ndata is invisible to per-document cleanup and becomes orphaned when its\ndocument is deleted or reprocessed.\n\nThis tool enumerates the whole graph (deliberately offline-only; the hot\npaths never do a graph-wide scan), maps every node/edge back to its owning\ndocuments via chunk `source_id` → `text_chunks` → `full_doc_id`, and:\n\n- reports per-document anchor gaps;\n- reports irrecoverable orphans (contributions whose source chunks no longer\n  exist) — these are never modified automatically;\n- with `--apply`, unions the missing entries into the anchor rows.\n\n## Usage\n\nLibrary (works with every configured backend combination):\n\n```python\nfrom lightrag.tools.kg_integrity_repair import audit_kg_integrity\n\nrag = LightRAG(...)\nawait rag.initialize_storages()\nreport = await audit_kg_integrity(rag)              # report only\nreport = await audit_kg_integrity(rag, apply=True)  # repair anchors\n```\n\nCLI (env-driven construction — `WORKING_DIR`, `WORKSPACE`, `EMBEDDING_DIM`,\n`LIGHTRAG_*` storage selectors; never calls the LLM or embedder):\n\n```bash\npython -m lightrag.tools.kg_integrity_repair             # report\npython -m lightrag.tools.kg_integrity_repair --apply     # repair\npython -m lightrag.tools.kg_integrity_repair --verbose   # per-doc details\n```\n\nRun it while the server is stopped (or the workspace is otherwise idle):\nthe audit reads a moving target if ingestion runs concurrently.\n\n## This tool is now the required remedy, not just a diagnostic\n\nPurge and deletion **fail closed** when a document has no recovery proof.\nPreviously, absent anchor rows were read as \"this document contributed\nnothing\", so graph cleanup was silently skipped while the chunks were deleted\nanyway — which destroyed the provenance chain (`source_id` → `text_chunks` →\n`full_doc_id`) that this tool needs, turning repairable data into permanent\norphans. So instead, the operation now refuses before deleting anything:\n\n```\nRecoveryAnchorMissingError: Refusing to purge document doc-…: recovery anchor\nrow(s) missing or unusable (full_entities, full_relations) …\n```\n\nOver the API this surfaces as **HTTP 409** with no data removed. Retrying\nunchanged will refuse again — run `--apply` first, then retry.\n\nThe rule is narrower than \"every purge needs a proof\". What must never happen\nis deleting something that *carries* attribution — a chunk row, or an anchor\nrow that names objects — while leaving those objects in the graph. A purge that\nremoves no such carrier cannot strand anything, so it needs no proof. Otherwise\none of these must hold:\n\n| Proof | Meaning |\n| --- | --- |\n| Both anchor rows present | The normal case. Presence is the test, **not** whether the lists are non-empty: a row holding an empty list is a document that extracted no entities, and is a perfectly good proof. |\n| `doc_status.metadata.kg_write_state == pre_graph` | Stamped at enqueue and advanced only once the anchors are durable, so it proves the document never reached its first graph mutation. Staged chunks are cleaned up with no graph access at all. |\n| `doc_status.metadata.kg_purge` past `prepared` | A previous purge attempt got far enough to have deleted the anchors itself. Without this, purge's own last step would make every retry refuse forever. |\n| No chunks and no populated anchor row | Nothing that carries attribution would be deleted. This is what lets a document that was queued before the `kg_write_state` marker existed, and never processed, be deleted directly — no scan, no audit. |\n\nThree states therefore need this tool:\n\n- documents ingested **before** #3416 landed the write-ahead anchors;\n- documents written through direct KG paths such as `ainsert_custom_kg`,\n  which are documented as sitting outside the document-level guarantee;\n- documents ingested with `skip_kg` (`process_options` `'!'`) **before** the\n  `kg_write_state` marker existed — see below.\n\n### Documents that legitimately own nothing\n\n`skip_kg` skips extraction and the merge entirely, so no anchor rows are ever\nwritten. Documents ingested that way *after* this change carry\n`kg_write_state=pre_graph` from enqueue and delete normally. Older ones have\nneither anchors nor a marker, and anchor repair has nothing to rebuild from —\nthey never appear in the graph scan, because they own nothing in it.\n\nThe audit settles that case, and it is the only thing that can: it enumerates\nthe **whole** graph, which the hot paths deliberately never do, so a document\nabsent from that scan is not merely unproven but *proven empty*. Such\ndocuments are listed under `anchorless_docs`, and `--apply` gives them empty\nanchor rows — the present-and-empty pair that is the normal proof for a\ndocument with no contributions.\n\nAbsence is only ever concluded from the completed scan. A document that does\nown graph objects is repaired with its real names instead; blanking it would\nmanufacture a false proof and license the exact deletion this work prevents.\n\n### What the certification assumes of the storage backends\n\n\"Proven empty\" is only as strong as the enumeration it is concluded from, so\nboth reads are held to a complete-or-raise contract and the audit **fails\nloudly instead of certifying on a partial view**:\n\n- the `doc_status` enumeration uses `strict=True`, so a transport or\n  deserialization failure raises rather than silently narrowing the set of\n  documents being certified;\n- the graph scan relies on `get_all_nodes()` / `get_all_edges()` never\n  silently truncating. All shipped graph backends satisfy this (errors\n  propagate; there is no result cap). Corrupt AGE properties raise a\n  `PGGraphQueryException` rather than dropping the node or blanking the\n  edge's `source_id` — a dropped node is an attribution the scan never sees,\n  which is exactly how a contributing document could be falsely certified.\n\nOne sharp edge remains: a graph backend whose indices/collections are missing\n(e.g. deleted out-of-band) reads as an **empty graph**, not as an error. The\naudit cannot distinguish that from a workspace that truly owns no graph data —\none more reason to run it only while the server is stopped and the workspace\nis healthy, as stated at the top.\n\nNote that `--apply` reports what it found under `missing_entity_anchors` /\n`missing_relation_anchors` (the diagnosis) and what it wrote under\n`repaired_docs` (the action) — the first two are not emptied by a repair.\n\nAnything listed under `orphan_entities` / `orphan_relations` cannot be\nrepaired by this tool: its source chunks are already gone, so no owning\ndocument is determinable. Those need external provenance, a re-ingest, or\nmanual deletion.\n","lightrag/tools/README_MIGRATE_LLM_CACHE.md":"# LLM Cache Migration Tool - User Guide\n\n## Overview\n\nThis tool migrates LightRAG's LLM response cache between different KV storage implementations. It specifically migrates caches generated during file extraction (mode `default`), including entity extraction, summary, and multimodal analysis caches.\n\n## Supported Storage Types\n\n1. **JsonKVStorage** - File-based JSON storage\n2. **RedisKVStorage** - Redis database storage\n3. **PGKVStorage** - PostgreSQL database storage\n4. **MongoKVStorage** - MongoDB database storage\n5. **OpenSearchKVStorage** - OpenSearch index storage\n\n## Cache Types\n\nThe tool migrates the following cache types:\n- `default:extract:*` - Entity and relationship extraction caches\n- `default:summary:*` - Entity and relationship summary caches\n- `default:analysis:*` - Multimodal image, table, and equation analysis caches\n\n**Note**: Query caches (modes like `mix`,`local`, `global`, etc.) are NOT migrated.\n\n## Prerequisites\n\nThe LLM Cache Migration Tool reads the storage configuration of the LightRAG Server and provides an LLM migration option to select source and destination storage. Ensure that both the source and destination storage have been correctly configured and are accessible via the LightRAG Server before cache migration.\n\n## Usage\n\n### Basic Usage\n\nRun from the LightRAG project root directory:\n\n```bash\npython -m lightrag.tools.migrate_llm_cache\n# or\npython lightrag/tools/migrate_llm_cache.py\n```\n\n### Interactive Workflow\n\nThe tool guides you through the following steps:\n\n#### 1. Select Source Storage Type\n```\nSupported KV Storage Types:\n[1] JsonKVStorage\n[2] RedisKVStorage\n[3] PGKVStorage\n[4] MongoKVStorage\n[5] OpenSearchKVStorage\n\nSelect Source storage type (1-5) (Press Enter to exit): 1\n```\n\n**Note**: You can press Enter or type `0` at any storage selection prompt to exit gracefully.\n\n#### 2. Source Storage Validation\nThe tool will:\n- Check required environment variables\n- Auto-detect workspace configuration\n- Initialize and connect to storage\n- Count cache records available for migration\n\n```\nChecking environment variables...\n✓ All required environment variables are set\n\nInitializing Source storage...\n- Storage Type: JsonKVStorage\n- Workspace: space1\n- Connection Status: ✓ Success\n\nCounting cache records...\n- Total: 8,734 records\n```\n\n**Progress Display by Storage Type:**\n- **JsonKVStorage**: Fast in-memory counting, displays final count without incremental progress\n  ```\n  Counting cache records...\n  - Total: 8,734 records\n  ```\n- **RedisKVStorage**: Real-time scanning progress with incremental counts\n  ```\n  Scanning Redis keys... found 8,734 records\n  ```\n- **PostgreSQL**: Quick COUNT(*) query, shows timing only if operation takes >1 second\n  ```\n  Counting PostgreSQL records... (took 2.3s)\n  ```\n- **MongoDB**: Fast count_documents(), shows timing only if operation takes >1 second\n  ```\n  Counting MongoDB documents... (took 1.8s)\n  ```\n- **OpenSearchKVStorage**: PIT-based scan with timing shown when noticeable\n  ```\n  Scanning OpenSearch documents... (took 1.5s)\n  ```\n\n#### 3. Select Target Storage Type\n\nThe tool automatically excludes the source storage type from the target selection and renumbers the remaining options sequentially:\n\n```\nAvailable Storage Types for Target (source: JsonKVStorage excluded):\n[1] RedisKVStorage\n[2] PGKVStorage\n[3] MongoKVStorage\n[4] OpenSearchKVStorage\n\nSelect Target storage type (1-4) (Press Enter or 0 to exit): 1\n```\n\n**Important Notes:**\n- You **cannot** select the same storage type for both source and target\n- Options are automatically renumbered (e.g., [1], [2], [3] instead of [2], [3], [4])\n- You can press Enter or type `0` to exit at this point as well\n\nThe tool then validates the target storage following the same process as the source (checking environment variables, initializing connection, counting records).\n\n#### 4. Confirm Migration\n\n```\n==================================================\nMigration Confirmation\nSource: JsonKVStorage (workspace: space1) - 8,734 records\nTarget: MongoKVStorage (workspace: space1) - 0 records\nBatch Size: 1,000 records/batch\nMemory Mode: Streaming (memory-optimized)\n\n⚠️  Warning: Target storage already has 0 records\nMigration will overwrite records with the same keys\n\nContinue? (y/n): y\n```\n\n#### 5. Execute Migration\n\nThe tool uses **streaming migration** by default for memory efficiency. Observe migration progress:\n\n```\n=== Starting Streaming Migration ===\n💡 Memory-optimized mode: Processing 1,000 records at a time\n\nBatch 1/9: ████████░░░░░░░░░░░░ 1000/8734 (11.4%) - default:extract ✓\nBatch 2/9: ████████████░░░░░░░░ 2000/8734 (22.9%) - default:extract ✓\n...\nBatch 9/9: ████████████████████ 8734/8734 (100.0%) - default:summary ✓\n\nPersisting data to disk...\n✓ Data persisted successfully\n```\n\n**Key Features:**\n- **Streaming mode**: Processes data in batches without loading entire dataset into memory\n- **Real-time progress**: Shows progress bar with precise percentage and cache type\n- **Success indicators**: ✓ for successful batches, ✗ for failed batches\n- **Constant memory usage**: Handles millions of records efficiently\n\n#### 6. Review Migration Report\n\nThe tool provides a comprehensive final report showing statistics and any errors encountered:\n\n**Successful Migration:**\n```\nMigration Complete - Final Report\n\n📊 Statistics:\n  Total source records:    8,734\n  Total batches:           9\n  Successful batches:      9\n  Failed batches:          0\n  Successfully migrated:   8,734\n  Failed to migrate:       0\n  Success rate:            100.00%\n\n✓ SUCCESS: All records migrated successfully!\n```\n\n**Migration with Errors:**\n```\nMigration Complete - Final Report\n\n📊 Statistics:\n  Total source records:    8,734\n  Total batches:           9\n  Successful batches:      8\n  Failed batches:          1\n  Successfully migrated:   7,734\n  Failed to migrate:       1,000\n  Success rate:            88.55%\n\n⚠️  Errors encountered: 1\n\nError Details:\n------------------------------------------------------------\n\nError Summary:\n  - ConnectionError: 1 occurrence(s)\n\nFirst 5 errors:\n\n  1. Batch 2\n     Type: ConnectionError\n     Message: Connection timeout after 30s\n     Records lost: 1,000\n\n⚠️  WARNING: Migration completed with errors!\n   Please review the error details above.\n```\n\n## Technical Details\n\n### Workspace Handling\n\nThe tool retrieves workspace in the following priority order:\n\n1. **Storage-specific workspace environment variables**\n   - PGKVStorage: `POSTGRES_WORKSPACE`\n   - MongoKVStorage: `MONGODB_WORKSPACE`\n   - RedisKVStorage: `REDIS_WORKSPACE`\n   - OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`\n\n2. **Generic workspace environment variable**\n   - `WORKSPACE`\n\n3. **Default value**\n   - Empty string (uses storage's default workspace)\n\n### Batch Migration\n\n- Default batch size: 1000 records/batch\n- Avoids memory overflow from loading too much data at once\n- Each batch is committed independently, supporting resume capability\n\n### Memory-Efficient Pagination\n\nFor large datasets, the tool implements storage-specific pagination strategies:\n\n- **JsonKVStorage**: Direct in-memory access (data already loaded in shared storage)\n- **RedisKVStorage**: Cursor-based SCAN with pipeline batching (1000 keys/batch)\n- **PGKVStorage**: SQL LIMIT/OFFSET pagination (1000 records/batch)\n- **MongoKVStorage**: Cursor streaming with batch_size (1000 documents/batch)\n- **OpenSearchKVStorage**: PIT + `search_after` scan of the KV index (1000 documents/batch)\n\nThis ensures the tool can handle millions of cache records without memory issues.\n\n### Prefix Filtering Implementation\n\nThe tool uses optimized filtering methods for different storage types:\n\n- **JsonKVStorage**: Direct dictionary iteration with lock protection\n- **RedisKVStorage**: SCAN command with namespace-prefixed patterns + pipeline for bulk GET\n- **PGKVStorage**: SQL LIKE queries with proper field mapping (id, return_value, etc.)\n- **MongoKVStorage**: MongoDB regex queries on `_id` field with cursor streaming\n- **OpenSearchKVStorage**: Full-index scan with `_id` prefix filtering and `_source` passthrough\n\n## Error Handling & Resilience\n\nThe tool implements comprehensive error tracking to ensure transparent and resilient migrations:\n\n### Batch-Level Error Tracking\n- Each batch is independently error-checked\n- Failed batches are logged but don't stop the migration\n- Successful batches are committed even if later batches fail\n- Real-time progress shows ✓ (success) or ✗ (failed) for each batch\n\n### Error Reporting\nAfter migration completes, a detailed report includes:\n- **Statistics**: Total records, success/failure counts, success rate\n- **Error Summary**: Grouped by error type with occurrence counts\n- **Error Details**: Batch number, error type, message, and records lost\n- **Recommendations**: Clear indication of success or need for review\n\n### No Double Data Loading\n- Unlike traditional verification approaches, the tool does NOT reload all target data\n- Errors are detected during migration, not after\n- This eliminates memory overhead and handles pre-existing target data correctly\n\n## Important Notes\n\n1. **Data Overwrite Warning**\n   - Migration will overwrite records with the same keys in the target storage\n   - Tool displays a warning if target storage already has data\n   - Data migration can be performed repeatedly\n   - Pre-existing data in target storage is handled correctly\n3. **Interrupt and Resume**\n   - Migration can be interrupted at any time (Ctrl+C)\n   - Already migrated data will remain in target storage\n   - Re-running will overwrite existing records\n   - Failed batches can be manually retried\n4. **Performance Considerations**\n   - Large data migration may take considerable time\n   - Recommend migrating during off-peak hours\n   - Ensure stable network connection (for remote databases)\n   - Memory usage stays constant regardless of dataset size\n\n## Storage Configuration\n\nThe tool supports multiple configuration methods with the following priority:\n\n1. **Environment variables** (highest priority)\n2. **Default values** (lowest priority)\n\n#### Option A: Environment Variable Configuration\n\nConfigure storage settings in your `.env` file:\n\n#### Workspace Configuration (Optional)\n\n```bash\n# Generic workspace (shared by all storages)\nWORKSPACE=space1\n\n# Or configure independent workspace for specific storage\nPOSTGRES_WORKSPACE=pg_space\nMONGODB_WORKSPACE=mongo_space\nREDIS_WORKSPACE=redis_space\nOPENSEARCH_WORKSPACE=os_space\n```\n\n**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string\n\n#### JsonKVStorage\n\n```bash\nWORKING_DIR=./rag_storage\n```\n\n#### RedisKVStorage\n\n```bash\nREDIS_URI=redis://localhost:6379\n```\n\n#### PGKVStorage\n\n```bash\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_DATABASE=your_database\n```\n\n#### MongoKVStorage\n\n```bash\nMONGO_URI=mongodb://root:root@localhost:27017/\nMONGO_DATABASE=LightRAG\n```\n\n#### OpenSearchKVStorage\n\n```bash\nOPENSEARCH_HOSTS=localhost:9200\nOPENSEARCH_WORKSPACE=os_space\n```\n\nIf environment variables are not provided, the tool falls back to built-in defaults where available. JsonKVStorage uses `WORKING_DIR` or defaults to `./rag_storage`.\n\n## Troubleshooting\n\n### Missing Environment Variables\n```\n✗ Missing required environment variables: POSTGRES_USER, POSTGRES_PASSWORD\n```\n**Solution**: Add missing variables to your `.env` file\n\n### Connection Failed\n```\n✗ Initialization failed: Connection refused\n```\n**Solutions**:\n- Check if database service is running\n- Verify connection parameters (host, port, credentials)\n- Check firewall settings\n\n**Solutions**:\n- Check migration process for error logs\n- Re-run migration tool\n- Check target storage capacity and permissions\n\n## Example Scenarios\n\n### Scenario 1: JSON to MongoDB Migration\n\nUse case: Migrating from single-machine development to production\n\n```bash\n# 1. Configure environment variables\nWORKSPACE=production\nMONGO_URI=mongodb://user:pass@prod-server:27017/\nMONGO_DATABASE=LightRAG\n\n# 2. Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# 3. Select: 1 (JsonKVStorage) -> 1 (MongoKVStorage - renumbered from 4)\n```\n\n**Note**: After selecting JsonKVStorage as source, MongoKVStorage will be shown as option [1] in the target selection since options are renumbered after excluding the source.\n\n### Scenario 2: Redis to PostgreSQL\n\nUse case: Migrating from cache storage to relational database\n\n```bash\n# 1. Ensure both databases are accessible\nREDIS_URI=redis://old-redis:6379\nPOSTGRES_HOST=new-postgres-server\n# ... Other PostgreSQL configs\n\n# 2. Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# 3. Select: 2 (RedisKVStorage) -> 2 (PGKVStorage - renumbered from 3)\n```\n\n**Note**: After selecting RedisKVStorage as source, PGKVStorage will be shown as option [2] in the target selection.\n\n### Scenario 3: Different Workspaces Migration\n\nUse case: Migrating data between different workspace environments\n\n```bash\n# Configure separate workspaces for source and target\nPOSTGRES_WORKSPACE=dev_workspace  # For development environment\nMONGODB_WORKSPACE=prod_workspace  # For production environment\n\n# Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# Select: 3 (PGKVStorage with dev_workspace) -> 3 (MongoKVStorage with prod_workspace)\n```\n\n**Note**: This allows you to migrate between different logical data partitions while changing storage backends.\n\n## Tool Limitations\n\n1. **Same Storage Type Not Allowed**\n   - You cannot migrate between the same storage type (e.g., PostgreSQL to PostgreSQL)\n   - This is enforced by the tool automatically excluding the source storage type from target selection\n   - For same-storage migrations (e.g., database switches), use database-native tools instead\n2. **Only Default Mode Caches**\n   - Only migrates `default:extract:*`, `default:summary:*`, and `default:analysis:*`\n   - Query caches are not included\n4. **Network Dependency**\n   - Tool requires stable network connection for remote databases\n   - Large datasets may fail if connection is interrupted\n\n## Best Practices\n\n1. **Backup Before Migration**\n   - Always backup your data before migration\n   - Test migration on non-production data first\n\n2. **Verify Results**\n   - Check the verification output after migration\n   - Manually verify a few cache entries if needed\n\n3. **Monitor Performance**\n   - Watch database resource usage during migration\n   - Consider migrating in smaller batches if needed\n\n4. **Clean Old Data**\n   - After successful migration, consider cleaning old cache data\n   - Keep backups for a reasonable period before deletion\n","lightrag/tools/README_MIGRATE_GRAPH_STORAGE.md":"# Graph Storage Migration (`PGGraphStorage` → `PGTableGraphStorage`)\n\nOffline tool for moving an already-extracted entity-relation graph from the\nApache AGE backend (`PGGraphStorage`) to the plain-table backend\n(`PGTableGraphStorage`) without re-indexing the source documents.\n\nChanging `LIGHTRAG_GRAPH_STORAGE` normally means the previously extracted\ngraph is simply not visible to the new backend, so the documents have to be\nre-processed — an LLM-cost-bearing operation for a graph that already exists.\nThis tool copies the graph itself, through the public storage API only\n(`get_all_nodes` / `get_all_edges` to read, `upsert_nodes_batch` /\n`upsert_edges_batch` to write, `remove_nodes` / `remove_edges` to undo). It\nissues no raw SQL, so canonicalization, endpoint handling and identity rules\nstay in the already-tested storage layer.\n\nOnly the graph moves. Vector and KV storages are untouched, and because the\nmigrated graph keeps the same entity and relation identities, existing vector\ndata stays valid — nothing is re-embedded and nothing is re-extracted. The LLM\ncache is a separate concern with its own tool (see\n[README_MIGRATE_LLM_CACHE.md](README_MIGRATE_LLM_CACHE.md)).\n\nRe-indexing remains the general guidance for changing storage backends. This\ntool is an advanced path for one specific pair.\n\n## Before you run it\n\n**Stop every LightRAG writer first — the tool cannot check this for you.**\nIt reads the source graph, writes it to the target, and then verifies the\nresult. A concurrent writer mutating either side inside that window\ninvalidates the comparison, and the tool can neither detect a live writer nor\nlock the backends. Every other precondition below is enforced; this one is\npurely operational, and it is the assumption the whole design rests on. The\nsame requirement applies to `rebuild_vdb`.\n\n**The target graph slice must be empty.** The migration is defined as\npopulating an empty slice; that invariant is also what makes its failure\nhandling exact (see below). A non-empty target is refused before anything is\nwritten.\n\nBoth backends read the usual `POSTGRES_USER`, `POSTGRES_PASSWORD`,\n`POSTGRES_DATABASE`, `POSTGRES_HOST` and `POSTGRES_PORT` variables.\n\n## Usage\n\n```bash\n# Dry run (default): reads and checks everything, migrates nothing.\npython -m lightrag.tools.migrate_graph_storage\n\n# Perform the migration.\npython -m lightrag.tools.migrate_graph_storage --apply\n```\n\n| Flag | Default | Meaning |\n| --- | --- | --- |\n| `--apply` | off | Perform the migration. Without it no graph data is written (see the schema caveat below). |\n| `--workspace` | `WORKSPACE` env | Workspace to migrate. |\n| `--force-empty-target` | off | **Dangerous.** `drop()` the whole non-empty target graph slice before migrating, instead of refusing. |\n| `--source-backend` | `PGGraphStorage` | Source graph storage class name. |\n| `--target-backend` | `PGTableGraphStorage` | Target graph storage class name. |\n\n`--source-backend` / `--target-backend` are an extension seam, not an open\ndoor: only allow-listed pairs run, and Phase 1 allow-lists exactly\n`PGGraphStorage` → `PGTableGraphStorage`. Any other pair is rejected before a\nbackend is even constructed.\n\nThe dry run performs the same enumeration and the same refusal checks as the\nreal run, so it predicts what `--apply` will do. With `--force-empty-target`\nit reports that the apply run *would* drop the pre-existing slice, but never\ndrops anything itself.\n\n**A dry run migrates nothing, but it is not read-only, and its blast radius is\nnot limited to one workspace.** Both backends have to be initialized before\ntheir contents can be inspected, and `initialize()` is not a passive step. Two\ndifferent scopes are involved:\n\n- *Scoped to the workspace being migrated:* `_normalize_legacy_edges`\n  **deletes and re-inserts edge rows** (reversed duplicates, legacy\n  orderings) for this workspace and namespace only; the AGE side creates the\n  graph and its indexes.\n- *Table-wide, across every workspace and namespace:* the schema migration in\n  `PGTableGraphStorage`'s DDL. On a table whose primary key predates the\n  `namespace` column it **DELETEs duplicate rows** from `lightrag_graph_nodes`\n  and `lightrag_graph_edges` (keeping the most recently updated row per key)\n  before rebuilding the primary keys, and when the endpoint foreign keys are\n  absent it then **DELETEs every orphan edge** — an edge whose endpoint node\n  is missing — from the whole table. That sweep cannot be narrowed: as\n  `pgtable_impl` puts it, \"the sweep is table-wide because the FK is a global\n  composite constraint and cannot be validated per workspace.\" Both paths are\n  one-time (first install, or the first run after the namespace-PK upgrade)\n  and are skipped once the constraints exist, but the run that performs them\n  touches rows belonging to workspaces `--workspace` never named.\n\nSo a dry run against one workspace can still modify others, on a legacy\nschema. What it never does is migrate data or copy anything between backends —\nthat is the guarantee, and it is narrower than \"writes nothing\". Take a backup\nof both tables before the first run against a pre-`namespace` schema.\n\n**`--workspace` is cross-checked, not trusted.** The backends resolve their\nworkspace as `POSTGRES_WORKSPACE` env > `config.ini` `[postgres] workspace` >\nthe value passed in > `\"default\"`, so either the environment or a checked-in\nconfig file can outrank the flag. The tool compares what was requested\nagainst what the backends actually resolved to, refuses when they differ, and\nreports the resolved workspace — the slice a destructive run acted on is never\nleft implicit.\n\nExit code is 0 on success and non-zero on any failure.\n\n## The report\n\nBoth modes print a dict to stdout, following the convention of the other\ntools:\n\n```python\n{'mode': 'dry-run', 'source_backend': 'PGGraphStorage',\n 'target_backend': 'PGTableGraphStorage', 'workspace': 'default',\n 'nodes': 8123, 'edges': 41022,\n 'verified': False, 'compensated': False, 'target_non_empty': False,\n 'would_drop_target_slice': False,\n 'written_node_count': 8123, 'written_edge_count': 41022}\n```\n\n`mode` is `dry-run` or `apply`. `verified` is true only when the post-write\ncomparison passed. The destructive-consequence field is mode-symmetric:\n`would_drop_target_slice` in a dry run, `dropped_target_slice` in an apply run\n— so the preview cannot look clean while hiding that `--force-empty-target`\nwill destroy the existing slice.\n\n`nodes` / `edges` in a dry run also double as a sizing signal for the apply\nrun: both modes hold the whole graph in memory (see **It does not stream**\nbelow), so a dry run against the real source is the cheapest way to gauge how\nmuch memory the real run will need.\n\n## What makes it refuse\n\nEvery check below runs *before* the first write, and each one aborts the run\nrather than migrating a graph it cannot faithfully reproduce:\n\n- the storage pair is not allow-listed;\n- the target graph slice is not empty (unless `--force-empty-target`);\n- a node has no usable identity — `id` or `entity_id` missing, not a string,\n  or the two disagreeing. Both backends can legitimately produce such a row: a\n  vertex created with no properties is legal in AGE and enumerates as\n  `{\"id\": None}`, and NetworkX creates an edge's unknown endpoint as an\n  attribute-less node;\n- the same node id appears more than once, **whether or not the payloads\n  match**. The target keeps one row per id, so two physical source vertices\n  become one node and the graph's node count changes;\n- `a→b` and `b→a` both exist, **whether or not the payloads match**. The\n  target stores one row per canonical pair, and AGE computes degree by\n  counting relationship rows — so collapsing an identical reciprocal preserves\n  every property while changing degree and traversal. LightRAG's own write\n  path matches edges undirected, so it does not create reciprocals: finding\n  one means the source violates the invariant this tool assumes;\n- a payload contains a value PostgreSQL `jsonb` cannot store faithfully:\n  `NaN`, `Infinity` and `-Infinity` (which `agtype` represents and `jsonb`\n  rejects outright), or `-0.0` (which `jsonb` accepts but normalises to `0.0`,\n  and this tool's comparison treats signed zeros as different — so the write\n  would succeed and then fail its own verification). The general rule: if the\n  target cannot round-trip a value, refuse before the destructive step rather\n  than discover it after the point of no return;\n- more than one directed row *reaches the tool* for the same ordered pair — a\n  violation of the invariant that makes the canonical merge lossless. Note the\n  limit of this backstop: AGE enumerates edges with `SELECT DISTINCT source,\n  target, properties`, so byte-identical parallel relationships are already\n  collapsed to one row before the tool sees them. It can therefore only catch\n  parallels whose payloads *differ*. **This is the one cardinality gap the tool\n  cannot close**: byte-identical same-direction parallels are invisible to it,\n  and collapsing them changes degree without losing any property. Everything\n  else that would change cardinality — reciprocals, duplicate ids — is refused\n  outright.\n\nVerification after the write re-runs the source-side checks and compares node\nids, canonical edges and properties. It is driven from the source because the\ntarget has already canonicalized and therefore cannot reveal what was lost.\nProperty comparison is type-strict: `1`, `1.0`, `True` and `\"1\"` are four\ndifferent values, and `-0.0` differs from `0.0` — all are distinguishable\ndownstream.\n\n**What `verified` claims — and what it cannot.** The tool only ever sees what\nthe source's `get_all_nodes` / `get_all_edges` return, so its claim is scoped\nto that enumerated view: **every node and edge the enumerator returned came\nacross with its identity, its payload, and its count intact.** Within that\nview nothing silently collapses — reciprocals and duplicate ids are refused\nbefore the write rather than merged and blessed.\n\n`verified` does **not** prove that the physical AGE graph was reproduced. Two\nthings are already lost or hidden before this tool is handed a row:\n\n- **Physical multiplicity.** AGE enumerates with `SELECT DISTINCT source,\n  target, properties`, so two byte-identical relationships between the same\n  ordered pair arrive as one row. AGE's own degree counts relationship rows, so\n  such a source has degree 2 where the target will have 1 — and `verified` will\n  still be true, because both sides match the one row the enumerator showed.\n  Reading raw multiplicity would mean going around the storage API, which this\n  tool deliberately does not do.\n- **Payload keys named `id`, `source` or `target`.** `PGGraphStorage`'s\n  enumerator overwrites them with identity values when flattening each row, so\n  a business property under one of those names is gone upstream and no\n  comparison here can notice.\n\nIf exact physical fidelity matters for your graph, check for duplicate\nrelationships directly in the AGE tables before migrating.\n\n## If a write fails\n\nThe tool removes exactly what the run wrote — the migrated node ids plus every\nedge endpoint, since the target auto-creates missing endpoints without\nreporting them — using `remove_edges` first and then `remove_nodes` to respect\nthe foreign key. It never uses `drop()` for this; whole-slice deletion is\nreachable only through `--force-empty-target`. The written set is computed\nbefore the first write, so a failure at any point still has the complete set,\nand removing it is exact precisely because the slice started empty.\n\n`--force-empty-target` has an irreducible window. Every source check runs\nbefore the drop, so a malformed source cannot cost you the slice — but nothing\nproves the *write* will succeed. If the migration fails after the drop, the\ncompensation removes what this run wrote and the target ends up empty: the\npre-existing data is gone and the new data never landed. That is the bargain\nthe flag asks for; take a backup before using it.\n\nA failed `drop()` under `--force-empty-target` aborts the run rather than\nproceeding: the backend reports that failure by returning an error status\ninstead of raising, so an unchecked call would let the migration write into,\nand later compensate against, rows it does not own.\n\nCompensation is not atomic. There is no transaction spanning two backends, so\na crash during compensation can leave the target partially populated. Two\nthings make that recoverable: re-running the tool refuses a non-empty target\nrather than compounding the mess, and if compensation itself fails the report\ncarries the full `written_node_ids` and `written_edge_keys` so the remaining\nrows can be removed by hand.\n\n## What this tool does not do\n\n- **It does not migrate any other pair.** `PRECONDITION_HOOKS` and\n  `COMPENSATION_HOOKS` are named placeholders — declared, but not yet consulted\n  anywhere; the Phase 1 behaviour is hardcoded. Adding a pair means wiring them\n  as well as allow-listing it.\n- **It does not merge into a populated target.** Empty-target is a\n  precondition, not a convenience.\n- **It does not run in a single transaction.** Two backends, no shared\n  transaction; the failure handling above is the substitute.\n- **It does not stream.** The storage API enumerates whole graphs, so a run\n  holds the source graph, the derived write plan and — during verification —\n  the target graph in memory at the same time. Peak memory grows with the\n  graph, roughly 2x its decoded size, and no setting bounds it: the fail-closed\n  checks (duplicate ids, reciprocals, parallel pairs) are global properties\n  that need the whole-graph view. For a very large graph, run the tool on a\n  host sized accordingly, or migrate workspace by workspace. Running out of\n  memory *inside* the process (`MemoryError`) is handled like any other\n  failure — compensation removes what the run wrote — but a kernel OOM kill\n  terminates the process outright: nothing compensates, the report never\n  prints, and under `--force-empty-target` it lands after the drop. Size the\n  host before reaching for that flag.\n- **It does not protect you from a live writer.** See the preconditions.\n- **It does not disambiguate workspaces that share an AGE graph.** AGE derives\n  its graph name by replacing every non-alphanumeric character with `_`, so\n  `team-a` and `team_a` name the same AGE graph while PGTable keeps them as\n  distinct workspaces. Check which graph a workspace actually resolves to\n  before migrating one whose name contains punctuation.\n- **It does not move vectors, KV data or document status.**\n","AGENTS.md":"# Repository Guidelines\n\n## Project Overview\n\nLightRAG is a Retrieval-Augmented Generation (RAG) framework that uses graph-based knowledge representation for enhanced information retrieval. The system extracts entities and relationships from documents, builds a knowledge graph, and uses multiple retrieval modes (`local`, `global`, `hybrid`, `mix`, `naive`) for queries.\n\n## Project Structure\n\nTop-level directories:\n\n- **lightrag/**: Core Python package — see *Module Layout* below.\n- **lightrag_webui/**: React 19 + TypeScript client (Bun + Vite + Tailwind). UI components in `src/`.\n- **scripts/**: `test.sh` (preferred test runner), `setup/` interactive environment wizard (use `make env-*` rather than calling `setup.sh` directly — see *Configuration > Setup Wizard Outputs*), and release tooling.\n- **tests/**: Pytest coverage, organized into subdirectories that mirror `lightrag/` (see *Testing* below for layout). Working datasets stay in `inputs/`, `rag_storage/`, and `temp/`; deployment collateral lives in `docs/`, `k8s-deploy/`, and compose files.\n\n### Module Layout (`lightrag/`)\n\n- **lightrag.py**: Main orchestrator class (`LightRAG`) — assembled from mixins (see *LightRAG class composition*). Hosts `ainsert_custom_kg`, `_insert_done`, `_process_extract_entities`, `_refresh_addon_params_cache`, and `addon_params` accessors. Critical: always call `await rag.initialize_storages()` after instantiation.\n- **pipeline.py**: `_PipelineMixin` — owns the document ingestion pipeline (`apipeline_enqueue_documents`, `apipeline_process_enqueue_documents`, `apipeline_process_error_documents`), the `parse_native` / `parse_mineru` / `parse_docling` parser dispatchers, multimodal analysis, validation, and the worker scaffolding.\n- **utils_pipeline.py**: Pure helpers shared by the pipeline mixin and other entry points: doc-status field access, document identity (source key, content hash), parsed-artifact path resolution, parser payload normalization, multimodal entity augmentation, and `make_lightrag_doc_content`.\n- **llm_roles.py**: `RoleSpec` / `RoleLLMConfig` / `_RoleLLMState` / `ROLES` registry plus `_RoleLLMMixin` — role normalization, builder registration, wrapper rebuild, runtime config update, queue cleanup, sanitized config export, queue status reporting. Route role-specific behavior here rather than into provider modules.\n- **storage_migrations.py**: `_StorageMigrationMixin` — `check_and_migrate_data`, `_migrate_entity_relation_data`, `_migrate_chunk_tracking_storage`.\n- **addon_params.py**: `ObservableAddonParams` plus `default_addon_params` / `normalize_addon_params` helpers.\n- **operate.py**: Core extraction and query operations including entity/relation extraction, chunking, and multi-mode retrieval logic.\n- **base.py**: Abstract base classes for storage backends (`BaseKVStorage`, `BaseVectorStorage`, `BaseGraphStorage`, `BaseDocStatusStorage`).\n- **kg/**: Storage implementations (JSON, NetworkX, Neo4j, PostgreSQL, MongoDB, Redis, Milvus, Qdrant, Faiss, Memgraph, OpenSearch, NanoVectorDB). The backend registry (`STORAGE_IMPLEMENTATIONS` / `STORAGES`) lives in `kg/__init__.py`; `kg/factory.py::get_storage_class()` resolves backend classes from configuration.\n- **llm/**: LLM and embedding provider bindings (OpenAI, Ollama, Azure, Gemini, Bedrock, Anthropic, etc.). All async with caching support.\n- **parser/**: Unified parsing layer. `parser/routing.py` resolves engine and filename hints for `legacy`, `native`, `mineru`, and `docling` flows; `parser/debug.py` provides an offline LightRAG stub for the `parser/cli.py` debug entry point (`python -m lightrag.parser.cli`). Native format parsers live as sibling sub-packages under `parser/` (currently `parser/docx/`); external HTTP-based adapters live under `parser/external/` (`mineru`, `docling`) with shared helpers in `parser/external/_common.py`, `_manifest.py`, `_zip.py`.\n- **chunker/**: Chunking strategies (token-size, recursive character, semantic vector, paragraph semantic).\n- **api/**: FastAPI service (`lightrag_server.py`) with REST endpoints and Ollama-compatible API; routers under `routers/`, static Swagger assets, packaged WebUI output, and Gunicorn launcher.\n\n## Core Architecture\n\n### LightRAG class composition\n\n`LightRAG` is assembled from focused mixins (split out of the previously monolithic `lightrag.py`):\n\n```\nLightRAG → _RoleLLMMixin → _StorageMigrationMixin → _PipelineMixin → object\n```\n\nThe `@final` decorator on `LightRAG` is preserved — the mixin layering is an internal implementation detail, not an external subclassing surface. The public API (`ainsert`, `aquery`, `ainsert_custom_kg`, `initialize_storages`, etc.) is unchanged. `ainsert_custom_kg` and its internal construction logic, `_insert_done`, `_process_extract_entities`, `_refresh_addon_params_cache`, and the `addon_params` property accessors stay on `LightRAG` itself because they cut across multiple flows or depend on prompt-profile state.\n\n### Storage Layer\n\nLightRAG uses 4 storage types with pluggable backends:\n- **KV_STORAGE**: LLM response cache, text chunks, document info\n- **VECTOR_STORAGE**: Entity/relation/chunk embeddings\n- **GRAPH_STORAGE**: Entity-relation graph structure\n- **DOC_STATUS_STORAGE**: Document processing status tracking\n\nEach `LightRAG` instance can pass a `workspace` parameter for data isolation. Implementation differs per storage type:\n- **File-based**: subdirectories under `working_dir`.\n- **Collection-based**: collection name prefixes.\n- **Relational DB**: workspace column filtering.\n- **Qdrant**: payload-based partitioning.\n\n### Consistency without transactions\n\nLightRAG writes to independent stores — graph, KV, vector, doc-status — with **no transaction across them**. Every multi-store operation therefore has intermediate states, and no ordering removes them; an ordering only chooses which one it keeps.\n\n- **The rule:** an inconsistency is acceptable when it **heals itself later** (a retry, a rebuild, or the next run rewrites it) or is **harmless in direction**. Losing data is never acceptable; retaining an object that could have been deleted, or surfacing a chunk a query did not need, is.\n- A change must **improve on an accepted residue**, not swap it for its mirror. \"An inconsistent state exists\" is not by itself a defect report — the questions are which state, how it heals, and whether the alternative is better.\n- Every accepted residue is **written down** with its reason and recovery path, next to the code or in the relevant contract. An undocumented residue is a defect; a documented one is a decision.\n- This licenses nothing for **silent failure**. A durable write must never be reported as one that did not happen, and a failure must never be swallowed: fail loud, then let the documented residue heal.\n\n### File-backed storage contracts\n\n**Full contracts: [docs/design/NetworkXSingleWriterContract.md](docs/design/NetworkXSingleWriterContract.md) — read it before touching `lightrag/kg/networkx_impl.py` or any caller of `index_done_callback` on the graph store; [docs/design/FileBackedSnapshotContract.md](docs/design/FileBackedSnapshotContract.md) — read it before touching `lightrag/kg/nano_vector_db_impl.py`, `lightrag/kg/faiss_impl.py`, `lightrag/kg/json_kv_impl.py`, `lightrag/kg/json_doc_status_impl.py` or `lightrag/kg/file_fingerprint.py`.**\n\nFive storages keep their data in memory and publish it by rewriting a whole file, and **they do not share one model** — `JsonKVStorage` and `JsonDocStatusStorage` are the odd ones out and the contracts say so at length. Read the right one before assuming.\n\n- **All five**: a commit publishes the WHOLE namespace, so any writer's flush also publishes every other writer's pending mutation there, half-finished ones included. All five are supported for **small-scale testing and validation only**; no change to them may be justified by write throughput.\n- **`NetworkXStorage`, `NanoVectorDBStorage`, `FaissVectorDBStorage`** keep one in-memory copy per process and reconcile by reloading the file. Visibility rests on a **two-channel fence**: the file's own `(st_mtime_ns, st_size)` (authoritative, state) OR-ed with the `storage_updated` flag (accelerator, a consumable event). Both are permanent — their blind spots do not overlap.\n- Those three diverge on a write conflict, and the reason is in the contracts: the graph store **declines** the commit (it has no buffer to replay, and graph payloads are accumulate-over-read), the vector stores **reload and replay** their pending buffers and redo logs. Do not reopen reload-then-replay for the graph store without addressing the accumulate-over-read argument.\n- **`JsonKVStorage` and `JsonDocStatusStorage` use none of that.** Their data is a `Manager().dict()` every worker shares, so a mutation is visible everywhere immediately and there is nothing to reload — adding a `_get_*` entry method would be wrong. Their `storage_updated` flag means the OPPOSITE of the other three's: `True` is \"dirty data still to flush\", never \"fresher data on disk to reload\". Do not read it as a peer notification. `JsonDocStatusStorage` reimplements this protocol rather than inheriting it, so a change to one of the pair is almost always a change the other needs too; where they diverge is the flush trigger — doc-status writes that change scheduling state flush synchronously because doc-status is the pipeline's recovery anchor.\n- `NetworkXStorage` is the only storage that declares `requires_single_writer`, which is what puts the admin flows under `LightRAG._admin_write_gate`.\n\n### Pipeline concurrency contract\n\n**Full contract: [docs/design/PipelineConcurrencyContract.md](docs/design/PipelineConcurrencyContract.md) — read it before touching `lightrag/pipeline.py`, `lightrag/kg/pipeline_ingress.py`, `pipeline_status` fields, or any `/documents/*` endpoint.**\n\n- Concurrent writers coordinate through `pipeline_status` (per-workspace shared dict in `lightrag.kg.shared_storage`), mutated under `get_namespace_lock(\"pipeline_status\", workspace=...)`.\n- `busy` alone does NOT block enqueue — enqueue + processing are allowed to run concurrently. Three states do refuse it: `destructive_busy` (clear / delete, which drops storages), `scanning_exclusive` (scan's classification phase), and `manual_freeze_requested` (a manual retry draining the pipeline to idle).\n- **Admin graph writes** (`acreate_*` / `aedit_*` / `adelete_by_*` / `amerge_entities` / `ainsert_custom_kg`) run inside `LightRAG._admin_write_gate` when the graph storage declares `requires_single_writer` (`NetworkXStorage` only): a workspace admin lock (waited for) then the `busy` reservation (`kind=\"admin\"`, refuses on `busy` / `scanning`), in that fixed order and OUTSIDE the per-entity keyed locks. A pipeline start during the hold is deferred into the ingress mailbox and driven once on release. The routes' `check_pipeline_busy_or_raise` preflight exempts an `admin`-owned `busy` so a second REST edit reaches the admin lock and queues. Never re-acquire the admin lock inside `lightrag/utils_graph.py`.\n- The workspace **ingress mailbox** (`get_pipeline_ingress(workspace)`) is the pipeline's only wake-up channel; `doc_status` stays the source of truth, so a dropped notification is recovered by the next strict scan.\n- FAILED documents never resume automatically: they re-enter only through a sticky manual retry request (`/documents/scan`, `/documents/reprocess_failed`), granting ONE attempt each.\n- All scheduling-control-plane `doc_status` queries use `get_docs_by_statuses(..., strict=True)`; scheduler `full_docs` reads must distinguish confirmed-absent (`None`) from backend errors (raise).\n\n### Purge recovery contract\n\n**Full contract: [docs/design/PurgeRecoveryContract.md](docs/design/PurgeRecoveryContract.md) — read it before touching `_purge_kg_contributions`, `adelete_by_doc_id`, the anchor writes in `merge_nodes_and_edges`, the `kg_write_state` / `kg_purge` metadata, or the cache write ordering in `use_llm_func_with_cache`.**\n\n- \"What did this document contribute?\" is answerable only from the per-document write-ahead anchors (`full_entities` / `full_relations`). The reverse lookup through `text_chunks` is not a fallback — purge deletes those chunks.\n- Governing invariant: **a purge must never delete something that CARRIES attribution — a chunk row or an anchor row that names objects — and leave those objects behind.** `_purge_kg_contributions` **fails closed** (`RecoveryAnchorMissingError` → HTTP 409, nothing deleted) unless one of four proofs holds: `anchors`, `pre_graph`, `journal`, `empty_scope`.\n- **`kg_write_state` must never be inferred or backfilled** — it is written once at enqueue and is monotonic. A backfill reproduces the original silent-skip defect.\n- `kg_write_state` and `kg_purge` must stay in both `_DOC_STATUS_METADATA_CARRY_OVER_KEYS` and `_DOC_STATUS_METADATA_DIRECTIVE_KEYS` (`lightrag/utils_pipeline.py`); dropping either turns a resumable purge into a permanent refusal.\n- Chunk tracking (`entity_chunks` / `relation_chunks`) outranks graph `source_id`; code folding a `source_id` delta back into tracking must append genuine additions only.\n- LLM extraction cache rows are reachable only through the owning chunk's `llm_cache_list`, which makes that list an attribution carrier too: [LLM extraction cache reachability](docs/design/PurgeRecoveryContract.md#llm-extraction-cache-reachability) states the reference-before-row ordering, why a reference that cannot be recorded skips the cache write instead, and what the ordering does not close.\n- Merge and rename apply *Consistency without transactions* above: [the failure model](docs/design/PurgeRecoveryContract.md#merge-and-rename-failure-model) lists their ordering invariants, accepted residues and already-rejected remedies. Read it before reordering `_merge_entities_impl` or the rename branch of `_edit_entity_impl`.\n\n### Relation weight contract\n\n**Full contract: [docs/ProgramingWithCore.md](docs/ProgramingWithCore.md#relation-weight-contract)** — keep it synchronized with the core API docstrings, REST graph documentation, and custom-KG examples whenever relation write behavior changes.\n\n- `weight >= len(distinct real source IDs)` on the graph edge; a larger value is an optional importance boost. Empty IDs and the legacy placeholders `manual_creation` / `UNKNOWN` are not evidence, so a source-less relation may use any non-negative fractional weight.\n- Public ingress paths (`create_relation`, `edit_relation`, `insert_custom_kg`) must validate the complete relation **before the first storage mutation**. To go below the current evidence count, creation callers omit `source_id`, edit callers set it to an empty string in the same operation.\n- Entity merges use `max(all input weights, distinct merged real source IDs)`.\n- Legacy-row repair is **opportunistic, not a sweep**: `_merge_edges_then_upsert`'s KEEP-cap skip branch returns the stored edge without writing graph or vector record, so an undersized legacy row stays undersized until a merge, an unrelated edit, or a rebuild rewrites it.\n\n### Query Modes\n\n- **local**: Context-dependent retrieval focused on specific entities\n- **global**: Community/summary-based broad knowledge retrieval\n- **hybrid**: Combines local and global\n- **naive**: Direct vector search without graph\n- **mix**: Integrates KG and vector retrieval (recommended with reranker)\n\n## Development Commands\n\n### Setup\n```bash\n# Install with uv\nuv sync\nsource .venv/bin/activate  # Or: .venv\\Scripts\\activate on Windows\n\n# Install with API support\nuv sync --extra api\n\n# Install specific extras\nuv sync --extra offline-storage  # Storage backends\nuv sync --extra offline-llm      # LLM providers\nuv sync --extra test             # Testing dependencies\n```\n\n### API Server\n```bash\n# Copy and configure environment\ncp env.example .env  # Edit with your LLM/embedding configs\n\n# Build WebUI\ncd lightrag_webui\nbun install --frozen-lockfile\nbun run build\ncd ..\n\n# Run server\nlightrag-server                                           # Production\nuvicorn lightrag.api.lightrag_server:app --reload        # Development\nlightrag-gunicorn                                         # Multi-worker (gunicorn)\n```\n\n### WebUI\n\nEvery command below is run **from `lightrag_webui/`**, not the repository root.\n`bun test` in particular resolves `bunfig.toml` — and the preload paths inside\nit — relative to the working directory, so running it from the root silently\nloads no DOM (see *React component tests*).\n\n```bash\ncd lightrag_webui\nbun install --frozen-lockfile      # REQUIRED after any change to package.json /\n                                   # bun.lock, including a branch switch across\n                                   # one: `git checkout` does not update\n                                   # node_modules, and a stale tree surfaces as\n                                   # a wall of TS2307 \"Cannot find module\".\nbun run dev                        # Dev server (Node + Vite)\nbun run dev:bun                    # Dev server (Bun native)\nbun run build                      # Production build\nbun run preview                    # Preview production build\nbun run lint                       # ESLint over *.ts/tsx/js/jsx\n\n# Testing — Bun built-in runner (NOT Vitest/Jest)\nbun test                           # All tests\nbun test --watch                   # Watch mode\nbun test --coverage                # With coverage report\nbun test src/api/lightrag.test.ts  # Single test file\nbunx tsc --noEmit                  # Typecheck (`bun run build` does NOT typecheck)\n```\n\n### Testing\n\n- Use mock-based tests for external services (Redis, httpx, etc.) — do not depend on live services in unit tests.\n- Add regression tests for every bug fix.\n- **Run only the test directories that mirror the modules you changed**, and report which subset you ran plus its pass count. The suite is ~7800 offline tests: about 5 minutes serially, ~70 s with `-n auto` on an 8-core machine — still too slow for the edit loop. Every PR's CI runs the full offline suite (and `tests/setup/` when its inputs change) — proving nothing else broke is its job, not yours.\n- Derive the subset from the mirror layout below: `lightrag/api/config.py` → `tests/api/config/`, `lightrag/kg/redis_impl.py` → `tests/kg/redis_impl/`, `lightrag/chunker/` → `tests/chunker/`. When a change spans several modules, run each of their directories rather than widening to `tests/`.\n- **`tests/setup/` is the setup wizard's suite and runs only when a wizard input changed**: `scripts/setup/**`, `env.example`, `docker-compose*.yml`, `Makefile`, or `tests/setup/` itself. It never imports `lightrag`, so a `lightrag/` change never needs it. `norecursedirs` in `pyproject.toml` already skips it in every recursive run (`pytest`, `pytest tests`, the full suite, CI's main job); run it by naming it: `./scripts/test.sh tests/setup`. The easy one to miss is `env.example`: adding a config knob for `lightrag` usually edits it, and the wizard parses it, so that change runs `tests/setup/` too.\n- Run the full suite locally only at a milestone, or when the change is genuinely cross-cutting (`lightrag/base.py`, `lightrag/utils.py`, `lightrag/kg/shared_storage.py`, or anything every backend inherits). **Run it with `-n auto --dist worksteal`** (pytest-xdist, in the `pytest` extra): on an 8-core machine it cuts ~5 min to ~1 min. CI stays serial because hosted 4-vCPU runners showed no gain. Do not add `-n` to subset runs: every worker re-collects and re-imports the suite (~12 s), which costs more than a single directory saves. Drop `-n` when you need `pdb`/`-s` or ordered output.\n- Backend tests use pytest; frontend unit tests use Bun's built-in runner — see *WebUI* above and *React component tests* below.\n- **A WebUI change runs the WHOLE frontend check set**, from `lightrag_webui/`: `bun install --frozen-lockfile` (see *WebUI* above — skip it after a branch switch and every later step fails on missing modules), then `bun test`, `bunx tsc --noEmit`, and `bun run lint`. The subsetting rule above is a backend rule and does not apply — all three together take well under a minute (test ~2 s, typecheck ~14 s, lint ~21 s), so there is nothing to save by running less. Report the pass count. `bun run build` transpiles WITHOUT checking types, so skipping `tsc --noEmit` means nothing checks them.\n\n```bash\n# Preferred for fresh shells and automation; resolves PYTHON, venv, uv, .venv, venv, python, python3\n# Default during development: only the directories mirroring the changed modules\n./scripts/test.sh tests/api/config\n./scripts/test.sh tests/kg/redis_impl\n\n# Run specific test file\n./scripts/test.sh tests/kg/test_graph_storage.py\n\n# Full suite — ~7800 offline tests; milestones and cross-cutting changes only.\n# Parallel via pytest-xdist: ~70 s on 8 cores (vs ~5 min serial).\n# tests/setup is skipped by default (norecursedirs); name it explicitly when\n# scripts/setup/, env.example, docker-compose*.yml or Makefile change.\n./scripts/test.sh tests -m offline -n auto --dist worksteal\n./scripts/test.sh tests/setup -m offline -n auto --dist worksteal\n\n# Integration-test concurrency (a LightRAG fixture, NOT pytest parallelism)\n./scripts/test.sh tests --test-workers 4\n```\n\n- `tests/`: main test suite, mirrors feature folders. Place new tests under the subdirectory matching the module under test:\n  - `tests/api/{auth,config,routes}/` for FastAPI server tests (auth/token, config loading, route handlers); top-level `tests/api/` for app-wide concerns (path prefixes, Ollama-compatible endpoint).\n  - `tests/chunker/`, `tests/evaluation/`, `tests/extraction/` for the like-named modules.\n  - `tests/kg/<backend>_impl/` for backend-specific storage tests, mirroring the `lightrag/kg/<backend>_impl.py` file naming. The `_impl` suffix on every subdirectory keeps the layout uniform and avoids `sys.path` shadowing on names that overlap with top-level PyPI/stdlib packages (`faiss`, `json`, `neo4j`, `networkx`, `redis`) when a test is launched directly via `python tests/kg/...`. Current backends: `faiss_impl/`, `json_impl/`, `memgraph_impl/`, `milvus_impl/`, `mongo_impl/`, `nano_impl/`, `neo4j_impl/`, `networkx_impl/`, `opensearch_impl/`, `postgres_impl/`, `qdrant_impl/`, `redis_impl/`. `tests/kg/` root holds cross-backend tests (`test_graph_storage`, `test_batch_graph_operations`, `test_unified_lock_safety`, `test_file_atomic`).\n  - `tests/llm/<provider>_impl/` for provider-specific behavior, same `_impl` convention: `bedrock_impl/`, `gemini_impl/`, `ollama_impl/`, `openai_impl/`, `voyageai_impl/`, `zhipu_impl/`. `tests/llm/` root holds cross-provider concerns (embedding, VLM, cache, role).\n  - `tests/parser/`, `tests/parser/docx/`, `tests/parser/external/{mineru,docling}/` for parser implementations.\n  - `tests/pipeline/` for ingestion pipeline and doc-status behavior (including `test_pipeline_*`, `test_doc_status_*`, `test_multimodal_*`, `test_graph_keyed_locks`).\n  - `tests/sidecar/`, `tests/setup/`, `tests/workspace/` for the like-named cross-cutting concerns. `tests/setup/` never imports `lightrag`; it runs in its own path-filtered CI job (`.github/workflows/setup-tests.yml`) and `norecursedirs` keeps it out of every recursive run, so a new input the wizard reads must be added to that workflow's `paths`.\n  - When adding a new backend or LLM provider, create a new subdirectory plus an empty `__init__.py` rather than dropping the file in the parent directory root.\n- Markers (registered in `[tool.pytest.ini_options]` in `pyproject.toml`): `offline`, `integration`, `requires_db`, `requires_api`, `pg_smoke`. Integration tests are skipped by default via `-m \"not integration\"`; opt in with `--run-integration`.\n- Integration env vars: `LIGHTRAG_RUN_INTEGRATION=true`, `LIGHTRAG_KEEP_ARTIFACTS=true`, `LIGHTRAG_TEST_WORKERS=4`, plus storage-specific connection strings.\n\n#### React component tests\n\n**Full guide: [docs/design/WebUITestingGuide.md](docs/design/WebUITestingGuide.md) — read it before writing or converting any test under `lightrag_webui/src/`.**\n\n- Tests are **colocated** next to the module they cover (`src/features/SiteHeader.test.ts`); the `tests/` mirror layout above is a backend rule. A test file containing JSX must be named `.test.tsx`.\n- **Run `bun test` from `lightrag_webui/`.** The `bunfig.toml` preload that installs the DOM is resolved against the CWD, and from the repository root the failure is silent — pure logic tests still pass, only component tests break.\n- **Test rendered behavior by rendering it**, through `renderWithProviders` (`src/test/render.tsx`). Do not write new tests that `readFileSync` a `.tsx` and match substrings.\n- **Never assert `toBeNull()` / `not.toBeInTheDocument()` / `toBe(element)` on a DOM node** — Bun serialises the whole happy-dom element and the run appears to HANG instead of failing. Assert a count or an extracted boolean/string instead.\n- **The DOM is process-wide**: never leave a bare `delete` of the `window` / `document` global behind; use `withoutDomGlobals()` from `src/test/domGlobals.ts`.\n- **Never import the test harness from production code** — one import from `src/` ships happy-dom to the browser.\n- **Prove the test can fail**: break the behavior it pins, see it go red, then restore.\n\n### Linting\n```bash\nruff check .\n```\n\n## Key Implementation Patterns\n\n### LightRAG Initialization (Critical)\n\nThe most common error is forgetting to initialize storages (manifests as `AttributeError: __aenter__` or `KeyError: 'history_messages'`):\n\n```python\nimport asyncio\nfrom lightrag import LightRAG\nfrom lightrag.llm.openai import gpt_4o_mini_complete, openai_embed\n\nasync def main():\n    rag = LightRAG(\n        working_dir=\"./rag_storage\",\n        llm_model_func=gpt_4o_mini_complete,\n        embedding_func=openai_embed\n    )\n\n    # REQUIRED: Initialize storage backends\n    await rag.initialize_storages()\n\n    # Now safe to use\n    await rag.ainsert(\"Your text here\")\n    result = await rag.aquery(\"Your question\", param=QueryParam(mode=\"hybrid\"))\n\n    # Cleanup\n    await rag.finalize_storages()\n\nasyncio.run(main())\n```\n\n### Custom Embedding Functions\n\n**Full guide: [docs/ProgramingWithCore.md](docs/ProgramingWithCore.md#custom-embedding-functions)** — the `.func` unwrapping rule, `max_token_size`, the `(len(texts), embedding_dim)` return contract that `EmbeddingFunc.__call__` enforces on every call, and the clear-the-data-directory pitfall when switching models. Read it before writing or wrapping an embedding function.\n\n### Storage Configuration\n\nConfigure via environment variables or constructor params:\n\n```python\n# Environment-based (recommended for production)\n# See env.example for full list\n\n# Constructor-based\nrag = LightRAG(\n    working_dir=\"./storage\",\n    workspace=\"project_name\",  # For data isolation\n    kv_storage=\"PGKVStorage\",\n    vector_storage=\"PGVectorStorage\",\n    graph_storage=\"Neo4JStorage\",\n    doc_status_storage=\"PGDocStatusStorage\",\n    vector_db_storage_cls_kwargs={\n        \"cosine_better_than_threshold\": 0.2\n    }\n)\n```\n\n### Document Insertion\n\n```python\n# Single document\nawait rag.ainsert(\"Text content\")\n\n# Batch insertion\nawait rag.ainsert([\"Text 1\", \"Text 2\", ...])\n\n# With custom IDs\nawait rag.ainsert(\"Text\", ids=[\"doc-123\"])\n\n# With file paths (for citation)\nawait rag.ainsert([\"Text 1\", \"Text 2\"], file_paths=[\"doc1.pdf\", \"doc2.pdf\"])\n\n# Configure batch size\nrag = LightRAG(..., max_parallel_insert=4)  # Default: 3, max recommended: 10\n```\n\n### Query Configuration\n\n```python\nfrom lightrag import QueryParam\n\nresult = await rag.aquery(\n    \"Your question\",\n    param=QueryParam(\n        mode=\"mix\",                    # Recommended with reranker\n        top_k=60,                      # KG entities/relations to retrieve\n        chunk_top_k=20,                # Text chunks to retrieve\n        max_entity_tokens=6000,\n        max_relation_tokens=8000,\n        max_total_tokens=30000,\n        enable_rerank=True,\n        user_prompt=\"Additional instructions for LLM\",\n        stream=False\n    )\n)\n```\n\n## Frontend Debugging via Playwright\n\nFor WebUI bugs whose symptoms only surface in the rendered DOM — layout/overflow/scrollbar issues, transient flashes, third-party libraries attaching helpers to `<body>` outside React's tree, or end-to-end verification of a fix — drive the running dev server (`http://localhost:5173`) with the `document-skills:webapp-testing` skill instead of reasoning from source alone. Seed state directly via `localStorage` (persist key `settings-storage`, schema in `lightrag_webui/src/stores/settings.ts`) to skip live LLM calls. Use `wait_until=\"domcontentloaded\"` plus a selector wait — Vite dev's long-lived polling makes `networkidle` time out.\n\n## Configuration\n\n### .env Configuration\nPrimary configuration file for API server. Generate it with `make env-base` or copy `env.example` manually. Key sections:\n- Server settings (HOST, PORT, CORS)\n- Storage backends (connection strings via environment variables)\n- Query parameters (TOP_K, MAX_TOTAL_TOKENS, etc.)\n- Reranking configuration (RERANK_BINDING, RERANK_MODEL)\n- Authentication (AUTH_ACCOUNTS, LIGHTRAG_API_KEY)\n\nSee `env.example` for comprehensive template.\n\n### Setup Wizard Outputs\n- Keep `.env` host-usable. Container-only hostnames and staged SSL paths belong in the wizard-managed compose layer, not persisted back into `.env`.\n- Treat `docker-compose.final.yml` as generated output assembled from `scripts/setup/templates/*.yml`.\n- For setup workflow changes, prefer `make env-*` targets over direct `scripts/setup/setup.sh` calls.\n\n## Code Style\n\n### Language\nComments, backend code, log messages, and Git commit messages in English. Frontend uses i18next for multi-language support.\n\n### Docstrings and comments\n\nDocstrings state the **rules**: what a caller must do, what it must not do, and the gotchas it will otherwise be caught by. The **mechanism** — how it works, the accepted residues, and the alternatives already rejected — goes in `docs/design/` with a pointer from the docstring. A docstring that has grown into a design document is the thing this separates: it buries the code, and the same facts in two places drift apart.\n\nTwo rules are enforced by `tests/test_docstring_budget.py` rather than by review, because both are properties of the tree rather than of any one change:\n\n- **No docstring over 80 lines** (100 for a class). The limit is generous on purpose — it catches a document, not a thorough docstring.\n- **No source line may cite a GitHub issue number.** The referent does not survive a fork, so a comment saying something is \"documented in #NNNN\" leaves nothing that documents it. Name the thing instead (\"the two-channel fence\"), or move the content into `docs/design/` and cite that. Never delete such a reference bare — migrate what it pointed at first. Enforced over `lightrag/` only: **tests may cite issue numbers**, and many do. There the number names the defect the test pins, and the test itself — its name, docstring and assertions — is the documentation, so the reference is provenance rather than the thing carrying the meaning.\n\nA third test requires every `docs/**.md` path named in the package to resolve; nothing imports those strings, so a typo is otherwise silent.\n\n### Python\n- Follow PEP 8 with 4-space indentation\n- Use type annotations\n- Prefer dataclasses for state management\n- Use `lightrag.utils.logger` instead of print\n- Async/await patterns throughout\n\n### TypeScript / React (incl. WebUI ESLint)\n- Functional components with hooks; PascalCase for components\n- 2-space indentation, single quotes (enforced by `@stylistic` rules)\n- Tailwind utility-first styling\n- ESLint stack: TypeScript-ESLint + React Hooks plugin + Prettier; `@typescript-eslint/no-explicit-any` is disabled (allowed)\n\n## Commit and Pull Request Guidance\n\n- If this repo is a fork of `HKUDS/LightRAG`. Target to `HKUDS/LightRAG` when creating PRs, not the fork's own repo.\n- PR descriptions should include: summary, motivation, linked issues if applyed, what's changed, what's broken and how it works.\n- Write commit messages (subject and body) in English. Commit messages are repository artifacts — like code comments and log messages — not conversational replies, so they follow the English code-style rule above regardless of any per-conversation working language.\n","lightrag/tools/README_REBUILD_VDB.md":"# Offline Vector Storage (VDB) Rebuild Tool\n\n`lightrag-rebuild-vdb` restores consistency between LightRAG's authoritative\ndata sources and its vector storages by dropping each vector storage and\nrebuilding it from scratch:\n\n| Vector storage | Authoritative source |\n|---|---|\n| `entities_vdb` | graph nodes |\n| `relationships_vdb` | graph edges |\n| `chunks_vdb` | `text_chunks` KV store |\n\n## When do I need this?\n\nLightRAG performs multi-step writes (graph + vector storage). If a vector\nstorage write fails at runtime — embedder outage, network timeout, context\noverflow on a high-degree entity — the graph and the vector storage drift\napart. The most common symptom is the edge-count drift reported in issue\n[#2917](https://github.com/HKUDS/LightRAG/issues/2917): graph edges with no\nvector counterpart, so `local`/`hybrid` queries miss relations that exist in\nthe graph.\n\nSince v(next), `amerge_entities` raises `VectorStorageConsistencyError` when\nthis happens, with a pointer to this tool. No data is lost in this situation:\nthe graph and the `text_chunks` KV store hold everything needed to rebuild\nthe vectors.\n\nA full drop + rebuild also clears *reverse* orphans (records present in the\nvector storage but absent from the graph), which incremental repair cannot\nreliably do.\n\nYou can also use this tool after changing the embedding model or embedding\ndimension. Existing vector records were generated in the old embedding space\n(and some vector backends bind collections/tables to the configured dimension),\nso run the tool with the updated `.env` and choose **Rebuild ALL vector\nstorages** to regenerate `entities_vdb`, `relationships_vdb`, and `chunks_vdb`\nfrom the authoritative graph/KV sources. The consistency check is not enough\nfor this case because it only detects missing graph → VDB records, not vectors\nthat exist but were embedded with a previous model or dimension.\n\nA vector storage whose container was written in a different embedding space\n**refuses to attach** rather than serving an empty or foreign index. That\nrefusal is the condition this tool clears, so it is the one startup failure the\ntool tolerates: the three vector targets are initialized individually, a typed\n`VectorSpaceMismatchError` is recorded instead of aborting the run, and the\nrebuild opens by dropping that container and re-provisioning it in the current\nembedding space. No out-of-band step (deleting the index through the backend's\nown API, `DROP TABLE`, `rm`) is needed.\n\nEverything else still aborts. A cluster outage, a bad credential or a corrupt\nfile is not a model change, and the recovery for a model change is destructive —\nso it is only ever applied to a refusal that says so in its type. The\nauthoritative sources (graph storage and `text_chunks`) likewise keep the\nserver-identical startup path, migrations included: rebuilding vectors from a\nhalf-migrated source is worse than not rebuilding.\n\nSee `docs/design/VectorSpaceProvenance.md` for the provenance marker and the\nrules that decide when a backend refuses.\n\nThe tool also completes a graph-only backfill performed with\n`NoopVectorDBStorage`. Stop every writer, preserve the same working directory,\nworkspace, graph storage, and KV storage, switch to the intended persistent\nvector backend, configure the production embedding model and dimension, and\nselect **Rebuild ALL vector storages**. Create a new `LightRAG` instance or\nrestart the process after changing the vector backend. Do not start the server\nuntil the rebuild succeeds; there is currently no persisted vector-index\nreadiness marker.\n\n## Usage\n\n```bash\n# Stop the LightRAG Server first!\nlightrag-rebuild-vdb\n# or\npython -m lightrag.tools.rebuild_vdb\n```\n\nThe tool reads the same `.env` / environment configuration as the server\n(`LIGHTRAG_GRAPH_STORAGE`, `LIGHTRAG_VECTOR_STORAGE`, `LIGHTRAG_KV_STORAGE`,\n`WORKSPACE`, `WORKING_DIR`, `EMBEDDING_*`, backend connection settings) and\nbuilds its embedding function through the exact factory the server uses —\nrun it with the same `.env` so rebuilt vectors live in the same embedding\nspace the server queries against.\n\nMenu options:\n\n1. **Consistency check (diagnose only)** — probes every graph entity/relation\n   for a vector counterpart and reports what is missing. Run this first to\n   decide whether a rebuild is worth the embedding cost. The check covers\n   the graph → VDB direction only; reverse orphans can only be cleared by a\n   full rebuild. Legacy reverse-order relation ids (from old custom-KG\n   imports) are recognized and not misreported as missing. The check issues\n   read queries only and does not run a rebuild (no drop + re-embed). It is\n   **not** strictly side-effect-free, though: the tool initializes every\n   storage on startup — exactly as the server does — and for some backends\n   that includes schema/DDL setup and one-time legacy migrations (e.g. Qdrant\n   upserts into the new collection, PostgreSQL batch-inserts into the new\n   table, Milvus may create a temp collection and drop/rename the original).\n   Treat running the tool — even just for a check — like starting the server:\n   stop other writers first.\n2. **Rebuild entities + relationships VDB** — sufficient for the #2917\n   merge-failure scenario.\n3. **Rebuild chunks VDB**.\n4. **Rebuild ALL vector storages** — use this after changing the embedding\n   model or embedding dimension.\n\n## Important notes\n\n- **Stop the server first.** The tool drops and rewrites vector storages;\n  concurrent writers (any backend, not just file-based ones) can corrupt\n  data or lose updates.\n- **Embedding model/dimension changes.** Run the tool with the new embedding\n  configuration and rebuild all vector storages. A consistency check can still\n  pass when every vector record exists but was created with the old embedding\n  model or dimension — unless the backend recorded its embedding-space\n  provenance, in which case the check reports the refusal and says the rebuild\n  is mandatory rather than reporting every record as missing.\n- **A refused target is dropped only after you confirm the rebuild.** Choosing\n  the consistency check never destroys anything, so a refusal can be diagnosed\n  before it is acted on. Options 2–4 recover exactly the targets they rebuild:\n  picking option 2 leaves a refused `chunks_vdb` refused until option 3 or 4\n  runs.\n- **Embedding cost.** A rebuild re-embeds every affected record. On large\n  datasets this means real API cost and time. Use the check mode first, and\n  rebuild only the storages that need it.\n- **Idempotent / crash-safe.** Sources (graph, `text_chunks`) are never\n  modified. If the tool crashes between drop and rewrite, just re-run it.\n- **`__created_at__` reset.** Backends that store creation timestamps in\n  vector records (nano, faiss) will show fresh timestamps after a rebuild.\n  No query logic depends on them.\n- **Custom-KG placeholder entities.** `UNKNOWN` placeholder nodes created by\n  `ainsert_custom_kg` are rebuilt faithfully from the graph; they may gain a\n  vector record they previously lacked (improving their retrievability).\n- **Chunk enumeration is backend-specific.** `BaseKVStorage` has no key\n  enumeration API, so the tool scans each KV backend directly (JsonKV,\n  Redis, PostgreSQL, MongoDB, OpenSearch). When a new KV backend is added,\n  `enumerate_kv_keys()` in `rebuild_vdb.py` must be extended.\n\n## Library usage\n\nThe core rebuild/check functions are plain async functions that accept your\nown initialized storage instances:\n\nRebuild targets must persist a queryable vector index. Passing a graph-only\nbackend such as `NoopVectorDBStorage` raises before source records are read or\nthe target is dropped; configure a persistent vector backend first.\n\n```python\nfrom lightrag.tools.rebuild_vdb import (\n    check_vdb_consistency,\n    rebuild_chunks_vdb,\n    rebuild_entities_vdb,\n    rebuild_relationships_vdb,\n)\n\nreport = await check_vdb_consistency(graph, entities_vdb, relationships_vdb)\nif not report[\"consistent\"]:\n    await rebuild_entities_vdb(graph, entities_vdb, global_config)\n    await rebuild_relationships_vdb(graph, relationships_vdb, global_config)\n```\n","lightrag/tools/README_SOURCE_CONFLICT_REPAIR.md":"# Source-Conflict Listing / Repair\n\nOffline companion to the typed source resolver (LR2 §5.5).\n\nA canonical source key — a filename with any parser `[hint]` stripped — is\nexpected to belong to exactly one **primary** document. History can break that:\ncustom-ID inserts, legacy ids, and basename collisions predate the strict\nresolver. When two or more primary rows claim one key, the resolver returns a\n*conflict* and `/documents/scan` refuses to act on that file: it is not\nenqueued, no record is deleted and nothing is archived, because picking a winner\nautomatically could silently retire the document you wanted to keep.\n\nThis tool (and the equivalent HTTP endpoints) is how an operator settles such a\nconflict explicitly.\n\n- The winner is never chosen for you: you name the `primary_doc_id` to keep.\n- Every other candidate is marked `metadata.is_duplicate=true` with\n  `original_doc_id=<primary>`. **No content is ever deleted** — the demoted rows\n  keep their status and their `full_docs` entry; they only lose their claim on\n  the canonical source, which is what makes the resolver return a single\n  primary afterwards.\n- Only `doc_status` and `full_docs` are opened. No vector store, graph store,\n  LLM or embedding model is touched, so a doc_status bookkeeping problem never\n  requires a reachable Milvus/Neo4j/LLM endpoint to fix. `full_docs` is read (not\n  written) to verify the primary you named — see *Safety*.\n\n## Usage\n\nCLI (honors `WORKING_DIR` / `WORKSPACE` and the `LIGHTRAG_*` storage variables\nfrom `.env`, exactly like the server):\n\n```bash\n# What is conflicting?\npython -m lightrag.tools.source_conflict_repair list [--limit 50] [--all]\n\n# Plan a repair (dry-run: nothing is modified)\npython -m lightrag.tools.source_conflict_repair repair \\\n    --source report.docx --primary doc-abc123\n\n# Commit it\npython -m lightrag.tools.source_conflict_repair repair \\\n    --source report.docx --primary doc-abc123 --apply\n```\n\n`--workspace` overrides the `WORKSPACE` env var. Exit code is `1` when a repair\nis refused (unknown primary, candidate set changed, backend cannot repair).\n\nLibrary (for a deployment that builds its own `LightRAG`):\n\n```python\nfrom lightrag.tools.source_conflict_repair import (\n    collect_source_conflicts,\n    repair_one_conflict,\n)\n\nconflicts = await collect_source_conflicts(rag.doc_status, limit=50)\nresult = await repair_one_conflict(\n    rag.doc_status,\n    \"report.docx\",\n    \"doc-abc123\",\n    workspace=rag.workspace,\n    # Required for a COMMIT: it is what verifies the primary you named can\n    # actually keep the source. A dry-run needs none.\n    full_docs=rag.full_docs,\n    apply=True,\n)\n```\n\n## Online equivalent\n\n```\nGET  /documents/source_conflicts?limit=50&cursor=<next_cursor>\nPOST /documents/source_conflicts/repair\n     {\"canonical_source_key\": \"...\", \"primary_doc_id\": \"...\",\n      \"expected_candidate_count\": N, \"expected_candidate_fingerprint\": \"...\",\n      \"dry_run\": false}\n```\n\nThe HTTP flow is two explicit calls: a dry-run returns the\n`candidate_count`/`fingerprint` pair, and the commit echoes both back. The CLI\nperforms the same two steps for you — it automates the copy-paste, not the\nguard.\n\n## Safety\n\n- Listing and a repair without `--apply` mutate nothing.\n- A commit is mutually exclusive with every in-deployment writer that can change\n  which documents claim the key: enqueue (the enqueue-serialize lock), the\n  processing stage's duplicate marking (a keyed lock on the canonical source key,\n  which the marking takes too), and clear/delete + scan classification + manual\n  reset (a pending-enqueue reservation). So the two operations are ordered rather\n  than interleaved: if the marking goes first the commit refuses before demoting\n  anything; if the commit goes first it verifies the key is settled and returns,\n  and a later marking is an ordinary state transition. Only the standalone CLI and\n  a second deployment writing the same database fall outside this.\n- A commit re-reads the candidate set under the backend's repair lock and\n  proceeds only when the count and fingerprint still match the dry-run\n  (compare-and-set), so a concurrent enqueue / delete / repair fails the commit\n  instead of being overwritten. A refused commit changes nothing; re-run it.\n- A repair interrupted half-way needs no reconciliation and leaves no marker\n  behind: the rows it finished already express `duplicate`, the rest still\n  resolve as a conflict. Run the command again.\n- Repeating a completed repair is safe: the same request is refused (its token\n  described the old candidate set) and a fresh run is a no-op with one\n  candidate and nothing to demote.\n- A commit refuses a primary that cannot keep the source, because the demotions\n  are irreversible — a repair only demotes, and a key left with no candidate has\n  no conflict left to settle:\n  - no `full_docs` content (an unprocessable stub a scan would delete) → 409;\n  - content that already exists under a **different** source, which the\n    processing stage will mark `FAILED [DUPLICATE:content_hash]` → 409;\n  - the content could not be *verified* at all — read failure, a backend without\n    strict point reads, or no `full_docs` handle → 503, retry later. Unverified\n    is not verified.\n- Not covered, by construction: a primary that has not been parsed yet has no\n  content hash, so nothing can predict whether its content will turn out to\n  duplicate another document. The commit itself still succeeds, and the key\n  really is unique when it returns — no marking can interleave with it. A\n  marking that lands afterwards is an ordinary state transition, not a\n  half-applied repair: the outcome is the ordinary content-dedup steady state\n  (the content lives under that document and this key ends up with no primary),\n  and no content is lost.\n"},"files":{"lightrag/tools/README_CLEAN_LLM_QUERY_CACHE.md":"# LLM Query Cache Cleanup Tool - User Guide\n\n## Overview\n\nThis tool cleans up LightRAG's LLM query cache from KV storage implementations. It specifically targets query caches generated during RAG query operations (modes: `mix`, `hybrid`, `local`, `global`), including both query and keywords caches.\n\n## Supported Storage Types\n\n1. **JsonKVStorage** - File-based JSON storage\n2. **RedisKVStorage** - Redis database storage\n3. **PGKVStorage** - PostgreSQL database storage\n4. **MongoKVStorage** - MongoDB database storage\n5. **OpenSearchKVStorage** - OpenSearch index storage\n\n## Cache Types\n\nThe tool cleans up the following query cache types:\n\n### Query Cache Modes (4 types)\n- `mix:*` - Mixed mode query caches\n- `hybrid:*` - Hybrid mode query caches\n- `local:*` - Local mode query caches\n- `global:*` - Global mode query caches\n\n### Cache Content Types (2 types)\n- `*:query:*` - Query result caches\n- `*:keywords:*` - Keywords extraction caches\n\n### Cache Key Format\n```\n<mode>:<cache_type>:<hash>\n```\n\nExamples:\n- `mix:query:5ce04d25e957c290216cee5bfe6344fa`\n- `mix:keywords:fee77b98244a0b047ce95e21060de60e`\n- `global:query:abc123def456...`\n- `local:keywords:789xyz...`\n\n**Important Note**: This tool does NOT clean extraction caches (`default:extract:*` and `default:summary:*`). Use the migration tool or manual deletion for those caches.\n\n## Prerequisites\n\n- The tool reads storage configuration from environment variables\n- Ensure the target storage is properly configured and accessible\n- Backup important data before running cleanup operations\n\n## Usage\n\n### Basic Usage\n\nRun from the LightRAG project root directory:\n\n```bash\npython -m lightrag.tools.clean_llm_query_cache\n# or\npython lightrag/tools/clean_llm_query_cache.py\n```\n\n### Interactive Workflow\n\nThe tool guides you through the following steps:\n\n#### 1. Select Storage Type\n```\n============================================================\nLLM Query Cache Cleanup Tool - LightRAG\n============================================================\n\n=== Storage Setup ===\n\nSupported KV Storage Types:\n[1] JsonKVStorage\n[2] RedisKVStorage\n[3] PGKVStorage\n[4] MongoKVStorage\n[5] OpenSearchKVStorage\n\nSelect storage type (1-5) (Press Enter to exit): 1\n```\n\n**Note**: You can press Enter or type `0` at any prompt to exit gracefully.\n\n#### 2. Storage Validation\nThe tool will:\n- Check required environment variables\n- Auto-detect workspace configuration\n- Initialize and connect to storage\n- Verify connection status\n\n```\nChecking configuration...\n✓ All required environment variables are set\n\nInitializing storage...\n- Storage Type: JsonKVStorage\n- Workspace: space1\n- Connection Status: ✓ Success\n```\n\n#### 3. View Cache Statistics\n\nThe tool displays a detailed breakdown of query caches by mode and type:\n\n```\nCounting query cache records...\n\n📊 Query Cache Statistics (Before Cleanup):\n┌────────────┬────────────┬────────────┬────────────┐\n│ Mode       │ Query      │ Keywords   │ Total      │\n├────────────┼────────────┼────────────┼────────────┤\n│ mix        │      1,234 │        567 │      1,801 │\n│ hybrid     │        890 │        423 │      1,313 │\n│ local      │      2,345 │      1,123 │      3,468 │\n│ global     │        678 │        345 │      1,023 │\n├────────────┼────────────┼────────────┼────────────┤\n│ Total      │      5,147 │      2,458 │      7,605 │\n└────────────┴────────────┴────────────┴────────────┘\n```\n\n#### 4. Select Cleanup Scope\n\nChoose what type of caches to delete:\n\n```\n=== Cleanup Options ===\n[1] Delete all query caches (both query and keywords)\n[2] Delete query caches only (keep keywords)\n[3] Delete keywords caches only (keep query)\n[0] Cancel\n\nSelect cleanup option (0-3): 1\n```\n\n**Cleanup Types:**\n- **Option 1 (all)**: Deletes both query and keywords caches across all modes\n- **Option 2 (query)**: Deletes only query caches, preserves keywords caches\n- **Option 3 (keywords)**: Deletes only keywords caches, preserves query caches\n\n#### 5. Confirm Deletion\n\nReview the cleanup plan and confirm:\n\n```\n============================================================\nCleanup Confirmation\n============================================================\nStorage: JsonKVStorage (workspace: space1)\nCleanup Type: all\nRecords to Delete: 7,605 / 7,605\n\n⚠️  WARNING: This will delete ALL query caches across all modes!\n\nContinue with deletion? (y/n): y\n```\n\n#### 6. Execute Cleanup\n\nThe tool performs batch deletion with real-time progress:\n\n**JsonKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing 1,000 records at a time from JsonKVStorage\n\nBatch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓\nBatch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓\n...\nBatch 8/8: ████████████████████ 7,605/7,605 (100.0%) ✓\n\nPersisting changes to storage...\n✓ Changes persisted successfully\n```\n\n**RedisKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing Redis keys in batches of 1,000\n\nBatch 1: Deleted 1,000 keys (Total: 1,000) ✓\nBatch 2: Deleted 1,000 keys (Total: 2,000) ✓\n...\n```\n\n**PostgreSQL Example:**\n```\n=== Starting Cleanup ===\n💡 Executing PostgreSQL DELETE query\n\n✓ Deleted 7,605 records in 0.45s\n```\n\n**MongoDB Example:**\n```\n=== Starting Cleanup ===\n💡 Executing MongoDB deleteMany operations\n\nPattern 1/8: Deleted 1,234 records ✓\nPattern 2/8: Deleted 567 records ✓\n...\nTotal deleted: 7,605 records\n```\n\n**OpenSearchKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing 1,000 records at a time from OpenSearchKVStorage\n\nBatch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓\nBatch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓\n...\n```\n\n#### 7. Review Cleanup Report\n\nThe tool provides a comprehensive final report:\n\n**Successful Cleanup:**\n```\n============================================================\nCleanup Complete - Final Report\n============================================================\n\n📊 Statistics:\n  Total records to delete:  7,605\n  Total batches:            8\n  Successful batches:       8\n  Failed batches:           0\n  Successfully deleted:     7,605\n  Failed to delete:         0\n  Success rate:             100.00%\n\n📈 Before/After Comparison:\n  Total caches before:      7,605\n  Total caches after:       0\n  Net reduction:            7,605\n\n============================================================\n✓ SUCCESS: All records cleaned up successfully!\n============================================================\n\n📊 Query Cache Statistics (After Cleanup):\n┌────────────┬────────────┬────────────┬────────────┐\n│ Mode       │ Query      │ Keywords   │ Total      │\n├────────────┼────────────┼────────────┼────────────┤\n│ mix        │          0 │          0 │          0 │\n│ hybrid     │          0 │          0 │          0 │\n│ local      │          0 │          0 │          0 │\n│ global     │          0 │          0 │          0 │\n├────────────┼────────────┼────────────┼────────────┤\n│ Total      │          0 │          0 │          0 │\n└────────────┴────────────┴────────────┴────────────┘\n```\n\n**Cleanup with Errors:**\n```\n============================================================\nCleanup Complete - Final Report\n============================================================\n\n📊 Statistics:\n  Total records to delete:  7,605\n  Total batches:            8\n  Successful batches:       7\n  Failed batches:           1\n  Successfully deleted:     6,605\n  Failed to delete:         1,000\n  Success rate:             86.85%\n\n📈 Before/After Comparison:\n  Total caches before:      7,605\n  Total caches after:       1,000\n  Net reduction:            6,605\n\n⚠️  Errors encountered: 1\n\nError Details:\n------------------------------------------------------------\n\nError Summary:\n  - ConnectionError: 1 occurrence(s)\n\nFirst 5 errors:\n\n  1. Batch 3\n     Type: ConnectionError\n     Message: Connection timeout after 30s\n     Records lost: 1,000\n\n============================================================\n⚠️  WARNING: Cleanup completed with errors!\n   Please review the error details above.\n============================================================\n```\n\n## Technical Details\n\n### Workspace Handling\n\nThe tool retrieves workspace in the following priority order:\n\n1. **Storage-specific workspace environment variables**\n   - PGKVStorage: `POSTGRES_WORKSPACE`\n   - MongoKVStorage: `MONGODB_WORKSPACE`\n   - RedisKVStorage: `REDIS_WORKSPACE`\n   - OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`\n\n2. **Generic workspace environment variable**\n   - `WORKSPACE`\n\n3. **Default value**\n   - Empty string (uses storage's default workspace)\n\n### Batch Deletion\n\n- Default batch size: 1000 records/batch\n- Prevents memory overflow and connection timeouts\n- Each batch is processed independently\n- Failed batches are logged but don't stop cleanup\n\n### Storage-Specific Deletion Strategies\n\n#### JsonKVStorage\n- Collects all matching keys first (snapshot approach)\n- Deletes in batches with lock protection\n- Fast in-memory operations\n\n#### RedisKVStorage\n- Uses SCAN with pattern matching\n- Pipeline DELETE for batch operations\n- Cursor-based iteration for large datasets\n\n#### PostgreSQL\n- Single DELETE query with OR conditions\n- Efficient server-side bulk deletion\n- Uses LIKE patterns for mode/type matching\n\n#### MongoDB\n- Multiple deleteMany operations (one per pattern)\n- Regex-based document matching\n- Returns exact deletion counts\n\n### Pattern Matching Implementation\n\n**JsonKVStorage:**\n```python\n# Direct key prefix matching\nif key.startswith(\"mix:query:\") or key.startswith(\"mix:keywords:\")\n```\n\n**RedisKVStorage:**\n```python\n# SCAN with namespace-prefixed patterns\npattern = f\"{namespace}:mix:query:*\"\ncursor, keys = await redis.scan(cursor, match=pattern)\n```\n\n**PostgreSQL:**\n```python\n# SQL LIKE conditions\nWHERE id LIKE 'mix:query:%' OR id LIKE 'mix:keywords:%'\n```\n\n**MongoDB:**\n```python\n# Regex queries on _id field\n{\"_id\": {\"$regex\": \"^mix:query:\"}}\n```\n\n**OpenSearchKVStorage:**\n```python\n# Scan raw hits, then match cache key prefixes in Python\nif hit[\"_id\"].startswith(\"mix:query:\"):\n```\n\n## Error Handling & Resilience\n\nThe tool implements comprehensive error tracking:\n\n### Batch-Level Error Tracking\n- Each batch is independently error-checked\n- Failed batches are logged with full details\n- Successful batches commit even if later batches fail\n- Real-time progress shows ✓ (success) or ✗ (failed)\n\n### Error Reporting\nAfter cleanup completes, a detailed report includes:\n- **Statistics**: Total records, success/failure counts, success rate\n- **Before/After Comparison**: Net reduction in cache count\n- **Error Summary**: Grouped by error type with occurrence counts\n- **Error Details**: Batch number, error type, message, and records lost\n- **Recommendations**: Clear indication of success or need for review\n\n### Verification\n- Post-cleanup count verification\n- Before/after statistics comparison\n- Identifies partial cleanup scenarios\n\n## Important Notes\n\n1. **Irreversible Operation**\n   - Deleted caches cannot be recovered\n   - Always backup important data before cleanup\n   - Test on non-production data first\n\n2. **Performance Impact**\n   - Query performance may degrade temporarily after cleanup\n   - Caches will rebuild on subsequent queries\n   - Consider cleanup during off-peak hours\n\n3. **Selective Cleanup**\n   - Choose cleanup scope carefully\n   - Keywords caches may be valuable for future queries\n   - Query caches rebuild faster than keywords caches\n\n4. **Workspace Isolation**\n   - Cleanup only affects the selected workspace\n   - Other workspaces remain untouched\n   - Verify workspace before confirming cleanup\n\n5. **Interrupt and Resume**\n   - Cleanup can be interrupted at any time (Ctrl+C)\n   - Already deleted records cannot be recovered\n   - No automatic resume - must run tool again\n\n## Storage Configuration\n\nThe tool supports multiple configuration methods with the following priority:\n\n1. **Environment variables** (highest priority)\n2. **Default values** (lowest priority)\n\n### Environment Variable Configuration\n\nConfigure storage settings in your `.env` file:\n\n#### Workspace Configuration (Optional)\n\n```bash\n# Generic workspace (shared by all storages)\nWORKSPACE=space1\n\n# Or configure independent workspace for specific storage\nPOSTGRES_WORKSPACE=pg_space\nMONGODB_WORKSPACE=mongo_space\nREDIS_WORKSPACE=redis_space\n```\n\n**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string\n\n#### JsonKVStorage\n\n```bash\nWORKING_DIR=./rag_storage\n```\n\n#### RedisKVStorage\n\n```bash\nREDIS_URI=redis://localhost:6379\n```\n\n#### PGKVStorage\n\n```bash\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_DATABASE=your_database\n```\n\n#### MongoKVStorage\n\n```bash\nMONGO_URI=mongodb://root:root@localhost:27017/\nMONGO_DATABASE=LightRAG\n```\n\n#### OpenSearchKVStorage\n\n```bash\nOPENSEARCH_HOSTS=localhost:9200\nOPENSEARCH_WORKSPACE=search_space\n```\n\nIf environment variables are not provided, the tool falls back to built-in defaults where available.\n\n## Troubleshooting\n\n### Missing Environment Variables\n```\n⚠️  Warning: Missing environment variables: POSTGRES_USER, POSTGRES_PASSWORD\n```\n**Solution**: Add missing variables to your `.env` file\n\n### Connection Failed\n```\n✗ Initialization failed: Connection refused\n```\n**Solutions**:\n- Check if database service is running\n- Verify connection parameters (host, port, credentials)\n- Check firewall settings\n- Ensure network connectivity for remote databases\n\n### No Caches Found\n```\n⚠️  No query caches found in storage\n```\n**Possible Reasons**:\n- No queries have been run yet\n- Caches were already cleaned\n- Wrong workspace selected\n- Different storage type was used for queries\n\n### Partial Cleanup\n```\n⚠️  WARNING: Cleanup completed with errors!\n```\n**Solutions**:\n- Check error details in the report\n- Verify storage connection stability\n- Re-run tool to clean remaining caches\n- Check storage capacity and permissions\n\n## Use Cases\n\n### Use Case 1: Clean All Query Caches\n\n**Scenario**: Free up storage space by removing all query caches\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 1 (all) -> Confirm (y)\n```\n\n**Result**: All query and keywords caches deleted, maximum storage freed\n\n### Use Case 2: Refresh Query Caches Only\n\n**Scenario**: Force query cache rebuild while keeping keywords\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 2 (query only) -> Confirm (y)\n```\n\n**Result**: Query caches deleted, keywords preserved for faster rebuild\n\n### Use Case 3: Clean Stale Keywords\n\n**Scenario**: Remove outdated keywords while keeping recent query results\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 3 (keywords only) -> Confirm (y)\n```\n\n**Result**: Keywords deleted, query caches preserved\n\n### Use Case 4: Workspace-Specific Cleanup\n\n**Scenario**: Clean caches for a specific workspace\n\n```bash\n# Configure workspace\nexport WORKSPACE=development\n\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Cleanup option -> Confirm (y)\n```\n\n**Result**: Only development workspace caches cleaned\n\n## Best Practices\n\n1. **Backup Before Cleanup**\n   - Always backup your storage before major cleanup\n   - Test cleanup on non-production data first\n   - Document cleanup decisions\n\n2. **Monitor Performance**\n   - Watch storage metrics during cleanup\n   - Monitor query performance after cleanup\n   - Allow time for cache rebuild\n\n3. **Scheduled Cleanup**\n   - Clean caches periodically (weekly/monthly)\n   - Automate cleanup for development environments\n   - Keep production cleanup manual for safety\n\n4. **Selective Deletion**\n   - Consider cleanup scope based on needs\n   - Keywords caches are harder to rebuild\n   - Query caches rebuild automatically\n\n5. **Storage Capacity**\n   - Monitor storage usage trends\n   - Clean caches before reaching capacity limits\n   - Archive old data if needed\n\n## Comparison with Migration Tool\n\n| Feature | Cleanup Tool | Migration Tool |\n|---------|-------------|----------------|\n| **Purpose** | Delete query caches | Migrate extraction caches |\n| **Cache Types** | mix/hybrid/local/global | default:extract/summary |\n| **Modes** | query, keywords | extract, summary |\n| **Operation** | Deletion | Copy between storages |\n| **Reversible** | No | Yes (source unchanged) |\n| **Use Case** | Free storage, refresh caches | Change storage backend |\n\n## Limitations\n\n1. **Single Storage Operation**\n   - Can only clean one storage type at a time\n   - To clean multiple storages, run tool multiple times\n\n2. **No Dry Run Mode**\n   - Deletion is immediate after confirmation\n   - No preview-only mode available\n   - Test on non-production first\n\n3. **No Selective Mode Cleanup**\n   - Cannot clean only specific modes (e.g., only `mix`)\n   - Cleanup applies to all modes for selected cache type\n   - All-or-nothing per cache type\n\n4. **No Scheduled Cleanup**\n   - Manual execution required\n   - No built-in scheduling\n   - Use cron/scheduler if automation needed\n\n5. **Verification Limitations**\n   - Post-cleanup verification may fail in error scenarios\n   - Manual verification recommended for critical operations\n\n## Future Enhancements\n\nPotential improvements for future versions:\n\n- Selective mode cleanup (e.g., clean only `mix` mode)\n- Age-based cleanup (delete caches older than X days)\n- Size-based cleanup (delete largest caches first)\n- Dry run mode for safe preview\n- Automated scheduling support\n- Cache statistics export\n- Incremental cleanup with pause/resume\n\n## Support\n\nFor issues, questions, or feature requests:\n- Check the error details in the cleanup report\n- Review storage configuration\n- Verify workspace settings\n- Test with a small dataset first\n- Report bugs through project issue tracker\n","prompts/UserCustomizePrompts.md":"# Folder for User-Defined Prompts\n\nThis folder is designated for storing customized prompt templates created by users.\n","lightrag/tools/README_KG_INTEGRITY_REPAIR.md":"# KG Integrity Audit / Recovery-Anchor Repair\n\nOffline companion to the issue #3400 recoverable-mutation work.\n\nSince #3400, ingestion prewrites per-document **recovery anchors**\n(`full_entities` / `full_relations`) before mutating the graph, so purge,\nretry, and scan rollback can always discover a document's contributions.\nInstallations with data ingested **before** that change (or written through\ndirect KG paths like `ainsert_custom_kg`, which sit outside the\ndocument-level guarantee) may hold graph contributions with no anchor — such\ndata is invisible to per-document cleanup and becomes orphaned when its\ndocument is deleted or reprocessed.\n\nThis tool enumerates the whole graph (deliberately offline-only; the hot\npaths never do a graph-wide scan), maps every node/edge back to its owning\ndocuments via chunk `source_id` → `text_chunks` → `full_doc_id`, and:\n\n- reports per-document anchor gaps;\n- reports irrecoverable orphans (contributions whose source chunks no longer\n  exist) — these are never modified automatically;\n- with `--apply`, unions the missing entries into the anchor rows.\n\n## Usage\n\nLibrary (works with every configured backend combination):\n\n```python\nfrom lightrag.tools.kg_integrity_repair import audit_kg_integrity\n\nrag = LightRAG(...)\nawait rag.initialize_storages()\nreport = await audit_kg_integrity(rag)              # report only\nreport = await audit_kg_integrity(rag, apply=True)  # repair anchors\n```\n\nCLI (env-driven construction — `WORKING_DIR`, `WORKSPACE`, `EMBEDDING_DIM`,\n`LIGHTRAG_*` storage selectors; never calls the LLM or embedder):\n\n```bash\npython -m lightrag.tools.kg_integrity_repair             # report\npython -m lightrag.tools.kg_integrity_repair --apply     # repair\npython -m lightrag.tools.kg_integrity_repair --verbose   # per-doc details\n```\n\nRun it while the server is stopped (or the workspace is otherwise idle):\nthe audit reads a moving target if ingestion runs concurrently.\n\n## This tool is now the required remedy, not just a diagnostic\n\nPurge and deletion **fail closed** when a document has no recovery proof.\nPreviously, absent anchor rows were read as \"this document contributed\nnothing\", so graph cleanup was silently skipped while the chunks were deleted\nanyway — which destroyed the provenance chain (`source_id` → `text_chunks` →\n`full_doc_id`) that this tool needs, turning repairable data into permanent\norphans. So instead, the operation now refuses before deleting anything:\n\n```\nRecoveryAnchorMissingError: Refusing to purge document doc-…: recovery anchor\nrow(s) missing or unusable (full_entities, full_relations) …\n```\n\nOver the API this surfaces as **HTTP 409** with no data removed. Retrying\nunchanged will refuse again — run `--apply` first, then retry.\n\nThe rule is narrower than \"every purge needs a proof\". What must never happen\nis deleting something that *carries* attribution — a chunk row, or an anchor\nrow that names objects — while leaving those objects in the graph. A purge that\nremoves no such carrier cannot strand anything, so it needs no proof. Otherwise\none of these must hold:\n\n| Proof | Meaning |\n| --- | --- |\n| Both anchor rows present | The normal case. Presence is the test, **not** whether the lists are non-empty: a row holding an empty list is a document that extracted no entities, and is a perfectly good proof. |\n| `doc_status.metadata.kg_write_state == pre_graph` | Stamped at enqueue and advanced only once the anchors are durable, so it proves the document never reached its first graph mutation. Staged chunks are cleaned up with no graph access at all. |\n| `doc_status.metadata.kg_purge` past `prepared` | A previous purge attempt got far enough to have deleted the anchors itself. Without this, purge's own last step would make every retry refuse forever. |\n| No chunks and no populated anchor row | Nothing that carries attribution would be deleted. This is what lets a document that was queued before the `kg_write_state` marker existed, and never processed, be deleted directly — no scan, no audit. |\n\nThree states therefore need this tool:\n\n- documents ingested **before** #3416 landed the write-ahead anchors;\n- documents written through direct KG paths such as `ainsert_custom_kg`,\n  which are documented as sitting outside the document-level guarantee;\n- documents ingested with `skip_kg` (`process_options` `'!'`) **before** the\n  `kg_write_state` marker existed — see below.\n\n### Documents that legitimately own nothing\n\n`skip_kg` skips extraction and the merge entirely, so no anchor rows are ever\nwritten. Documents ingested that way *after* this change carry\n`kg_write_state=pre_graph` from enqueue and delete normally. Older ones have\nneither anchors nor a marker, and anchor repair has nothing to rebuild from —\nthey never appear in the graph scan, because they own nothing in it.\n\nThe audit settles that case, and it is the only thing that can: it enumerates\nthe **whole** graph, which the hot paths deliberately never do, so a document\nabsent from that scan is not merely unproven but *proven empty*. Such\ndocuments are listed under `anchorless_docs`, and `--apply` gives them empty\nanchor rows — the present-and-empty pair that is the normal proof for a\ndocument with no contributions.\n\nAbsence is only ever concluded from the completed scan. A document that does\nown graph objects is repaired with its real names instead; blanking it would\nmanufacture a false proof and license the exact deletion this work prevents.\n\n### What the certification assumes of the storage backends\n\n\"Proven empty\" is only as strong as the enumeration it is concluded from, so\nboth reads are held to a complete-or-raise contract and the audit **fails\nloudly instead of certifying on a partial view**:\n\n- the `doc_status` enumeration uses `strict=True`, so a transport or\n  deserialization failure raises rather than silently narrowing the set of\n  documents being certified;\n- the graph scan relies on `get_all_nodes()` / `get_all_edges()` never\n  silently truncating. All shipped graph backends satisfy this (errors\n  propagate; there is no result cap). Corrupt AGE properties raise a\n  `PGGraphQueryException` rather than dropping the node or blanking the\n  edge's `source_id` — a dropped node is an attribution the scan never sees,\n  which is exactly how a contributing document could be falsely certified.\n\nOne sharp edge remains: a graph backend whose indices/collections are missing\n(e.g. deleted out-of-band) reads as an **empty graph**, not as an error. The\naudit cannot distinguish that from a workspace that truly owns no graph data —\none more reason to run it only while the server is stopped and the workspace\nis healthy, as stated at the top.\n\nNote that `--apply` reports what it found under `missing_entity_anchors` /\n`missing_relation_anchors` (the diagnosis) and what it wrote under\n`repaired_docs` (the action) — the first two are not emptied by a repair.\n\nAnything listed under `orphan_entities` / `orphan_relations` cannot be\nrepaired by this tool: its source chunks are already gone, so no owning\ndocument is determinable. Those need external provenance, a re-ingest, or\nmanual deletion.\n","lightrag/tools/README_MIGRATE_LLM_CACHE.md":"# LLM Cache Migration Tool - User Guide\n\n## Overview\n\nThis tool migrates LightRAG's LLM response cache between different KV storage implementations. It specifically migrates caches generated during file extraction (mode `default`), including entity extraction, summary, and multimodal analysis caches.\n\n## Supported Storage Types\n\n1. **JsonKVStorage** - File-based JSON storage\n2. **RedisKVStorage** - Redis database storage\n3. **PGKVStorage** - PostgreSQL database storage\n4. **MongoKVStorage** - MongoDB database storage\n5. **OpenSearchKVStorage** - OpenSearch index storage\n\n## Cache Types\n\nThe tool migrates the following cache types:\n- `default:extract:*` - Entity and relationship extraction caches\n- `default:summary:*` - Entity and relationship summary caches\n- `default:analysis:*` - Multimodal image, table, and equation analysis caches\n\n**Note**: Query caches (modes like `mix`,`local`, `global`, etc.) are NOT migrated.\n\n## Prerequisites\n\nThe LLM Cache Migration Tool reads the storage configuration of the LightRAG Server and provides an LLM migration option to select source and destination storage. Ensure that both the source and destination storage have been correctly configured and are accessible via the LightRAG Server before cache migration.\n\n## Usage\n\n### Basic Usage\n\nRun from the LightRAG project root directory:\n\n```bash\npython -m lightrag.tools.migrate_llm_cache\n# or\npython lightrag/tools/migrate_llm_cache.py\n```\n\n### Interactive Workflow\n\nThe tool guides you through the following steps:\n\n#### 1. Select Source Storage Type\n```\nSupported KV Storage Types:\n[1] JsonKVStorage\n[2] RedisKVStorage\n[3] PGKVStorage\n[4] MongoKVStorage\n[5] OpenSearchKVStorage\n\nSelect Source storage type (1-5) (Press Enter to exit): 1\n```\n\n**Note**: You can press Enter or type `0` at any storage selection prompt to exit gracefully.\n\n#### 2. Source Storage Validation\nThe tool will:\n- Check required environment variables\n- Auto-detect workspace configuration\n- Initialize and connect to storage\n- Count cache records available for migration\n\n```\nChecking environment variables...\n✓ All required environment variables are set\n\nInitializing Source storage...\n- Storage Type: JsonKVStorage\n- Workspace: space1\n- Connection Status: ✓ Success\n\nCounting cache records...\n- Total: 8,734 records\n```\n\n**Progress Display by Storage Type:**\n- **JsonKVStorage**: Fast in-memory counting, displays final count without incremental progress\n  ```\n  Counting cache records...\n  - Total: 8,734 records\n  ```\n- **RedisKVStorage**: Real-time scanning progress with incremental counts\n  ```\n  Scanning Redis keys... found 8,734 records\n  ```\n- **PostgreSQL**: Quick COUNT(*) query, shows timing only if operation takes >1 second\n  ```\n  Counting PostgreSQL records... (took 2.3s)\n  ```\n- **MongoDB**: Fast count_documents(), shows timing only if operation takes >1 second\n  ```\n  Counting MongoDB documents... (took 1.8s)\n  ```\n- **OpenSearchKVStorage**: PIT-based scan with timing shown when noticeable\n  ```\n  Scanning OpenSearch documents... (took 1.5s)\n  ```\n\n#### 3. Select Target Storage Type\n\nThe tool automatically excludes the source storage type from the target selection and renumbers the remaining options sequentially:\n\n```\nAvailable Storage Types for Target (source: JsonKVStorage excluded):\n[1] RedisKVStorage\n[2] PGKVStorage\n[3] MongoKVStorage\n[4] OpenSearchKVStorage\n\nSelect Target storage type (1-4) (Press Enter or 0 to exit): 1\n```\n\n**Important Notes:**\n- You **cannot** select the same storage type for both source and target\n- Options are automatically renumbered (e.g., [1], [2], [3] instead of [2], [3], [4])\n- You can press Enter or type `0` to exit at this point as well\n\nThe tool then validates the target storage following the same process as the source (checking environment variables, initializing connection, counting records).\n\n#### 4. Confirm Migration\n\n```\n==================================================\nMigration Confirmation\nSource: JsonKVStorage (workspace: space1) - 8,734 records\nTarget: MongoKVStorage (workspace: space1) - 0 records\nBatch Size: 1,000 records/batch\nMemory Mode: Streaming (memory-optimized)\n\n⚠️  Warning: Target storage already has 0 records\nMigration will overwrite records with the same keys\n\nContinue? (y/n): y\n```\n\n#### 5. Execute Migration\n\nThe tool uses **streaming migration** by default for memory efficiency. Observe migration progress:\n\n```\n=== Starting Streaming Migration ===\n💡 Memory-optimized mode: Processing 1,000 records at a time\n\nBatch 1/9: ████████░░░░░░░░░░░░ 1000/8734 (11.4%) - default:extract ✓\nBatch 2/9: ████████████░░░░░░░░ 2000/8734 (22.9%) - default:extract ✓\n...\nBatch 9/9: ████████████████████ 8734/8734 (100.0%) - default:summary ✓\n\nPersisting data to disk...\n✓ Data persisted successfully\n```\n\n**Key Features:**\n- **Streaming mode**: Processes data in batches without loading entire dataset into memory\n- **Real-time progress**: Shows progress bar with precise percentage and cache type\n- **Success indicators**: ✓ for successful batches, ✗ for failed batches\n- **Constant memory usage**: Handles millions of records efficiently\n\n#### 6. Review Migration Report\n\nThe tool provides a comprehensive final report showing statistics and any errors encountered:\n\n**Successful Migration:**\n```\nMigration Complete - Final Report\n\n📊 Statistics:\n  Total source records:    8,734\n  Total batches:           9\n  Successful batches:      9\n  Failed batches:          0\n  Successfully migrated:   8,734\n  Failed to migrate:       0\n  Success rate:            100.00%\n\n✓ SUCCESS: All records migrated successfully!\n```\n\n**Migration with Errors:**\n```\nMigration Complete - Final Report\n\n📊 Statistics:\n  Total source records:    8,734\n  Total batches:           9\n  Successful batches:      8\n  Failed batches:          1\n  Successfully migrated:   7,734\n  Failed to migrate:       1,000\n  Success rate:            88.55%\n\n⚠️  Errors encountered: 1\n\nError Details:\n------------------------------------------------------------\n\nError Summary:\n  - ConnectionError: 1 occurrence(s)\n\nFirst 5 errors:\n\n  1. Batch 2\n     Type: ConnectionError\n     Message: Connection timeout after 30s\n     Records lost: 1,000\n\n⚠️  WARNING: Migration completed with errors!\n   Please review the error details above.\n```\n\n## Technical Details\n\n### Workspace Handling\n\nThe tool retrieves workspace in the following priority order:\n\n1. **Storage-specific workspace environment variables**\n   - PGKVStorage: `POSTGRES_WORKSPACE`\n   - MongoKVStorage: `MONGODB_WORKSPACE`\n   - RedisKVStorage: `REDIS_WORKSPACE`\n   - OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`\n\n2. **Generic workspace environment variable**\n   - `WORKSPACE`\n\n3. **Default value**\n   - Empty string (uses storage's default workspace)\n\n### Batch Migration\n\n- Default batch size: 1000 records/batch\n- Avoids memory overflow from loading too much data at once\n- Each batch is committed independently, supporting resume capability\n\n### Memory-Efficient Pagination\n\nFor large datasets, the tool implements storage-specific pagination strategies:\n\n- **JsonKVStorage**: Direct in-memory access (data already loaded in shared storage)\n- **RedisKVStorage**: Cursor-based SCAN with pipeline batching (1000 keys/batch)\n- **PGKVStorage**: SQL LIMIT/OFFSET pagination (1000 records/batch)\n- **MongoKVStorage**: Cursor streaming with batch_size (1000 documents/batch)\n- **OpenSearchKVStorage**: PIT + `search_after` scan of the KV index (1000 documents/batch)\n\nThis ensures the tool can handle millions of cache records without memory issues.\n\n### Prefix Filtering Implementation\n\nThe tool uses optimized filtering methods for different storage types:\n\n- **JsonKVStorage**: Direct dictionary iteration with lock protection\n- **RedisKVStorage**: SCAN command with namespace-prefixed patterns + pipeline for bulk GET\n- **PGKVStorage**: SQL LIKE queries with proper field mapping (id, return_value, etc.)\n- **MongoKVStorage**: MongoDB regex queries on `_id` field with cursor streaming\n- **OpenSearchKVStorage**: Full-index scan with `_id` prefix filtering and `_source` passthrough\n\n## Error Handling & Resilience\n\nThe tool implements comprehensive error tracking to ensure transparent and resilient migrations:\n\n### Batch-Level Error Tracking\n- Each batch is independently error-checked\n- Failed batches are logged but don't stop the migration\n- Successful batches are committed even if later batches fail\n- Real-time progress shows ✓ (success) or ✗ (failed) for each batch\n\n### Error Reporting\nAfter migration completes, a detailed report includes:\n- **Statistics**: Total records, success/failure counts, success rate\n- **Error Summary**: Grouped by error type with occurrence counts\n- **Error Details**: Batch number, error type, message, and records lost\n- **Recommendations**: Clear indication of success or need for review\n\n### No Double Data Loading\n- Unlike traditional verification approaches, the tool does NOT reload all target data\n- Errors are detected during migration, not after\n- This eliminates memory overhead and handles pre-existing target data correctly\n\n## Important Notes\n\n1. **Data Overwrite Warning**\n   - Migration will overwrite records with the same keys in the target storage\n   - Tool displays a warning if target storage already has data\n   - Data migration can be performed repeatedly\n   - Pre-existing data in target storage is handled correctly\n3. **Interrupt and Resume**\n   - Migration can be interrupted at any time (Ctrl+C)\n   - Already migrated data will remain in target storage\n   - Re-running will overwrite existing records\n   - Failed batches can be manually retried\n4. **Performance Considerations**\n   - Large data migration may take considerable time\n   - Recommend migrating during off-peak hours\n   - Ensure stable network connection (for remote databases)\n   - Memory usage stays constant regardless of dataset size\n\n## Storage Configuration\n\nThe tool supports multiple configuration methods with the following priority:\n\n1. **Environment variables** (highest priority)\n2. **Default values** (lowest priority)\n\n#### Option A: Environment Variable Configuration\n\nConfigure storage settings in your `.env` file:\n\n#### Workspace Configuration (Optional)\n\n```bash\n# Generic workspace (shared by all storages)\nWORKSPACE=space1\n\n# Or configure independent workspace for specific storage\nPOSTGRES_WORKSPACE=pg_space\nMONGODB_WORKSPACE=mongo_space\nREDIS_WORKSPACE=redis_space\nOPENSEARCH_WORKSPACE=os_space\n```\n\n**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string\n\n#### JsonKVStorage\n\n```bash\nWORKING_DIR=./rag_storage\n```\n\n#### RedisKVStorage\n\n```bash\nREDIS_URI=redis://localhost:6379\n```\n\n#### PGKVStorage\n\n```bash\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_DATABASE=your_database\n```\n\n#### MongoKVStorage\n\n```bash\nMONGO_URI=mongodb://root:root@localhost:27017/\nMONGO_DATABASE=LightRAG\n```\n\n#### OpenSearchKVStorage\n\n```bash\nOPENSEARCH_HOSTS=localhost:9200\nOPENSEARCH_WORKSPACE=os_space\n```\n\nIf environment variables are not provided, the tool falls back to built-in defaults where available. JsonKVStorage uses `WORKING_DIR` or defaults to `./rag_storage`.\n\n## Troubleshooting\n\n### Missing Environment Variables\n```\n✗ Missing required environment variables: POSTGRES_USER, POSTGRES_PASSWORD\n```\n**Solution**: Add missing variables to your `.env` file\n\n### Connection Failed\n```\n✗ Initialization failed: Connection refused\n```\n**Solutions**:\n- Check if database service is running\n- Verify connection parameters (host, port, credentials)\n- Check firewall settings\n\n**Solutions**:\n- Check migration process for error logs\n- Re-run migration tool\n- Check target storage capacity and permissions\n\n## Example Scenarios\n\n### Scenario 1: JSON to MongoDB Migration\n\nUse case: Migrating from single-machine development to production\n\n```bash\n# 1. Configure environment variables\nWORKSPACE=production\nMONGO_URI=mongodb://user:pass@prod-server:27017/\nMONGO_DATABASE=LightRAG\n\n# 2. Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# 3. Select: 1 (JsonKVStorage) -> 1 (MongoKVStorage - renumbered from 4)\n```\n\n**Note**: After selecting JsonKVStorage as source, MongoKVStorage will be shown as option [1] in the target selection since options are renumbered after excluding the source.\n\n### Scenario 2: Redis to PostgreSQL\n\nUse case: Migrating from cache storage to relational database\n\n```bash\n# 1. Ensure both databases are accessible\nREDIS_URI=redis://old-redis:6379\nPOSTGRES_HOST=new-postgres-server\n# ... Other PostgreSQL configs\n\n# 2. Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# 3. Select: 2 (RedisKVStorage) -> 2 (PGKVStorage - renumbered from 3)\n```\n\n**Note**: After selecting RedisKVStorage as source, PGKVStorage will be shown as option [2] in the target selection.\n\n### Scenario 3: Different Workspaces Migration\n\nUse case: Migrating data between different workspace environments\n\n```bash\n# Configure separate workspaces for source and target\nPOSTGRES_WORKSPACE=dev_workspace  # For development environment\nMONGODB_WORKSPACE=prod_workspace  # For production environment\n\n# Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# Select: 3 (PGKVStorage with dev_workspace) -> 3 (MongoKVStorage with prod_workspace)\n```\n\n**Note**: This allows you to migrate between different logical data partitions while changing storage backends.\n\n## Tool Limitations\n\n1. **Same Storage Type Not Allowed**\n   - You cannot migrate between the same storage type (e.g., PostgreSQL to PostgreSQL)\n   - This is enforced by the tool automatically excluding the source storage type from target selection\n   - For same-storage migrations (e.g., database switches), use database-native tools instead\n2. **Only Default Mode Caches**\n   - Only migrates `default:extract:*`, `default:summary:*`, and `default:analysis:*`\n   - Query caches are not included\n4. **Network Dependency**\n   - Tool requires stable network connection for remote databases\n   - Large datasets may fail if connection is interrupted\n\n## Best Practices\n\n1. **Backup Before Migration**\n   - Always backup your data before migration\n   - Test migration on non-production data first\n\n2. **Verify Results**\n   - Check the verification output after migration\n   - Manually verify a few cache entries if needed\n\n3. **Monitor Performance**\n   - Watch database resource usage during migration\n   - Consider migrating in smaller batches if needed\n\n4. **Clean Old Data**\n   - After successful migration, consider cleaning old cache data\n   - Keep backups for a reasonable period before deletion\n","lightrag/tools/README_MIGRATE_GRAPH_STORAGE.md":"# Graph Storage Migration (`PGGraphStorage` → `PGTableGraphStorage`)\n\nOffline tool for moving an already-extracted entity-relation graph from the\nApache AGE backend (`PGGraphStorage`) to the plain-table backend\n(`PGTableGraphStorage`) without re-indexing the source documents.\n\nChanging `LIGHTRAG_GRAPH_STORAGE` normally means the previously extracted\ngraph is simply not visible to the new backend, so the documents have to be\nre-processed — an LLM-cost-bearing operation for a graph that already exists.\nThis tool copies the graph itself, through the public storage API only\n(`get_all_nodes` / `get_all_edges` to read, `upsert_nodes_batch` /\n`upsert_edges_batch` to write, `remove_nodes` / `remove_edges` to undo). It\nissues no raw SQL, so canonicalization, endpoint handling and identity rules\nstay in the already-tested storage layer.\n\nOnly the graph moves. Vector and KV storages are untouched, and because the\nmigrated graph keeps the same entity and relation identities, existing vector\ndata stays valid — nothing is re-embedded and nothing is re-extracted. The LLM\ncache is a separate concern with its own tool (see\n[README_MIGRATE_LLM_CACHE.md](README_MIGRATE_LLM_CACHE.md)).\n\nRe-indexing remains the general guidance for changing storage backends. This\ntool is an advanced path for one specific pair.\n\n## Before you run it\n\n**Stop every LightRAG writer first — the tool cannot check this for you.**\nIt reads the source graph, writes it to the target, and then verifies the\nresult. A concurrent writer mutating either side inside that window\ninvalidates the comparison, and the tool can neither detect a live writer nor\nlock the backends. Every other precondition below is enforced; this one is\npurely operational, and it is the assumption the whole design rests on. The\nsame requirement applies to `rebuild_vdb`.\n\n**The target graph slice must be empty.** The migration is defined as\npopulating an empty slice; that invariant is also what makes its failure\nhandling exact (see below). A non-empty target is refused before anything is\nwritten.\n\nBoth backends read the usual `POSTGRES_USER`, `POSTGRES_PASSWORD`,\n`POSTGRES_DATABASE`, `POSTGRES_HOST` and `POSTGRES_PORT` variables.\n\n## Usage\n\n```bash\n# Dry run (default): reads and checks everything, migrates nothing.\npython -m lightrag.tools.migrate_graph_storage\n\n# Perform the migration.\npython -m lightrag.tools.migrate_graph_storage --apply\n```\n\n| Flag | Default | Meaning |\n| --- | --- | --- |\n| `--apply` | off | Perform the migration. Without it no graph data is written (see the schema caveat below). |\n| `--workspace` | `WORKSPACE` env | Workspace to migrate. |\n| `--force-empty-target` | off | **Dangerous.** `drop()` the whole non-empty target graph slice before migrating, instead of refusing. |\n| `--source-backend` | `PGGraphStorage` | Source graph storage class name. |\n| `--target-backend` | `PGTableGraphStorage` | Target graph storage class name. |\n\n`--source-backend` / `--target-backend` are an extension seam, not an open\ndoor: only allow-listed pairs run, and Phase 1 allow-lists exactly\n`PGGraphStorage` → `PGTableGraphStorage`. Any other pair is rejected before a\nbackend is even constructed.\n\nThe dry run performs the same enumeration and the same refusal checks as the\nreal run, so it predicts what `--apply` will do. With `--force-empty-target`\nit reports that the apply run *would* drop the pre-existing slice, but never\ndrops anything itself.\n\n**A dry run migrates nothing, but it is not read-only, and its blast radius is\nnot limited to one workspace.** Both backends have to be initialized before\ntheir contents can be inspected, and `initialize()` is not a passive step. Two\ndifferent scopes are involved:\n\n- *Scoped to the workspace being migrated:* `_normalize_legacy_edges`\n  **deletes and re-inserts edge rows** (reversed duplicates, legacy\n  orderings) for this workspace and namespace only; the AGE side creates the\n  graph and its indexes.\n- *Table-wide, across every workspace and namespace:* the schema migration in\n  `PGTableGraphStorage`'s DDL. On a table whose primary key predates the\n  `namespace` column it **DELETEs duplicate rows** from `lightrag_graph_nodes`\n  and `lightrag_graph_edges` (keeping the most recently updated row per key)\n  before rebuilding the primary keys, and when the endpoint foreign keys are\n  absent it then **DELETEs every orphan edge** — an edge whose endpoint node\n  is missing — from the whole table. That sweep cannot be narrowed: as\n  `pgtable_impl` puts it, \"the sweep is table-wide because the FK is a global\n  composite constraint and cannot be validated per workspace.\" Both paths are\n  one-time (first install, or the first run after the namespace-PK upgrade)\n  and are skipped once the constraints exist, but the run that performs them\n  touches rows belonging to workspaces `--workspace` never named.\n\nSo a dry run against one workspace can still modify others, on a legacy\nschema. What it never does is migrate data or copy anything between backends —\nthat is the guarantee, and it is narrower than \"writes nothing\". Take a backup\nof both tables before the first run against a pre-`namespace` schema.\n\n**`--workspace` is cross-checked, not trusted.** The backends resolve their\nworkspace as `POSTGRES_WORKSPACE` env > `config.ini` `[postgres] workspace` >\nthe value passed in > `\"default\"`, so either the environment or a checked-in\nconfig file can outrank the flag. The tool compares what was requested\nagainst what the backends actually resolved to, refuses when they differ, and\nreports the resolved workspace — the slice a destructive run acted on is never\nleft implicit.\n\nExit code is 0 on success and non-zero on any failure.\n\n## The report\n\nBoth modes print a dict to stdout, following the convention of the other\ntools:\n\n```python\n{'mode': 'dry-run', 'source_backend': 'PGGraphStorage',\n 'target_backend': 'PGTableGraphStorage', 'workspace': 'default',\n 'nodes': 8123, 'edges': 41022,\n 'verified': False, 'compensated': False, 'target_non_empty': False,\n 'would_drop_target_slice': False,\n 'written_node_count': 8123, 'written_edge_count': 41022}\n```\n\n`mode` is `dry-run` or `apply`. `verified` is true only when the post-write\ncomparison passed. The destructive-consequence field is mode-symmetric:\n`would_drop_target_slice` in a dry run, `dropped_target_slice` in an apply run\n— so the preview cannot look clean while hiding that `--force-empty-target`\nwill destroy the existing slice.\n\n`nodes` / `edges` in a dry run also double as a sizing signal for the apply\nrun: both modes hold the whole graph in memory (see **It does not stream**\nbelow), so a dry run against the real source is the cheapest way to gauge how\nmuch memory the real run will need.\n\n## What makes it refuse\n\nEvery check below runs *before* the first write, and each one aborts the run\nrather than migrating a graph it cannot faithfully reproduce:\n\n- the storage pair is not allow-listed;\n- the target graph slice is not empty (unless `--force-empty-target`);\n- a node has no usable identity — `id` or `entity_id` missing, not a string,\n  or the two disagreeing. Both backends can legitimately produce such a row: a\n  vertex created with no properties is legal in AGE and enumerates as\n  `{\"id\": None}`, and NetworkX creates an edge's unknown endpoint as an\n  attribute-less node;\n- the same node id appears more than once, **whether or not the payloads\n  match**. The target keeps one row per id, so two physical source vertices\n  become one node and the graph's node count changes;\n- `a→b` and `b→a` both exist, **whether or not the payloads match**. The\n  target stores one row per canonical pair, and AGE computes degree by\n  counting relationship rows — so collapsing an identical reciprocal preserves\n  every property while changing degree and traversal. LightRAG's own write\n  path matches edges undirected, so it does not create reciprocals: finding\n  one means the source violates the invariant this tool assumes;\n- a payload contains a value PostgreSQL `jsonb` cannot store faithfully:\n  `NaN`, `Infinity` and `-Infinity` (which `agtype` represents and `jsonb`\n  rejects outright), or `-0.0` (which `jsonb` accepts but normalises to `0.0`,\n  and this tool's comparison treats signed zeros as different — so the write\n  would succeed and then fail its own verification). The general rule: if the\n  target cannot round-trip a value, refuse before the destructive step rather\n  than discover it after the point of no return;\n- more than one directed row *reaches the tool* for the same ordered pair — a\n  violation of the invariant that makes the canonical merge lossless. Note the\n  limit of this backstop: AGE enumerates edges with `SELECT DISTINCT source,\n  target, properties`, so byte-identical parallel relationships are already\n  collapsed to one row before the tool sees them. It can therefore only catch\n  parallels whose payloads *differ*. **This is the one cardinality gap the tool\n  cannot close**: byte-identical same-direction parallels are invisible to it,\n  and collapsing them changes degree without losing any property. Everything\n  else that would change cardinality — reciprocals, duplicate ids — is refused\n  outright.\n\nVerification after the write re-runs the source-side checks and compares node\nids, canonical edges and properties. It is driven from the source because the\ntarget has already canonicalized and therefore cannot reveal what was lost.\nProperty comparison is type-strict: `1`, `1.0`, `True` and `\"1\"` are four\ndifferent values, and `-0.0` differs from `0.0` — all are distinguishable\ndownstream.\n\n**What `verified` claims — and what it cannot.** The tool only ever sees what\nthe source's `get_all_nodes` / `get_all_edges` return, so its claim is scoped\nto that enumerated view: **every node and edge the enumerator returned came\nacross with its identity, its payload, and its count intact.** Within that\nview nothing silently collapses — reciprocals and duplicate ids are refused\nbefore the write rather than merged and blessed.\n\n`verified` does **not** prove that the physical AGE graph was reproduced. Two\nthings are already lost or hidden before this tool is handed a row:\n\n- **Physical multiplicity.** AGE enumerates with `SELECT DISTINCT source,\n  target, properties`, so two byte-identical relationships between the same\n  ordered pair arrive as one row. AGE's own degree counts relationship rows, so\n  such a source has degree 2 where the target will have 1 — and `verified` will\n  still be true, because both sides match the one row the enumerator showed.\n  Reading raw multiplicity would mean going around the storage API, which this\n  tool deliberately does not do.\n- **Payload keys named `id`, `source` or `target`.** `PGGraphStorage`'s\n  enumerator overwrites them with identity values when flattening each row, so\n  a business property under one of those names is gone upstream and no\n  comparison here can notice.\n\nIf exact physical fidelity matters for your graph, check for duplicate\nrelationships directly in the AGE tables before migrating.\n\n## If a write fails\n\nThe tool removes exactly what the run wrote — the migrated node ids plus every\nedge endpoint, since the target auto-creates missing endpoints without\nreporting them — using `remove_edges` first and then `remove_nodes` to respect\nthe foreign key. It never uses `drop()` for this; whole-slice deletion is\nreachable only through `--force-empty-target`. The written set is computed\nbefore the first write, so a failure at any point still has the complete set,\nand removing it is exact precisely because the slice started empty.\n\n`--force-empty-target` has an irreducible window. Every source check runs\nbefore the drop, so a malformed source cannot cost you the slice — but nothing\nproves the *write* will succeed. If the migration fails after the drop, the\ncompensation removes what this run wrote and the target ends up empty: the\npre-existing data is gone and the new data never landed. That is the bargain\nthe flag asks for; take a backup before using it.\n\nA failed `drop()` under `--force-empty-target` aborts the run rather than\nproceeding: the backend reports that failure by returning an error status\ninstead of raising, so an unchecked call would let the migration write into,\nand later compensate against, rows it does not own.\n\nCompensation is not atomic. There is no transaction spanning two backends, so\na crash during compensation can leave the target partially populated. Two\nthings make that recoverable: re-running the tool refuses a non-empty target\nrather than compounding the mess, and if compensation itself fails the report\ncarries the full `written_node_ids` and `written_edge_keys` so the remaining\nrows can be removed by hand.\n\n## What this tool does not do\n\n- **It does not migrate any other pair.** `PRECONDITION_HOOKS` and\n  `COMPENSATION_HOOKS` are named placeholders — declared, but not yet consulted\n  anywhere; the Phase 1 behaviour is hardcoded. Adding a pair means wiring them\n  as well as allow-listing it.\n- **It does not merge into a populated target.** Empty-target is a\n  precondition, not a convenience.\n- **It does not run in a single transaction.** Two backends, no shared\n  transaction; the failure handling above is the substitute.\n- **It does not stream.** The storage API enumerates whole graphs, so a run\n  holds the source graph, the derived write plan and — during verification —\n  the target graph in memory at the same time. Peak memory grows with the\n  graph, roughly 2x its decoded size, and no setting bounds it: the fail-closed\n  checks (duplicate ids, reciprocals, parallel pairs) are global properties\n  that need the whole-graph view. For a very large graph, run the tool on a\n  host sized accordingly, or migrate workspace by workspace. Running out of\n  memory *inside* the process (`MemoryError`) is handled like any other\n  failure — compensation removes what the run wrote — but a kernel OOM kill\n  terminates the process outright: nothing compensates, the report never\n  prints, and under `--force-empty-target` it lands after the drop. Size the\n  host before reaching for that flag.\n- **It does not protect you from a live writer.** See the preconditions.\n- **It does not disambiguate workspaces that share an AGE graph.** AGE derives\n  its graph name by replacing every non-alphanumeric character with `_`, so\n  `team-a` and `team_a` name the same AGE graph while PGTable keeps them as\n  distinct workspaces. Check which graph a workspace actually resolves to\n  before migrating one whose name contains punctuation.\n- **It does not move vectors, KV data or document status.**\n","AGENTS.md":"# Repository Guidelines\n\n## Project Overview\n\nLightRAG is a Retrieval-Augmented Generation (RAG) framework that uses graph-based knowledge representation for enhanced information retrieval. The system extracts entities and relationships from documents, builds a knowledge graph, and uses multiple retrieval modes (`local`, `global`, `hybrid`, `mix`, `naive`) for queries.\n\n## Project Structure\n\nTop-level directories:\n\n- **lightrag/**: Core Python package — see *Module Layout* below.\n- **lightrag_webui/**: React 19 + TypeScript client (Bun + Vite + Tailwind). UI components in `src/`.\n- **scripts/**: `test.sh` (preferred test runner), `setup/` interactive environment wizard (use `make env-*` rather than calling `setup.sh` directly — see *Configuration > Setup Wizard Outputs*), and release tooling.\n- **tests/**: Pytest coverage, organized into subdirectories that mirror `lightrag/` (see *Testing* below for layout). Working datasets stay in `inputs/`, `rag_storage/`, and `temp/`; deployment collateral lives in `docs/`, `k8s-deploy/`, and compose files.\n\n### Module Layout (`lightrag/`)\n\n- **lightrag.py**: Main orchestrator class (`LightRAG`) — assembled from mixins (see *LightRAG class composition*). Hosts `ainsert_custom_kg`, `_insert_done`, `_process_extract_entities`, `_refresh_addon_params_cache`, and `addon_params` accessors. Critical: always call `await rag.initialize_storages()` after instantiation.\n- **pipeline.py**: `_PipelineMixin` — owns the document ingestion pipeline (`apipeline_enqueue_documents`, `apipeline_process_enqueue_documents`, `apipeline_process_error_documents`), the `parse_native` / `parse_mineru` / `parse_docling` parser dispatchers, multimodal analysis, validation, and the worker scaffolding.\n- **utils_pipeline.py**: Pure helpers shared by the pipeline mixin and other entry points: doc-status field access, document identity (source key, content hash), parsed-artifact path resolution, parser payload normalization, multimodal entity augmentation, and `make_lightrag_doc_content`.\n- **llm_roles.py**: `RoleSpec` / `RoleLLMConfig` / `_RoleLLMState` / `ROLES` registry plus `_RoleLLMMixin` — role normalization, builder registration, wrapper rebuild, runtime config update, queue cleanup, sanitized config export, queue status reporting. Route role-specific behavior here rather than into provider modules.\n- **storage_migrations.py**: `_StorageMigrationMixin` — `check_and_migrate_data`, `_migrate_entity_relation_data`, `_migrate_chunk_tracking_storage`.\n- **addon_params.py**: `ObservableAddonParams` plus `default_addon_params` / `normalize_addon_params` helpers.\n- **operate.py**: Core extraction and query operations including entity/relation extraction, chunking, and multi-mode retrieval logic.\n- **base.py**: Abstract base classes for storage backends (`BaseKVStorage`, `BaseVectorStorage`, `BaseGraphStorage`, `BaseDocStatusStorage`).\n- **kg/**: Storage implementations (JSON, NetworkX, Neo4j, PostgreSQL, MongoDB, Redis, Milvus, Qdrant, Faiss, Memgraph, OpenSearch, NanoVectorDB). The backend registry (`STORAGE_IMPLEMENTATIONS` / `STORAGES`) lives in `kg/__init__.py`; `kg/factory.py::get_storage_class()` resolves backend classes from configuration.\n- **llm/**: LLM and embedding provider bindings (OpenAI, Ollama, Azure, Gemini, Bedrock, Anthropic, etc.). All async with caching support.\n- **parser/**: Unified parsing layer. `parser/routing.py` resolves engine and filename hints for `legacy`, `native`, `mineru`, and `docling` flows; `parser/debug.py` provides an offline LightRAG stub for the `parser/cli.py` debug entry point (`python -m lightrag.parser.cli`). Native format parsers live as sibling sub-packages under `parser/` (currently `parser/docx/`); external HTTP-based adapters live under `parser/external/` (`mineru`, `docling`) with shared helpers in `parser/external/_common.py`, `_manifest.py`, `_zip.py`.\n- **chunker/**: Chunking strategies (token-size, recursive character, semantic vector, paragraph semantic).\n- **api/**: FastAPI service (`lightrag_server.py`) with REST endpoints and Ollama-compatible API; routers under `routers/`, static Swagger assets, packaged WebUI output, and Gunicorn launcher.\n\n## Core Architecture\n\n### LightRAG class composition\n\n`LightRAG` is assembled from focused mixins (split out of the previously monolithic `lightrag.py`):\n\n```\nLightRAG → _RoleLLMMixin → _StorageMigrationMixin → _PipelineMixin → object\n```\n\nThe `@final` decorator on `LightRAG` is preserved — the mixin layering is an internal implementation detail, not an external subclassing surface. The public API (`ainsert`, `aquery`, `ainsert_custom_kg`, `initialize_storages`, etc.) is unchanged. `ainsert_custom_kg` and its internal construction logic, `_insert_done`, `_process_extract_entities`, `_refresh_addon_params_cache`, and the `addon_params` property accessors stay on `LightRAG` itself because they cut across multiple flows or depend on prompt-profile state.\n\n### Storage Layer\n\nLightRAG uses 4 storage types with pluggable backends:\n- **KV_STORAGE**: LLM response cache, text chunks, document info\n- **VECTOR_STORAGE**: Entity/relation/chunk embeddings\n- **GRAPH_STORAGE**: Entity-relation graph structure\n- **DOC_STATUS_STORAGE**: Document processing status tracking\n\nEach `LightRAG` instance can pass a `workspace` parameter for data isolation. Implementation differs per storage type:\n- **File-based**: subdirectories under `working_dir`.\n- **Collection-based**: collection name prefixes.\n- **Relational DB**: workspace column filtering.\n- **Qdrant**: payload-based partitioning.\n\n### Consistency without transactions\n\nLightRAG writes to independent stores — graph, KV, vector, doc-status — with **no transaction across them**. Every multi-store operation therefore has intermediate states, and no ordering removes them; an ordering only chooses which one it keeps.\n\n- **The rule:** an inconsistency is acceptable when it **heals itself later** (a retry, a rebuild, or the next run rewrites it) or is **harmless in direction**. Losing data is never acceptable; retaining an object that could have been deleted, or surfacing a chunk a query did not need, is.\n- A change must **improve on an accepted residue**, not swap it for its mirror. \"An inconsistent state exists\" is not by itself a defect report — the questions are which state, how it heals, and whether the alternative is better.\n- Every accepted residue is **written down** with its reason and recovery path, next to the code or in the relevant contract. An undocumented residue is a defect; a documented one is a decision.\n- This licenses nothing for **silent failure**. A durable write must never be reported as one that did not happen, and a failure must never be swallowed: fail loud, then let the documented residue heal.\n\n### File-backed storage contracts\n\n**Full contracts: [docs/design/NetworkXSingleWriterContract.md](docs/design/NetworkXSingleWriterContract.md) — read it before touching `lightrag/kg/networkx_impl.py` or any caller of `index_done_callback` on the graph store; [docs/design/FileBackedSnapshotContract.md](docs/design/FileBackedSnapshotContract.md) — read it before touching `lightrag/kg/nano_vector_db_impl.py`, `lightrag/kg/faiss_impl.py`, `lightrag/kg/json_kv_impl.py`, `lightrag/kg/json_doc_status_impl.py` or `lightrag/kg/file_fingerprint.py`.**\n\nFive storages keep their data in memory and publish it by rewriting a whole file, and **they do not share one model** — `JsonKVStorage` and `JsonDocStatusStorage` are the odd ones out and the contracts say so at length. Read the right one before assuming.\n\n- **All five**: a commit publishes the WHOLE namespace, so any writer's flush also publishes every other writer's pending mutation there, half-finished ones included. All five are supported for **small-scale testing and validation only**; no change to them may be justified by write throughput.\n- **`NetworkXStorage`, `NanoVectorDBStorage`, `FaissVectorDBStorage`** keep one in-memory copy per process and reconcile by reloading the file. Visibility rests on a **two-channel fence**: the file's own `(st_mtime_ns, st_size)` (authoritative, state) OR-ed with the `storage_updated` flag (accelerator, a consumable event). Both are permanent — their blind spots do not overlap.\n- Those three diverge on a write conflict, and the reason is in the contracts: the graph store **declines** the commit (it has no buffer to replay, and graph payloads are accumulate-over-read), the vector stores **reload and replay** their pending buffers and redo logs. Do not reopen reload-then-replay for the graph store without addressing the accumulate-over-read argument.\n- **`JsonKVStorage` and `JsonDocStatusStorage` use none of that.** Their data is a `Manager().dict()` every worker shares, so a mutation is visible everywhere immediately and there is nothing to reload — adding a `_get_*` entry method would be wrong. Their `storage_updated` flag means the OPPOSITE of the other three's: `True` is \"dirty data still to flush\", never \"fresher data on disk to reload\". Do not read it as a peer notification. `JsonDocStatusStorage` reimplements this protocol rather than inheriting it, so a change to one of the pair is almost always a change the other needs too; where they diverge is the flush trigger — doc-status writes that change scheduling state flush synchronously because doc-status is the pipeline's recovery anchor.\n- `NetworkXStorage` is the only storage that declares `requires_single_writer`, which is what puts the admin flows under `LightRAG._admin_write_gate`.\n\n### Pipeline concurrency contract\n\n**Full contract: [docs/design/PipelineConcurrencyContract.md](docs/design/PipelineConcurrencyContract.md) — read it before touching `lightrag/pipeline.py`, `lightrag/kg/pipeline_ingress.py`, `pipeline_status` fields, or any `/documents/*` endpoint.**\n\n- Concurrent writers coordinate through `pipeline_status` (per-workspace shared dict in `lightrag.kg.shared_storage`), mutated under `get_namespace_lock(\"pipeline_status\", workspace=...)`.\n- `busy` alone does NOT block enqueue — enqueue + processing are allowed to run concurrently. Three states do refuse it: `destructive_busy` (clear / delete, which drops storages), `scanning_exclusive` (scan's classification phase), and `manual_freeze_requested` (a manual retry draining the pipeline to idle).\n- **Admin graph writes** (`acreate_*` / `aedit_*` / `adelete_by_*` / `amerge_entities` / `ainsert_custom_kg`) run inside `LightRAG._admin_write_gate` when the graph storage declares `requires_single_writer` (`NetworkXStorage` only): a workspace admin lock (waited for) then the `busy` reservation (`kind=\"admin\"`, refuses on `busy` / `scanning`), in that fixed order and OUTSIDE the per-entity keyed locks. A pipeline start during the hold is deferred into the ingress mailbox and driven once on release. The routes' `check_pipeline_busy_or_raise` preflight exempts an `admin`-owned `busy` so a second REST edit reaches the admin lock and queues. Never re-acquire the admin lock inside `lightrag/utils_graph.py`.\n- The workspace **ingress mailbox** (`get_pipeline_ingress(workspace)`) is the pipeline's only wake-up channel; `doc_status` stays the source of truth, so a dropped notification is recovered by the next strict scan.\n- FAILED documents never resume automatically: they re-enter only through a sticky manual retry request (`/documents/scan`, `/documents/reprocess_failed`), granting ONE attempt each.\n- All scheduling-control-plane `doc_status` queries use `get_docs_by_statuses(..., strict=True)`; scheduler `full_docs` reads must distinguish confirmed-absent (`None`) from backend errors (raise).\n\n### Purge recovery contract\n\n**Full contract: [docs/design/PurgeRecoveryContract.md](docs/design/PurgeRecoveryContract.md) — read it before touching `_purge_kg_contributions`, `adelete_by_doc_id`, the anchor writes in `merge_nodes_and_edges`, the `kg_write_state` / `kg_purge` metadata, or the cache write ordering in `use_llm_func_with_cache`.**\n\n- \"What did this document contribute?\" is answerable only from the per-document write-ahead anchors (`full_entities` / `full_relations`). The reverse lookup through `text_chunks` is not a fallback — purge deletes those chunks.\n- Governing invariant: **a purge must never delete something that CARRIES attribution — a chunk row or an anchor row that names objects — and leave those objects behind.** `_purge_kg_contributions` **fails closed** (`RecoveryAnchorMissingError` → HTTP 409, nothing deleted) unless one of four proofs holds: `anchors`, `pre_graph`, `journal`, `empty_scope`.\n- **`kg_write_state` must never be inferred or backfilled** — it is written once at enqueue and is monotonic. A backfill reproduces the original silent-skip defect.\n- `kg_write_state` and `kg_purge` must stay in both `_DOC_STATUS_METADATA_CARRY_OVER_KEYS` and `_DOC_STATUS_METADATA_DIRECTIVE_KEYS` (`lightrag/utils_pipeline.py`); dropping either turns a resumable purge into a permanent refusal.\n- Chunk tracking (`entity_chunks` / `relation_chunks`) outranks graph `source_id`; code folding a `source_id` delta back into tracking must append genuine additions only.\n- LLM extraction cache rows are reachable only through the owning chunk's `llm_cache_list`, which makes that list an attribution carrier too: [LLM extraction cache reachability](docs/design/PurgeRecoveryContract.md#llm-extraction-cache-reachability) states the reference-before-row ordering, why a reference that cannot be recorded skips the cache write instead, and what the ordering does not close.\n- Merge and rename apply *Consistency without transactions* above: [the failure model](docs/design/PurgeRecoveryContract.md#merge-and-rename-failure-model) lists their ordering invariants, accepted residues and already-rejected remedies. Read it before reordering `_merge_entities_impl` or the rename branch of `_edit_entity_impl`.\n\n### Relation weight contract\n\n**Full contract: [docs/ProgramingWithCore.md](docs/ProgramingWithCore.md#relation-weight-contract)** — keep it synchronized with the core API docstrings, REST graph documentation, and custom-KG examples whenever relation write behavior changes.\n\n- `weight >= len(distinct real source IDs)` on the graph edge; a larger value is an optional importance boost. Empty IDs and the legacy placeholders `manual_creation` / `UNKNOWN` are not evidence, so a source-less relation may use any non-negative fractional weight.\n- Public ingress paths (`create_relation`, `edit_relation`, `insert_custom_kg`) must validate the complete relation **before the first storage mutation**. To go below the current evidence count, creation callers omit `source_id`, edit callers set it to an empty string in the same operation.\n- Entity merges use `max(all input weights, distinct merged real source IDs)`.\n- Legacy-row repair is **opportunistic, not a sweep**: `_merge_edges_then_upsert`'s KEEP-cap skip branch returns the stored edge without writing graph or vector record, so an undersized legacy row stays undersized until a merge, an unrelated edit, or a rebuild rewrites it.\n\n### Query Modes\n\n- **local**: Context-dependent retrieval focused on specific entities\n- **global**: Community/summary-based broad knowledge retrieval\n- **hybrid**: Combines local and global\n- **naive**: Direct vector search without graph\n- **mix**: Integrates KG and vector retrieval (recommended with reranker)\n\n## Development Commands\n\n### Setup\n```bash\n# Install with uv\nuv sync\nsource .venv/bin/activate  # Or: .venv\\Scripts\\activate on Windows\n\n# Install with API support\nuv sync --extra api\n\n# Install specific extras\nuv sync --extra offline-storage  # Storage backends\nuv sync --extra offline-llm      # LLM providers\nuv sync --extra test             # Testing dependencies\n```\n\n### API Server\n```bash\n# Copy and configure environment\ncp env.example .env  # Edit with your LLM/embedding configs\n\n# Build WebUI\ncd lightrag_webui\nbun install --frozen-lockfile\nbun run build\ncd ..\n\n# Run server\nlightrag-server                                           # Production\nuvicorn lightrag.api.lightrag_server:app --reload        # Development\nlightrag-gunicorn                                         # Multi-worker (gunicorn)\n```\n\n### WebUI\n\nEvery command below is run **from `lightrag_webui/`**, not the repository root.\n`bun test` in particular resolves `bunfig.toml` — and the preload paths inside\nit — relative to the working directory, so running it from the root silently\nloads no DOM (see *React component tests*).\n\n```bash\ncd lightrag_webui\nbun install --frozen-lockfile      # REQUIRED after any change to package.json /\n                                   # bun.lock, including a branch switch across\n                                   # one: `git checkout` does not update\n                                   # node_modules, and a stale tree surfaces as\n                                   # a wall of TS2307 \"Cannot find module\".\nbun run dev                        # Dev server (Node + Vite)\nbun run dev:bun                    # Dev server (Bun native)\nbun run build                      # Production build\nbun run preview                    # Preview production build\nbun run lint                       # ESLint over *.ts/tsx/js/jsx\n\n# Testing — Bun built-in runner (NOT Vitest/Jest)\nbun test                           # All tests\nbun test --watch                   # Watch mode\nbun test --coverage                # With coverage report\nbun test src/api/lightrag.test.ts  # Single test file\nbunx tsc --noEmit                  # Typecheck (`bun run build` does NOT typecheck)\n```\n\n### Testing\n\n- Use mock-based tests for external services (Redis, httpx, etc.) — do not depend on live services in unit tests.\n- Add regression tests for every bug fix.\n- **Run only the test directories that mirror the modules you changed**, and report which subset you ran plus its pass count. The suite is ~7800 offline tests: about 5 minutes serially, ~70 s with `-n auto` on an 8-core machine — still too slow for the edit loop. Every PR's CI runs the full offline suite (and `tests/setup/` when its inputs change) — proving nothing else broke is its job, not yours.\n- Derive the subset from the mirror layout below: `lightrag/api/config.py` → `tests/api/config/`, `lightrag/kg/redis_impl.py` → `tests/kg/redis_impl/`, `lightrag/chunker/` → `tests/chunker/`. When a change spans several modules, run each of their directories rather than widening to `tests/`.\n- **`tests/setup/` is the setup wizard's suite and runs only when a wizard input changed**: `scripts/setup/**`, `env.example`, `docker-compose*.yml`, `Makefile`, or `tests/setup/` itself. It never imports `lightrag`, so a `lightrag/` change never needs it. `norecursedirs` in `pyproject.toml` already skips it in every recursive run (`pytest`, `pytest tests`, the full suite, CI's main job); run it by naming it: `./scripts/test.sh tests/setup`. The easy one to miss is `env.example`: adding a config knob for `lightrag` usually edits it, and the wizard parses it, so that change runs `tests/setup/` too.\n- Run the full suite locally only at a milestone, or when the change is genuinely cross-cutting (`lightrag/base.py`, `lightrag/utils.py`, `lightrag/kg/shared_storage.py`, or anything every backend inherits). **Run it with `-n auto --dist worksteal`** (pytest-xdist, in the `pytest` extra): on an 8-core machine it cuts ~5 min to ~1 min. CI stays serial because hosted 4-vCPU runners showed no gain. Do not add `-n` to subset runs: every worker re-collects and re-imports the suite (~12 s), which costs more than a single directory saves. Drop `-n` when you need `pdb`/`-s` or ordered output.\n- Backend tests use pytest; frontend unit tests use Bun's built-in runner — see *WebUI* above and *React component tests* below.\n- **A WebUI change runs the WHOLE frontend check set**, from `lightrag_webui/`: `bun install --frozen-lockfile` (see *WebUI* above — skip it after a branch switch and every later step fails on missing modules), then `bun test`, `bunx tsc --noEmit`, and `bun run lint`. The subsetting rule above is a backend rule and does not apply — all three together take well under a minute (test ~2 s, typecheck ~14 s, lint ~21 s), so there is nothing to save by running less. Report the pass count. `bun run build` transpiles WITHOUT checking types, so skipping `tsc --noEmit` means nothing checks them.\n\n```bash\n# Preferred for fresh shells and automation; resolves PYTHON, venv, uv, .venv, venv, python, python3\n# Default during development: only the directories mirroring the changed modules\n./scripts/test.sh tests/api/config\n./scripts/test.sh tests/kg/redis_impl\n\n# Run specific test file\n./scripts/test.sh tests/kg/test_graph_storage.py\n\n# Full suite — ~7800 offline tests; milestones and cross-cutting changes only.\n# Parallel via pytest-xdist: ~70 s on 8 cores (vs ~5 min serial).\n# tests/setup is skipped by default (norecursedirs); name it explicitly when\n# scripts/setup/, env.example, docker-compose*.yml or Makefile change.\n./scripts/test.sh tests -m offline -n auto --dist worksteal\n./scripts/test.sh tests/setup -m offline -n auto --dist worksteal\n\n# Integration-test concurrency (a LightRAG fixture, NOT pytest parallelism)\n./scripts/test.sh tests --test-workers 4\n```\n\n- `tests/`: main test suite, mirrors feature folders. Place new tests under the subdirectory matching the module under test:\n  - `tests/api/{auth,config,routes}/` for FastAPI server tests (auth/token, config loading, route handlers); top-level `tests/api/` for app-wide concerns (path prefixes, Ollama-compatible endpoint).\n  - `tests/chunker/`, `tests/evaluation/`, `tests/extraction/` for the like-named modules.\n  - `tests/kg/<backend>_impl/` for backend-specific storage tests, mirroring the `lightrag/kg/<backend>_impl.py` file naming. The `_impl` suffix on every subdirectory keeps the layout uniform and avoids `sys.path` shadowing on names that overlap with top-level PyPI/stdlib packages (`faiss`, `json`, `neo4j`, `networkx`, `redis`) when a test is launched directly via `python tests/kg/...`. Current backends: `faiss_impl/`, `json_impl/`, `memgraph_impl/`, `milvus_impl/`, `mongo_impl/`, `nano_impl/`, `neo4j_impl/`, `networkx_impl/`, `opensearch_impl/`, `postgres_impl/`, `qdrant_impl/`, `redis_impl/`. `tests/kg/` root holds cross-backend tests (`test_graph_storage`, `test_batch_graph_operations`, `test_unified_lock_safety`, `test_file_atomic`).\n  - `tests/llm/<provider>_impl/` for provider-specific behavior, same `_impl` convention: `bedrock_impl/`, `gemini_impl/`, `ollama_impl/`, `openai_impl/`, `voyageai_impl/`, `zhipu_impl/`. `tests/llm/` root holds cross-provider concerns (embedding, VLM, cache, role).\n  - `tests/parser/`, `tests/parser/docx/`, `tests/parser/external/{mineru,docling}/` for parser implementations.\n  - `tests/pipeline/` for ingestion pipeline and doc-status behavior (including `test_pipeline_*`, `test_doc_status_*`, `test_multimodal_*`, `test_graph_keyed_locks`).\n  - `tests/sidecar/`, `tests/setup/`, `tests/workspace/` for the like-named cross-cutting concerns. `tests/setup/` never imports `lightrag`; it runs in its own path-filtered CI job (`.github/workflows/setup-tests.yml`) and `norecursedirs` keeps it out of every recursive run, so a new input the wizard reads must be added to that workflow's `paths`.\n  - When adding a new backend or LLM provider, create a new subdirectory plus an empty `__init__.py` rather than dropping the file in the parent directory root.\n- Markers (registered in `[tool.pytest.ini_options]` in `pyproject.toml`): `offline`, `integration`, `requires_db`, `requires_api`, `pg_smoke`. Integration tests are skipped by default via `-m \"not integration\"`; opt in with `--run-integration`.\n- Integration env vars: `LIGHTRAG_RUN_INTEGRATION=true`, `LIGHTRAG_KEEP_ARTIFACTS=true`, `LIGHTRAG_TEST_WORKERS=4`, plus storage-specific connection strings.\n\n#### React component tests\n\n**Full guide: [docs/design/WebUITestingGuide.md](docs/design/WebUITestingGuide.md) — read it before writing or converting any test under `lightrag_webui/src/`.**\n\n- Tests are **colocated** next to the module they cover (`src/features/SiteHeader.test.ts`); the `tests/` mirror layout above is a backend rule. A test file containing JSX must be named `.test.tsx`.\n- **Run `bun test` from `lightrag_webui/`.** The `bunfig.toml` preload that installs the DOM is resolved against the CWD, and from the repository root the failure is silent — pure logic tests still pass, only component tests break.\n- **Test rendered behavior by rendering it**, through `renderWithProviders` (`src/test/render.tsx`). Do not write new tests that `readFileSync` a `.tsx` and match substrings.\n- **Never assert `toBeNull()` / `not.toBeInTheDocument()` / `toBe(element)` on a DOM node** — Bun serialises the whole happy-dom element and the run appears to HANG instead of failing. Assert a count or an extracted boolean/string instead.\n- **The DOM is process-wide**: never leave a bare `delete` of the `window` / `document` global behind; use `withoutDomGlobals()` from `src/test/domGlobals.ts`.\n- **Never import the test harness from production code** — one import from `src/` ships happy-dom to the browser.\n- **Prove the test can fail**: break the behavior it pins, see it go red, then restore.\n\n### Linting\n```bash\nruff check .\n```\n\n## Key Implementation Patterns\n\n### LightRAG Initialization (Critical)\n\nThe most common error is forgetting to initialize storages (manifests as `AttributeError: __aenter__` or `KeyError: 'history_messages'`):\n\n```python\nimport asyncio\nfrom lightrag import LightRAG\nfrom lightrag.llm.openai import gpt_4o_mini_complete, openai_embed\n\nasync def main():\n    rag = LightRAG(\n        working_dir=\"./rag_storage\",\n        llm_model_func=gpt_4o_mini_complete,\n        embedding_func=openai_embed\n    )\n\n    # REQUIRED: Initialize storage backends\n    await rag.initialize_storages()\n\n    # Now safe to use\n    await rag.ainsert(\"Your text here\")\n    result = await rag.aquery(\"Your question\", param=QueryParam(mode=\"hybrid\"))\n\n    # Cleanup\n    await rag.finalize_storages()\n\nasyncio.run(main())\n```\n\n### Custom Embedding Functions\n\n**Full guide: [docs/ProgramingWithCore.md](docs/ProgramingWithCore.md#custom-embedding-functions)** — the `.func` unwrapping rule, `max_token_size`, the `(len(texts), embedding_dim)` return contract that `EmbeddingFunc.__call__` enforces on every call, and the clear-the-data-directory pitfall when switching models. Read it before writing or wrapping an embedding function.\n\n### Storage Configuration\n\nConfigure via environment variables or constructor params:\n\n```python\n# Environment-based (recommended for production)\n# See env.example for full list\n\n# Constructor-based\nrag = LightRAG(\n    working_dir=\"./storage\",\n    workspace=\"project_name\",  # For data isolation\n    kv_storage=\"PGKVStorage\",\n    vector_storage=\"PGVectorStorage\",\n    graph_storage=\"Neo4JStorage\",\n    doc_status_storage=\"PGDocStatusStorage\",\n    vector_db_storage_cls_kwargs={\n        \"cosine_better_than_threshold\": 0.2\n    }\n)\n```\n\n### Document Insertion\n\n```python\n# Single document\nawait rag.ainsert(\"Text content\")\n\n# Batch insertion\nawait rag.ainsert([\"Text 1\", \"Text 2\", ...])\n\n# With custom IDs\nawait rag.ainsert(\"Text\", ids=[\"doc-123\"])\n\n# With file paths (for citation)\nawait rag.ainsert([\"Text 1\", \"Text 2\"], file_paths=[\"doc1.pdf\", \"doc2.pdf\"])\n\n# Configure batch size\nrag = LightRAG(..., max_parallel_insert=4)  # Default: 3, max recommended: 10\n```\n\n### Query Configuration\n\n```python\nfrom lightrag import QueryParam\n\nresult = await rag.aquery(\n    \"Your question\",\n    param=QueryParam(\n        mode=\"mix\",                    # Recommended with reranker\n        top_k=60,                      # KG entities/relations to retrieve\n        chunk_top_k=20,                # Text chunks to retrieve\n        max_entity_tokens=6000,\n        max_relation_tokens=8000,\n        max_total_tokens=30000,\n        enable_rerank=True,\n        user_prompt=\"Additional instructions for LLM\",\n        stream=False\n    )\n)\n```\n\n## Frontend Debugging via Playwright\n\nFor WebUI bugs whose symptoms only surface in the rendered DOM — layout/overflow/scrollbar issues, transient flashes, third-party libraries attaching helpers to `<body>` outside React's tree, or end-to-end verification of a fix — drive the running dev server (`http://localhost:5173`) with the `document-skills:webapp-testing` skill instead of reasoning from source alone. Seed state directly via `localStorage` (persist key `settings-storage`, schema in `lightrag_webui/src/stores/settings.ts`) to skip live LLM calls. Use `wait_until=\"domcontentloaded\"` plus a selector wait — Vite dev's long-lived polling makes `networkidle` time out.\n\n## Configuration\n\n### .env Configuration\nPrimary configuration file for API server. Generate it with `make env-base` or copy `env.example` manually. Key sections:\n- Server settings (HOST, PORT, CORS)\n- Storage backends (connection strings via environment variables)\n- Query parameters (TOP_K, MAX_TOTAL_TOKENS, etc.)\n- Reranking configuration (RERANK_BINDING, RERANK_MODEL)\n- Authentication (AUTH_ACCOUNTS, LIGHTRAG_API_KEY)\n\nSee `env.example` for comprehensive template.\n\n### Setup Wizard Outputs\n- Keep `.env` host-usable. Container-only hostnames and staged SSL paths belong in the wizard-managed compose layer, not persisted back into `.env`.\n- Treat `docker-compose.final.yml` as generated output assembled from `scripts/setup/templates/*.yml`.\n- For setup workflow changes, prefer `make env-*` targets over direct `scripts/setup/setup.sh` calls.\n\n## Code Style\n\n### Language\nComments, backend code, log messages, and Git commit messages in English. Frontend uses i18next for multi-language support.\n\n### Docstrings and comments\n\nDocstrings state the **rules**: what a caller must do, what it must not do, and the gotchas it will otherwise be caught by. The **mechanism** — how it works, the accepted residues, and the alternatives already rejected — goes in `docs/design/` with a pointer from the docstring. A docstring that has grown into a design document is the thing this separates: it buries the code, and the same facts in two places drift apart.\n\nTwo rules are enforced by `tests/test_docstring_budget.py` rather than by review, because both are properties of the tree rather than of any one change:\n\n- **No docstring over 80 lines** (100 for a class). The limit is generous on purpose — it catches a document, not a thorough docstring.\n- **No source line may cite a GitHub issue number.** The referent does not survive a fork, so a comment saying something is \"documented in #NNNN\" leaves nothing that documents it. Name the thing instead (\"the two-channel fence\"), or move the content into `docs/design/` and cite that. Never delete such a reference bare — migrate what it pointed at first. Enforced over `lightrag/` only: **tests may cite issue numbers**, and many do. There the number names the defect the test pins, and the test itself — its name, docstring and assertions — is the documentation, so the reference is provenance rather than the thing carrying the meaning.\n\nA third test requires every `docs/**.md` path named in the package to resolve; nothing imports those strings, so a typo is otherwise silent.\n\n### Python\n- Follow PEP 8 with 4-space indentation\n- Use type annotations\n- Prefer dataclasses for state management\n- Use `lightrag.utils.logger` instead of print\n- Async/await patterns throughout\n\n### TypeScript / React (incl. WebUI ESLint)\n- Functional components with hooks; PascalCase for components\n- 2-space indentation, single quotes (enforced by `@stylistic` rules)\n- Tailwind utility-first styling\n- ESLint stack: TypeScript-ESLint + React Hooks plugin + Prettier; `@typescript-eslint/no-explicit-any` is disabled (allowed)\n\n## Commit and Pull Request Guidance\n\n- If this repo is a fork of `HKUDS/LightRAG`. Target to `HKUDS/LightRAG` when creating PRs, not the fork's own repo.\n- PR descriptions should include: summary, motivation, linked issues if applyed, what's changed, what's broken and how it works.\n- Write commit messages (subject and body) in English. Commit messages are repository artifacts — like code comments and log messages — not conversational replies, so they follow the English code-style rule above regardless of any per-conversation working language.\n","lightrag/tools/README_REBUILD_VDB.md":"# Offline Vector Storage (VDB) Rebuild Tool\n\n`lightrag-rebuild-vdb` restores consistency between LightRAG's authoritative\ndata sources and its vector storages by dropping each vector storage and\nrebuilding it from scratch:\n\n| Vector storage | Authoritative source |\n|---|---|\n| `entities_vdb` | graph nodes |\n| `relationships_vdb` | graph edges |\n| `chunks_vdb` | `text_chunks` KV store |\n\n## When do I need this?\n\nLightRAG performs multi-step writes (graph + vector storage). If a vector\nstorage write fails at runtime — embedder outage, network timeout, context\noverflow on a high-degree entity — the graph and the vector storage drift\napart. The most common symptom is the edge-count drift reported in issue\n[#2917](https://github.com/HKUDS/LightRAG/issues/2917): graph edges with no\nvector counterpart, so `local`/`hybrid` queries miss relations that exist in\nthe graph.\n\nSince v(next), `amerge_entities` raises `VectorStorageConsistencyError` when\nthis happens, with a pointer to this tool. No data is lost in this situation:\nthe graph and the `text_chunks` KV store hold everything needed to rebuild\nthe vectors.\n\nA full drop + rebuild also clears *reverse* orphans (records present in the\nvector storage but absent from the graph), which incremental repair cannot\nreliably do.\n\nYou can also use this tool after changing the embedding model or embedding\ndimension. Existing vector records were generated in the old embedding space\n(and some vector backends bind collections/tables to the configured dimension),\nso run the tool with the updated `.env` and choose **Rebuild ALL vector\nstorages** to regenerate `entities_vdb`, `relationships_vdb`, and `chunks_vdb`\nfrom the authoritative graph/KV sources. The consistency check is not enough\nfor this case because it only detects missing graph → VDB records, not vectors\nthat exist but were embedded with a previous model or dimension.\n\nA vector storage whose container was written in a different embedding space\n**refuses to attach** rather than serving an empty or foreign index. That\nrefusal is the condition this tool clears, so it is the one startup failure the\ntool tolerates: the three vector targets are initialized individually, a typed\n`VectorSpaceMismatchError` is recorded instead of aborting the run, and the\nrebuild opens by dropping that container and re-provisioning it in the current\nembedding space. No out-of-band step (deleting the index through the backend's\nown API, `DROP TABLE`, `rm`) is needed.\n\nEverything else still aborts. A cluster outage, a bad credential or a corrupt\nfile is not a model change, and the recovery for a model change is destructive —\nso it is only ever applied to a refusal that says so in its type. The\nauthoritative sources (graph storage and `text_chunks`) likewise keep the\nserver-identical startup path, migrations included: rebuilding vectors from a\nhalf-migrated source is worse than not rebuilding.\n\nSee `docs/design/VectorSpaceProvenance.md` for the provenance marker and the\nrules that decide when a backend refuses.\n\nThe tool also completes a graph-only backfill performed with\n`NoopVectorDBStorage`. Stop every writer, preserve the same working directory,\nworkspace, graph storage, and KV storage, switch to the intended persistent\nvector backend, configure the production embedding model and dimension, and\nselect **Rebuild ALL vector storages**. Create a new `LightRAG` instance or\nrestart the process after changing the vector backend. Do not start the server\nuntil the rebuild succeeds; there is currently no persisted vector-index\nreadiness marker.\n\n## Usage\n\n```bash\n# Stop the LightRAG Server first!\nlightrag-rebuild-vdb\n# or\npython -m lightrag.tools.rebuild_vdb\n```\n\nThe tool reads the same `.env` / environment configuration as the server\n(`LIGHTRAG_GRAPH_STORAGE`, `LIGHTRAG_VECTOR_STORAGE`, `LIGHTRAG_KV_STORAGE`,\n`WORKSPACE`, `WORKING_DIR`, `EMBEDDING_*`, backend connection settings) and\nbuilds its embedding function through the exact factory the server uses —\nrun it with the same `.env` so rebuilt vectors live in the same embedding\nspace the server queries against.\n\nMenu options:\n\n1. **Consistency check (diagnose only)** — probes every graph entity/relation\n   for a vector counterpart and reports what is missing. Run this first to\n   decide whether a rebuild is worth the embedding cost. The check covers\n   the graph → VDB direction only; reverse orphans can only be cleared by a\n   full rebuild. Legacy reverse-order relation ids (from old custom-KG\n   imports) are recognized and not misreported as missing. The check issues\n   read queries only and does not run a rebuild (no drop + re-embed). It is\n   **not** strictly side-effect-free, though: the tool initializes every\n   storage on startup — exactly as the server does — and for some backends\n   that includes schema/DDL setup and one-time legacy migrations (e.g. Qdrant\n   upserts into the new collection, PostgreSQL batch-inserts into the new\n   table, Milvus may create a temp collection and drop/rename the original).\n   Treat running the tool — even just for a check — like starting the server:\n   stop other writers first.\n2. **Rebuild entities + relationships VDB** — sufficient for the #2917\n   merge-failure scenario.\n3. **Rebuild chunks VDB**.\n4. **Rebuild ALL vector storages** — use this after changing the embedding\n   model or embedding dimension.\n\n## Important notes\n\n- **Stop the server first.** The tool drops and rewrites vector storages;\n  concurrent writers (any backend, not just file-based ones) can corrupt\n  data or lose updates.\n- **Embedding model/dimension changes.** Run the tool with the new embedding\n  configuration and rebuild all vector storages. A consistency check can still\n  pass when every vector record exists but was created with the old embedding\n  model or dimension — unless the backend recorded its embedding-space\n  provenance, in which case the check reports the refusal and says the rebuild\n  is mandatory rather than reporting every record as missing.\n- **A refused target is dropped only after you confirm the rebuild.** Choosing\n  the consistency check never destroys anything, so a refusal can be diagnosed\n  before it is acted on. Options 2–4 recover exactly the targets they rebuild:\n  picking option 2 leaves a refused `chunks_vdb` refused until option 3 or 4\n  runs.\n- **Embedding cost.** A rebuild re-embeds every affected record. On large\n  datasets this means real API cost and time. Use the check mode first, and\n  rebuild only the storages that need it.\n- **Idempotent / crash-safe.** Sources (graph, `text_chunks`) are never\n  modified. If the tool crashes between drop and rewrite, just re-run it.\n- **`__created_at__` reset.** Backends that store creation timestamps in\n  vector records (nano, faiss) will show fresh timestamps after a rebuild.\n  No query logic depends on them.\n- **Custom-KG placeholder entities.** `UNKNOWN` placeholder nodes created by\n  `ainsert_custom_kg` are rebuilt faithfully from the graph; they may gain a\n  vector record they previously lacked (improving their retrievability).\n- **Chunk enumeration is backend-specific.** `BaseKVStorage` has no key\n  enumeration API, so the tool scans each KV backend directly (JsonKV,\n  Redis, PostgreSQL, MongoDB, OpenSearch). When a new KV backend is added,\n  `enumerate_kv_keys()` in `rebuild_vdb.py` must be extended.\n\n## Library usage\n\nThe core rebuild/check functions are plain async functions that accept your\nown initialized storage instances:\n\nRebuild targets must persist a queryable vector index. Passing a graph-only\nbackend such as `NoopVectorDBStorage` raises before source records are read or\nthe target is dropped; configure a persistent vector backend first.\n\n```python\nfrom lightrag.tools.rebuild_vdb import (\n    check_vdb_consistency,\n    rebuild_chunks_vdb,\n    rebuild_entities_vdb,\n    rebuild_relationships_vdb,\n)\n\nreport = await check_vdb_consistency(graph, entities_vdb, relationships_vdb)\nif not report[\"consistent\"]:\n    await rebuild_entities_vdb(graph, entities_vdb, global_config)\n    await rebuild_relationships_vdb(graph, relationships_vdb, global_config)\n```\n","lightrag/tools/README_SOURCE_CONFLICT_REPAIR.md":"# Source-Conflict Listing / Repair\n\nOffline companion to the typed source resolver (LR2 §5.5).\n\nA canonical source key — a filename with any parser `[hint]` stripped — is\nexpected to belong to exactly one **primary** document. History can break that:\ncustom-ID inserts, legacy ids, and basename collisions predate the strict\nresolver. When two or more primary rows claim one key, the resolver returns a\n*conflict* and `/documents/scan` refuses to act on that file: it is not\nenqueued, no record is deleted and nothing is archived, because picking a winner\nautomatically could silently retire the document you wanted to keep.\n\nThis tool (and the equivalent HTTP endpoints) is how an operator settles such a\nconflict explicitly.\n\n- The winner is never chosen for you: you name the `primary_doc_id` to keep.\n- Every other candidate is marked `metadata.is_duplicate=true` with\n  `original_doc_id=<primary>`. **No content is ever deleted** — the demoted rows\n  keep their status and their `full_docs` entry; they only lose their claim on\n  the canonical source, which is what makes the resolver return a single\n  primary afterwards.\n- Only `doc_status` and `full_docs` are opened. No vector store, graph store,\n  LLM or embedding model is touched, so a doc_status bookkeeping problem never\n  requires a reachable Milvus/Neo4j/LLM endpoint to fix. `full_docs` is read (not\n  written) to verify the primary you named — see *Safety*.\n\n## Usage\n\nCLI (honors `WORKING_DIR` / `WORKSPACE` and the `LIGHTRAG_*` storage variables\nfrom `.env`, exactly like the server):\n\n```bash\n# What is conflicting?\npython -m lightrag.tools.source_conflict_repair list [--limit 50] [--all]\n\n# Plan a repair (dry-run: nothing is modified)\npython -m lightrag.tools.source_conflict_repair repair \\\n    --source report.docx --primary doc-abc123\n\n# Commit it\npython -m lightrag.tools.source_conflict_repair repair \\\n    --source report.docx --primary doc-abc123 --apply\n```\n\n`--workspace` overrides the `WORKSPACE` env var. Exit code is `1` when a repair\nis refused (unknown primary, candidate set changed, backend cannot repair).\n\nLibrary (for a deployment that builds its own `LightRAG`):\n\n```python\nfrom lightrag.tools.source_conflict_repair import (\n    collect_source_conflicts,\n    repair_one_conflict,\n)\n\nconflicts = await collect_source_conflicts(rag.doc_status, limit=50)\nresult = await repair_one_conflict(\n    rag.doc_status,\n    \"report.docx\",\n    \"doc-abc123\",\n    workspace=rag.workspace,\n    # Required for a COMMIT: it is what verifies the primary you named can\n    # actually keep the source. A dry-run needs none.\n    full_docs=rag.full_docs,\n    apply=True,\n)\n```\n\n## Online equivalent\n\n```\nGET  /documents/source_conflicts?limit=50&cursor=<next_cursor>\nPOST /documents/source_conflicts/repair\n     {\"canonical_source_key\": \"...\", \"primary_doc_id\": \"...\",\n      \"expected_candidate_count\": N, \"expected_candidate_fingerprint\": \"...\",\n      \"dry_run\": false}\n```\n\nThe HTTP flow is two explicit calls: a dry-run returns the\n`candidate_count`/`fingerprint` pair, and the commit echoes both back. The CLI\nperforms the same two steps for you — it automates the copy-paste, not the\nguard.\n\n## Safety\n\n- Listing and a repair without `--apply` mutate nothing.\n- A commit is mutually exclusive with every in-deployment writer that can change\n  which documents claim the key: enqueue (the enqueue-serialize lock), the\n  processing stage's duplicate marking (a keyed lock on the canonical source key,\n  which the marking takes too), and clear/delete + scan classification + manual\n  reset (a pending-enqueue reservation). So the two operations are ordered rather\n  than interleaved: if the marking goes first the commit refuses before demoting\n  anything; if the commit goes first it verifies the key is settled and returns,\n  and a later marking is an ordinary state transition. Only the standalone CLI and\n  a second deployment writing the same database fall outside this.\n- A commit re-reads the candidate set under the backend's repair lock and\n  proceeds only when the count and fingerprint still match the dry-run\n  (compare-and-set), so a concurrent enqueue / delete / repair fails the commit\n  instead of being overwritten. A refused commit changes nothing; re-run it.\n- A repair interrupted half-way needs no reconciliation and leaves no marker\n  behind: the rows it finished already express `duplicate`, the rest still\n  resolve as a conflict. Run the command again.\n- Repeating a completed repair is safe: the same request is refused (its token\n  described the old candidate set) and a fresh run is a no-op with one\n  candidate and nothing to demote.\n- A commit refuses a primary that cannot keep the source, because the demotions\n  are irreversible — a repair only demotes, and a key left with no candidate has\n  no conflict left to settle:\n  - no `full_docs` content (an unprocessable stub a scan would delete) → 409;\n  - content that already exists under a **different** source, which the\n    processing stage will mark `FAILED [DUPLICATE:content_hash]` → 409;\n  - the content could not be *verified* at all — read failure, a backend without\n    strict point reads, or no `full_docs` handle → 503, retry later. Unverified\n    is not verified.\n- Not covered, by construction: a primary that has not been parsed yet has no\n  content hash, so nothing can predict whether its content will turn out to\n  duplicate another document. The commit itself still succeeds, and the key\n  really is unique when it returns — no marking can interleave with it. A\n  marking that lands afterwards is an ordinary state transition, not a\n  half-applied repair: the outcome is the ordinary content-dedup steady state\n  (the content lives under that document and this key ends up with no primary),\n  and no content is lost.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/LightRAG/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# Repository Guidelines\n\n## Project Overview\n\nLightRAG is a Retrieval-Augmented Generation (RAG) framework that uses graph-based knowledge representation for enhanced information retrieval. The system extracts entities and relationships from documents, builds a knowledge graph, and uses multiple retrieval modes (`local`, `global`, `hybrid`, `mix`, `naive`) for queries.\n\n## Project Structure\n\nTop-level directories:\n\n- **lightrag/**: Core Python package — see *Module Layout* below.\n- **lightrag_webui/**: React 19 + TypeScript client (Bun + Vite + Tailwind). UI components in `src/`.\n- **scripts/**: `test.sh` (preferred test runner), `setup/` interactive environment wizard (use `make env-*` rather than calling `setup.sh` directly — see *Configuration > Setup Wizard Outputs*), and release tooling.\n- **tests/**: Pytest coverage, organized into subdirectories that mirror `lightrag/` (see *Testing* below for layout). Working datasets stay in `inputs/`, `rag_storage/`, and `temp/`; deployment collateral lives in `docs/`, `k8s-deploy/`, and compose files.\n\n### Module Layout (`lightrag/`)\n\n- **lightrag.py**: Main orchestrator class (`LightRAG`) — assembled from mixins (see *LightRAG class composition*). Hosts `ainsert_custom_kg`, `_insert_done`, `_process_extract_entities`, `_refresh_addon_params_cache`, and `addon_params` accessors. Critical: always call `await rag.initialize_storages()` after instantiation.\n- **pipeline.py**: `_PipelineMixin` — owns the document ingestion pipeline (`apipeline_enqueue_documents`, `apipeline_process_enqueue_documents`, `apipeline_process_error_documents`), the `parse_native` / `parse_mineru` / `parse_docling` parser dispatchers, multimodal analysis, validation, and the worker scaffolding.\n- **utils_pipeline.py**: Pure helpers shared by the pipeline mixin and other entry points: doc-status field access, document identity (source key, content hash), parsed-artifact path resolution, parser payload normalization, multimodal entity augmentation, and `make_lightrag_doc_content`.\n- **llm_roles.py**: `RoleSpec` / `RoleLLMConfig` / `_RoleLLMState` / `ROLES` registry plus `_RoleLLMMixin` — role normalization, builder registration, wrapper rebuild, runtime config update, queue cleanup, sanitized config export, queue status reporting. Route role-specific behavior here rather than into provider modules.\n- **storage_migrations.py**: `_StorageMigrationMixin` — `check_and_migrate_data`, `_migrate_entity_relation_data`, `_migrate_chunk_tracking_storage`.\n- **addon_params.py**: `ObservableAddonParams` plus `default_addon_params` / `normalize_addon_params` helpers.\n- **operate.py**: Core extraction and query operations including entity/relation extraction, chunking, and multi-mode retrieval logic.\n- **base.py**: Abstract base classes for storage backends (`BaseKVStorage`, `BaseVectorStorage`, `BaseGraphStorage`, `BaseDocStatusStorage`).\n- **kg/**: Storage implementations (JSON, NetworkX, Neo4j, PostgreSQL, MongoDB, Redis, Milvus, Qdrant, Faiss, Memgraph, OpenSearch, NanoVectorDB). The backend registry (`STORAGE_IMPLEMENTATIONS` / `STORAGES`) lives in `kg/__init__.py`; `kg/factory.py::get_storage_class()` resolves backend classes from configuration.\n- **llm/**: LLM and embedding provider bindings (OpenAI, Ollama, Azure, Gemini, Bedrock, Anthropic, etc.). All async with caching support.\n- **parser/**: Unified parsing layer. `parser/routing.py` resolves engine and filename hints for `legacy`, `native`, `mineru`, and `docling` flows; `parser/debug.py` provides an offline LightRAG stub for the `parser/cli.py` debug entry point (`python -m lightrag.parser.cli`). Native format parsers live as sibling sub-packages under `parser/` (currently `parser/docx/`); external HTTP-based adapters live under `parser/external/` (`mineru`, `docling`) with shared helpers in `parser/external/_common.py`, `_manifest.py`, `_zip.py`.\n- **chunker/**: Chunking strategies (token-size, recursive character, semantic vector, paragraph semantic).\n- **api/**: FastAPI service (`lightrag_server.py`) with REST endpoints and Ollama-compatible API; routers under `routers/`, static Swagger assets, packaged WebUI output, and Gunicorn launcher.\n\n## Core Architecture\n\n### LightRAG class composition\n\n`LightRAG` is assembled from focused mixins (split out of the previously monolithic `lightrag.py`):\n\n```\nLightRAG → _RoleLLMMixin → _StorageMigrationMixin → _PipelineMixin → object\n```\n\nThe `@final` decorator on `LightRAG` is preserved — the mixin layering is an internal implementation detail, not an external subclassing surface. The public API (`ainsert`, `aquery`, `ainsert_custom_kg`, `initialize_storages`, etc.) is unchanged. `ainsert_custom_kg` and its internal construction logic, `_insert_done`, `_process_extract_entities`, `_refresh_addon_params_cache`, and the `addon_params` property accessors stay on `LightRAG` itself because they cut across multiple flows or depend on prompt-profile state.\n\n### Storage Layer\n\nLightRAG uses 4 storage types with pluggable backends:\n- **KV_STORAGE**: LLM response cache, text chunks, document info\n- **VECTOR_STORAGE**: Entity/relation/chunk embeddings\n- **GRAPH_STORAGE**: Entity-relation graph structure\n- **DOC_STATUS_STORAGE**: Document processing status tracking\n\nEach `LightRAG` instance can pass a `workspace` parameter for data isolation. Implementation differs per storage type:\n- **File-based**: subdirectories under `working_dir`.\n- **Collection-based**: collection name prefixes.\n- **Relational DB**: workspace column filtering.\n- **Qdrant**: payload-based partitioning.\n\n### Consistency without transactions\n\nLightRAG writes to independent stores — graph, KV, vector, doc-status — with **no transaction across them**. Every multi-store operation therefore has intermediate states, and no ordering removes them; an ordering only chooses which one it keeps.\n\n- **The rule:** an inconsistency is acceptable when it **heals itself later** (a retry, a rebuild, or the next run rewrites it) or is **harmless in direction**. Losing data is never acceptable; retaining an object that could have been deleted, or surfacing a chunk a query did not need, is.\n- A change must **improve on an accepted residue**, not swap it for its mirror. \"An inconsistent state exists\" is not by itself a defect report — the questions are which state, how it heals, and whether the alternative is better.\n- Every accepted residue is **written down** with its reason and recovery path, next to the code or in the relevant contract. An undocumented residue is a defect; a documented one is a decision.\n- This licenses nothing for **silent failure**. A durable write must never be reported as one that did not happen, and a failure must never be swallowed: fail loud, then let the documented residue heal.\n\n### File-backed storage contracts\n\n**Full contracts: [docs/design/NetworkXSingleWriterContract.md](docs/design/NetworkXSingleWriterContract.md) — read it before touching `lightrag/kg/networkx_impl.py` or any caller of `index_done_callback` on the graph store; [docs/design/FileBackedSnapshotContract.md](docs/design/FileBackedSnapshotContract.md) — read it before touching `lightrag/kg/nano_vector_db_impl.py`, `lightrag/kg/faiss_impl.py`, `lightrag/kg/json_kv_impl.py`, `lightrag/kg/json_doc_status_impl.py` or `lightrag/kg/file_fingerprint.py`.**\n\nFive storages keep their data in memory and publish it by rewriting a whole file, and **they do not share one model** — `JsonKVStorage` and `JsonDocStatusStorage` are the odd ones out and the contracts say so at length. Read the right one before assuming.\n\n- **All five**: a commit publishes the WHOLE namespace, so any writer's flush also publishes every other writer's pending mutation there, half-finished ones included. All five are supported for **small-scale testing and validation only**; no change to them may be justified by write throughput.\n- **`NetworkXStorage`, `NanoVectorDBStorage`, `FaissVectorDBStorage`** keep one in-memory copy per process and reconcile by reloading the file. Visibility rests on a **two-channel fence**: the file's own `(st_mtime_ns, st_size)` (authoritative, state) OR-ed with the `storage_updated` flag (accelerator, a consumable event). Both are permanent — their blind spots do not overlap.\n- Those three diverge on a write conflict, and the reason is in the contracts: the graph store **declines** the commit (it has no buffer to replay, and graph payloads are accumulate-over-read), the vector stores **reload and replay** their pending buffers and redo logs. Do not reopen reload-then-replay for the graph store without addressing the accumulate-over-read argument.\n- **`JsonKVStorage` and `JsonDocStatusStorage` use none of that.** Their data is a `Manager().dict()` every worker shares, so a mutation is visible everywhere immediately and there is nothing to reload — adding a `_get_*` entry method would be wrong. Their `storage_updated` flag means the OPPOSITE of the other three's: `True` is \"dirty data still to flush\", never \"fresher data on disk to reload\". Do not read it as a peer notification. `JsonDocStatusStorage` reimplements this protocol rather than inheriting it, so a change to one of the pair is almost always a change the other needs too; where they diverge is the flush trigger — doc-status writes that change scheduling state flush synchronously because doc-status is the pipeline's recovery anchor.\n- `NetworkXStorage` is the only storage that declares `requires_single_writer`, which is what puts the admin flows under `LightRAG._admin_write_gate`.\n\n### Pipeline concurrency contract\n\n**Full contract: [docs/design/PipelineConcurrencyContract.md](docs/design/PipelineConcurrencyContract.md) — read it before touching `lightrag/pipeline.py`, `lightrag/kg/pipeline_ingress.py`, `pipeline_status` fields, or any `/documents/*` endpoint.**\n\n- Concurrent writers coordinate through `pipeline_status` (per-workspace shared dict in `lightrag.kg.shared_storage`), mutated under `get_namespace_lock(\"pipeline_status\", workspace=...)`.\n- `busy` alone does NOT block enqueue — enqueue + processing are allowed to run concurrently. Three states do refuse it: `destructive_busy` (clear / delete, which drops storages), `scanning_exclusive` (scan's classification phase), and `manual_freeze_requested` (a manual retry draining the pipeline to idle).\n- **Admin graph writes** (`acreate_*` / `aedit_*` / `adelete_by_*` / `amerge_entities` / `ainsert_custom_kg`) run inside `LightRAG._admin_write_gate` when the graph storage declares `requires_single_writer` (`NetworkXStorage` only): a workspace admin lock (waited for) then the `busy` reservation (`kind=\"admin\"`, refuses on `busy` / `scanning`), in that fixed order and OUTSIDE the per-entity keyed locks. A pipeline start during the hold is deferred into the ingress mailbox and driven once on release. The routes' `check_pipeline_busy_or_raise` preflight exempts an `admin`-owned `busy` so a second REST edit reaches the admin lock and queues. Never re-acquire the admin lock inside `lightrag/utils_graph.py`.\n- The workspace **ingress mailbox** (`get_pipeline_ingress(workspace)`) is the pipeline's only wake-up channel; `doc_status` stays the source of truth, so a dropped notification is recovered by the next strict scan.\n- FAILED documents never resume automatically: they re-enter only through a sticky manual retry request (`/documents/scan`, `/documents/reprocess_failed`), granting ONE attempt each.\n- All scheduling-control-plane `doc_status` queries use `get_docs_by_statuses(..., strict=True)`; scheduler `full_docs` reads must distinguish confirmed-absent (`None`) from backend errors (raise).\n\n### Purge recovery contract\n\n**Full contract: [docs/design/PurgeRecoveryContract.md](docs/design/PurgeRecoveryContract.md) — read it before touching `_purge_kg_contributions`, `adelete_by_doc_id`, the anchor writes in `merge_nodes_and_edges`, the `kg_write_state` / `kg_purge` metadata, or the cache write ordering in `use_llm_func_with_cache`.**\n\n- \"What did this document contribute?\" is answerable only from the per-document write-ahead anchors (`full_entities` / `full_relations`). The reverse lookup through `text_chunks` is not a fallback — purge deletes those chunks.\n- Governing invariant: **a purge must never delete something that CARRIES attribution — a chunk row or an anchor row that names objects — and leave those objects behind.** `_purge_kg_contributions` **fails closed** (`RecoveryAnchorMissingError` → HTTP 409, nothing deleted) unless one of four proofs holds: `anchors`, `pre_graph`, `journal`, `empty_scope`.\n- **`kg_write_state` must never be inferred or backfilled** — it is written once at enqueue and is monotonic. A backfill reproduces the original silent-skip defect.\n- `kg_write_state` and `kg_purge` must stay in both `_DOC_STATUS_METADATA_CARRY_OVER_KEYS` and `_DOC_STATUS_METADATA_DIRECTIVE_KEYS` (`lightrag/utils_pipeline.py`); dropping either turns a resumable purge into a permanent refusal.\n- Chunk tracking (`entity_chunks` / `relation_chunks`) outranks graph `source_id`; code folding a `source_id` delta back into tracking must append genuine additions only.\n- LLM extraction cache rows are reachable only through the owning chunk's `llm_cache_list`, which makes that list an attribution carrier too: [LLM extraction cache reachability](docs/design/PurgeRecoveryContract.md#llm-extraction-cache-reachability) states the reference-before-row ordering, why a reference that cannot be recorded skips the cache write instead, and what the ordering does not close.\n- Merge and rename apply *Consistency without transactions* above: [the failure model](docs/design/PurgeRecoveryContract.md#merge-and-rename-failure-model) lists their ordering invariants, accepted residues and already-rejected remedies. Read it before reordering `_merge_entities_impl` or the rename branch of `_edit_entity_impl`.\n\n### Relation weight contract\n\n**Full contract: [docs/ProgramingWithCore.md](docs/ProgramingWithCore.md#relation-weight-contract)** — keep it synchronized with the core API docstrings, REST graph documentation, and custom-KG examples whenever relation write behavior changes.\n\n- `weight >= len(distinct real source IDs)` on the graph edge; a larger value is an optional importance boost. Empty IDs and the legacy placeholders `manual_creation` / `UNKNOWN` are not evidence, so a source-less relation may use any non-negative fractional weight.\n- Public ingress paths (`create_relation`, `edit_relation`, `insert_custom_kg`) must validate the complete relation **before the first storage mutation**. To go below the current evidence count, creation callers omit `source_id`, edit callers set it to an empty string in the same operation.\n- Entity merges use `max(all input weights, distinct merged real source IDs)`.\n- Legacy-row repair is **opportunistic, not a sweep**: `_merge_edges_then_upsert`'s KEEP-cap skip branch returns the stored edge without writing graph or vector record, so an undersized legacy row stays undersized until a merge, an unrelated edit, or a rebuild rewrites it.\n\n### Query Modes\n\n- **local**: Context-dependent retrieval focused on specific entities\n- **global**: Community/summary-based broad knowledge retrieval\n- **hybrid**: Combines local and global\n- **naive**: Direct vector search without graph\n- **mix**: Integrates KG and vector retrieval (recommended with reranker)\n\n## Development Commands\n\n### Setup\n```bash\n# Install with uv\nuv sync\nsource .venv/bin/activate  # Or: .venv\\Scripts\\activate on Windows\n\n# Install with API support\nuv sync --extra api\n\n# Install specific extras\nuv sync --extra offline-storage  # Storage backends\nuv sync --extra offline-llm      # LLM providers\nuv sync --extra test             # Testing dependencies\n```\n\n### API Server\n```bash\n# Copy and configure environment\ncp env.example .env  # Edit with your LLM/embedding configs\n\n# Build WebUI\ncd lightrag_webui\nbun install --frozen-lockfile\nbun run build\ncd ..\n\n# Run server\nlightrag-server                                           # Production\nuvicorn lightrag.api.lightrag_server:app --reload        # Development\nlightrag-gunicorn                                         # Multi-worker (gunicorn)\n```\n\n### WebUI\n\nEvery command below is run **from `lightrag_webui/`**, not the repository root.\n`bun test` in particular resolves `bunfig.toml` — and the preload paths inside\nit — relative to the working directory, so running it from the root silently\nloads no DOM (see *React component tests*).\n\n```bash\ncd lightrag_webui\nbun install --frozen-lockfile      # REQUIRED after any change to package.json /\n                                   # bun.lock, including a branch switch across\n                                   # one: `git checkout` does not update\n                                   # node_modules, and a stale tree surfaces as\n                                   # a wall of TS2307 \"Cannot find module\".\nbun run dev                        # Dev server (Node + Vite)\nbun run dev:bun                    # Dev server (Bun native)\nbun run build                      # Production build\nbun run preview                    # Preview production build\nbun run lint                       # ESLint over *.ts/tsx/js/jsx\n\n# Testing — Bun built-in runner (NOT Vitest/Jest)\nbun test                           # All tests\nbun test --watch                   # Watch mode\nbun test --coverage                # With coverage report\nbun test src/api/lightrag.test.ts  # Single test file\nbunx tsc --noEmit                  # Typecheck (`bun run build` does NOT typecheck)\n```\n\n### Testing\n\n- Use mock-based tests for external services (Redis, httpx, etc.) — do not depend on live services in unit tests.\n- Add regression tests for every bug fix.\n- **Run only the test directories that mirror the modules you changed**, and report which subset you ran plus its pass count. The suite is ~7800 offline tests: about 5 minutes serially, ~70 s with `-n auto` on an 8-core machine — still too slow for the edit loop. Every PR's CI runs the full offline suite (and `tests/setup/` when its inputs change) — proving nothing else broke is its job, not yours.\n- Derive the subset from the mirror layout below: `lightrag/api/config.py` → `tests/api/config/`, `lightrag/kg/redis_impl.py` → `tests/kg/redis_impl/`, `lightrag/chunker/` → `tests/chunker/`. When a change spans several modules, run each of their directories rather than widening to `tests/`.\n- **`tests/setup/` is the setup wizard's suite and runs only when a wizard input changed**: `scripts/setup/**`, `env.example`, `docker-compose*.yml`, `Makefile`, or `tests/setup/` itself. It never imports `lightrag`, so a `lightrag/` change never needs it. `norecursedirs` in `pyproject.toml` already skips it in every recursive run (`pytest`, `pytest tests`, the full suite, CI's main job); run it by naming it: `./scripts/test.sh tests/setup`. The easy one to miss is `env.example`: adding a config knob for `lightrag` usually edits it, and the wizard parses it, so that change runs `tests/setup/` too.\n- Run the full suite locally only at a milestone, or when the change is genuinely cross-cutting (`lightrag/base.py`, `lightrag/utils.py`, `lightrag/kg/shared_storage.py`, or anything every backend inherits). **Run it with `-n auto --dist worksteal`** (pytest-xdist, in the `pytest` extra): on an 8-core machine it cuts ~5 min to ~1 min. CI stays serial because hosted 4-vCPU runners showed no gain. Do not add `-n` to subset runs: every worker re-collects and re-imports the suite (~12 s), which costs more than a single directory saves. Drop `-n` when you need `pdb`/`-s` or ordered output.\n- Backend tests use pytest; frontend unit tests use Bun's built-in runner — see *WebUI* above and *React component tests* below.\n- **A WebUI change runs the WHOLE frontend check set**, from `lightrag_webui/`: `bun install --frozen-lockfile` (see *WebUI* above — skip it after a branch switch and every later step fails on missing modules), then `bun test`, `bunx tsc --noEmit`, and `bun run lint`. The subsetting rule above is a backend rule and does not apply — all three together take well under a minute (test ~2 s, typecheck ~14 s, lint ~21 s), so there is nothing to save by running less. Report the pass count. `bun run build` transpiles WITHOUT checking types, so skipping `tsc --noEmit` means nothing checks them.\n\n```bash\n# Preferred for fresh shells and automation; resolves PYTHON, venv, uv, .venv, venv, python, python3\n# Default during development: only the directories mirroring the changed modules\n./scripts/test.sh tests/api/config\n./scripts/test.sh tests/kg/redis_impl\n\n# Run specific test file\n./scripts/test.sh tests/kg/test_graph_storage.py\n\n# Full suite — ~7800 offline tests; milestones and cross-cutting changes only.\n# Parallel via pytest-xdist: ~70 s on 8 cores (vs ~5 min serial).\n# tests/setup is skipped by default (norecursedirs); name it explicitly when\n# scripts/setup/, env.example, docker-compose*.yml or Makefile change.\n./scripts/test.sh tests -m offline -n auto --dist worksteal\n./scripts/test.sh tests/setup -m offline -n auto --dist worksteal\n\n# Integration-test concurrency (a LightRAG fixture, NOT pytest parallelism)\n./scripts/test.sh tests --test-workers 4\n```\n\n- `tests/`: main test suite, mirrors feature folders. Place new tests under the subdirectory matching the module under test:\n  - `tests/api/{auth,config,routes}/` for FastAPI server tests (auth/token, config loading, route handlers); top-level `tests/api/` for app-wide concerns (path prefixes, Ollama-compatible endpoint).\n  - `tests/chunker/`, `tests/evaluation/`, `tests/extraction/` for the like-named modules.\n  - `tests/kg/<backend>_impl/` for backend-specific storage tests, mirroring the `lightrag/kg/<backend>_impl.py` file naming. The `_impl` suffix on every subdirectory keeps the layout uniform and avoids `sys.path` shadowing on names that overlap with top-level PyPI/stdlib packages (`faiss`, `json`, `neo4j`, `networkx`, `redis`) when a test is launched directly via `python tests/kg/...`. Current backends: `faiss_impl/`, `json_impl/`, `memgraph_impl/`, `milvus_impl/`, `mongo_impl/`, `nano_impl/`, `neo4j_impl/`, `networkx_impl/`, `opensearch_impl/`, `postgres_impl/`, `qdrant_impl/`, `redis_impl/`. `tests/kg/` root holds cross-backend tests (`test_graph_storage`, `test_batch_graph_operations`, `test_unified_lock_safety`, `test_file_atomic`).\n  - `tests/llm/<provider>_impl/` for provider-specific behavior, same `_impl` convention: `bedrock_impl/`, `gemini_impl/`, `ollama_impl/`, `openai_impl/`, `voyageai_impl/`, `zhipu_impl/`. `tests/llm/` root holds cross-provider concerns (embedding, VLM, cache, role).\n  - `tests/parser/`, `tests/parser/docx/`, `tests/parser/external/{mineru,docling}/` for parser implementations.\n  - `tests/pipeline/` for ingestion pipeline and doc-status behavior (including `test_pipeline_*`, `test_doc_status_*`, `test_multimodal_*`, `test_graph_keyed_locks`).\n  - `tests/sidecar/`, `tests/setup/`, `tests/workspace/` for the like-named cross-cutting concerns. `tests/setup/` never imports `lightrag`; it runs in its own path-filtered CI job (`.github/workflows/setup-tests.yml`) and `norecursedirs` keeps it out of every recursive run, so a new input the wizard reads must be added to that workflow's `paths`.\n  - When adding a new backend or LLM provider, create a new subdirectory plus an empty `__init__.py` rather than dropping the file in the parent directory root.\n- Markers (registered in `[tool.pytest.ini_options]` in `pyproject.toml`): `offline`, `integration`, `requires_db`, `requires_api`, `pg_smoke`. Integration tests are skipped by default via `-m \"not integration\"`; opt in with `--run-integration`.\n- Integration env vars: `LIGHTRAG_RUN_INTEGRATION=true`, `LIGHTRAG_KEEP_ARTIFACTS=true`, `LIGHTRAG_TEST_WORKERS=4`, plus storage-specific connection strings.\n\n#### React component tests\n\n**Full guide: [docs/design/WebUITestingGuide.md](docs/design/WebUITestingGuide.md) — read it before writing or converting any test under `lightrag_webui/src/`.**\n\n- Tests are **colocated** next to the module they cover (`src/features/SiteHeader.test.ts`); the `tests/` mirror layout above is a backend rule. A test file containing JSX must be named `.test.tsx`.\n- **Run `bun test` from `lightrag_webui/`.** The `bunfig.toml` preload that installs the DOM is resolved against the CWD, and from the repository root the failure is silent — pure logic tests still pass, only component tests break.\n- **Test rendered behavior by rendering it**, through `renderWithProviders` (`src/test/render.tsx`). Do not write new tests that `readFileSync` a `.tsx` and match substrings.\n- **Never assert `toBeNull()` / `not.toBeInTheDocument()` / `toBe(element)` on a DOM node** — Bun serialises the whole happy-dom element and the run appears to HANG instead of failing. Assert a count or an extracted boolean/string instead.\n- **The DOM is process-wide**: never leave a bare `delete` of the `window` / `document` global behind; use `withoutDomGlobals()` from `src/test/domGlobals.ts`.\n- **Never import the test harness from production code** — one import from `src/` ships happy-dom to the browser.\n- **Prove the test can fail**: break the behavior it pins, see it go red, then restore.\n\n### Linting\n```bash\nruff check .\n```\n\n## Key Implementation Patterns\n\n### LightRAG Initialization (Critical)\n\nThe most common error is forgetting to initialize storages (manifests as `AttributeError: __aenter__` or `KeyError: 'history_messages'`):\n\n```python\nimport asyncio\nfrom lightrag import LightRAG\nfrom lightrag.llm.openai import gpt_4o_mini_complete, openai_embed\n\nasync def main():\n    rag = LightRAG(\n        working_dir=\"./rag_storage\",\n        llm_model_func=gpt_4o_mini_complete,\n        embedding_func=openai_embed\n    )\n\n    # REQUIRED: Initialize storage backends\n    await rag.initialize_storages()\n\n    # Now safe to use\n    await rag.ainsert(\"Your text here\")\n    result = await rag.aquery(\"Your question\", param=QueryParam(mode=\"hybrid\"))\n\n    # Cleanup\n    await rag.finalize_storages()\n\nasyncio.run(main())\n```\n\n### Custom Embedding Functions\n\n**Full guide: [docs/ProgramingWithCore.md](docs/ProgramingWithCore.md#custom-embedding-functions)** — the `.func` unwrapping rule, `max_token_size`, the `(len(texts), embedding_dim)` return contract that `EmbeddingFunc.__call__` enforces on every call, and the clear-the-data-directory pitfall when switching models. Read it before writing or wrapping an embedding function.\n\n### Storage Configuration\n\nConfigure via environment variables or constructor params:\n\n```python\n# Environment-based (recommended for production)\n# See env.example for full list\n\n# Constructor-based\nrag = LightRAG(\n    working_dir=\"./storage\",\n    workspace=\"project_name\",  # For data isolation\n    kv_storage=\"PGKVStorage\",\n    vector_storage=\"PGVectorStorage\",\n    graph_storage=\"Neo4JStorage\",\n    doc_status_storage=\"PGDocStatusStorage\",\n    vector_db_storage_cls_kwargs={\n        \"cosine_better_than_threshold\": 0.2\n    }\n)\n```\n\n### Document Insertion\n\n```python\n# Single document\nawait rag.ainsert(\"Text content\")\n\n# Batch insertion\nawait rag.ainsert([\"Text 1\", \"Text 2\", ...])\n\n# With custom IDs\nawait rag.ainsert(\"Text\", ids=[\"doc-123\"])\n\n# With file paths (for citation)\nawait rag.ainsert([\"Text 1\", \"Text 2\"], file_paths=[\"doc1.pdf\", \"doc2.pdf\"])\n\n# Configure batch size\nrag = LightRAG(..., max_parallel_insert=4)  # Default: 3, max recommended: 10\n```\n\n### Query Configuration\n\n```python\nfrom lightrag import QueryParam\n\nresult = await rag.aquery(\n    \"Your question\",\n    param=QueryParam(\n        mode=\"mix\",                    # Recommended with reranker\n        top_k=60,                      # KG entities/relations to retrieve\n        chunk_top_k=20,                # Text chunks to retrieve\n        max_entity_tokens=6000,\n        max_relation_tokens=8000,\n        max_total_tokens=30000,\n        enable_rerank=True,\n        user_prompt=\"Additional instructions for LLM\",\n        stream=False\n    )\n)\n```\n\n## Frontend Debugging via Playwright\n\nFor WebUI bugs whose symptoms only surface in the rendered DOM — layout/overflow/scrollbar issues, transient flashes, third-party libraries attaching helpers to `<body>` outside React's tree, or end-to-end verification of a fix — drive the running dev server (`http://localhost:5173`) with the `document-skills:webapp-testing` skill instead of reasoning from source alone. Seed state directly via `localStorage` (persist key `settings-storage`, schema in `lightrag_webui/src/stores/settings.ts`) to skip live LLM calls. Use `wait_until=\"domcontentloaded\"` plus a selector wait — Vite dev's long-lived polling makes `networkidle` time out.\n\n## Configuration\n\n### .env Configuration\nPrimary configuration file for API server. Generate it with `make env-base` or copy `env.example` manually. Key sections:\n- Server settings (HOST, PORT, CORS)\n- Storage backends (connection strings via environment variables)\n- Query parameters (TOP_K, MAX_TOTAL_TOKENS, etc.)\n- Reranking configuration (RERANK_BINDING, RERANK_MODEL)\n- Authentication (AUTH_ACCOUNTS, LIGHTRAG_API_KEY)\n\nSee `env.example` for comprehensive template.\n\n### Setup Wizard Outputs\n- Keep `.env` host-usable. Container-only hostnames and staged SSL paths belong in the wizard-managed compose layer, not persisted back into `.env`.\n- Treat `docker-compose.final.yml` as generated output assembled from `scripts/setup/templates/*.yml`.\n- For setup workflow changes, prefer `make env-*` targets over direct `scripts/setup/setup.sh` calls.\n\n## Code Style\n\n### Language\nComments, backend code, log messages, and Git commit messages in English. Frontend uses i18next for multi-language support.\n\n### Docstrings and comments\n\nDocstrings state the **rules**: what a caller must do, what it must not do, and the gotchas it will otherwise be caught by. The **mechanism** — how it works, the accepted residues, and the alternatives already rejected — goes in `docs/design/` with a pointer from the docstring. A docstring that has grown into a design document is the thing this separates: it buries the code, and the same facts in two places drift apart.\n\nTwo rules are enforced by `tests/test_docstring_budget.py` rather than by review, because both are properties of the tree rather than of any one change:\n\n- **No docstring over 80 lines** (100 for a class). The limit is generous on purpose — it catches a document, not a thorough docstring.\n- **No source line may cite a GitHub issue number.** The referent does not survive a fork, so a comment saying something is \"documented in #NNNN\" leaves nothing that documents it. Name the thing instead (\"the two-channel fence\"), or move the content into `docs/design/` and cite that. Never delete such a reference bare — migrate what it pointed at first. Enforced over `lightrag/` only: **tests may cite issue numbers**, and many do. There the number names the defect the test pins, and the test itself — its name, docstring and assertions — is the documentation, so the reference is provenance rather than the thing carrying the meaning.\n\nA third test requires every `docs/**.md` path named in the package to resolve; nothing imports those strings, so a typo is otherwise silent.\n\n### Python\n- Follow PEP 8 with 4-space indentation\n- Use type annotations\n- Prefer dataclasses for state management\n- Use `lightrag.utils.logger` instead of print\n- Async/await patterns throughout\n\n### TypeScript / React (incl. WebUI ESLint)\n- Functional components with hooks; PascalCase for components\n- 2-space indentation, single quotes (enforced by `@stylistic` rules)\n- Tailwind utility-first styling\n- ESLint stack: TypeScript-ESLint + React Hooks plugin + Prettier; `@typescript-eslint/no-explicit-any` is disabled (allowed)\n\n## Commit and Pull Request Guidance\n\n- If this repo is a fork of `HKUDS/LightRAG`. Target to `HKUDS/LightRAG` when creating PRs, not the fork's own repo.\n- PR descriptions should include: summary, motivation, linked issues if applyed, what's changed, what's broken and how it works.\n- Write commit messages (subject and body) in English. Commit messages are repository artifacts — like code comments and log messages — not conversational replies, so they follow the English code-style rule above regardless of any per-conversation working language.\n","isInternal":false,"tokens":7619,"sizeBytes":32425},{"name":"README_CLEAN_LLM_QUERY_CACHE.md","path":"lightrag/tools/README_CLEAN_LLM_QUERY_CACHE.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/LightRAG/HEAD/lightrag/tools/README_CLEAN_LLM_QUERY_CACHE.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# LLM Query Cache Cleanup Tool - User Guide\n\n## Overview\n\nThis tool cleans up LightRAG's LLM query cache from KV storage implementations. It specifically targets query caches generated during RAG query operations (modes: `mix`, `hybrid`, `local`, `global`), including both query and keywords caches.\n\n## Supported Storage Types\n\n1. **JsonKVStorage** - File-based JSON storage\n2. **RedisKVStorage** - Redis database storage\n3. **PGKVStorage** - PostgreSQL database storage\n4. **MongoKVStorage** - MongoDB database storage\n5. **OpenSearchKVStorage** - OpenSearch index storage\n\n## Cache Types\n\nThe tool cleans up the following query cache types:\n\n### Query Cache Modes (4 types)\n- `mix:*` - Mixed mode query caches\n- `hybrid:*` - Hybrid mode query caches\n- `local:*` - Local mode query caches\n- `global:*` - Global mode query caches\n\n### Cache Content Types (2 types)\n- `*:query:*` - Query result caches\n- `*:keywords:*` - Keywords extraction caches\n\n### Cache Key Format\n```\n<mode>:<cache_type>:<hash>\n```\n\nExamples:\n- `mix:query:5ce04d25e957c290216cee5bfe6344fa`\n- `mix:keywords:fee77b98244a0b047ce95e21060de60e`\n- `global:query:abc123def456...`\n- `local:keywords:789xyz...`\n\n**Important Note**: This tool does NOT clean extraction caches (`default:extract:*` and `default:summary:*`). Use the migration tool or manual deletion for those caches.\n\n## Prerequisites\n\n- The tool reads storage configuration from environment variables\n- Ensure the target storage is properly configured and accessible\n- Backup important data before running cleanup operations\n\n## Usage\n\n### Basic Usage\n\nRun from the LightRAG project root directory:\n\n```bash\npython -m lightrag.tools.clean_llm_query_cache\n# or\npython lightrag/tools/clean_llm_query_cache.py\n```\n\n### Interactive Workflow\n\nThe tool guides you through the following steps:\n\n#### 1. Select Storage Type\n```\n============================================================\nLLM Query Cache Cleanup Tool - LightRAG\n============================================================\n\n=== Storage Setup ===\n\nSupported KV Storage Types:\n[1] JsonKVStorage\n[2] RedisKVStorage\n[3] PGKVStorage\n[4] MongoKVStorage\n[5] OpenSearchKVStorage\n\nSelect storage type (1-5) (Press Enter to exit): 1\n```\n\n**Note**: You can press Enter or type `0` at any prompt to exit gracefully.\n\n#### 2. Storage Validation\nThe tool will:\n- Check required environment variables\n- Auto-detect workspace configuration\n- Initialize and connect to storage\n- Verify connection status\n\n```\nChecking configuration...\n✓ All required environment variables are set\n\nInitializing storage...\n- Storage Type: JsonKVStorage\n- Workspace: space1\n- Connection Status: ✓ Success\n```\n\n#### 3. View Cache Statistics\n\nThe tool displays a detailed breakdown of query caches by mode and type:\n\n```\nCounting query cache records...\n\n📊 Query Cache Statistics (Before Cleanup):\n┌────────────┬────────────┬────────────┬────────────┐\n│ Mode       │ Query      │ Keywords   │ Total      │\n├────────────┼────────────┼────────────┼────────────┤\n│ mix        │      1,234 │        567 │      1,801 │\n│ hybrid     │        890 │        423 │      1,313 │\n│ local      │      2,345 │      1,123 │      3,468 │\n│ global     │        678 │        345 │      1,023 │\n├────────────┼────────────┼────────────┼────────────┤\n│ Total      │      5,147 │      2,458 │      7,605 │\n└────────────┴────────────┴────────────┴────────────┘\n```\n\n#### 4. Select Cleanup Scope\n\nChoose what type of caches to delete:\n\n```\n=== Cleanup Options ===\n[1] Delete all query caches (both query and keywords)\n[2] Delete query caches only (keep keywords)\n[3] Delete keywords caches only (keep query)\n[0] Cancel\n\nSelect cleanup option (0-3): 1\n```\n\n**Cleanup Types:**\n- **Option 1 (all)**: Deletes both query and keywords caches across all modes\n- **Option 2 (query)**: Deletes only query caches, preserves keywords caches\n- **Option 3 (keywords)**: Deletes only keywords caches, preserves query caches\n\n#### 5. Confirm Deletion\n\nReview the cleanup plan and confirm:\n\n```\n============================================================\nCleanup Confirmation\n============================================================\nStorage: JsonKVStorage (workspace: space1)\nCleanup Type: all\nRecords to Delete: 7,605 / 7,605\n\n⚠️  WARNING: This will delete ALL query caches across all modes!\n\nContinue with deletion? (y/n): y\n```\n\n#### 6. Execute Cleanup\n\nThe tool performs batch deletion with real-time progress:\n\n**JsonKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing 1,000 records at a time from JsonKVStorage\n\nBatch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓\nBatch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓\n...\nBatch 8/8: ████████████████████ 7,605/7,605 (100.0%) ✓\n\nPersisting changes to storage...\n✓ Changes persisted successfully\n```\n\n**RedisKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing Redis keys in batches of 1,000\n\nBatch 1: Deleted 1,000 keys (Total: 1,000) ✓\nBatch 2: Deleted 1,000 keys (Total: 2,000) ✓\n...\n```\n\n**PostgreSQL Example:**\n```\n=== Starting Cleanup ===\n💡 Executing PostgreSQL DELETE query\n\n✓ Deleted 7,605 records in 0.45s\n```\n\n**MongoDB Example:**\n```\n=== Starting Cleanup ===\n💡 Executing MongoDB deleteMany operations\n\nPattern 1/8: Deleted 1,234 records ✓\nPattern 2/8: Deleted 567 records ✓\n...\nTotal deleted: 7,605 records\n```\n\n**OpenSearchKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing 1,000 records at a time from OpenSearchKVStorage\n\nBatch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓\nBatch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓\n...\n```\n\n#### 7. Review Cleanup Report\n\nThe tool provides a comprehensive final report:\n\n**Successful Cleanup:**\n```\n============================================================\nCleanup Complete - Final Report\n============================================================\n\n📊 Statistics:\n  Total records to delete:  7,605\n  Total batches:            8\n  Successful batches:       8\n  Failed batches:           0\n  Successfully deleted:     7,605\n  Failed to delete:         0\n  Success rate:             100.00%\n\n📈 Before/After Comparison:\n  Total caches before:      7,605\n  Total caches after:       0\n  Net reduction:            7,605\n\n============================================================\n✓ SUCCESS: All records cleaned up successfully!\n============================================================\n\n📊 Query Cache Statistics (After Cleanup):\n┌────────────┬────────────┬────────────┬────────────┐\n│ Mode       │ Query      │ Keywords   │ Total      │\n├────────────┼────────────┼────────────┼────────────┤\n│ mix        │          0 │          0 │          0 │\n│ hybrid     │          0 │          0 │          0 │\n│ local      │          0 │          0 │          0 │\n│ global     │          0 │          0 │          0 │\n├────────────┼────────────┼────────────┼────────────┤\n│ Total      │          0 │          0 │          0 │\n└────────────┴────────────┴────────────┴────────────┘\n```\n\n**Cleanup with Errors:**\n```\n============================================================\nCleanup Complete - Final Report\n============================================================\n\n📊 Statistics:\n  Total records to delete:  7,605\n  Total batches:            8\n  Successful batches:       7\n  Failed batches:           1\n  Successfully deleted:     6,605\n  Failed to delete:         1,000\n  Success rate:             86.85%\n\n📈 Before/After Comparison:\n  Total caches before:      7,605\n  Total caches after:       1,000\n  Net reduction:            6,605\n\n⚠️  Errors encountered: 1\n\nError Details:\n------------------------------------------------------------\n\nError Summary:\n  - ConnectionError: 1 occurrence(s)\n\nFirst 5 errors:\n\n  1. Batch 3\n     Type: ConnectionError\n     Message: Connection timeout after 30s\n     Records lost: 1,000\n\n============================================================\n⚠️  WARNING: Cleanup completed with errors!\n   Please review the error details above.\n============================================================\n```\n\n## Technical Details\n\n### Workspace Handling\n\nThe tool retrieves workspace in the following priority order:\n\n1. **Storage-specific workspace environment variables**\n   - PGKVStorage: `POSTGRES_WORKSPACE`\n   - MongoKVStorage: `MONGODB_WORKSPACE`\n   - RedisKVStorage: `REDIS_WORKSPACE`\n   - OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`\n\n2. **Generic workspace environment variable**\n   - `WORKSPACE`\n\n3. **Default value**\n   - Empty string (uses storage's default workspace)\n\n### Batch Deletion\n\n- Default batch size: 1000 records/batch\n- Prevents memory overflow and connection timeouts\n- Each batch is processed independently\n- Failed batches are logged but don't stop cleanup\n\n### Storage-Specific Deletion Strategies\n\n#### JsonKVStorage\n- Collects all matching keys first (snapshot approach)\n- Deletes in batches with lock protection\n- Fast in-memory operations\n\n#### RedisKVStorage\n- Uses SCAN with pattern matching\n- Pipeline DELETE for batch operations\n- Cursor-based iteration for large datasets\n\n#### PostgreSQL\n- Single DELETE query with OR conditions\n- Efficient server-side bulk deletion\n- Uses LIKE patterns for mode/type matching\n\n#### MongoDB\n- Multiple deleteMany operations (one per pattern)\n- Regex-based document matching\n- Returns exact deletion counts\n\n### Pattern Matching Implementation\n\n**JsonKVStorage:**\n```python\n# Direct key prefix matching\nif key.startswith(\"mix:query:\") or key.startswith(\"mix:keywords:\")\n```\n\n**RedisKVStorage:**\n```python\n# SCAN with namespace-prefixed patterns\npattern = f\"{namespace}:mix:query:*\"\ncursor, keys = await redis.scan(cursor, match=pattern)\n```\n\n**PostgreSQL:**\n```python\n# SQL LIKE conditions\nWHERE id LIKE 'mix:query:%' OR id LIKE 'mix:keywords:%'\n```\n\n**MongoDB:**\n```python\n# Regex queries on _id field\n{\"_id\": {\"$regex\": \"^mix:query:\"}}\n```\n\n**OpenSearchKVStorage:**\n```python\n# Scan raw hits, then match cache key prefixes in Python\nif hit[\"_id\"].startswith(\"mix:query:\"):\n```\n\n## Error Handling & Resilience\n\nThe tool implements comprehensive error tracking:\n\n### Batch-Level Error Tracking\n- Each batch is independently error-checked\n- Failed batches are logged with full details\n- Successful batches commit even if later batches fail\n- Real-time progress shows ✓ (success) or ✗ (failed)\n\n### Error Reporting\nAfter cleanup completes, a detailed report includes:\n- **Statistics**: Total records, success/failure counts, success rate\n- **Before/After Comparison**: Net reduction in cache count\n- **Error Summary**: Grouped by error type with occurrence counts\n- **Error Details**: Batch number, error type, message, and records lost\n- **Recommendations**: Clear indication of success or need for review\n\n### Verification\n- Post-cleanup count verification\n- Before/after statistics comparison\n- Identifies partial cleanup scenarios\n\n## Important Notes\n\n1. **Irreversible Operation**\n   - Deleted caches cannot be recovered\n   - Always backup important data before cleanup\n   - Test on non-production data first\n\n2. **Performance Impact**\n   - Query performance may degrade temporarily after cleanup\n   - Caches will rebuild on subsequent queries\n   - Consider cleanup during off-peak hours\n\n3. **Selective Cleanup**\n   - Choose cleanup scope carefully\n   - Keywords caches may be valuable for future queries\n   - Query caches rebuild faster than keywords caches\n\n4. **Workspace Isolation**\n   - Cleanup only affects the selected workspace\n   - Other workspaces remain untouched\n   - Verify workspace before confirming cleanup\n\n5. **Interrupt and Resume**\n   - Cleanup can be interrupted at any time (Ctrl+C)\n   - Already deleted records cannot be recovered\n   - No automatic resume - must run tool again\n\n## Storage Configuration\n\nThe tool supports multiple configuration methods with the following priority:\n\n1. **Environment variables** (highest priority)\n2. **Default values** (lowest priority)\n\n### Environment Variable Configuration\n\nConfigure storage settings in your `.env` file:\n\n#### Workspace Configuration (Optional)\n\n```bash\n# Generic workspace (shared by all storages)\nWORKSPACE=space1\n\n# Or configure independent workspace for specific storage\nPOSTGRES_WORKSPACE=pg_space\nMONGODB_WORKSPACE=mongo_space\nREDIS_WORKSPACE=redis_space\n```\n\n**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string\n\n#### JsonKVStorage\n\n```bash\nWORKING_DIR=./rag_storage\n```\n\n#### RedisKVStorage\n\n```bash\nREDIS_URI=redis://localhost:6379\n```\n\n#### PGKVStorage\n\n```bash\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_DATABASE=your_database\n```\n\n#### MongoKVStorage\n\n```bash\nMONGO_URI=mongodb://root:root@localhost:27017/\nMONGO_DATABASE=LightRAG\n```\n\n#### OpenSearchKVStorage\n\n```bash\nOPENSEARCH_HOSTS=localhost:9200\nOPENSEARCH_WORKSPACE=search_space\n```\n\nIf environment variables are not provided, the tool falls back to built-in defaults where available.\n\n## Troubleshooting\n\n### Missing Environment Variables\n```\n⚠️  Warning: Missing environment variables: POSTGRES_USER, POSTGRES_PASSWORD\n```\n**Solution**: Add missing variables to your `.env` file\n\n### Connection Failed\n```\n✗ Initialization failed: Connection refused\n```\n**Solutions**:\n- Check if database service is running\n- Verify connection parameters (host, port, credentials)\n- Check firewall settings\n- Ensure network connectivity for remote databases\n\n### No Caches Found\n```\n⚠️  No query caches found in storage\n```\n**Possible Reasons**:\n- No queries have been run yet\n- Caches were already cleaned\n- Wrong workspace selected\n- Different storage type was used for queries\n\n### Partial Cleanup\n```\n⚠️  WARNING: Cleanup completed with errors!\n```\n**Solutions**:\n- Check error details in the report\n- Verify storage connection stability\n- Re-run tool to clean remaining caches\n- Check storage capacity and permissions\n\n## Use Cases\n\n### Use Case 1: Clean All Query Caches\n\n**Scenario**: Free up storage space by removing all query caches\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 1 (all) -> Confirm (y)\n```\n\n**Result**: All query and keywords caches deleted, maximum storage freed\n\n### Use Case 2: Refresh Query Caches Only\n\n**Scenario**: Force query cache rebuild while keeping keywords\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 2 (query only) -> Confirm (y)\n```\n\n**Result**: Query caches deleted, keywords preserved for faster rebuild\n\n### Use Case 3: Clean Stale Keywords\n\n**Scenario**: Remove outdated keywords while keeping recent query results\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 3 (keywords only) -> Confirm (y)\n```\n\n**Result**: Keywords deleted, query caches preserved\n\n### Use Case 4: Workspace-Specific Cleanup\n\n**Scenario**: Clean caches for a specific workspace\n\n```bash\n# Configure workspace\nexport WORKSPACE=development\n\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Cleanup option -> Confirm (y)\n```\n\n**Result**: Only development workspace caches cleaned\n\n## Best Practices\n\n1. **Backup Before Cleanup**\n   - Always backup your storage before major cleanup\n   - Test cleanup on non-production data first\n   - Document cleanup decisions\n\n2. **Monitor Performance**\n   - Watch storage metrics during cleanup\n   - Monitor query performance after cleanup\n   - Allow time for cache rebuild\n\n3. **Scheduled Cleanup**\n   - Clean caches periodically (weekly/monthly)\n   - Automate cleanup for development environments\n   - Keep production cleanup manual for safety\n\n4. **Selective Deletion**\n   - Consider cleanup scope based on needs\n   - Keywords caches are harder to rebuild\n   - Query caches rebuild automatically\n\n5. **Storage Capacity**\n   - Monitor storage usage trends\n   - Clean caches before reaching capacity limits\n   - Archive old data if needed\n\n## Comparison with Migration Tool\n\n| Feature | Cleanup Tool | Migration Tool |\n|---------|-------------|----------------|\n| **Purpose** | Delete query caches | Migrate extraction caches |\n| **Cache Types** | mix/hybrid/local/global | default:extract/summary |\n| **Modes** | query, keywords | extract, summary |\n| **Operation** | Deletion | Copy between storages |\n| **Reversible** | No | Yes (source unchanged) |\n| **Use Case** | Free storage, refresh caches | Change storage backend |\n\n## Limitations\n\n1. **Single Storage Operation**\n   - Can only clean one storage type at a time\n   - To clean multiple storages, run tool multiple times\n\n2. **No Dry Run Mode**\n   - Deletion is immediate after confirmation\n   - No preview-only mode available\n   - Test on non-production first\n\n3. **No Selective Mode Cleanup**\n   - Cannot clean only specific modes (e.g., only `mix`)\n   - Cleanup applies to all modes for selected cache type\n   - All-or-nothing per cache type\n\n4. **No Scheduled Cleanup**\n   - Manual execution required\n   - No built-in scheduling\n   - Use cron/scheduler if automation needed\n\n5. **Verification Limitations**\n   - Post-cleanup verification may fail in error scenarios\n   - Manual verification recommended for critical operations\n\n## Future Enhancements\n\nPotential improvements for future versions:\n\n- Selective mode cleanup (e.g., clean only `mix` mode)\n- Age-based cleanup (delete caches older than X days)\n- Size-based cleanup (delete largest caches first)\n- Dry run mode for safe preview\n- Automated scheduling support\n- Cache statistics export\n- Incremental cleanup with pause/resume\n\n## Support\n\nFor issues, questions, or feature requests:\n- Check the error details in the cleanup report\n- Review storage configuration\n- Verify workspace settings\n- Test with a small dataset first\n- Report bugs through project issue tracker\n","isInternal":false,"tokens":4284,"sizeBytes":18992},{"name":"README_KG_INTEGRITY_REPAIR.md","path":"lightrag/tools/README_KG_INTEGRITY_REPAIR.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/LightRAG/HEAD/lightrag/tools/README_KG_INTEGRITY_REPAIR.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# KG Integrity Audit / Recovery-Anchor Repair\n\nOffline companion to the issue #3400 recoverable-mutation work.\n\nSince #3400, ingestion prewrites per-document **recovery anchors**\n(`full_entities` / `full_relations`) before mutating the graph, so purge,\nretry, and scan rollback can always discover a document's contributions.\nInstallations with data ingested **before** that change (or written through\ndirect KG paths like `ainsert_custom_kg`, which sit outside the\ndocument-level guarantee) may hold graph contributions with no anchor — such\ndata is invisible to per-document cleanup and becomes orphaned when its\ndocument is deleted or reprocessed.\n\nThis tool enumerates the whole graph (deliberately offline-only; the hot\npaths never do a graph-wide scan), maps every node/edge back to its owning\ndocuments via chunk `source_id` → `text_chunks` → `full_doc_id`, and:\n\n- reports per-document anchor gaps;\n- reports irrecoverable orphans (contributions whose source chunks no longer\n  exist) — these are never modified automatically;\n- with `--apply`, unions the missing entries into the anchor rows.\n\n## Usage\n\nLibrary (works with every configured backend combination):\n\n```python\nfrom lightrag.tools.kg_integrity_repair import audit_kg_integrity\n\nrag = LightRAG(...)\nawait rag.initialize_storages()\nreport = await audit_kg_integrity(rag)              # report only\nreport = await audit_kg_integrity(rag, apply=True)  # repair anchors\n```\n\nCLI (env-driven construction — `WORKING_DIR`, `WORKSPACE`, `EMBEDDING_DIM`,\n`LIGHTRAG_*` storage selectors; never calls the LLM or embedder):\n\n```bash\npython -m lightrag.tools.kg_integrity_repair             # report\npython -m lightrag.tools.kg_integrity_repair --apply     # repair\npython -m lightrag.tools.kg_integrity_repair --verbose   # per-doc details\n```\n\nRun it while the server is stopped (or the workspace is otherwise idle):\nthe audit reads a moving target if ingestion runs concurrently.\n\n## This tool is now the required remedy, not just a diagnostic\n\nPurge and deletion **fail closed** when a document has no recovery proof.\nPreviously, absent anchor rows were read as \"this document contributed\nnothing\", so graph cleanup was silently skipped while the chunks were deleted\nanyway — which destroyed the provenance chain (`source_id` → `text_chunks` →\n`full_doc_id`) that this tool needs, turning repairable data into permanent\norphans. So instead, the operation now refuses before deleting anything:\n\n```\nRecoveryAnchorMissingError: Refusing to purge document doc-…: recovery anchor\nrow(s) missing or unusable (full_entities, full_relations) …\n```\n\nOver the API this surfaces as **HTTP 409** with no data removed. Retrying\nunchanged will refuse again — run `--apply` first, then retry.\n\nThe rule is narrower than \"every purge needs a proof\". What must never happen\nis deleting something that *carries* attribution — a chunk row, or an anchor\nrow that names objects — while leaving those objects in the graph. A purge that\nremoves no such carrier cannot strand anything, so it needs no proof. Otherwise\none of these must hold:\n\n| Proof | Meaning |\n| --- | --- |\n| Both anchor rows present | The normal case. Presence is the test, **not** whether the lists are non-empty: a row holding an empty list is a document that extracted no entities, and is a perfectly good proof. |\n| `doc_status.metadata.kg_write_state == pre_graph` | Stamped at enqueue and advanced only once the anchors are durable, so it proves the document never reached its first graph mutation. Staged chunks are cleaned up with no graph access at all. |\n| `doc_status.metadata.kg_purge` past `prepared` | A previous purge attempt got far enough to have deleted the anchors itself. Without this, purge's own last step would make every retry refuse forever. |\n| No chunks and no populated anchor row | Nothing that carries attribution would be deleted. This is what lets a document that was queued before the `kg_write_state` marker existed, and never processed, be deleted directly — no scan, no audit. |\n\nThree states therefore need this tool:\n\n- documents ingested **before** #3416 landed the write-ahead anchors;\n- documents written through direct KG paths such as `ainsert_custom_kg`,\n  which are documented as sitting outside the document-level guarantee;\n- documents ingested with `skip_kg` (`process_options` `'!'`) **before** the\n  `kg_write_state` marker existed — see below.\n\n### Documents that legitimately own nothing\n\n`skip_kg` skips extraction and the merge entirely, so no anchor rows are ever\nwritten. Documents ingested that way *after* this change carry\n`kg_write_state=pre_graph` from enqueue and delete normally. Older ones have\nneither anchors nor a marker, and anchor repair has nothing to rebuild from —\nthey never appear in the graph scan, because they own nothing in it.\n\nThe audit settles that case, and it is the only thing that can: it enumerates\nthe **whole** graph, which the hot paths deliberately never do, so a document\nabsent from that scan is not merely unproven but *proven empty*. Such\ndocuments are listed under `anchorless_docs`, and `--apply` gives them empty\nanchor rows — the present-and-empty pair that is the normal proof for a\ndocument with no contributions.\n\nAbsence is only ever concluded from the completed scan. A document that does\nown graph objects is repaired with its real names instead; blanking it would\nmanufacture a false proof and license the exact deletion this work prevents.\n\n### What the certification assumes of the storage backends\n\n\"Proven empty\" is only as strong as the enumeration it is concluded from, so\nboth reads are held to a complete-or-raise contract and the audit **fails\nloudly instead of certifying on a partial view**:\n\n- the `doc_status` enumeration uses `strict=True`, so a transport or\n  deserialization failure raises rather than silently narrowing the set of\n  documents being certified;\n- the graph scan relies on `get_all_nodes()` / `get_all_edges()` never\n  silently truncating. All shipped graph backends satisfy this (errors\n  propagate; there is no result cap). Corrupt AGE properties raise a\n  `PGGraphQueryException` rather than dropping the node or blanking the\n  edge's `source_id` — a dropped node is an attribution the scan never sees,\n  which is exactly how a contributing document could be falsely certified.\n\nOne sharp edge remains: a graph backend whose indices/collections are missing\n(e.g. deleted out-of-band) reads as an **empty graph**, not as an error. The\naudit cannot distinguish that from a workspace that truly owns no graph data —\none more reason to run it only while the server is stopped and the workspace\nis healthy, as stated at the top.\n\nNote that `--apply` reports what it found under `missing_entity_anchors` /\n`missing_relation_anchors` (the diagnosis) and what it wrote under\n`repaired_docs` (the action) — the first two are not emptied by a repair.\n\nAnything listed under `orphan_entities` / `orphan_relations` cannot be\nrepaired by this tool: its source chunks are already gone, so no owning\ndocument is determinable. Those need external provenance, a re-ingest, or\nmanual deletion.\n","isInternal":false,"tokens":1608,"sizeBytes":7166},{"name":"README_MIGRATE_GRAPH_STORAGE.md","path":"lightrag/tools/README_MIGRATE_GRAPH_STORAGE.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/LightRAG/HEAD/lightrag/tools/README_MIGRATE_GRAPH_STORAGE.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Graph Storage Migration (`PGGraphStorage` → `PGTableGraphStorage`)\n\nOffline tool for moving an already-extracted entity-relation graph from the\nApache AGE backend (`PGGraphStorage`) to the plain-table backend\n(`PGTableGraphStorage`) without re-indexing the source documents.\n\nChanging `LIGHTRAG_GRAPH_STORAGE` normally means the previously extracted\ngraph is simply not visible to the new backend, so the documents have to be\nre-processed — an LLM-cost-bearing operation for a graph that already exists.\nThis tool copies the graph itself, through the public storage API only\n(`get_all_nodes` / `get_all_edges` to read, `upsert_nodes_batch` /\n`upsert_edges_batch` to write, `remove_nodes` / `remove_edges` to undo). It\nissues no raw SQL, so canonicalization, endpoint handling and identity rules\nstay in the already-tested storage layer.\n\nOnly the graph moves. Vector and KV storages are untouched, and because the\nmigrated graph keeps the same entity and relation identities, existing vector\ndata stays valid — nothing is re-embedded and nothing is re-extracted. The LLM\ncache is a separate concern with its own tool (see\n[README_MIGRATE_LLM_CACHE.md](README_MIGRATE_LLM_CACHE.md)).\n\nRe-indexing remains the general guidance for changing storage backends. This\ntool is an advanced path for one specific pair.\n\n## Before you run it\n\n**Stop every LightRAG writer first — the tool cannot check this for you.**\nIt reads the source graph, writes it to the target, and then verifies the\nresult. A concurrent writer mutating either side inside that window\ninvalidates the comparison, and the tool can neither detect a live writer nor\nlock the backends. Every other precondition below is enforced; this one is\npurely operational, and it is the assumption the whole design rests on. The\nsame requirement applies to `rebuild_vdb`.\n\n**The target graph slice must be empty.** The migration is defined as\npopulating an empty slice; that invariant is also what makes its failure\nhandling exact (see below). A non-empty target is refused before anything is\nwritten.\n\nBoth backends read the usual `POSTGRES_USER`, `POSTGRES_PASSWORD`,\n`POSTGRES_DATABASE`, `POSTGRES_HOST` and `POSTGRES_PORT` variables.\n\n## Usage\n\n```bash\n# Dry run (default): reads and checks everything, migrates nothing.\npython -m lightrag.tools.migrate_graph_storage\n\n# Perform the migration.\npython -m lightrag.tools.migrate_graph_storage --apply\n```\n\n| Flag | Default | Meaning |\n| --- | --- | --- |\n| `--apply` | off | Perform the migration. Without it no graph data is written (see the schema caveat below). |\n| `--workspace` | `WORKSPACE` env | Workspace to migrate. |\n| `--force-empty-target` | off | **Dangerous.** `drop()` the whole non-empty target graph slice before migrating, instead of refusing. |\n| `--source-backend` | `PGGraphStorage` | Source graph storage class name. |\n| `--target-backend` | `PGTableGraphStorage` | Target graph storage class name. |\n\n`--source-backend` / `--target-backend` are an extension seam, not an open\ndoor: only allow-listed pairs run, and Phase 1 allow-lists exactly\n`PGGraphStorage` → `PGTableGraphStorage`. Any other pair is rejected before a\nbackend is even constructed.\n\nThe dry run performs the same enumeration and the same refusal checks as the\nreal run, so it predicts what `--apply` will do. With `--force-empty-target`\nit reports that the apply run *would* drop the pre-existing slice, but never\ndrops anything itself.\n\n**A dry run migrates nothing, but it is not read-only, and its blast radius is\nnot limited to one workspace.** Both backends have to be initialized before\ntheir contents can be inspected, and `initialize()` is not a passive step. Two\ndifferent scopes are involved:\n\n- *Scoped to the workspace being migrated:* `_normalize_legacy_edges`\n  **deletes and re-inserts edge rows** (reversed duplicates, legacy\n  orderings) for this workspace and namespace only; the AGE side creates the\n  graph and its indexes.\n- *Table-wide, across every workspace and namespace:* the schema migration in\n  `PGTableGraphStorage`'s DDL. On a table whose primary key predates the\n  `namespace` column it **DELETEs duplicate rows** from `lightrag_graph_nodes`\n  and `lightrag_graph_edges` (keeping the most recently updated row per key)\n  before rebuilding the primary keys, and when the endpoint foreign keys are\n  absent it then **DELETEs every orphan edge** — an edge whose endpoint node\n  is missing — from the whole table. That sweep cannot be narrowed: as\n  `pgtable_impl` puts it, \"the sweep is table-wide because the FK is a global\n  composite constraint and cannot be validated per workspace.\" Both paths are\n  one-time (first install, or the first run after the namespace-PK upgrade)\n  and are skipped once the constraints exist, but the run that performs them\n  touches rows belonging to workspaces `--workspace` never named.\n\nSo a dry run against one workspace can still modify others, on a legacy\nschema. What it never does is migrate data or copy anything between backends —\nthat is the guarantee, and it is narrower than \"writes nothing\". Take a backup\nof both tables before the first run against a pre-`namespace` schema.\n\n**`--workspace` is cross-checked, not trusted.** The backends resolve their\nworkspace as `POSTGRES_WORKSPACE` env > `config.ini` `[postgres] workspace` >\nthe value passed in > `\"default\"`, so either the environment or a checked-in\nconfig file can outrank the flag. The tool compares what was requested\nagainst what the backends actually resolved to, refuses when they differ, and\nreports the resolved workspace — the slice a destructive run acted on is never\nleft implicit.\n\nExit code is 0 on success and non-zero on any failure.\n\n## The report\n\nBoth modes print a dict to stdout, following the convention of the other\ntools:\n\n```python\n{'mode': 'dry-run', 'source_backend': 'PGGraphStorage',\n 'target_backend': 'PGTableGraphStorage', 'workspace': 'default',\n 'nodes': 8123, 'edges': 41022,\n 'verified': False, 'compensated': False, 'target_non_empty': False,\n 'would_drop_target_slice': False,\n 'written_node_count': 8123, 'written_edge_count': 41022}\n```\n\n`mode` is `dry-run` or `apply`. `verified` is true only when the post-write\ncomparison passed. The destructive-consequence field is mode-symmetric:\n`would_drop_target_slice` in a dry run, `dropped_target_slice` in an apply run\n— so the preview cannot look clean while hiding that `--force-empty-target`\nwill destroy the existing slice.\n\n`nodes` / `edges` in a dry run also double as a sizing signal for the apply\nrun: both modes hold the whole graph in memory (see **It does not stream**\nbelow), so a dry run against the real source is the cheapest way to gauge how\nmuch memory the real run will need.\n\n## What makes it refuse\n\nEvery check below runs *before* the first write, and each one aborts the run\nrather than migrating a graph it cannot faithfully reproduce:\n\n- the storage pair is not allow-listed;\n- the target graph slice is not empty (unless `--force-empty-target`);\n- a node has no usable identity — `id` or `entity_id` missing, not a string,\n  or the two disagreeing. Both backends can legitimately produce such a row: a\n  vertex created with no properties is legal in AGE and enumerates as\n  `{\"id\": None}`, and NetworkX creates an edge's unknown endpoint as an\n  attribute-less node;\n- the same node id appears more than once, **whether or not the payloads\n  match**. The target keeps one row per id, so two physical source vertices\n  become one node and the graph's node count changes;\n- `a→b` and `b→a` both exist, **whether or not the payloads match**. The\n  target stores one row per canonical pair, and AGE computes degree by\n  counting relationship rows — so collapsing an identical reciprocal preserves\n  every property while changing degree and traversal. LightRAG's own write\n  path matches edges undirected, so it does not create reciprocals: finding\n  one means the source violates the invariant this tool assumes;\n- a payload contains a value PostgreSQL `jsonb` cannot store faithfully:\n  `NaN`, `Infinity` and `-Infinity` (which `agtype` represents and `jsonb`\n  rejects outright), or `-0.0` (which `jsonb` accepts but normalises to `0.0`,\n  and this tool's comparison treats signed zeros as different — so the write\n  would succeed and then fail its own verification). The general rule: if the\n  target cannot round-trip a value, refuse before the destructive step rather\n  than discover it after the point of no return;\n- more than one directed row *reaches the tool* for the same ordered pair — a\n  violation of the invariant that makes the canonical merge lossless. Note the\n  limit of this backstop: AGE enumerates edges with `SELECT DISTINCT source,\n  target, properties`, so byte-identical parallel relationships are already\n  collapsed to one row before the tool sees them. It can therefore only catch\n  parallels whose payloads *differ*. **This is the one cardinality gap the tool\n  cannot close**: byte-identical same-direction parallels are invisible to it,\n  and collapsing them changes degree without losing any property. Everything\n  else that would change cardinality — reciprocals, duplicate ids — is refused\n  outright.\n\nVerification after the write re-runs the source-side checks and compares node\nids, canonical edges and properties. It is driven from the source because the\ntarget has already canonicalized and therefore cannot reveal what was lost.\nProperty comparison is type-strict: `1`, `1.0`, `True` and `\"1\"` are four\ndifferent values, and `-0.0` differs from `0.0` — all are distinguishable\ndownstream.\n\n**What `verified` claims — and what it cannot.** The tool only ever sees what\nthe source's `get_all_nodes` / `get_all_edges` return, so its claim is scoped\nto that enumerated view: **every node and edge the enumerator returned came\nacross with its identity, its payload, and its count intact.** Within that\nview nothing silently collapses — reciprocals and duplicate ids are refused\nbefore the write rather than merged and blessed.\n\n`verified` does **not** prove that the physical AGE graph was reproduced. Two\nthings are already lost or hidden before this tool is handed a row:\n\n- **Physical multiplicity.** AGE enumerates with `SELECT DISTINCT source,\n  target, properties`, so two byte-identical relationships between the same\n  ordered pair arrive as one row. AGE's own degree counts relationship rows, so\n  such a source has degree 2 where the target will have 1 — and `verified` will\n  still be true, because both sides match the one row the enumerator showed.\n  Reading raw multiplicity would mean going around the storage API, which this\n  tool deliberately does not do.\n- **Payload keys named `id`, `source` or `target`.** `PGGraphStorage`'s\n  enumerator overwrites them with identity values when flattening each row, so\n  a business property under one of those names is gone upstream and no\n  comparison here can notice.\n\nIf exact physical fidelity matters for your graph, check for duplicate\nrelationships directly in the AGE tables before migrating.\n\n## If a write fails\n\nThe tool removes exactly what the run wrote — the migrated node ids plus every\nedge endpoint, since the target auto-creates missing endpoints without\nreporting them — using `remove_edges` first and then `remove_nodes` to respect\nthe foreign key. It never uses `drop()` for this; whole-slice deletion is\nreachable only through `--force-empty-target`. The written set is computed\nbefore the first write, so a failure at any point still has the complete set,\nand removing it is exact precisely because the slice started empty.\n\n`--force-empty-target` has an irreducible window. Every source check runs\nbefore the drop, so a malformed source cannot cost you the slice — but nothing\nproves the *write* will succeed. If the migration fails after the drop, the\ncompensation removes what this run wrote and the target ends up empty: the\npre-existing data is gone and the new data never landed. That is the bargain\nthe flag asks for; take a backup before using it.\n\nA failed `drop()` under `--force-empty-target` aborts the run rather than\nproceeding: the backend reports that failure by returning an error status\ninstead of raising, so an unchecked call would let the migration write into,\nand later compensate against, rows it does not own.\n\nCompensation is not atomic. There is no transaction spanning two backends, so\na crash during compensation can leave the target partially populated. Two\nthings make that recoverable: re-running the tool refuses a non-empty target\nrather than compounding the mess, and if compensation itself fails the report\ncarries the full `written_node_ids` and `written_edge_keys` so the remaining\nrows can be removed by hand.\n\n## What this tool does not do\n\n- **It does not migrate any other pair.** `PRECONDITION_HOOKS` and\n  `COMPENSATION_HOOKS` are named placeholders — declared, but not yet consulted\n  anywhere; the Phase 1 behaviour is hardcoded. Adding a pair means wiring them\n  as well as allow-listing it.\n- **It does not merge into a populated target.** Empty-target is a\n  precondition, not a convenience.\n- **It does not run in a single transaction.** Two backends, no shared\n  transaction; the failure handling above is the substitute.\n- **It does not stream.** The storage API enumerates whole graphs, so a run\n  holds the source graph, the derived write plan and — during verification —\n  the target graph in memory at the same time. Peak memory grows with the\n  graph, roughly 2x its decoded size, and no setting bounds it: the fail-closed\n  checks (duplicate ids, reciprocals, parallel pairs) are global properties\n  that need the whole-graph view. For a very large graph, run the tool on a\n  host sized accordingly, or migrate workspace by workspace. Running out of\n  memory *inside* the process (`MemoryError`) is handled like any other\n  failure — compensation removes what the run wrote — but a kernel OOM kill\n  terminates the process outright: nothing compensates, the report never\n  prints, and under `--force-empty-target` it lands after the drop. Size the\n  host before reaching for that flag.\n- **It does not protect you from a live writer.** See the preconditions.\n- **It does not disambiguate workspaces that share an AGE graph.** AGE derives\n  its graph name by replacing every non-alphanumeric character with `_`, so\n  `team-a` and `team_a` name the same AGE graph while PGTable keeps them as\n  distinct workspaces. Check which graph a workspace actually resolves to\n  before migrating one whose name contains punctuation.\n- **It does not move vectors, KV data or document status.**\n","isInternal":false,"tokens":3365,"sizeBytes":14646},{"name":"README_MIGRATE_LLM_CACHE.md","path":"lightrag/tools/README_MIGRATE_LLM_CACHE.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/LightRAG/HEAD/lightrag/tools/README_MIGRATE_LLM_CACHE.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# LLM Cache Migration Tool - User Guide\n\n## Overview\n\nThis tool migrates LightRAG's LLM response cache between different KV storage implementations. It specifically migrates caches generated during file extraction (mode `default`), including entity extraction, summary, and multimodal analysis caches.\n\n## Supported Storage Types\n\n1. **JsonKVStorage** - File-based JSON storage\n2. **RedisKVStorage** - Redis database storage\n3. **PGKVStorage** - PostgreSQL database storage\n4. **MongoKVStorage** - MongoDB database storage\n5. **OpenSearchKVStorage** - OpenSearch index storage\n\n## Cache Types\n\nThe tool migrates the following cache types:\n- `default:extract:*` - Entity and relationship extraction caches\n- `default:summary:*` - Entity and relationship summary caches\n- `default:analysis:*` - Multimodal image, table, and equation analysis caches\n\n**Note**: Query caches (modes like `mix`,`local`, `global`, etc.) are NOT migrated.\n\n## Prerequisites\n\nThe LLM Cache Migration Tool reads the storage configuration of the LightRAG Server and provides an LLM migration option to select source and destination storage. Ensure that both the source and destination storage have been correctly configured and are accessible via the LightRAG Server before cache migration.\n\n## Usage\n\n### Basic Usage\n\nRun from the LightRAG project root directory:\n\n```bash\npython -m lightrag.tools.migrate_llm_cache\n# or\npython lightrag/tools/migrate_llm_cache.py\n```\n\n### Interactive Workflow\n\nThe tool guides you through the following steps:\n\n#### 1. Select Source Storage Type\n```\nSupported KV Storage Types:\n[1] JsonKVStorage\n[2] RedisKVStorage\n[3] PGKVStorage\n[4] MongoKVStorage\n[5] OpenSearchKVStorage\n\nSelect Source storage type (1-5) (Press Enter to exit): 1\n```\n\n**Note**: You can press Enter or type `0` at any storage selection prompt to exit gracefully.\n\n#### 2. Source Storage Validation\nThe tool will:\n- Check required environment variables\n- Auto-detect workspace configuration\n- Initialize and connect to storage\n- Count cache records available for migration\n\n```\nChecking environment variables...\n✓ All required environment variables are set\n\nInitializing Source storage...\n- Storage Type: JsonKVStorage\n- Workspace: space1\n- Connection Status: ✓ Success\n\nCounting cache records...\n- Total: 8,734 records\n```\n\n**Progress Display by Storage Type:**\n- **JsonKVStorage**: Fast in-memory counting, displays final count without incremental progress\n  ```\n  Counting cache records...\n  - Total: 8,734 records\n  ```\n- **RedisKVStorage**: Real-time scanning progress with incremental counts\n  ```\n  Scanning Redis keys... found 8,734 records\n  ```\n- **PostgreSQL**: Quick COUNT(*) query, shows timing only if operation takes >1 second\n  ```\n  Counting PostgreSQL records... (took 2.3s)\n  ```\n- **MongoDB**: Fast count_documents(), shows timing only if operation takes >1 second\n  ```\n  Counting MongoDB documents... (took 1.8s)\n  ```\n- **OpenSearchKVStorage**: PIT-based scan with timing shown when noticeable\n  ```\n  Scanning OpenSearch documents... (took 1.5s)\n  ```\n\n#### 3. Select Target Storage Type\n\nThe tool automatically excludes the source storage type from the target selection and renumbers the remaining options sequentially:\n\n```\nAvailable Storage Types for Target (source: JsonKVStorage excluded):\n[1] RedisKVStorage\n[2] PGKVStorage\n[3] MongoKVStorage\n[4] OpenSearchKVStorage\n\nSelect Target storage type (1-4) (Press Enter or 0 to exit): 1\n```\n\n**Important Notes:**\n- You **cannot** select the same storage type for both source and target\n- Options are automatically renumbered (e.g., [1], [2], [3] instead of [2], [3], [4])\n- You can press Enter or type `0` to exit at this point as well\n\nThe tool then validates the target storage following the same process as the source (checking environment variables, initializing connection, counting records).\n\n#### 4. Confirm Migration\n\n```\n==================================================\nMigration Confirmation\nSource: JsonKVStorage (workspace: space1) - 8,734 records\nTarget: MongoKVStorage (workspace: space1) - 0 records\nBatch Size: 1,000 records/batch\nMemory Mode: Streaming (memory-optimized)\n\n⚠️  Warning: Target storage already has 0 records\nMigration will overwrite records with the same keys\n\nContinue? (y/n): y\n```\n\n#### 5. Execute Migration\n\nThe tool uses **streaming migration** by default for memory efficiency. Observe migration progress:\n\n```\n=== Starting Streaming Migration ===\n💡 Memory-optimized mode: Processing 1,000 records at a time\n\nBatch 1/9: ████████░░░░░░░░░░░░ 1000/8734 (11.4%) - default:extract ✓\nBatch 2/9: ████████████░░░░░░░░ 2000/8734 (22.9%) - default:extract ✓\n...\nBatch 9/9: ████████████████████ 8734/8734 (100.0%) - default:summary ✓\n\nPersisting data to disk...\n✓ Data persisted successfully\n```\n\n**Key Features:**\n- **Streaming mode**: Processes data in batches without loading entire dataset into memory\n- **Real-time progress**: Shows progress bar with precise percentage and cache type\n- **Success indicators**: ✓ for successful batches, ✗ for failed batches\n- **Constant memory usage**: Handles millions of records efficiently\n\n#### 6. Review Migration Report\n\nThe tool provides a comprehensive final report showing statistics and any errors encountered:\n\n**Successful Migration:**\n```\nMigration Complete - Final Report\n\n📊 Statistics:\n  Total source records:    8,734\n  Total batches:           9\n  Successful batches:      9\n  Failed batches:          0\n  Successfully migrated:   8,734\n  Failed to migrate:       0\n  Success rate:            100.00%\n\n✓ SUCCESS: All records migrated successfully!\n```\n\n**Migration with Errors:**\n```\nMigration Complete - Final Report\n\n📊 Statistics:\n  Total source records:    8,734\n  Total batches:           9\n  Successful batches:      8\n  Failed batches:          1\n  Successfully migrated:   7,734\n  Failed to migrate:       1,000\n  Success rate:            88.55%\n\n⚠️  Errors encountered: 1\n\nError Details:\n------------------------------------------------------------\n\nError Summary:\n  - ConnectionError: 1 occurrence(s)\n\nFirst 5 errors:\n\n  1. Batch 2\n     Type: ConnectionError\n     Message: Connection timeout after 30s\n     Records lost: 1,000\n\n⚠️  WARNING: Migration completed with errors!\n   Please review the error details above.\n```\n\n## Technical Details\n\n### Workspace Handling\n\nThe tool retrieves workspace in the following priority order:\n\n1. **Storage-specific workspace environment variables**\n   - PGKVStorage: `POSTGRES_WORKSPACE`\n   - MongoKVStorage: `MONGODB_WORKSPACE`\n   - RedisKVStorage: `REDIS_WORKSPACE`\n   - OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`\n\n2. **Generic workspace environment variable**\n   - `WORKSPACE`\n\n3. **Default value**\n   - Empty string (uses storage's default workspace)\n\n### Batch Migration\n\n- Default batch size: 1000 records/batch\n- Avoids memory overflow from loading too much data at once\n- Each batch is committed independently, supporting resume capability\n\n### Memory-Efficient Pagination\n\nFor large datasets, the tool implements storage-specific pagination strategies:\n\n- **JsonKVStorage**: Direct in-memory access (data already loaded in shared storage)\n- **RedisKVStorage**: Cursor-based SCAN with pipeline batching (1000 keys/batch)\n- **PGKVStorage**: SQL LIMIT/OFFSET pagination (1000 records/batch)\n- **MongoKVStorage**: Cursor streaming with batch_size (1000 documents/batch)\n- **OpenSearchKVStorage**: PIT + `search_after` scan of the KV index (1000 documents/batch)\n\nThis ensures the tool can handle millions of cache records without memory issues.\n\n### Prefix Filtering Implementation\n\nThe tool uses optimized filtering methods for different storage types:\n\n- **JsonKVStorage**: Direct dictionary iteration with lock protection\n- **RedisKVStorage**: SCAN command with namespace-prefixed patterns + pipeline for bulk GET\n- **PGKVStorage**: SQL LIKE queries with proper field mapping (id, return_value, etc.)\n- **MongoKVStorage**: MongoDB regex queries on `_id` field with cursor streaming\n- **OpenSearchKVStorage**: Full-index scan with `_id` prefix filtering and `_source` passthrough\n\n## Error Handling & Resilience\n\nThe tool implements comprehensive error tracking to ensure transparent and resilient migrations:\n\n### Batch-Level Error Tracking\n- Each batch is independently error-checked\n- Failed batches are logged but don't stop the migration\n- Successful batches are committed even if later batches fail\n- Real-time progress shows ✓ (success) or ✗ (failed) for each batch\n\n### Error Reporting\nAfter migration completes, a detailed report includes:\n- **Statistics**: Total records, success/failure counts, success rate\n- **Error Summary**: Grouped by error type with occurrence counts\n- **Error Details**: Batch number, error type, message, and records lost\n- **Recommendations**: Clear indication of success or need for review\n\n### No Double Data Loading\n- Unlike traditional verification approaches, the tool does NOT reload all target data\n- Errors are detected during migration, not after\n- This eliminates memory overhead and handles pre-existing target data correctly\n\n## Important Notes\n\n1. **Data Overwrite Warning**\n   - Migration will overwrite records with the same keys in the target storage\n   - Tool displays a warning if target storage already has data\n   - Data migration can be performed repeatedly\n   - Pre-existing data in target storage is handled correctly\n3. **Interrupt and Resume**\n   - Migration can be interrupted at any time (Ctrl+C)\n   - Already migrated data will remain in target storage\n   - Re-running will overwrite existing records\n   - Failed batches can be manually retried\n4. **Performance Considerations**\n   - Large data migration may take considerable time\n   - Recommend migrating during off-peak hours\n   - Ensure stable network connection (for remote databases)\n   - Memory usage stays constant regardless of dataset size\n\n## Storage Configuration\n\nThe tool supports multiple configuration methods with the following priority:\n\n1. **Environment variables** (highest priority)\n2. **Default values** (lowest priority)\n\n#### Option A: Environment Variable Configuration\n\nConfigure storage settings in your `.env` file:\n\n#### Workspace Configuration (Optional)\n\n```bash\n# Generic workspace (shared by all storages)\nWORKSPACE=space1\n\n# Or configure independent workspace for specific storage\nPOSTGRES_WORKSPACE=pg_space\nMONGODB_WORKSPACE=mongo_space\nREDIS_WORKSPACE=redis_space\nOPENSEARCH_WORKSPACE=os_space\n```\n\n**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string\n\n#### JsonKVStorage\n\n```bash\nWORKING_DIR=./rag_storage\n```\n\n#### RedisKVStorage\n\n```bash\nREDIS_URI=redis://localhost:6379\n```\n\n#### PGKVStorage\n\n```bash\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_DATABASE=your_database\n```\n\n#### MongoKVStorage\n\n```bash\nMONGO_URI=mongodb://root:root@localhost:27017/\nMONGO_DATABASE=LightRAG\n```\n\n#### OpenSearchKVStorage\n\n```bash\nOPENSEARCH_HOSTS=localhost:9200\nOPENSEARCH_WORKSPACE=os_space\n```\n\nIf environment variables are not provided, the tool falls back to built-in defaults where available. JsonKVStorage uses `WORKING_DIR` or defaults to `./rag_storage`.\n\n## Troubleshooting\n\n### Missing Environment Variables\n```\n✗ Missing required environment variables: POSTGRES_USER, POSTGRES_PASSWORD\n```\n**Solution**: Add missing variables to your `.env` file\n\n### Connection Failed\n```\n✗ Initialization failed: Connection refused\n```\n**Solutions**:\n- Check if database service is running\n- Verify connection parameters (host, port, credentials)\n- Check firewall settings\n\n**Solutions**:\n- Check migration process for error logs\n- Re-run migration tool\n- Check target storage capacity and permissions\n\n## Example Scenarios\n\n### Scenario 1: JSON to MongoDB Migration\n\nUse case: Migrating from single-machine development to production\n\n```bash\n# 1. Configure environment variables\nWORKSPACE=production\nMONGO_URI=mongodb://user:pass@prod-server:27017/\nMONGO_DATABASE=LightRAG\n\n# 2. Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# 3. Select: 1 (JsonKVStorage) -> 1 (MongoKVStorage - renumbered from 4)\n```\n\n**Note**: After selecting JsonKVStorage as source, MongoKVStorage will be shown as option [1] in the target selection since options are renumbered after excluding the source.\n\n### Scenario 2: Redis to PostgreSQL\n\nUse case: Migrating from cache storage to relational database\n\n```bash\n# 1. Ensure both databases are accessible\nREDIS_URI=redis://old-redis:6379\nPOSTGRES_HOST=new-postgres-server\n# ... Other PostgreSQL configs\n\n# 2. Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# 3. Select: 2 (RedisKVStorage) -> 2 (PGKVStorage - renumbered from 3)\n```\n\n**Note**: After selecting RedisKVStorage as source, PGKVStorage will be shown as option [2] in the target selection.\n\n### Scenario 3: Different Workspaces Migration\n\nUse case: Migrating data between different workspace environments\n\n```bash\n# Configure separate workspaces for source and target\nPOSTGRES_WORKSPACE=dev_workspace  # For development environment\nMONGODB_WORKSPACE=prod_workspace  # For production environment\n\n# Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# Select: 3 (PGKVStorage with dev_workspace) -> 3 (MongoKVStorage with prod_workspace)\n```\n\n**Note**: This allows you to migrate between different logical data partitions while changing storage backends.\n\n## Tool Limitations\n\n1. **Same Storage Type Not Allowed**\n   - You cannot migrate between the same storage type (e.g., PostgreSQL to PostgreSQL)\n   - This is enforced by the tool automatically excluding the source storage type from target selection\n   - For same-storage migrations (e.g., database switches), use database-native tools instead\n2. **Only Default Mode Caches**\n   - Only migrates `default:extract:*`, `default:summary:*`, and `default:analysis:*`\n   - Query caches are not included\n4. **Network Dependency**\n   - Tool requires stable network connection for remote databases\n   - Large datasets may fail if connection is interrupted\n\n## Best Practices\n\n1. **Backup Before Migration**\n   - Always backup your data before migration\n   - Test migration on non-production data first\n\n2. **Verify Results**\n   - Check the verification output after migration\n   - Manually verify a few cache entries if needed\n\n3. **Monitor Performance**\n   - Watch database resource usage during migration\n   - Consider migrating in smaller batches if needed\n\n4. **Clean Old Data**\n   - After successful migration, consider cleaning old cache data\n   - Keep backups for a reasonable period before deletion\n","isInternal":false,"tokens":3359,"sizeBytes":14732},{"name":"README_REBUILD_VDB.md","path":"lightrag/tools/README_REBUILD_VDB.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/LightRAG/HEAD/lightrag/tools/README_REBUILD_VDB.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Offline Vector Storage (VDB) Rebuild Tool\n\n`lightrag-rebuild-vdb` restores consistency between LightRAG's authoritative\ndata sources and its vector storages by dropping each vector storage and\nrebuilding it from scratch:\n\n| Vector storage | Authoritative source |\n|---|---|\n| `entities_vdb` | graph nodes |\n| `relationships_vdb` | graph edges |\n| `chunks_vdb` | `text_chunks` KV store |\n\n## When do I need this?\n\nLightRAG performs multi-step writes (graph + vector storage). If a vector\nstorage write fails at runtime — embedder outage, network timeout, context\noverflow on a high-degree entity — the graph and the vector storage drift\napart. The most common symptom is the edge-count drift reported in issue\n[#2917](https://github.com/HKUDS/LightRAG/issues/2917): graph edges with no\nvector counterpart, so `local`/`hybrid` queries miss relations that exist in\nthe graph.\n\nSince v(next), `amerge_entities` raises `VectorStorageConsistencyError` when\nthis happens, with a pointer to this tool. No data is lost in this situation:\nthe graph and the `text_chunks` KV store hold everything needed to rebuild\nthe vectors.\n\nA full drop + rebuild also clears *reverse* orphans (records present in the\nvector storage but absent from the graph), which incremental repair cannot\nreliably do.\n\nYou can also use this tool after changing the embedding model or embedding\ndimension. Existing vector records were generated in the old embedding space\n(and some vector backends bind collections/tables to the configured dimension),\nso run the tool with the updated `.env` and choose **Rebuild ALL vector\nstorages** to regenerate `entities_vdb`, `relationships_vdb`, and `chunks_vdb`\nfrom the authoritative graph/KV sources. The consistency check is not enough\nfor this case because it only detects missing graph → VDB records, not vectors\nthat exist but were embedded with a previous model or dimension.\n\nA vector storage whose container was written in a different embedding space\n**refuses to attach** rather than serving an empty or foreign index. That\nrefusal is the condition this tool clears, so it is the one startup failure the\ntool tolerates: the three vector targets are initialized individually, a typed\n`VectorSpaceMismatchError` is recorded instead of aborting the run, and the\nrebuild opens by dropping that container and re-provisioning it in the current\nembedding space. No out-of-band step (deleting the index through the backend's\nown API, `DROP TABLE`, `rm`) is needed.\n\nEverything else still aborts. A cluster outage, a bad credential or a corrupt\nfile is not a model change, and the recovery for a model change is destructive —\nso it is only ever applied to a refusal that says so in its type. The\nauthoritative sources (graph storage and `text_chunks`) likewise keep the\nserver-identical startup path, migrations included: rebuilding vectors from a\nhalf-migrated source is worse than not rebuilding.\n\nSee `docs/design/VectorSpaceProvenance.md` for the provenance marker and the\nrules that decide when a backend refuses.\n\nThe tool also completes a graph-only backfill performed with\n`NoopVectorDBStorage`. Stop every writer, preserve the same working directory,\nworkspace, graph storage, and KV storage, switch to the intended persistent\nvector backend, configure the production embedding model and dimension, and\nselect **Rebuild ALL vector storages**. Create a new `LightRAG` instance or\nrestart the process after changing the vector backend. Do not start the server\nuntil the rebuild succeeds; there is currently no persisted vector-index\nreadiness marker.\n\n## Usage\n\n```bash\n# Stop the LightRAG Server first!\nlightrag-rebuild-vdb\n# or\npython -m lightrag.tools.rebuild_vdb\n```\n\nThe tool reads the same `.env` / environment configuration as the server\n(`LIGHTRAG_GRAPH_STORAGE`, `LIGHTRAG_VECTOR_STORAGE`, `LIGHTRAG_KV_STORAGE`,\n`WORKSPACE`, `WORKING_DIR`, `EMBEDDING_*`, backend connection settings) and\nbuilds its embedding function through the exact factory the server uses —\nrun it with the same `.env` so rebuilt vectors live in the same embedding\nspace the server queries against.\n\nMenu options:\n\n1. **Consistency check (diagnose only)** — probes every graph entity/relation\n   for a vector counterpart and reports what is missing. Run this first to\n   decide whether a rebuild is worth the embedding cost. The check covers\n   the graph → VDB direction only; reverse orphans can only be cleared by a\n   full rebuild. Legacy reverse-order relation ids (from old custom-KG\n   imports) are recognized and not misreported as missing. The check issues\n   read queries only and does not run a rebuild (no drop + re-embed). It is\n   **not** strictly side-effect-free, though: the tool initializes every\n   storage on startup — exactly as the server does — and for some backends\n   that includes schema/DDL setup and one-time legacy migrations (e.g. Qdrant\n   upserts into the new collection, PostgreSQL batch-inserts into the new\n   table, Milvus may create a temp collection and drop/rename the original).\n   Treat running the tool — even just for a check — like starting the server:\n   stop other writers first.\n2. **Rebuild entities + relationships VDB** — sufficient for the #2917\n   merge-failure scenario.\n3. **Rebuild chunks VDB**.\n4. **Rebuild ALL vector storages** — use this after changing the embedding\n   model or embedding dimension.\n\n## Important notes\n\n- **Stop the server first.** The tool drops and rewrites vector storages;\n  concurrent writers (any backend, not just file-based ones) can corrupt\n  data or lose updates.\n- **Embedding model/dimension changes.** Run the tool with the new embedding\n  configuration and rebuild all vector storages. A consistency check can still\n  pass when every vector record exists but was created with the old embedding\n  model or dimension — unless the backend recorded its embedding-space\n  provenance, in which case the check reports the refusal and says the rebuild\n  is mandatory rather than reporting every record as missing.\n- **A refused target is dropped only after you confirm the rebuild.** Choosing\n  the consistency check never destroys anything, so a refusal can be diagnosed\n  before it is acted on. Options 2–4 recover exactly the targets they rebuild:\n  picking option 2 leaves a refused `chunks_vdb` refused until option 3 or 4\n  runs.\n- **Embedding cost.** A rebuild re-embeds every affected record. On large\n  datasets this means real API cost and time. Use the check mode first, and\n  rebuild only the storages that need it.\n- **Idempotent / crash-safe.** Sources (graph, `text_chunks`) are never\n  modified. If the tool crashes between drop and rewrite, just re-run it.\n- **`__created_at__` reset.** Backends that store creation timestamps in\n  vector records (nano, faiss) will show fresh timestamps after a rebuild.\n  No query logic depends on them.\n- **Custom-KG placeholder entities.** `UNKNOWN` placeholder nodes created by\n  `ainsert_custom_kg` are rebuilt faithfully from the graph; they may gain a\n  vector record they previously lacked (improving their retrievability).\n- **Chunk enumeration is backend-specific.** `BaseKVStorage` has no key\n  enumeration API, so the tool scans each KV backend directly (JsonKV,\n  Redis, PostgreSQL, MongoDB, OpenSearch). When a new KV backend is added,\n  `enumerate_kv_keys()` in `rebuild_vdb.py` must be extended.\n\n## Library usage\n\nThe core rebuild/check functions are plain async functions that accept your\nown initialized storage instances:\n\nRebuild targets must persist a queryable vector index. Passing a graph-only\nbackend such as `NoopVectorDBStorage` raises before source records are read or\nthe target is dropped; configure a persistent vector backend first.\n\n```python\nfrom lightrag.tools.rebuild_vdb import (\n    check_vdb_consistency,\n    rebuild_chunks_vdb,\n    rebuild_entities_vdb,\n    rebuild_relationships_vdb,\n)\n\nreport = await check_vdb_consistency(graph, entities_vdb, relationships_vdb)\nif not report[\"consistent\"]:\n    await rebuild_entities_vdb(graph, entities_vdb, global_config)\n    await rebuild_relationships_vdb(graph, relationships_vdb, global_config)\n```\n","isInternal":false,"tokens":1853,"sizeBytes":8141},{"name":"README_SOURCE_CONFLICT_REPAIR.md","path":"lightrag/tools/README_SOURCE_CONFLICT_REPAIR.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/LightRAG/HEAD/lightrag/tools/README_SOURCE_CONFLICT_REPAIR.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Source-Conflict Listing / Repair\n\nOffline companion to the typed source resolver (LR2 §5.5).\n\nA canonical source key — a filename with any parser `[hint]` stripped — is\nexpected to belong to exactly one **primary** document. History can break that:\ncustom-ID inserts, legacy ids, and basename collisions predate the strict\nresolver. When two or more primary rows claim one key, the resolver returns a\n*conflict* and `/documents/scan` refuses to act on that file: it is not\nenqueued, no record is deleted and nothing is archived, because picking a winner\nautomatically could silently retire the document you wanted to keep.\n\nThis tool (and the equivalent HTTP endpoints) is how an operator settles such a\nconflict explicitly.\n\n- The winner is never chosen for you: you name the `primary_doc_id` to keep.\n- Every other candidate is marked `metadata.is_duplicate=true` with\n  `original_doc_id=<primary>`. **No content is ever deleted** — the demoted rows\n  keep their status and their `full_docs` entry; they only lose their claim on\n  the canonical source, which is what makes the resolver return a single\n  primary afterwards.\n- Only `doc_status` and `full_docs` are opened. No vector store, graph store,\n  LLM or embedding model is touched, so a doc_status bookkeeping problem never\n  requires a reachable Milvus/Neo4j/LLM endpoint to fix. `full_docs` is read (not\n  written) to verify the primary you named — see *Safety*.\n\n## Usage\n\nCLI (honors `WORKING_DIR` / `WORKSPACE` and the `LIGHTRAG_*` storage variables\nfrom `.env`, exactly like the server):\n\n```bash\n# What is conflicting?\npython -m lightrag.tools.source_conflict_repair list [--limit 50] [--all]\n\n# Plan a repair (dry-run: nothing is modified)\npython -m lightrag.tools.source_conflict_repair repair \\\n    --source report.docx --primary doc-abc123\n\n# Commit it\npython -m lightrag.tools.source_conflict_repair repair \\\n    --source report.docx --primary doc-abc123 --apply\n```\n\n`--workspace` overrides the `WORKSPACE` env var. Exit code is `1` when a repair\nis refused (unknown primary, candidate set changed, backend cannot repair).\n\nLibrary (for a deployment that builds its own `LightRAG`):\n\n```python\nfrom lightrag.tools.source_conflict_repair import (\n    collect_source_conflicts,\n    repair_one_conflict,\n)\n\nconflicts = await collect_source_conflicts(rag.doc_status, limit=50)\nresult = await repair_one_conflict(\n    rag.doc_status,\n    \"report.docx\",\n    \"doc-abc123\",\n    workspace=rag.workspace,\n    # Required for a COMMIT: it is what verifies the primary you named can\n    # actually keep the source. A dry-run needs none.\n    full_docs=rag.full_docs,\n    apply=True,\n)\n```\n\n## Online equivalent\n\n```\nGET  /documents/source_conflicts?limit=50&cursor=<next_cursor>\nPOST /documents/source_conflicts/repair\n     {\"canonical_source_key\": \"...\", \"primary_doc_id\": \"...\",\n      \"expected_candidate_count\": N, \"expected_candidate_fingerprint\": \"...\",\n      \"dry_run\": false}\n```\n\nThe HTTP flow is two explicit calls: a dry-run returns the\n`candidate_count`/`fingerprint` pair, and the commit echoes both back. The CLI\nperforms the same two steps for you — it automates the copy-paste, not the\nguard.\n\n## Safety\n\n- Listing and a repair without `--apply` mutate nothing.\n- A commit is mutually exclusive with every in-deployment writer that can change\n  which documents claim the key: enqueue (the enqueue-serialize lock), the\n  processing stage's duplicate marking (a keyed lock on the canonical source key,\n  which the marking takes too), and clear/delete + scan classification + manual\n  reset (a pending-enqueue reservation). So the two operations are ordered rather\n  than interleaved: if the marking goes first the commit refuses before demoting\n  anything; if the commit goes first it verifies the key is settled and returns,\n  and a later marking is an ordinary state transition. Only the standalone CLI and\n  a second deployment writing the same database fall outside this.\n- A commit re-reads the candidate set under the backend's repair lock and\n  proceeds only when the count and fingerprint still match the dry-run\n  (compare-and-set), so a concurrent enqueue / delete / repair fails the commit\n  instead of being overwritten. A refused commit changes nothing; re-run it.\n- A repair interrupted half-way needs no reconciliation and leaves no marker\n  behind: the rows it finished already express `duplicate`, the rest still\n  resolve as a conflict. Run the command again.\n- Repeating a completed repair is safe: the same request is refused (its token\n  described the old candidate set) and a fresh run is a no-op with one\n  candidate and nothing to demote.\n- A commit refuses a primary that cannot keep the source, because the demotions\n  are irreversible — a repair only demotes, and a key left with no candidate has\n  no conflict left to settle:\n  - no `full_docs` content (an unprocessable stub a scan would delete) → 409;\n  - content that already exists under a **different** source, which the\n    processing stage will mark `FAILED [DUPLICATE:content_hash]` → 409;\n  - the content could not be *verified* at all — read failure, a backend without\n    strict point reads, or no `full_docs` handle → 503, retry later. Unverified\n    is not verified.\n- Not covered, by construction: a primary that has not been parsed yet has no\n  content hash, so nothing can predict whether its content will turn out to\n  duplicate another document. The commit itself still succeeds, and the key\n  really is unique when it returns — no marking can interleave with it. A\n  marking that lands afterwards is an ordinary state transition, not a\n  half-applied repair: the outcome is the ordinary content-dedup steady state\n  (the content lives under that document and this key ends up with no primary),\n  and no content is lost.\n","isInternal":false,"tokens":1360,"sizeBytes":5813},{"name":"UserCustomizePrompts.md","path":"prompts/UserCustomizePrompts.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/LightRAG/HEAD/prompts/UserCustomizePrompts.md","title":"Agent Directive: usercustomizeprompts","category":"generic","format":"markdown","content":"# Folder for User-Defined Prompts\n\nThis folder is designated for storing customized prompt templates created by users.\n","isInternal":false,"tokens":22,"sizeBytes":119}],"systemPromptSnippet":"<yakaai_skills repo=\"HKUDS/LightRAG\">\n<!-- File: AGENTS.md (Tokens: ~7619 | Category: root-instruction) -->\n# Repository Guidelines\n\n## Project Overview\n\nLightRAG is a Retrieval-Augmented Generation (RAG) framework that uses graph-based knowledge representation for enhanced information retrieval. The system extracts entities and relationships from documents, builds a knowledge graph, and uses multiple retrieval modes (`local`, `global`, `hybrid`, `mix`, `naive`) for queries.\n\n## Project Structure\n\nTop-level directories:\n\n- **lightrag/**: Core Python package — see *Module Layout* below.\n- **lightrag_webui/**: React 19 + TypeScript client (Bun + Vite + Tailwind). UI components in `src/`.\n- **scripts/**: `test.sh` (preferred test runner), `setup/` interactive environment wizard (use `make env-*` rather than calling `setup.sh` directly — see *Configuration > Setup Wizard Outputs*), and release tooling.\n- **tests/**: Pytest coverage, organized into subdirectories that mirror `lightrag/` (see *Testing* below for layout). Working datasets stay in `inputs/`, `rag_storage/`, and `temp/`; deployment collateral lives in `docs/`, `k8s-deploy/`, and compose files.\n\n### Module Layout (`lightrag/`)\n\n- **lightrag.py**: Main orchestrator class (`LightRAG`) — assembled from mixins (see *LightRAG class composition*). Hosts `ainsert_custom_kg`, `_insert_done`, `_process_extract_entities`, `_refresh_addon_params_cache`, and `addon_params` accessors. Critical: always call `await rag.initialize_storages()` after instantiation.\n- **pipeline.py**: `_PipelineMixin` — owns the document ingestion pipeline (`apipeline_enqueue_documents`, `apipeline_process_enqueue_documents`, `apipeline_process_error_documents`), the `parse_native` / `parse_mineru` / `parse_docling` parser dispatchers, multimodal analysis, validation, and the worker scaffolding.\n- **utils_pipeline.py**: Pure helpers shared by the pipeline mixin and other entry points: doc-status field access, document identity (source key, content hash), parsed-artifact path resolution, parser payload normalization, multimodal entity augmentation, and `make_lightrag_doc_content`.\n- **llm_roles.py**: `RoleSpec` / `RoleLLMConfig` / `_RoleLLMState` / `ROLES` registry plus `_RoleLLMMixin` — role normalization, builder registration, wrapper rebuild, runtime config update, queue cleanup, sanitized config export, queue status reporting. Route role-specific behavior here rather than into provider modules.\n- **storage_migrations.py**: `_StorageMigrationMixin` — `check_and_migrate_data`, `_migrate_entity_relation_data`, `_migrate_chunk_tracking_storage`.\n- **addon_params.py**: `ObservableAddonParams` plus `default_addon_params` / `normalize_addon_params` helpers.\n- **operate.py**: Core extraction and query operations including entity/relation extraction, chunking, and multi-mode retrieval logic.\n- **base.py**: Abstract base classes for storage backends (`BaseKVStorage`, `BaseVectorStorage`, `BaseGraphStorage`, `BaseDocStatusStorage`).\n- **kg/**: Storage implementations (JSON, NetworkX, Neo4j, PostgreSQL, MongoDB, Redis, Milvus, Qdrant, Faiss, Memgraph, OpenSearch, NanoVectorDB). The backend registry (`STORAGE_IMPLEMENTATIONS` / `STORAGES`) lives in `kg/__init__.py`; `kg/factory.py::get_storage_class()` resolves backend classes from configuration.\n- **llm/**: LLM and embedding provider bindings (OpenAI, Ollama, Azure, Gemini, Bedrock, Anthropic, etc.). All async with caching support.\n- **parser/**: Unified parsing layer. `parser/routing.py` resolves engine and filename hints for `legacy`, `native`, `mineru`, and `docling` flows; `parser/debug.py` provides an offline LightRAG stub for the `parser/cli.py` debug entry point (`python -m lightrag.parser.cli`). Native format parsers live as sibling sub-packages under `parser/` (currently `parser/docx/`); external HTTP-based adapters live under `parser/external/` (`mineru`, `docling`) with shared helpers in `parser/external/_common.py`, `_manifest.py`, `_zip.py`.\n- **chunker/**: Chunking strategies (token-size, recursive character, semantic vector, paragraph semantic).\n- **api/**: FastAPI service (`lightrag_server.py`) with REST endpoints and Ollama-compatible API; routers under `routers/`, static Swagger assets, packaged WebUI output, and Gunicorn launcher.\n\n## Core Architecture\n\n### LightRAG class composition\n\n`LightRAG` is assembled from focused mixins (split out of the previously monolithic `lightrag.py`):\n\n```\nLightRAG → _RoleLLMMixin → _StorageMigrationMixin → _PipelineMixin → object\n```\n\nThe `@final` decorator on `LightRAG` is preserved — the mixin layering is an internal implementation detail, not an external subclassing surface. The public API (`ainsert`, `aquery`, `ainsert_custom_kg`, `initialize_storages`, etc.) is unchanged. `ainsert_custom_kg` and its internal construction logic, `_insert_done`, `_process_extract_entities`, `_refresh_addon_params_cache`, and the `addon_params` property accessors stay on `LightRAG` itself because they cut across multiple flows or depend on prompt-profile state.\n\n### Storage Layer\n\nLightRAG uses 4 storage types with pluggable backends:\n- **KV_STORAGE**: LLM response cache, text chunks, document info\n- **VECTOR_STORAGE**: Entity/relation/chunk embeddings\n- **GRAPH_STORAGE**: Entity-relation graph structure\n- **DOC_STATUS_STORAGE**: Document processing status tracking\n\nEach `LightRAG` instance can pass a `workspace` parameter for data isolation. Implementation differs per storage type:\n- **File-based**: subdirectories under `working_dir`.\n- **Collection-based**: collection name prefixes.\n- **Relational DB**: workspace column filtering.\n- **Qdrant**: payload-based partitioning.\n\n### Consistency without transactions\n\nLightRAG writes to independent stores — graph, KV, vector, doc-status — with **no transaction across them**. Every multi-store operation therefore has intermediate states, and no ordering removes them; an ordering only chooses which one it keeps.\n\n- **The rule:** an inconsistency is acceptable when it **heals itself later** (a retry, a rebuild, or the next run rewrites it) or is **harmless in direction**. Losing data is never acceptable; retaining an object that could have been deleted, or surfacing a chunk a query did not need, is.\n- A change must **improve on an accepted residue**, not swap it for its mirror. \"An inconsistent state exists\" is not by itself a defect report — the questions are which state, how it heals, and whether the alternative is better.\n- Every accepted residue is **written down** with its reason and recovery path, next to the code or in the relevant contract. An undocumented residue is a defect; a documented one is a decision.\n- This licenses nothing for **silent failure**. A durable write must never be reported as one that did not happen, and a failure must never be swallowed: fail loud, then let the documented residue heal.\n\n### File-backed storage contracts\n\n**Full contracts: [docs/design/NetworkXSingleWriterContract.md](docs/design/NetworkXSingleWriterContract.md) — read it before touching `lightrag/kg/networkx_impl.py` or any caller of `index_done_callback` on the graph store; [docs/design/FileBackedSnapshotContract.md](docs/design/FileBackedSnapshotContract.md) — read it before touching `lightrag/kg/nano_vector_db_impl.py`, `lightrag/kg/faiss_impl.py`, `lightrag/kg/json_kv_impl.py`, `lightrag/kg/json_doc_status_impl.py` or `lightrag/kg/file_fingerprint.py`.**\n\nFive storages keep their data in memory and publish it by rewriting a whole file, and **they do not share one model** — `JsonKVStorage` and `JsonDocStatusStorage` are the odd ones out and the contracts say so at length. Read the right one before assuming.\n\n- **All five**: a commit publishes the WHOLE namespace, so any writer's flush also publishes every other writer's pending mutation there, half-finished ones included. All five are supported for **small-scale testing and validation only**; no change to them may be justified by write throughput.\n- **`NetworkXStorage`, `NanoVectorDBStorage`, `FaissVectorDBStorage`** keep one in-memory copy per process and reconcile by reloading the file. Visibility rests on a **two-channel fence**: the file's own `(st_mtime_ns, st_size)` (authoritative, state) OR-ed with the `storage_updated` flag (accelerator, a consumable event). Both are permanent — their blind spots do not overlap.\n- Those three diverge on a write conflict, and the reason is in the contracts: the graph store **declines** the commit (it has no buffer to replay, and graph payloads are accumulate-over-read), the vector stores **reload and replay** their pending buffers and redo logs. Do not reopen reload-then-replay for the graph store without addressing the accumulate-over-read argument.\n- **`JsonKVStorage` and `JsonDocStatusStorage` use none of that.** Their data is a `Manager().dict()` every worker shares, so a mutation is visible everywhere immediately and there is nothing to reload — adding a `_get_*` entry method would be wrong. Their `storage_updated` flag means the OPPOSITE of the other three's: `True` is \"dirty data still to flush\", never \"fresher data on disk to reload\". Do not read it as a peer notification. `JsonDocStatusStorage` reimplements this protocol rather than inheriting it, so a change to one of the pair is almost always a change the other needs too; where they diverge is the flush trigger — doc-status writes that change scheduling state flush synchronously because doc-status is the pipeline's recovery anchor.\n- `NetworkXStorage` is the only storage that declares `requires_single_writer`, which is what puts the admin flows under `LightRAG._admin_write_gate`.\n\n### Pipeline concurrency contract\n\n**Full contract: [docs/design/PipelineConcurrencyContract.md](docs/design/PipelineConcurrencyContract.md) — read it before touching `lightrag/pipeline.py`, `lightrag/kg/pipeline_ingress.py`, `pipeline_status` fields, or any `/documents/*` endpoint.**\n\n- Concurrent writers coordinate through `pipeline_status` (per-workspace shared dict in `lightrag.kg.shared_storage`), mutated under `get_namespace_lock(\"pipeline_status\", workspace=...)`.\n- `busy` alone does NOT block enqueue — enqueue + processing are allowed to run concurrently. Three states do refuse it: `destructive_busy` (clear / delete, which drops storages), `scanning_exclusive` (scan's classification phase), and `manual_freeze_requested` (a manual retry draining the pipeline to idle).\n- **Admin graph writes** (`acreate_*` / `aedit_*` / `adelete_by_*` / `amerge_entities` / `ainsert_custom_kg`) run inside `LightRAG._admin_write_gate` when the graph storage declares `requires_single_writer` (`NetworkXStorage` only): a workspace admin lock (waited for) then the `busy` reservation (`kind=\"admin\"`, refuses on `busy` / `scanning`), in that fixed order and OUTSIDE the per-entity keyed locks. A pipeline start during the hold is deferred into the ingress mailbox and driven once on release. The routes' `check_pipeline_busy_or_raise` preflight exempts an `admin`-owned `busy` so a second REST edit reaches the admin lock and queues. Never re-acquire the admin lock inside `lightrag/utils_graph.py`.\n- The workspace **ingress mailbox** (`get_pipeline_ingress(workspace)`) is the pipeline's only wake-up channel; `doc_status` stays the source of truth, so a dropped notification is recovered by the next strict scan.\n- FAILED documents never resume automatically: they re-enter only through a sticky manual retry request (`/documents/scan`, `/documents/reprocess_failed`), granting ONE attempt each.\n- All scheduling-control-plane `doc_status` queries use `get_docs_by_statuses(..., strict=True)`; scheduler `full_docs` reads must distinguish confirmed-absent (`None`) from backend errors (raise).\n\n### Purge recovery contract\n\n**Full contract: [docs/design/PurgeRecoveryContract.md](docs/design/PurgeRecoveryContract.md) — read it before touching `_purge_kg_contributions`, `adelete_by_doc_id`, the anchor writes in `merge_nodes_and_edges`, the `kg_write_state` / `kg_purge` metadata, or the cache write ordering in `use_llm_func_with_cache`.**\n\n- \"What did this document contribute?\" is answerable only from the per-document write-ahead anchors (`full_entities` / `full_relations`). The reverse lookup through `text_chunks` is not a fallback — purge deletes those chunks.\n- Governing invariant: **a purge must never delete something that CARRIES attribution — a chunk row or an anchor row that names objects — and leave those objects behind.** `_purge_kg_contributions` **fails closed** (`RecoveryAnchorMissingError` → HTTP 409, nothing deleted) unless one of four proofs holds: `anchors`, `pre_graph`, `journal`, `empty_scope`.\n- **`kg_write_state` must never be inferred or backfilled** — it is written once at enqueue and is monotonic. A backfill reproduces the original silent-skip defect.\n- `kg_write_state` and `kg_purge` must stay in both `_DOC_STATUS_METADATA_CARRY_OVER_KEYS` and `_DOC_STATUS_METADATA_DIRECTIVE_KEYS` (`lightrag/utils_pipeline.py`); dropping either turns a resumable purge into a permanent refusal.\n- Chunk tracking (`entity_chunks` / `relation_chunks`) outranks graph `source_id`; code folding a `source_id` delta back into tracking must append genuine additions only.\n- LLM extraction cache rows are reachable only through the owning chunk's `llm_cache_list`, which makes that list an attribution carrier too: [LLM extraction cache reachability](docs/design/PurgeRecoveryContract.md#llm-extraction-cache-reachability) states the reference-before-row ordering, why a reference that cannot be recorded skips the cache write instead, and what the ordering does not close.\n- Merge and rename apply *Consistency without transactions* above: [the failure model](docs/design/PurgeRecoveryContract.md#merge-and-rename-failure-model) lists their ordering invariants, accepted residues and already-rejected remedies. Read it before reordering `_merge_entities_impl` or the rename branch of `_edit_entity_impl`.\n\n### Relation weight contract\n\n**Full contract: [docs/ProgramingWithCore.md](docs/ProgramingWithCore.md#relation-weight-contract)** — keep it synchronized with the core API docstrings, REST graph documentation, and custom-KG examples whenever relation write behavior changes.\n\n- `weight >= len(distinct real source IDs)` on the graph edge; a larger value is an optional importance boost. Empty IDs and the legacy placeholders `manual_creation` / `UNKNOWN` are not evidence, so a source-less relation may use any non-negative fractional weight.\n- Public ingress paths (`create_relation`, `edit_relation`, `insert_custom_kg`) must validate the complete relation **before the first storage mutation**. To go below the current evidence count, creation callers omit `source_id`, edit callers set it to an empty string in the same operation.\n- Entity merges use `max(all input weights, distinct merged real source IDs)`.\n- Legacy-row repair is **opportunistic, not a sweep**: `_merge_edges_then_upsert`'s KEEP-cap skip branch returns the stored edge without writing graph or vector record, so an undersized legacy row stays undersized until a merge, an unrelated edit, or a rebuild rewrites it.\n\n### Query Modes\n\n- **local**: Context-dependent retrieval focused on specific entities\n- **global**: Community/summary-based broad knowledge retrieval\n- **hybrid**: Combines local and global\n- **naive**: Direct vector search without graph\n- **mix**: Integrates KG and vector retrieval (recommended with reranker)\n\n## Development Commands\n\n### Setup\n```bash\n# Install with uv\nuv sync\nsource .venv/bin/activate  # Or: .venv\\Scripts\\activate on Windows\n\n# Install with API support\nuv sync --extra api\n\n# Install specific extras\nuv sync --extra offline-storage  # Storage backends\nuv sync --extra offline-llm      # LLM providers\nuv sync --extra test             # Testing dependencies\n```\n\n### API Server\n```bash\n# Copy and configure environment\ncp env.example .env  # Edit with your LLM/embedding configs\n\n# Build WebUI\ncd lightrag_webui\nbun install --frozen-lockfile\nbun run build\ncd ..\n\n# Run server\nlightrag-server                                           # Production\nuvicorn lightrag.api.lightrag_server:app --reload        # Development\nlightrag-gunicorn                                         # Multi-worker (gunicorn)\n```\n\n### WebUI\n\nEvery command below is run **from `lightrag_webui/`**, not the repository root.\n`bun test` in particular resolves `bunfig.toml` — and the preload paths inside\nit — relative to the working directory, so running it from the root silently\nloads no DOM (see *React component tests*).\n\n```bash\ncd lightrag_webui\nbun install --frozen-lockfile      # REQUIRED after any change to package.json /\n                                   # bun.lock, including a branch switch across\n                                   # one: `git checkout` does not update\n                                   # node_modules, and a stale tree surfaces as\n                                   # a wall of TS2307 \"Cannot find module\".\nbun run dev                        # Dev server (Node + Vite)\nbun run dev:bun                    # Dev server (Bun native)\nbun run build                      # Production build\nbun run preview                    # Preview production build\nbun run lint                       # ESLint over *.ts/tsx/js/jsx\n\n# Testing — Bun built-in runner (NOT Vitest/Jest)\nbun test                           # All tests\nbun test --watch                   # Watch mode\nbun test --coverage                # With coverage report\nbun test src/api/lightrag.test.ts  # Single test file\nbunx tsc --noEmit                  # Typecheck (`bun run build` does NOT typecheck)\n```\n\n### Testing\n\n- Use mock-based tests for external services (Redis, httpx, etc.) — do not depend on live services in unit tests.\n- Add regression tests for every bug fix.\n- **Run only the test directories that mirror the modules you changed**, and report which subset you ran plus its pass count. The suite is ~7800 offline tests: about 5 minutes serially, ~70 s with `-n auto` on an 8-core machine — still too slow for the edit loop. Every PR's CI runs the full offline suite (and `tests/setup/` when its inputs change) — proving nothing else broke is its job, not yours.\n- Derive the subset from the mirror layout below: `lightrag/api/config.py` → `tests/api/config/`, `lightrag/kg/redis_impl.py` → `tests/kg/redis_impl/`, `lightrag/chunker/` → `tests/chunker/`. When a change spans several modules, run each of their directories rather than widening to `tests/`.\n- **`tests/setup/` is the setup wizard's suite and runs only when a wizard input changed**: `scripts/setup/**`, `env.example`, `docker-compose*.yml`, `Makefile`, or `tests/setup/` itself. It never imports `lightrag`, so a `lightrag/` change never needs it. `norecursedirs` in `pyproject.toml` already skips it in every recursive run (`pytest`, `pytest tests`, the full suite, CI's main job); run it by naming it: `./scripts/test.sh tests/setup`. The easy one to miss is `env.example`: adding a config knob for `lightrag` usually edits it, and the wizard parses it, so that change runs `tests/setup/` too.\n- Run the full suite locally only at a milestone, or when the change is genuinely cross-cutting (`lightrag/base.py`, `lightrag/utils.py`, `lightrag/kg/shared_storage.py`, or anything every backend inherits). **Run it with `-n auto --dist worksteal`** (pytest-xdist, in the `pytest` extra): on an 8-core machine it cuts ~5 min to ~1 min. CI stays serial because hosted 4-vCPU runners showed no gain. Do not add `-n` to subset runs: every worker re-collects and re-imports the suite (~12 s), which costs more than a single directory saves. Drop `-n` when you need `pdb`/`-s` or ordered output.\n- Backend tests use pytest; frontend unit tests use Bun's built-in runner — see *WebUI* above and *React component tests* below.\n- **A WebUI change runs the WHOLE frontend check set**, from `lightrag_webui/`: `bun install --frozen-lockfile` (see *WebUI* above — skip it after a branch switch and every later step fails on missing modules), then `bun test`, `bunx tsc --noEmit`, and `bun run lint`. The subsetting rule above is a backend rule and does not apply — all three together take well under a minute (test ~2 s, typecheck ~14 s, lint ~21 s), so there is nothing to save by running less. Report the pass count. `bun run build` transpiles WITHOUT checking types, so skipping `tsc --noEmit` means nothing checks them.\n\n```bash\n# Preferred for fresh shells and automation; resolves PYTHON, venv, uv, .venv, venv, python, python3\n# Default during development: only the directories mirroring the changed modules\n./scripts/test.sh tests/api/config\n./scripts/test.sh tests/kg/redis_impl\n\n# Run specific test file\n./scripts/test.sh tests/kg/test_graph_storage.py\n\n# Full suite — ~7800 offline tests; milestones and cross-cutting changes only.\n# Parallel via pytest-xdist: ~70 s on 8 cores (vs ~5 min serial).\n# tests/setup is skipped by default (norecursedirs); name it explicitly when\n# scripts/setup/, env.example, docker-compose*.yml or Makefile change.\n./scripts/test.sh tests -m offline -n auto --dist worksteal\n./scripts/test.sh tests/setup -m offline -n auto --dist worksteal\n\n# Integration-test concurrency (a LightRAG fixture, NOT pytest parallelism)\n./scripts/test.sh tests --test-workers 4\n```\n\n- `tests/`: main test suite, mirrors feature folders. Place new tests under the subdirectory matching the module under test:\n  - `tests/api/{auth,config,routes}/` for FastAPI server tests (auth/token, config loading, route handlers); top-level `tests/api/` for app-wide concerns (path prefixes, Ollama-compatible endpoint).\n  - `tests/chunker/`, `tests/evaluation/`, `tests/extraction/` for the like-named modules.\n  - `tests/kg/<backend>_impl/` for backend-specific storage tests, mirroring the `lightrag/kg/<backend>_impl.py` file naming. The `_impl` suffix on every subdirectory keeps the layout uniform and avoids `sys.path` shadowing on names that overlap with top-level PyPI/stdlib packages (`faiss`, `json`, `neo4j`, `networkx`, `redis`) when a test is launched directly via `python tests/kg/...`. Current backends: `faiss_impl/`, `json_impl/`, `memgraph_impl/`, `milvus_impl/`, `mongo_impl/`, `nano_impl/`, `neo4j_impl/`, `networkx_impl/`, `opensearch_impl/`, `postgres_impl/`, `qdrant_impl/`, `redis_impl/`. `tests/kg/` root holds cross-backend tests (`test_graph_storage`, `test_batch_graph_operations`, `test_unified_lock_safety`, `test_file_atomic`).\n  - `tests/llm/<provider>_impl/` for provider-specific behavior, same `_impl` convention: `bedrock_impl/`, `gemini_impl/`, `ollama_impl/`, `openai_impl/`, `voyageai_impl/`, `zhipu_impl/`. `tests/llm/` root holds cross-provider concerns (embedding, VLM, cache, role).\n  - `tests/parser/`, `tests/parser/docx/`, `tests/parser/external/{mineru,docling}/` for parser implementations.\n  - `tests/pipeline/` for ingestion pipeline and doc-status behavior (including `test_pipeline_*`, `test_doc_status_*`, `test_multimodal_*`, `test_graph_keyed_locks`).\n  - `tests/sidecar/`, `tests/setup/`, `tests/workspace/` for the like-named cross-cutting concerns. `tests/setup/` never imports `lightrag`; it runs in its own path-filtered CI job (`.github/workflows/setup-tests.yml`) and `norecursedirs` keeps it out of every recursive run, so a new input the wizard reads must be added to that workflow's `paths`.\n  - When adding a new backend or LLM provider, create a new subdirectory plus an empty `__init__.py` rather than dropping the file in the parent directory root.\n- Markers (registered in `[tool.pytest.ini_options]` in `pyproject.toml`): `offline`, `integration`, `requires_db`, `requires_api`, `pg_smoke`. Integration tests are skipped by default via `-m \"not integration\"`; opt in with `--run-integration`.\n- Integration env vars: `LIGHTRAG_RUN_INTEGRATION=true`, `LIGHTRAG_KEEP_ARTIFACTS=true`, `LIGHTRAG_TEST_WORKERS=4`, plus storage-specific connection strings.\n\n#### React component tests\n\n**Full guide: [docs/design/WebUITestingGuide.md](docs/design/WebUITestingGuide.md) — read it before writing or converting any test under `lightrag_webui/src/`.**\n\n- Tests are **colocated** next to the module they cover (`src/features/SiteHeader.test.ts`); the `tests/` mirror layout above is a backend rule. A test file containing JSX must be named `.test.tsx`.\n- **Run `bun test` from `lightrag_webui/`.** The `bunfig.toml` preload that installs the DOM is resolved against the CWD, and from the repository root the failure is silent — pure logic tests still pass, only component tests break.\n- **Test rendered behavior by rendering it**, through `renderWithProviders` (`src/test/render.tsx`). Do not write new tests that `readFileSync` a `.tsx` and match substrings.\n- **Never assert `toBeNull()` / `not.toBeInTheDocument()` / `toBe(element)` on a DOM node** — Bun serialises the whole happy-dom element and the run appears to HANG instead of failing. Assert a count or an extracted boolean/string instead.\n- **The DOM is process-wide**: never leave a bare `delete` of the `window` / `document` global behind; use `withoutDomGlobals()` from `src/test/domGlobals.ts`.\n- **Never import the test harness from production code** — one import from `src/` ships happy-dom to the browser.\n- **Prove the test can fail**: break the behavior it pins, see it go red, then restore.\n\n### Linting\n```bash\nruff check .\n```\n\n## Key Implementation Patterns\n\n### LightRAG Initialization (Critical)\n\nThe most common error is forgetting to initialize storages (manifests as `AttributeError: __aenter__` or `KeyError: 'history_messages'`):\n\n```python\nimport asyncio\nfrom lightrag import LightRAG\nfrom lightrag.llm.openai import gpt_4o_mini_complete, openai_embed\n\nasync def main():\n    rag = LightRAG(\n        working_dir=\"./rag_storage\",\n        llm_model_func=gpt_4o_mini_complete,\n        embedding_func=openai_embed\n    )\n\n    # REQUIRED: Initialize storage backends\n    await rag.initialize_storages()\n\n    # Now safe to use\n    await rag.ainsert(\"Your text here\")\n    result = await rag.aquery(\"Your question\", param=QueryParam(mode=\"hybrid\"))\n\n    # Cleanup\n    await rag.finalize_storages()\n\nasyncio.run(main())\n```\n\n### Custom Embedding Functions\n\n**Full guide: [docs/ProgramingWithCore.md](docs/ProgramingWithCore.md#custom-embedding-functions)** — the `.func` unwrapping rule, `max_token_size`, the `(len(texts), embedding_dim)` return contract that `EmbeddingFunc.__call__` enforces on every call, and the clear-the-data-directory pitfall when switching models. Read it before writing or wrapping an embedding function.\n\n### Storage Configuration\n\nConfigure via environment variables or constructor params:\n\n```python\n# Environment-based (recommended for production)\n# See env.example for full list\n\n# Constructor-based\nrag = LightRAG(\n    working_dir=\"./storage\",\n    workspace=\"project_name\",  # For data isolation\n    kv_storage=\"PGKVStorage\",\n    vector_storage=\"PGVectorStorage\",\n    graph_storage=\"Neo4JStorage\",\n    doc_status_storage=\"PGDocStatusStorage\",\n    vector_db_storage_cls_kwargs={\n        \"cosine_better_than_threshold\": 0.2\n    }\n)\n```\n\n### Document Insertion\n\n```python\n# Single document\nawait rag.ainsert(\"Text content\")\n\n# Batch insertion\nawait rag.ainsert([\"Text 1\", \"Text 2\", ...])\n\n# With custom IDs\nawait rag.ainsert(\"Text\", ids=[\"doc-123\"])\n\n# With file paths (for citation)\nawait rag.ainsert([\"Text 1\", \"Text 2\"], file_paths=[\"doc1.pdf\", \"doc2.pdf\"])\n\n# Configure batch size\nrag = LightRAG(..., max_parallel_insert=4)  # Default: 3, max recommended: 10\n```\n\n### Query Configuration\n\n```python\nfrom lightrag import QueryParam\n\nresult = await rag.aquery(\n    \"Your question\",\n    param=QueryParam(\n        mode=\"mix\",                    # Recommended with reranker\n        top_k=60,                      # KG entities/relations to retrieve\n        chunk_top_k=20,                # Text chunks to retrieve\n        max_entity_tokens=6000,\n        max_relation_tokens=8000,\n        max_total_tokens=30000,\n        enable_rerank=True,\n        user_prompt=\"Additional instructions for LLM\",\n        stream=False\n    )\n)\n```\n\n## Frontend Debugging via Playwright\n\nFor WebUI bugs whose symptoms only surface in the rendered DOM — layout/overflow/scrollbar issues, transient flashes, third-party libraries attaching helpers to `<body>` outside React's tree, or end-to-end verification of a fix — drive the running dev server (`http://localhost:5173`) with the `document-skills:webapp-testing` skill instead of reasoning from source alone. Seed state directly via `localStorage` (persist key `settings-storage`, schema in `lightrag_webui/src/stores/settings.ts`) to skip live LLM calls. Use `wait_until=\"domcontentloaded\"` plus a selector wait — Vite dev's long-lived polling makes `networkidle` time out.\n\n## Configuration\n\n### .env Configuration\nPrimary configuration file for API server. Generate it with `make env-base` or copy `env.example` manually. Key sections:\n- Server settings (HOST, PORT, CORS)\n- Storage backends (connection strings via environment variables)\n- Query parameters (TOP_K, MAX_TOTAL_TOKENS, etc.)\n- Reranking configuration (RERANK_BINDING, RERANK_MODEL)\n- Authentication (AUTH_ACCOUNTS, LIGHTRAG_API_KEY)\n\nSee `env.example` for comprehensive template.\n\n### Setup Wizard Outputs\n- Keep `.env` host-usable. Container-only hostnames and staged SSL paths belong in the wizard-managed compose layer, not persisted back into `.env`.\n- Treat `docker-compose.final.yml` as generated output assembled from `scripts/setup/templates/*.yml`.\n- For setup workflow changes, prefer `make env-*` targets over direct `scripts/setup/setup.sh` calls.\n\n## Code Style\n\n### Language\nComments, backend code, log messages, and Git commit messages in English. Frontend uses i18next for multi-language support.\n\n### Docstrings and comments\n\nDocstrings state the **rules**: what a caller must do, what it must not do, and the gotchas it will otherwise be caught by. The **mechanism** — how it works, the accepted residues, and the alternatives already rejected — goes in `docs/design/` with a pointer from the docstring. A docstring that has grown into a design document is the thing this separates: it buries the code, and the same facts in two places drift apart.\n\nTwo rules are enforced by `tests/test_docstring_budget.py` rather than by review, because both are properties of the tree rather than of any one change:\n\n- **No docstring over 80 lines** (100 for a class). The limit is generous on purpose — it catches a document, not a thorough docstring.\n- **No source line may cite a GitHub issue number.** The referent does not survive a fork, so a comment saying something is \"documented in #NNNN\" leaves nothing that documents it. Name the thing instead (\"the two-channel fence\"), or move the content into `docs/design/` and cite that. Never delete such a reference bare — migrate what it pointed at first. Enforced over `lightrag/` only: **tests may cite issue numbers**, and many do. There the number names the defect the test pins, and the test itself — its name, docstring and assertions — is the documentation, so the reference is provenance rather than the thing carrying the meaning.\n\nA third test requires every `docs/**.md` path named in the package to resolve; nothing imports those strings, so a typo is otherwise silent.\n\n### Python\n- Follow PEP 8 with 4-space indentation\n- Use type annotations\n- Prefer dataclasses for state management\n- Use `lightrag.utils.logger` instead of print\n- Async/await patterns throughout\n\n### TypeScript / React (incl. WebUI ESLint)\n- Functional components with hooks; PascalCase for components\n- 2-space indentation, single quotes (enforced by `@stylistic` rules)\n- Tailwind utility-first styling\n- ESLint stack: TypeScript-ESLint + React Hooks plugin + Prettier; `@typescript-eslint/no-explicit-any` is disabled (allowed)\n\n## Commit and Pull Request Guidance\n\n- If this repo is a fork of `HKUDS/LightRAG`. Target to `HKUDS/LightRAG` when creating PRs, not the fork's own repo.\n- PR descriptions should include: summary, motivation, linked issues if applyed, what's changed, what's broken and how it works.\n- Write commit messages (subject and body) in English. Commit messages are repository artifacts — like code comments and log messages — not conversational replies, so they follow the English code-style rule above regardless of any per-conversation working language.\n\n<!-- File: lightrag/tools/README_CLEAN_LLM_QUERY_CACHE.md (Tokens: ~4284 | Category: anthropic-skill) -->\n# LLM Query Cache Cleanup Tool - User Guide\n\n## Overview\n\nThis tool cleans up LightRAG's LLM query cache from KV storage implementations. It specifically targets query caches generated during RAG query operations (modes: `mix`, `hybrid`, `local`, `global`), including both query and keywords caches.\n\n## Supported Storage Types\n\n1. **JsonKVStorage** - File-based JSON storage\n2. **RedisKVStorage** - Redis database storage\n3. **PGKVStorage** - PostgreSQL database storage\n4. **MongoKVStorage** - MongoDB database storage\n5. **OpenSearchKVStorage** - OpenSearch index storage\n\n## Cache Types\n\nThe tool cleans up the following query cache types:\n\n### Query Cache Modes (4 types)\n- `mix:*` - Mixed mode query caches\n- `hybrid:*` - Hybrid mode query caches\n- `local:*` - Local mode query caches\n- `global:*` - Global mode query caches\n\n### Cache Content Types (2 types)\n- `*:query:*` - Query result caches\n- `*:keywords:*` - Keywords extraction caches\n\n### Cache Key Format\n```\n<mode>:<cache_type>:<hash>\n```\n\nExamples:\n- `mix:query:5ce04d25e957c290216cee5bfe6344fa`\n- `mix:keywords:fee77b98244a0b047ce95e21060de60e`\n- `global:query:abc123def456...`\n- `local:keywords:789xyz...`\n\n**Important Note**: This tool does NOT clean extraction caches (`default:extract:*` and `default:summary:*`). Use the migration tool or manual deletion for those caches.\n\n## Prerequisites\n\n- The tool reads storage configuration from environment variables\n- Ensure the target storage is properly configured and accessible\n- Backup important data before running cleanup operations\n\n## Usage\n\n### Basic Usage\n\nRun from the LightRAG project root directory:\n\n```bash\npython -m lightrag.tools.clean_llm_query_cache\n# or\npython lightrag/tools/clean_llm_query_cache.py\n```\n\n### Interactive Workflow\n\nThe tool guides you through the following steps:\n\n#### 1. Select Storage Type\n```\n============================================================\nLLM Query Cache Cleanup Tool - LightRAG\n============================================================\n\n=== Storage Setup ===\n\nSupported KV Storage Types:\n[1] JsonKVStorage\n[2] RedisKVStorage\n[3] PGKVStorage\n[4] MongoKVStorage\n[5] OpenSearchKVStorage\n\nSelect storage type (1-5) (Press Enter to exit): 1\n```\n\n**Note**: You can press Enter or type `0` at any prompt to exit gracefully.\n\n#### 2. Storage Validation\nThe tool will:\n- Check required environment variables\n- Auto-detect workspace configuration\n- Initialize and connect to storage\n- Verify connection status\n\n```\nChecking configuration...\n✓ All required environment variables are set\n\nInitializing storage...\n- Storage Type: JsonKVStorage\n- Workspace: space1\n- Connection Status: ✓ Success\n```\n\n#### 3. View Cache Statistics\n\nThe tool displays a detailed breakdown of query caches by mode and type:\n\n```\nCounting query cache records...\n\n📊 Query Cache Statistics (Before Cleanup):\n┌────────────┬────────────┬────────────┬────────────┐\n│ Mode       │ Query      │ Keywords   │ Total      │\n├────────────┼────────────┼────────────┼────────────┤\n│ mix        │      1,234 │        567 │      1,801 │\n│ hybrid     │        890 │        423 │      1,313 │\n│ local      │      2,345 │      1,123 │      3,468 │\n│ global     │        678 │        345 │      1,023 │\n├────────────┼────────────┼────────────┼────────────┤\n│ Total      │      5,147 │      2,458 │      7,605 │\n└────────────┴────────────┴────────────┴────────────┘\n```\n\n#### 4. Select Cleanup Scope\n\nChoose what type of caches to delete:\n\n```\n=== Cleanup Options ===\n[1] Delete all query caches (both query and keywords)\n[2] Delete query caches only (keep keywords)\n[3] Delete keywords caches only (keep query)\n[0] Cancel\n\nSelect cleanup option (0-3): 1\n```\n\n**Cleanup Types:**\n- **Option 1 (all)**: Deletes both query and keywords caches across all modes\n- **Option 2 (query)**: Deletes only query caches, preserves keywords caches\n- **Option 3 (keywords)**: Deletes only keywords caches, preserves query caches\n\n#### 5. Confirm Deletion\n\nReview the cleanup plan and confirm:\n\n```\n============================================================\nCleanup Confirmation\n============================================================\nStorage: JsonKVStorage (workspace: space1)\nCleanup Type: all\nRecords to Delete: 7,605 / 7,605\n\n⚠️  WARNING: This will delete ALL query caches across all modes!\n\nContinue with deletion? (y/n): y\n```\n\n#### 6. Execute Cleanup\n\nThe tool performs batch deletion with real-time progress:\n\n**JsonKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing 1,000 records at a time from JsonKVStorage\n\nBatch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓\nBatch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓\n...\nBatch 8/8: ████████████████████ 7,605/7,605 (100.0%) ✓\n\nPersisting changes to storage...\n✓ Changes persisted successfully\n```\n\n**RedisKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing Redis keys in batches of 1,000\n\nBatch 1: Deleted 1,000 keys (Total: 1,000) ✓\nBatch 2: Deleted 1,000 keys (Total: 2,000) ✓\n...\n```\n\n**PostgreSQL Example:**\n```\n=== Starting Cleanup ===\n💡 Executing PostgreSQL DELETE query\n\n✓ Deleted 7,605 records in 0.45s\n```\n\n**MongoDB Example:**\n```\n=== Starting Cleanup ===\n💡 Executing MongoDB deleteMany operations\n\nPattern 1/8: Deleted 1,234 records ✓\nPattern 2/8: Deleted 567 records ✓\n...\nTotal deleted: 7,605 records\n```\n\n**OpenSearchKVStorage Example:**\n```\n=== Starting Cleanup ===\n💡 Processing 1,000 records at a time from OpenSearchKVStorage\n\nBatch 1/8: ████░░░░░░░░░░░░░░░░ 1,000/7,605 (13.1%) ✓\nBatch 2/8: ████████░░░░░░░░░░░░ 2,000/7,605 (26.3%) ✓\n...\n```\n\n#### 7. Review Cleanup Report\n\nThe tool provides a comprehensive final report:\n\n**Successful Cleanup:**\n```\n============================================================\nCleanup Complete - Final Report\n============================================================\n\n📊 Statistics:\n  Total records to delete:  7,605\n  Total batches:            8\n  Successful batches:       8\n  Failed batches:           0\n  Successfully deleted:     7,605\n  Failed to delete:         0\n  Success rate:             100.00%\n\n📈 Before/After Comparison:\n  Total caches before:      7,605\n  Total caches after:       0\n  Net reduction:            7,605\n\n============================================================\n✓ SUCCESS: All records cleaned up successfully!\n============================================================\n\n📊 Query Cache Statistics (After Cleanup):\n┌────────────┬────────────┬────────────┬────────────┐\n│ Mode       │ Query      │ Keywords   │ Total      │\n├────────────┼────────────┼────────────┼────────────┤\n│ mix        │          0 │          0 │          0 │\n│ hybrid     │          0 │          0 │          0 │\n│ local      │          0 │          0 │          0 │\n│ global     │          0 │          0 │          0 │\n├────────────┼────────────┼────────────┼────────────┤\n│ Total      │          0 │          0 │          0 │\n└────────────┴────────────┴────────────┴────────────┘\n```\n\n**Cleanup with Errors:**\n```\n============================================================\nCleanup Complete - Final Report\n============================================================\n\n📊 Statistics:\n  Total records to delete:  7,605\n  Total batches:            8\n  Successful batches:       7\n  Failed batches:           1\n  Successfully deleted:     6,605\n  Failed to delete:         1,000\n  Success rate:             86.85%\n\n📈 Before/After Comparison:\n  Total caches before:      7,605\n  Total caches after:       1,000\n  Net reduction:            6,605\n\n⚠️  Errors encountered: 1\n\nError Details:\n------------------------------------------------------------\n\nError Summary:\n  - ConnectionError: 1 occurrence(s)\n\nFirst 5 errors:\n\n  1. Batch 3\n     Type: ConnectionError\n     Message: Connection timeout after 30s\n     Records lost: 1,000\n\n============================================================\n⚠️  WARNING: Cleanup completed with errors!\n   Please review the error details above.\n============================================================\n```\n\n## Technical Details\n\n### Workspace Handling\n\nThe tool retrieves workspace in the following priority order:\n\n1. **Storage-specific workspace environment variables**\n   - PGKVStorage: `POSTGRES_WORKSPACE`\n   - MongoKVStorage: `MONGODB_WORKSPACE`\n   - RedisKVStorage: `REDIS_WORKSPACE`\n   - OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`\n\n2. **Generic workspace environment variable**\n   - `WORKSPACE`\n\n3. **Default value**\n   - Empty string (uses storage's default workspace)\n\n### Batch Deletion\n\n- Default batch size: 1000 records/batch\n- Prevents memory overflow and connection timeouts\n- Each batch is processed independently\n- Failed batches are logged but don't stop cleanup\n\n### Storage-Specific Deletion Strategies\n\n#### JsonKVStorage\n- Collects all matching keys first (snapshot approach)\n- Deletes in batches with lock protection\n- Fast in-memory operations\n\n#### RedisKVStorage\n- Uses SCAN with pattern matching\n- Pipeline DELETE for batch operations\n- Cursor-based iteration for large datasets\n\n#### PostgreSQL\n- Single DELETE query with OR conditions\n- Efficient server-side bulk deletion\n- Uses LIKE patterns for mode/type matching\n\n#### MongoDB\n- Multiple deleteMany operations (one per pattern)\n- Regex-based document matching\n- Returns exact deletion counts\n\n### Pattern Matching Implementation\n\n**JsonKVStorage:**\n```python\n# Direct key prefix matching\nif key.startswith(\"mix:query:\") or key.startswith(\"mix:keywords:\")\n```\n\n**RedisKVStorage:**\n```python\n# SCAN with namespace-prefixed patterns\npattern = f\"{namespace}:mix:query:*\"\ncursor, keys = await redis.scan(cursor, match=pattern)\n```\n\n**PostgreSQL:**\n```python\n# SQL LIKE conditions\nWHERE id LIKE 'mix:query:%' OR id LIKE 'mix:keywords:%'\n```\n\n**MongoDB:**\n```python\n# Regex queries on _id field\n{\"_id\": {\"$regex\": \"^mix:query:\"}}\n```\n\n**OpenSearchKVStorage:**\n```python\n# Scan raw hits, then match cache key prefixes in Python\nif hit[\"_id\"].startswith(\"mix:query:\"):\n```\n\n## Error Handling & Resilience\n\nThe tool implements comprehensive error tracking:\n\n### Batch-Level Error Tracking\n- Each batch is independently error-checked\n- Failed batches are logged with full details\n- Successful batches commit even if later batches fail\n- Real-time progress shows ✓ (success) or ✗ (failed)\n\n### Error Reporting\nAfter cleanup completes, a detailed report includes:\n- **Statistics**: Total records, success/failure counts, success rate\n- **Before/After Comparison**: Net reduction in cache count\n- **Error Summary**: Grouped by error type with occurrence counts\n- **Error Details**: Batch number, error type, message, and records lost\n- **Recommendations**: Clear indication of success or need for review\n\n### Verification\n- Post-cleanup count verification\n- Before/after statistics comparison\n- Identifies partial cleanup scenarios\n\n## Important Notes\n\n1. **Irreversible Operation**\n   - Deleted caches cannot be recovered\n   - Always backup important data before cleanup\n   - Test on non-production data first\n\n2. **Performance Impact**\n   - Query performance may degrade temporarily after cleanup\n   - Caches will rebuild on subsequent queries\n   - Consider cleanup during off-peak hours\n\n3. **Selective Cleanup**\n   - Choose cleanup scope carefully\n   - Keywords caches may be valuable for future queries\n   - Query caches rebuild faster than keywords caches\n\n4. **Workspace Isolation**\n   - Cleanup only affects the selected workspace\n   - Other workspaces remain untouched\n   - Verify workspace before confirming cleanup\n\n5. **Interrupt and Resume**\n   - Cleanup can be interrupted at any time (Ctrl+C)\n   - Already deleted records cannot be recovered\n   - No automatic resume - must run tool again\n\n## Storage Configuration\n\nThe tool supports multiple configuration methods with the following priority:\n\n1. **Environment variables** (highest priority)\n2. **Default values** (lowest priority)\n\n### Environment Variable Configuration\n\nConfigure storage settings in your `.env` file:\n\n#### Workspace Configuration (Optional)\n\n```bash\n# Generic workspace (shared by all storages)\nWORKSPACE=space1\n\n# Or configure independent workspace for specific storage\nPOSTGRES_WORKSPACE=pg_space\nMONGODB_WORKSPACE=mongo_space\nREDIS_WORKSPACE=redis_space\n```\n\n**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string\n\n#### JsonKVStorage\n\n```bash\nWORKING_DIR=./rag_storage\n```\n\n#### RedisKVStorage\n\n```bash\nREDIS_URI=redis://localhost:6379\n```\n\n#### PGKVStorage\n\n```bash\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_DATABASE=your_database\n```\n\n#### MongoKVStorage\n\n```bash\nMONGO_URI=mongodb://root:root@localhost:27017/\nMONGO_DATABASE=LightRAG\n```\n\n#### OpenSearchKVStorage\n\n```bash\nOPENSEARCH_HOSTS=localhost:9200\nOPENSEARCH_WORKSPACE=search_space\n```\n\nIf environment variables are not provided, the tool falls back to built-in defaults where available.\n\n## Troubleshooting\n\n### Missing Environment Variables\n```\n⚠️  Warning: Missing environment variables: POSTGRES_USER, POSTGRES_PASSWORD\n```\n**Solution**: Add missing variables to your `.env` file\n\n### Connection Failed\n```\n✗ Initialization failed: Connection refused\n```\n**Solutions**:\n- Check if database service is running\n- Verify connection parameters (host, port, credentials)\n- Check firewall settings\n- Ensure network connectivity for remote databases\n\n### No Caches Found\n```\n⚠️  No query caches found in storage\n```\n**Possible Reasons**:\n- No queries have been run yet\n- Caches were already cleaned\n- Wrong workspace selected\n- Different storage type was used for queries\n\n### Partial Cleanup\n```\n⚠️  WARNING: Cleanup completed with errors!\n```\n**Solutions**:\n- Check error details in the report\n- Verify storage connection stability\n- Re-run tool to clean remaining caches\n- Check storage capacity and permissions\n\n## Use Cases\n\n### Use Case 1: Clean All Query Caches\n\n**Scenario**: Free up storage space by removing all query caches\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 1 (all) -> Confirm (y)\n```\n\n**Result**: All query and keywords caches deleted, maximum storage freed\n\n### Use Case 2: Refresh Query Caches Only\n\n**Scenario**: Force query cache rebuild while keeping keywords\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 2 (query only) -> Confirm (y)\n```\n\n**Result**: Query caches deleted, keywords preserved for faster rebuild\n\n### Use Case 3: Clean Stale Keywords\n\n**Scenario**: Remove outdated keywords while keeping recent query results\n\n```bash\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Option 3 (keywords only) -> Confirm (y)\n```\n\n**Result**: Keywords deleted, query caches preserved\n\n### Use Case 4: Workspace-Specific Cleanup\n\n**Scenario**: Clean caches for a specific workspace\n\n```bash\n# Configure workspace\nexport WORKSPACE=development\n\n# Run tool\npython -m lightrag.tools.clean_llm_query_cache\n\n# Select: Storage type -> Cleanup option -> Confirm (y)\n```\n\n**Result**: Only development workspace caches cleaned\n\n## Best Practices\n\n1. **Backup Before Cleanup**\n   - Always backup your storage before major cleanup\n   - Test cleanup on non-production data first\n   - Document cleanup decisions\n\n2. **Monitor Performance**\n   - Watch storage metrics during cleanup\n   - Monitor query performance after cleanup\n   - Allow time for cache rebuild\n\n3. **Scheduled Cleanup**\n   - Clean caches periodically (weekly/monthly)\n   - Automate cleanup for development environments\n   - Keep production cleanup manual for safety\n\n4. **Selective Deletion**\n   - Consider cleanup scope based on needs\n   - Keywords caches are harder to rebuild\n   - Query caches rebuild automatically\n\n5. **Storage Capacity**\n   - Monitor storage usage trends\n   - Clean caches before reaching capacity limits\n   - Archive old data if needed\n\n## Comparison with Migration Tool\n\n| Feature | Cleanup Tool | Migration Tool |\n|---------|-------------|----------------|\n| **Purpose** | Delete query caches | Migrate extraction caches |\n| **Cache Types** | mix/hybrid/local/global | default:extract/summary |\n| **Modes** | query, keywords | extract, summary |\n| **Operation** | Deletion | Copy between storages |\n| **Reversible** | No | Yes (source unchanged) |\n| **Use Case** | Free storage, refresh caches | Change storage backend |\n\n## Limitations\n\n1. **Single Storage Operation**\n   - Can only clean one storage type at a time\n   - To clean multiple storages, run tool multiple times\n\n2. **No Dry Run Mode**\n   - Deletion is immediate after confirmation\n   - No preview-only mode available\n   - Test on non-production first\n\n3. **No Selective Mode Cleanup**\n   - Cannot clean only specific modes (e.g., only `mix`)\n   - Cleanup applies to all modes for selected cache type\n   - All-or-nothing per cache type\n\n4. **No Scheduled Cleanup**\n   - Manual execution required\n   - No built-in scheduling\n   - Use cron/scheduler if automation needed\n\n5. **Verification Limitations**\n   - Post-cleanup verification may fail in error scenarios\n   - Manual verification recommended for critical operations\n\n## Future Enhancements\n\nPotential improvements for future versions:\n\n- Selective mode cleanup (e.g., clean only `mix` mode)\n- Age-based cleanup (delete caches older than X days)\n- Size-based cleanup (delete largest caches first)\n- Dry run mode for safe preview\n- Automated scheduling support\n- Cache statistics export\n- Incremental cleanup with pause/resume\n\n## Support\n\nFor issues, questions, or feature requests:\n- Check the error details in the cleanup report\n- Review storage configuration\n- Verify workspace settings\n- Test with a small dataset first\n- Report bugs through project issue tracker\n\n<!-- File: lightrag/tools/README_KG_INTEGRITY_REPAIR.md (Tokens: ~1608 | Category: anthropic-skill) -->\n# KG Integrity Audit / Recovery-Anchor Repair\n\nOffline companion to the issue #3400 recoverable-mutation work.\n\nSince #3400, ingestion prewrites per-document **recovery anchors**\n(`full_entities` / `full_relations`) before mutating the graph, so purge,\nretry, and scan rollback can always discover a document's contributions.\nInstallations with data ingested **before** that change (or written through\ndirect KG paths like `ainsert_custom_kg`, which sit outside the\ndocument-level guarantee) may hold graph contributions with no anchor — such\ndata is invisible to per-document cleanup and becomes orphaned when its\ndocument is deleted or reprocessed.\n\nThis tool enumerates the whole graph (deliberately offline-only; the hot\npaths never do a graph-wide scan), maps every node/edge back to its owning\ndocuments via chunk `source_id` → `text_chunks` → `full_doc_id`, and:\n\n- reports per-document anchor gaps;\n- reports irrecoverable orphans (contributions whose source chunks no longer\n  exist) — these are never modified automatically;\n- with `--apply`, unions the missing entries into the anchor rows.\n\n## Usage\n\nLibrary (works with every configured backend combination):\n\n```python\nfrom lightrag.tools.kg_integrity_repair import audit_kg_integrity\n\nrag = LightRAG(...)\nawait rag.initialize_storages()\nreport = await audit_kg_integrity(rag)              # report only\nreport = await audit_kg_integrity(rag, apply=True)  # repair anchors\n```\n\nCLI (env-driven construction — `WORKING_DIR`, `WORKSPACE`, `EMBEDDING_DIM`,\n`LIGHTRAG_*` storage selectors; never calls the LLM or embedder):\n\n```bash\npython -m lightrag.tools.kg_integrity_repair             # report\npython -m lightrag.tools.kg_integrity_repair --apply     # repair\npython -m lightrag.tools.kg_integrity_repair --verbose   # per-doc details\n```\n\nRun it while the server is stopped (or the workspace is otherwise idle):\nthe audit reads a moving target if ingestion runs concurrently.\n\n## This tool is now the required remedy, not just a diagnostic\n\nPurge and deletion **fail closed** when a document has no recovery proof.\nPreviously, absent anchor rows were read as \"this document contributed\nnothing\", so graph cleanup was silently skipped while the chunks were deleted\nanyway — which destroyed the provenance chain (`source_id` → `text_chunks` →\n`full_doc_id`) that this tool needs, turning repairable data into permanent\norphans. So instead, the operation now refuses before deleting anything:\n\n```\nRecoveryAnchorMissingError: Refusing to purge document doc-…: recovery anchor\nrow(s) missing or unusable (full_entities, full_relations) …\n```\n\nOver the API this surfaces as **HTTP 409** with no data removed. Retrying\nunchanged will refuse again — run `--apply` first, then retry.\n\nThe rule is narrower than \"every purge needs a proof\". What must never happen\nis deleting something that *carries* attribution — a chunk row, or an anchor\nrow that names objects — while leaving those objects in the graph. A purge that\nremoves no such carrier cannot strand anything, so it needs no proof. Otherwise\none of these must hold:\n\n| Proof | Meaning |\n| --- | --- |\n| Both anchor rows present | The normal case. Presence is the test, **not** whether the lists are non-empty: a row holding an empty list is a document that extracted no entities, and is a perfectly good proof. |\n| `doc_status.metadata.kg_write_state == pre_graph` | Stamped at enqueue and advanced only once the anchors are durable, so it proves the document never reached its first graph mutation. Staged chunks are cleaned up with no graph access at all. |\n| `doc_status.metadata.kg_purge` past `prepared` | A previous purge attempt got far enough to have deleted the anchors itself. Without this, purge's own last step would make every retry refuse forever. |\n| No chunks and no populated anchor row | Nothing that carries attribution would be deleted. This is what lets a document that was queued before the `kg_write_state` marker existed, and never processed, be deleted directly — no scan, no audit. |\n\nThree states therefore need this tool:\n\n- documents ingested **before** #3416 landed the write-ahead anchors;\n- documents written through direct KG paths such as `ainsert_custom_kg`,\n  which are documented as sitting outside the document-level guarantee;\n- documents ingested with `skip_kg` (`process_options` `'!'`) **before** the\n  `kg_write_state` marker existed — see below.\n\n### Documents that legitimately own nothing\n\n`skip_kg` skips extraction and the merge entirely, so no anchor rows are ever\nwritten. Documents ingested that way *after* this change carry\n`kg_write_state=pre_graph` from enqueue and delete normally. Older ones have\nneither anchors nor a marker, and anchor repair has nothing to rebuild from —\nthey never appear in the graph scan, because they own nothing in it.\n\nThe audit settles that case, and it is the only thing that can: it enumerates\nthe **whole** graph, which the hot paths deliberately never do, so a document\nabsent from that scan is not merely unproven but *proven empty*. Such\ndocuments are listed under `anchorless_docs`, and `--apply` gives them empty\nanchor rows — the present-and-empty pair that is the normal proof for a\ndocument with no contributions.\n\nAbsence is only ever concluded from the completed scan. A document that does\nown graph objects is repaired with its real names instead; blanking it would\nmanufacture a false proof and license the exact deletion this work prevents.\n\n### What the certification assumes of the storage backends\n\n\"Proven empty\" is only as strong as the enumeration it is concluded from, so\nboth reads are held to a complete-or-raise contract and the audit **fails\nloudly instead of certifying on a partial view**:\n\n- the `doc_status` enumeration uses `strict=True`, so a transport or\n  deserialization failure raises rather than silently narrowing the set of\n  documents being certified;\n- the graph scan relies on `get_all_nodes()` / `get_all_edges()` never\n  silently truncating. All shipped graph backends satisfy this (errors\n  propagate; there is no result cap). Corrupt AGE properties raise a\n  `PGGraphQueryException` rather than dropping the node or blanking the\n  edge's `source_id` — a dropped node is an attribution the scan never sees,\n  which is exactly how a contributing document could be falsely certified.\n\nOne sharp edge remains: a graph backend whose indices/collections are missing\n(e.g. deleted out-of-band) reads as an **empty graph**, not as an error. The\naudit cannot distinguish that from a workspace that truly owns no graph data —\none more reason to run it only while the server is stopped and the workspace\nis healthy, as stated at the top.\n\nNote that `--apply` reports what it found under `missing_entity_anchors` /\n`missing_relation_anchors` (the diagnosis) and what it wrote under\n`repaired_docs` (the action) — the first two are not emptied by a repair.\n\nAnything listed under `orphan_entities` / `orphan_relations` cannot be\nrepaired by this tool: its source chunks are already gone, so no owning\ndocument is determinable. Those need external provenance, a re-ingest, or\nmanual deletion.\n\n<!-- File: lightrag/tools/README_MIGRATE_GRAPH_STORAGE.md (Tokens: ~3365 | Category: anthropic-skill) -->\n# Graph Storage Migration (`PGGraphStorage` → `PGTableGraphStorage`)\n\nOffline tool for moving an already-extracted entity-relation graph from the\nApache AGE backend (`PGGraphStorage`) to the plain-table backend\n(`PGTableGraphStorage`) without re-indexing the source documents.\n\nChanging `LIGHTRAG_GRAPH_STORAGE` normally means the previously extracted\ngraph is simply not visible to the new backend, so the documents have to be\nre-processed — an LLM-cost-bearing operation for a graph that already exists.\nThis tool copies the graph itself, through the public storage API only\n(`get_all_nodes` / `get_all_edges` to read, `upsert_nodes_batch` /\n`upsert_edges_batch` to write, `remove_nodes` / `remove_edges` to undo). It\nissues no raw SQL, so canonicalization, endpoint handling and identity rules\nstay in the already-tested storage layer.\n\nOnly the graph moves. Vector and KV storages are untouched, and because the\nmigrated graph keeps the same entity and relation identities, existing vector\ndata stays valid — nothing is re-embedded and nothing is re-extracted. The LLM\ncache is a separate concern with its own tool (see\n[README_MIGRATE_LLM_CACHE.md](README_MIGRATE_LLM_CACHE.md)).\n\nRe-indexing remains the general guidance for changing storage backends. This\ntool is an advanced path for one specific pair.\n\n## Before you run it\n\n**Stop every LightRAG writer first — the tool cannot check this for you.**\nIt reads the source graph, writes it to the target, and then verifies the\nresult. A concurrent writer mutating either side inside that window\ninvalidates the comparison, and the tool can neither detect a live writer nor\nlock the backends. Every other precondition below is enforced; this one is\npurely operational, and it is the assumption the whole design rests on. The\nsame requirement applies to `rebuild_vdb`.\n\n**The target graph slice must be empty.** The migration is defined as\npopulating an empty slice; that invariant is also what makes its failure\nhandling exact (see below). A non-empty target is refused before anything is\nwritten.\n\nBoth backends read the usual `POSTGRES_USER`, `POSTGRES_PASSWORD`,\n`POSTGRES_DATABASE`, `POSTGRES_HOST` and `POSTGRES_PORT` variables.\n\n## Usage\n\n```bash\n# Dry run (default): reads and checks everything, migrates nothing.\npython -m lightrag.tools.migrate_graph_storage\n\n# Perform the migration.\npython -m lightrag.tools.migrate_graph_storage --apply\n```\n\n| Flag | Default | Meaning |\n| --- | --- | --- |\n| `--apply` | off | Perform the migration. Without it no graph data is written (see the schema caveat below). |\n| `--workspace` | `WORKSPACE` env | Workspace to migrate. |\n| `--force-empty-target` | off | **Dangerous.** `drop()` the whole non-empty target graph slice before migrating, instead of refusing. |\n| `--source-backend` | `PGGraphStorage` | Source graph storage class name. |\n| `--target-backend` | `PGTableGraphStorage` | Target graph storage class name. |\n\n`--source-backend` / `--target-backend` are an extension seam, not an open\ndoor: only allow-listed pairs run, and Phase 1 allow-lists exactly\n`PGGraphStorage` → `PGTableGraphStorage`. Any other pair is rejected before a\nbackend is even constructed.\n\nThe dry run performs the same enumeration and the same refusal checks as the\nreal run, so it predicts what `--apply` will do. With `--force-empty-target`\nit reports that the apply run *would* drop the pre-existing slice, but never\ndrops anything itself.\n\n**A dry run migrates nothing, but it is not read-only, and its blast radius is\nnot limited to one workspace.** Both backends have to be initialized before\ntheir contents can be inspected, and `initialize()` is not a passive step. Two\ndifferent scopes are involved:\n\n- *Scoped to the workspace being migrated:* `_normalize_legacy_edges`\n  **deletes and re-inserts edge rows** (reversed duplicates, legacy\n  orderings) for this workspace and namespace only; the AGE side creates the\n  graph and its indexes.\n- *Table-wide, across every workspace and namespace:* the schema migration in\n  `PGTableGraphStorage`'s DDL. On a table whose primary key predates the\n  `namespace` column it **DELETEs duplicate rows** from `lightrag_graph_nodes`\n  and `lightrag_graph_edges` (keeping the most recently updated row per key)\n  before rebuilding the primary keys, and when the endpoint foreign keys are\n  absent it then **DELETEs every orphan edge** — an edge whose endpoint node\n  is missing — from the whole table. That sweep cannot be narrowed: as\n  `pgtable_impl` puts it, \"the sweep is table-wide because the FK is a global\n  composite constraint and cannot be validated per workspace.\" Both paths are\n  one-time (first install, or the first run after the namespace-PK upgrade)\n  and are skipped once the constraints exist, but the run that performs them\n  touches rows belonging to workspaces `--workspace` never named.\n\nSo a dry run against one workspace can still modify others, on a legacy\nschema. What it never does is migrate data or copy anything between backends —\nthat is the guarantee, and it is narrower than \"writes nothing\". Take a backup\nof both tables before the first run against a pre-`namespace` schema.\n\n**`--workspace` is cross-checked, not trusted.** The backends resolve their\nworkspace as `POSTGRES_WORKSPACE` env > `config.ini` `[postgres] workspace` >\nthe value passed in > `\"default\"`, so either the environment or a checked-in\nconfig file can outrank the flag. The tool compares what was requested\nagainst what the backends actually resolved to, refuses when they differ, and\nreports the resolved workspace — the slice a destructive run acted on is never\nleft implicit.\n\nExit code is 0 on success and non-zero on any failure.\n\n## The report\n\nBoth modes print a dict to stdout, following the convention of the other\ntools:\n\n```python\n{'mode': 'dry-run', 'source_backend': 'PGGraphStorage',\n 'target_backend': 'PGTableGraphStorage', 'workspace': 'default',\n 'nodes': 8123, 'edges': 41022,\n 'verified': False, 'compensated': False, 'target_non_empty': False,\n 'would_drop_target_slice': False,\n 'written_node_count': 8123, 'written_edge_count': 41022}\n```\n\n`mode` is `dry-run` or `apply`. `verified` is true only when the post-write\ncomparison passed. The destructive-consequence field is mode-symmetric:\n`would_drop_target_slice` in a dry run, `dropped_target_slice` in an apply run\n— so the preview cannot look clean while hiding that `--force-empty-target`\nwill destroy the existing slice.\n\n`nodes` / `edges` in a dry run also double as a sizing signal for the apply\nrun: both modes hold the whole graph in memory (see **It does not stream**\nbelow), so a dry run against the real source is the cheapest way to gauge how\nmuch memory the real run will need.\n\n## What makes it refuse\n\nEvery check below runs *before* the first write, and each one aborts the run\nrather than migrating a graph it cannot faithfully reproduce:\n\n- the storage pair is not allow-listed;\n- the target graph slice is not empty (unless `--force-empty-target`);\n- a node has no usable identity — `id` or `entity_id` missing, not a string,\n  or the two disagreeing. Both backends can legitimately produce such a row: a\n  vertex created with no properties is legal in AGE and enumerates as\n  `{\"id\": None}`, and NetworkX creates an edge's unknown endpoint as an\n  attribute-less node;\n- the same node id appears more than once, **whether or not the payloads\n  match**. The target keeps one row per id, so two physical source vertices\n  become one node and the graph's node count changes;\n- `a→b` and `b→a` both exist, **whether or not the payloads match**. The\n  target stores one row per canonical pair, and AGE computes degree by\n  counting relationship rows — so collapsing an identical reciprocal preserves\n  every property while changing degree and traversal. LightRAG's own write\n  path matches edges undirected, so it does not create reciprocals: finding\n  one means the source violates the invariant this tool assumes;\n- a payload contains a value PostgreSQL `jsonb` cannot store faithfully:\n  `NaN`, `Infinity` and `-Infinity` (which `agtype` represents and `jsonb`\n  rejects outright), or `-0.0` (which `jsonb` accepts but normalises to `0.0`,\n  and this tool's comparison treats signed zeros as different — so the write\n  would succeed and then fail its own verification). The general rule: if the\n  target cannot round-trip a value, refuse before the destructive step rather\n  than discover it after the point of no return;\n- more than one directed row *reaches the tool* for the same ordered pair — a\n  violation of the invariant that makes the canonical merge lossless. Note the\n  limit of this backstop: AGE enumerates edges with `SELECT DISTINCT source,\n  target, properties`, so byte-identical parallel relationships are already\n  collapsed to one row before the tool sees them. It can therefore only catch\n  parallels whose payloads *differ*. **This is the one cardinality gap the tool\n  cannot close**: byte-identical same-direction parallels are invisible to it,\n  and collapsing them changes degree without losing any property. Everything\n  else that would change cardinality — reciprocals, duplicate ids — is refused\n  outright.\n\nVerification after the write re-runs the source-side checks and compares node\nids, canonical edges and properties. It is driven from the source because the\ntarget has already canonicalized and therefore cannot reveal what was lost.\nProperty comparison is type-strict: `1`, `1.0`, `True` and `\"1\"` are four\ndifferent values, and `-0.0` differs from `0.0` — all are distinguishable\ndownstream.\n\n**What `verified` claims — and what it cannot.** The tool only ever sees what\nthe source's `get_all_nodes` / `get_all_edges` return, so its claim is scoped\nto that enumerated view: **every node and edge the enumerator returned came\nacross with its identity, its payload, and its count intact.** Within that\nview nothing silently collapses — reciprocals and duplicate ids are refused\nbefore the write rather than merged and blessed.\n\n`verified` does **not** prove that the physical AGE graph was reproduced. Two\nthings are already lost or hidden before this tool is handed a row:\n\n- **Physical multiplicity.** AGE enumerates with `SELECT DISTINCT source,\n  target, properties`, so two byte-identical relationships between the same\n  ordered pair arrive as one row. AGE's own degree counts relationship rows, so\n  such a source has degree 2 where the target will have 1 — and `verified` will\n  still be true, because both sides match the one row the enumerator showed.\n  Reading raw multiplicity would mean going around the storage API, which this\n  tool deliberately does not do.\n- **Payload keys named `id`, `source` or `target`.** `PGGraphStorage`'s\n  enumerator overwrites them with identity values when flattening each row, so\n  a business property under one of those names is gone upstream and no\n  comparison here can notice.\n\nIf exact physical fidelity matters for your graph, check for duplicate\nrelationships directly in the AGE tables before migrating.\n\n## If a write fails\n\nThe tool removes exactly what the run wrote — the migrated node ids plus every\nedge endpoint, since the target auto-creates missing endpoints without\nreporting them — using `remove_edges` first and then `remove_nodes` to respect\nthe foreign key. It never uses `drop()` for this; whole-slice deletion is\nreachable only through `--force-empty-target`. The written set is computed\nbefore the first write, so a failure at any point still has the complete set,\nand removing it is exact precisely because the slice started empty.\n\n`--force-empty-target` has an irreducible window. Every source check runs\nbefore the drop, so a malformed source cannot cost you the slice — but nothing\nproves the *write* will succeed. If the migration fails after the drop, the\ncompensation removes what this run wrote and the target ends up empty: the\npre-existing data is gone and the new data never landed. That is the bargain\nthe flag asks for; take a backup before using it.\n\nA failed `drop()` under `--force-empty-target` aborts the run rather than\nproceeding: the backend reports that failure by returning an error status\ninstead of raising, so an unchecked call would let the migration write into,\nand later compensate against, rows it does not own.\n\nCompensation is not atomic. There is no transaction spanning two backends, so\na crash during compensation can leave the target partially populated. Two\nthings make that recoverable: re-running the tool refuses a non-empty target\nrather than compounding the mess, and if compensation itself fails the report\ncarries the full `written_node_ids` and `written_edge_keys` so the remaining\nrows can be removed by hand.\n\n## What this tool does not do\n\n- **It does not migrate any other pair.** `PRECONDITION_HOOKS` and\n  `COMPENSATION_HOOKS` are named placeholders — declared, but not yet consulted\n  anywhere; the Phase 1 behaviour is hardcoded. Adding a pair means wiring them\n  as well as allow-listing it.\n- **It does not merge into a populated target.** Empty-target is a\n  precondition, not a convenience.\n- **It does not run in a single transaction.** Two backends, no shared\n  transaction; the failure handling above is the substitute.\n- **It does not stream.** The storage API enumerates whole graphs, so a run\n  holds the source graph, the derived write plan and — during verification —\n  the target graph in memory at the same time. Peak memory grows with the\n  graph, roughly 2x its decoded size, and no setting bounds it: the fail-closed\n  checks (duplicate ids, reciprocals, parallel pairs) are global properties\n  that need the whole-graph view. For a very large graph, run the tool on a\n  host sized accordingly, or migrate workspace by workspace. Running out of\n  memory *inside* the process (`MemoryError`) is handled like any other\n  failure — compensation removes what the run wrote — but a kernel OOM kill\n  terminates the process outright: nothing compensates, the report never\n  prints, and under `--force-empty-target` it lands after the drop. Size the\n  host before reaching for that flag.\n- **It does not protect you from a live writer.** See the preconditions.\n- **It does not disambiguate workspaces that share an AGE graph.** AGE derives\n  its graph name by replacing every non-alphanumeric character with `_`, so\n  `team-a` and `team_a` name the same AGE graph while PGTable keeps them as\n  distinct workspaces. Check which graph a workspace actually resolves to\n  before migrating one whose name contains punctuation.\n- **It does not move vectors, KV data or document status.**\n\n<!-- File: lightrag/tools/README_MIGRATE_LLM_CACHE.md (Tokens: ~3359 | Category: anthropic-skill) -->\n# LLM Cache Migration Tool - User Guide\n\n## Overview\n\nThis tool migrates LightRAG's LLM response cache between different KV storage implementations. It specifically migrates caches generated during file extraction (mode `default`), including entity extraction, summary, and multimodal analysis caches.\n\n## Supported Storage Types\n\n1. **JsonKVStorage** - File-based JSON storage\n2. **RedisKVStorage** - Redis database storage\n3. **PGKVStorage** - PostgreSQL database storage\n4. **MongoKVStorage** - MongoDB database storage\n5. **OpenSearchKVStorage** - OpenSearch index storage\n\n## Cache Types\n\nThe tool migrates the following cache types:\n- `default:extract:*` - Entity and relationship extraction caches\n- `default:summary:*` - Entity and relationship summary caches\n- `default:analysis:*` - Multimodal image, table, and equation analysis caches\n\n**Note**: Query caches (modes like `mix`,`local`, `global`, etc.) are NOT migrated.\n\n## Prerequisites\n\nThe LLM Cache Migration Tool reads the storage configuration of the LightRAG Server and provides an LLM migration option to select source and destination storage. Ensure that both the source and destination storage have been correctly configured and are accessible via the LightRAG Server before cache migration.\n\n## Usage\n\n### Basic Usage\n\nRun from the LightRAG project root directory:\n\n```bash\npython -m lightrag.tools.migrate_llm_cache\n# or\npython lightrag/tools/migrate_llm_cache.py\n```\n\n### Interactive Workflow\n\nThe tool guides you through the following steps:\n\n#### 1. Select Source Storage Type\n```\nSupported KV Storage Types:\n[1] JsonKVStorage\n[2] RedisKVStorage\n[3] PGKVStorage\n[4] MongoKVStorage\n[5] OpenSearchKVStorage\n\nSelect Source storage type (1-5) (Press Enter to exit): 1\n```\n\n**Note**: You can press Enter or type `0` at any storage selection prompt to exit gracefully.\n\n#### 2. Source Storage Validation\nThe tool will:\n- Check required environment variables\n- Auto-detect workspace configuration\n- Initialize and connect to storage\n- Count cache records available for migration\n\n```\nChecking environment variables...\n✓ All required environment variables are set\n\nInitializing Source storage...\n- Storage Type: JsonKVStorage\n- Workspace: space1\n- Connection Status: ✓ Success\n\nCounting cache records...\n- Total: 8,734 records\n```\n\n**Progress Display by Storage Type:**\n- **JsonKVStorage**: Fast in-memory counting, displays final count without incremental progress\n  ```\n  Counting cache records...\n  - Total: 8,734 records\n  ```\n- **RedisKVStorage**: Real-time scanning progress with incremental counts\n  ```\n  Scanning Redis keys... found 8,734 records\n  ```\n- **PostgreSQL**: Quick COUNT(*) query, shows timing only if operation takes >1 second\n  ```\n  Counting PostgreSQL records... (took 2.3s)\n  ```\n- **MongoDB**: Fast count_documents(), shows timing only if operation takes >1 second\n  ```\n  Counting MongoDB documents... (took 1.8s)\n  ```\n- **OpenSearchKVStorage**: PIT-based scan with timing shown when noticeable\n  ```\n  Scanning OpenSearch documents... (took 1.5s)\n  ```\n\n#### 3. Select Target Storage Type\n\nThe tool automatically excludes the source storage type from the target selection and renumbers the remaining options sequentially:\n\n```\nAvailable Storage Types for Target (source: JsonKVStorage excluded):\n[1] RedisKVStorage\n[2] PGKVStorage\n[3] MongoKVStorage\n[4] OpenSearchKVStorage\n\nSelect Target storage type (1-4) (Press Enter or 0 to exit): 1\n```\n\n**Important Notes:**\n- You **cannot** select the same storage type for both source and target\n- Options are automatically renumbered (e.g., [1], [2], [3] instead of [2], [3], [4])\n- You can press Enter or type `0` to exit at this point as well\n\nThe tool then validates the target storage following the same process as the source (checking environment variables, initializing connection, counting records).\n\n#### 4. Confirm Migration\n\n```\n==================================================\nMigration Confirmation\nSource: JsonKVStorage (workspace: space1) - 8,734 records\nTarget: MongoKVStorage (workspace: space1) - 0 records\nBatch Size: 1,000 records/batch\nMemory Mode: Streaming (memory-optimized)\n\n⚠️  Warning: Target storage already has 0 records\nMigration will overwrite records with the same keys\n\nContinue? (y/n): y\n```\n\n#### 5. Execute Migration\n\nThe tool uses **streaming migration** by default for memory efficiency. Observe migration progress:\n\n```\n=== Starting Streaming Migration ===\n💡 Memory-optimized mode: Processing 1,000 records at a time\n\nBatch 1/9: ████████░░░░░░░░░░░░ 1000/8734 (11.4%) - default:extract ✓\nBatch 2/9: ████████████░░░░░░░░ 2000/8734 (22.9%) - default:extract ✓\n...\nBatch 9/9: ████████████████████ 8734/8734 (100.0%) - default:summary ✓\n\nPersisting data to disk...\n✓ Data persisted successfully\n```\n\n**Key Features:**\n- **Streaming mode**: Processes data in batches without loading entire dataset into memory\n- **Real-time progress**: Shows progress bar with precise percentage and cache type\n- **Success indicators**: ✓ for successful batches, ✗ for failed batches\n- **Constant memory usage**: Handles millions of records efficiently\n\n#### 6. Review Migration Report\n\nThe tool provides a comprehensive final report showing statistics and any errors encountered:\n\n**Successful Migration:**\n```\nMigration Complete - Final Report\n\n📊 Statistics:\n  Total source records:    8,734\n  Total batches:           9\n  Successful batches:      9\n  Failed batches:          0\n  Successfully migrated:   8,734\n  Failed to migrate:       0\n  Success rate:            100.00%\n\n✓ SUCCESS: All records migrated successfully!\n```\n\n**Migration with Errors:**\n```\nMigration Complete - Final Report\n\n📊 Statistics:\n  Total source records:    8,734\n  Total batches:           9\n  Successful batches:      8\n  Failed batches:          1\n  Successfully migrated:   7,734\n  Failed to migrate:       1,000\n  Success rate:            88.55%\n\n⚠️  Errors encountered: 1\n\nError Details:\n------------------------------------------------------------\n\nError Summary:\n  - ConnectionError: 1 occurrence(s)\n\nFirst 5 errors:\n\n  1. Batch 2\n     Type: ConnectionError\n     Message: Connection timeout after 30s\n     Records lost: 1,000\n\n⚠️  WARNING: Migration completed with errors!\n   Please review the error details above.\n```\n\n## Technical Details\n\n### Workspace Handling\n\nThe tool retrieves workspace in the following priority order:\n\n1. **Storage-specific workspace environment variables**\n   - PGKVStorage: `POSTGRES_WORKSPACE`\n   - MongoKVStorage: `MONGODB_WORKSPACE`\n   - RedisKVStorage: `REDIS_WORKSPACE`\n   - OpenSearchKVStorage: `OPENSEARCH_WORKSPACE`\n\n2. **Generic workspace environment variable**\n   - `WORKSPACE`\n\n3. **Default value**\n   - Empty string (uses storage's default workspace)\n\n### Batch Migration\n\n- Default batch size: 1000 records/batch\n- Avoids memory overflow from loading too much data at once\n- Each batch is committed independently, supporting resume capability\n\n### Memory-Efficient Pagination\n\nFor large datasets, the tool implements storage-specific pagination strategies:\n\n- **JsonKVStorage**: Direct in-memory access (data already loaded in shared storage)\n- **RedisKVStorage**: Cursor-based SCAN with pipeline batching (1000 keys/batch)\n- **PGKVStorage**: SQL LIMIT/OFFSET pagination (1000 records/batch)\n- **MongoKVStorage**: Cursor streaming with batch_size (1000 documents/batch)\n- **OpenSearchKVStorage**: PIT + `search_after` scan of the KV index (1000 documents/batch)\n\nThis ensures the tool can handle millions of cache records without memory issues.\n\n### Prefix Filtering Implementation\n\nThe tool uses optimized filtering methods for different storage types:\n\n- **JsonKVStorage**: Direct dictionary iteration with lock protection\n- **RedisKVStorage**: SCAN command with namespace-prefixed patterns + pipeline for bulk GET\n- **PGKVStorage**: SQL LIKE queries with proper field mapping (id, return_value, etc.)\n- **MongoKVStorage**: MongoDB regex queries on `_id` field with cursor streaming\n- **OpenSearchKVStorage**: Full-index scan with `_id` prefix filtering and `_source` passthrough\n\n## Error Handling & Resilience\n\nThe tool implements comprehensive error tracking to ensure transparent and resilient migrations:\n\n### Batch-Level Error Tracking\n- Each batch is independently error-checked\n- Failed batches are logged but don't stop the migration\n- Successful batches are committed even if later batches fail\n- Real-time progress shows ✓ (success) or ✗ (failed) for each batch\n\n### Error Reporting\nAfter migration completes, a detailed report includes:\n- **Statistics**: Total records, success/failure counts, success rate\n- **Error Summary**: Grouped by error type with occurrence counts\n- **Error Details**: Batch number, error type, message, and records lost\n- **Recommendations**: Clear indication of success or need for review\n\n### No Double Data Loading\n- Unlike traditional verification approaches, the tool does NOT reload all target data\n- Errors are detected during migration, not after\n- This eliminates memory overhead and handles pre-existing target data correctly\n\n## Important Notes\n\n1. **Data Overwrite Warning**\n   - Migration will overwrite records with the same keys in the target storage\n   - Tool displays a warning if target storage already has data\n   - Data migration can be performed repeatedly\n   - Pre-existing data in target storage is handled correctly\n3. **Interrupt and Resume**\n   - Migration can be interrupted at any time (Ctrl+C)\n   - Already migrated data will remain in target storage\n   - Re-running will overwrite existing records\n   - Failed batches can be manually retried\n4. **Performance Considerations**\n   - Large data migration may take considerable time\n   - Recommend migrating during off-peak hours\n   - Ensure stable network connection (for remote databases)\n   - Memory usage stays constant regardless of dataset size\n\n## Storage Configuration\n\nThe tool supports multiple configuration methods with the following priority:\n\n1. **Environment variables** (highest priority)\n2. **Default values** (lowest priority)\n\n#### Option A: Environment Variable Configuration\n\nConfigure storage settings in your `.env` file:\n\n#### Workspace Configuration (Optional)\n\n```bash\n# Generic workspace (shared by all storages)\nWORKSPACE=space1\n\n# Or configure independent workspace for specific storage\nPOSTGRES_WORKSPACE=pg_space\nMONGODB_WORKSPACE=mongo_space\nREDIS_WORKSPACE=redis_space\nOPENSEARCH_WORKSPACE=os_space\n```\n\n**Workspace Priority**: Storage-specific > Generic WORKSPACE > Empty string\n\n#### JsonKVStorage\n\n```bash\nWORKING_DIR=./rag_storage\n```\n\n#### RedisKVStorage\n\n```bash\nREDIS_URI=redis://localhost:6379\n```\n\n#### PGKVStorage\n\n```bash\nPOSTGRES_HOST=localhost\nPOSTGRES_PORT=5432\nPOSTGRES_USER=your_username\nPOSTGRES_PASSWORD=your_password\nPOSTGRES_DATABASE=your_database\n```\n\n#### MongoKVStorage\n\n```bash\nMONGO_URI=mongodb://root:root@localhost:27017/\nMONGO_DATABASE=LightRAG\n```\n\n#### OpenSearchKVStorage\n\n```bash\nOPENSEARCH_HOSTS=localhost:9200\nOPENSEARCH_WORKSPACE=os_space\n```\n\nIf environment variables are not provided, the tool falls back to built-in defaults where available. JsonKVStorage uses `WORKING_DIR` or defaults to `./rag_storage`.\n\n## Troubleshooting\n\n### Missing Environment Variables\n```\n✗ Missing required environment variables: POSTGRES_USER, POSTGRES_PASSWORD\n```\n**Solution**: Add missing variables to your `.env` file\n\n### Connection Failed\n```\n✗ Initialization failed: Connection refused\n```\n**Solutions**:\n- Check if database service is running\n- Verify connection parameters (host, port, credentials)\n- Check firewall settings\n\n**Solutions**:\n- Check migration process for error logs\n- Re-run migration tool\n- Check target storage capacity and permissions\n\n## Example Scenarios\n\n### Scenario 1: JSON to MongoDB Migration\n\nUse case: Migrating from single-machine development to production\n\n```bash\n# 1. Configure environment variables\nWORKSPACE=production\nMONGO_URI=mongodb://user:pass@prod-server:27017/\nMONGO_DATABASE=LightRAG\n\n# 2. Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# 3. Select: 1 (JsonKVStorage) -> 1 (MongoKVStorage - renumbered from 4)\n```\n\n**Note**: After selecting JsonKVStorage as source, MongoKVStorage will be shown as option [1] in the target selection since options are renumbered after excluding the source.\n\n### Scenario 2: Redis to PostgreSQL\n\nUse case: Migrating from cache storage to relational database\n\n```bash\n# 1. Ensure both databases are accessible\nREDIS_URI=redis://old-redis:6379\nPOSTGRES_HOST=new-postgres-server\n# ... Other PostgreSQL configs\n\n# 2. Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# 3. Select: 2 (RedisKVStorage) -> 2 (PGKVStorage - renumbered from 3)\n```\n\n**Note**: After selecting RedisKVStorage as source, PGKVStorage will be shown as option [2] in the target selection.\n\n### Scenario 3: Different Workspaces Migration\n\nUse case: Migrating data between different workspace environments\n\n```bash\n# Configure separate workspaces for source and target\nPOSTGRES_WORKSPACE=dev_workspace  # For development environment\nMONGODB_WORKSPACE=prod_workspace  # For production environment\n\n# Run tool\npython -m lightrag.tools.migrate_llm_cache\n\n# Select: 3 (PGKVStorage with dev_workspace) -> 3 (MongoKVStorage with prod_workspace)\n```\n\n**Note**: This allows you to migrate between different logical data partitions while changing storage backends.\n\n## Tool Limitations\n\n1. **Same Storage Type Not Allowed**\n   - You cannot migrate between the same storage type (e.g., PostgreSQL to PostgreSQL)\n   - This is enforced by the tool automatically excluding the source storage type from target selection\n   - For same-storage migrations (e.g., database switches), use database-native tools instead\n2. **Only Default Mode Caches**\n   - Only migrates `default:extract:*`, `default:summary:*`, and `default:analysis:*`\n   - Query caches are not included\n4. **Network Dependency**\n   - Tool requires stable network connection for remote databases\n   - Large datasets may fail if connection is interrupted\n\n## Best Practices\n\n1. **Backup Before Migration**\n   - Always backup your data before migration\n   - Test migration on non-production data first\n\n2. **Verify Results**\n   - Check the verification output after migration\n   - Manually verify a few cache entries if needed\n\n3. **Monitor Performance**\n   - Watch database resource usage during migration\n   - Consider migrating in smaller batches if needed\n\n4. **Clean Old Data**\n   - After successful migration, consider cleaning old cache data\n   - Keep backups for a reasonable period before deletion\n\n<!-- File: lightrag/tools/README_REBUILD_VDB.md (Tokens: ~1853 | Category: anthropic-skill) -->\n# Offline Vector Storage (VDB) Rebuild Tool\n\n`lightrag-rebuild-vdb` restores consistency between LightRAG's authoritative\ndata sources and its vector storages by dropping each vector storage and\nrebuilding it from scratch:\n\n| Vector storage | Authoritative source |\n|---|---|\n| `entities_vdb` | graph nodes |\n| `relationships_vdb` | graph edges |\n| `chunks_vdb` | `text_chunks` KV store |\n\n## When do I need this?\n\nLightRAG performs multi-step writes (graph + vector storage). If a vector\nstorage write fails at runtime — embedder outage, network timeout, context\noverflow on a high-degree entity — the graph and the vector storage drift\napart. The most common symptom is the edge-count drift reported in issue\n[#2917](https://github.com/HKUDS/LightRAG/issues/2917): graph edges with no\nvector counterpart, so `local`/`hybrid` queries miss relations that exist in\nthe graph.\n\nSince v(next), `amerge_entities` raises `VectorStorageConsistencyError` when\nthis happens, with a pointer to this tool. No data is lost in this situation:\nthe graph and the `text_chunks` KV store hold everything needed to rebuild\nthe vectors.\n\nA full drop + rebuild also clears *reverse* orphans (records present in the\nvector storage but absent from the graph), which incremental repair cannot\nreliably do.\n\nYou can also use this tool after changing the embedding model or embedding\ndimension. Existing vector records were generated in the old embedding space\n(and some vector backends bind collections/tables to the configured dimension),\nso run the tool with the updated `.env` and choose **Rebuild ALL vector\nstorages** to regenerate `entities_vdb`, `relationships_vdb`, and `chunks_vdb`\nfrom the authoritative graph/KV sources. The consistency check is not enough\nfor this case because it only detects missing graph → VDB records, not vectors\nthat exist but were embedded with a previous model or dimension.\n\nA vector storage whose container was written in a different embedding space\n**refuses to attach** rather than serving an empty or foreign index. That\nrefusal is the condition this tool clears, so it is the one startup failure the\ntool tolerates: the three vector targets are initialized individually, a typed\n`VectorSpaceMismatchError` is recorded instead of aborting the run, and the\nrebuild opens by dropping that container and re-provisioning it in the current\nembedding space. No out-of-band step (deleting the index through the backend's\nown API, `DROP TABLE`, `rm`) is needed.\n\nEverything else still aborts. A cluster outage, a bad credential or a corrupt\nfile is not a model change, and the recovery for a model change is destructive —\nso it is only ever applied to a refusal that says so in its type. The\nauthoritative sources (graph storage and `text_chunks`) likewise keep the\nserver-identical startup path, migrations included: rebuilding vectors from a\nhalf-migrated source is worse than not rebuilding.\n\nSee `docs/design/VectorSpaceProvenance.md` for the provenance marker and the\nrules that decide when a backend refuses.\n\nThe tool also completes a graph-only backfill performed with\n`NoopVectorDBStorage`. Stop every writer, preserve the same working directory,\nworkspace, graph storage, and KV storage, switch to the intended persistent\nvector backend, configure the production embedding model and dimension, and\nselect **Rebuild ALL vector storages**. Create a new `LightRAG` instance or\nrestart the process after changing the vector backend. Do not start the server\nuntil the rebuild succeeds; there is currently no persisted vector-index\nreadiness marker.\n\n## Usage\n\n```bash\n# Stop the LightRAG Server first!\nlightrag-rebuild-vdb\n# or\npython -m lightrag.tools.rebuild_vdb\n```\n\nThe tool reads the same `.env` / environment configuration as the server\n(`LIGHTRAG_GRAPH_STORAGE`, `LIGHTRAG_VECTOR_STORAGE`, `LIGHTRAG_KV_STORAGE`,\n`WORKSPACE`, `WORKING_DIR`, `EMBEDDING_*`, backend connection settings) and\nbuilds its embedding function through the exact factory the server uses —\nrun it with the same `.env` so rebuilt vectors live in the same embedding\nspace the server queries against.\n\nMenu options:\n\n1. **Consistency check (diagnose only)** — probes every graph entity/relation\n   for a vector counterpart and reports what is missing. Run this first to\n   decide whether a rebuild is worth the embedding cost. The check covers\n   the graph → VDB direction only; reverse orphans can only be cleared by a\n   full rebuild. Legacy reverse-order relation ids (from old custom-KG\n   imports) are recognized and not misreported as missing. The check issues\n   read queries only and does not run a rebuild (no drop + re-embed). It is\n   **not** strictly side-effect-free, though: the tool initializes every\n   storage on startup — exactly as the server does — and for some backends\n   that includes schema/DDL setup and one-time legacy migrations (e.g. Qdrant\n   upserts into the new collection, PostgreSQL batch-inserts into the new\n   table, Milvus may create a temp collection and drop/rename the original).\n   Treat running the tool — even just for a check — like starting the server:\n   stop other writers first.\n2. **Rebuild entities + relationships VDB** — sufficient for the #2917\n   merge-failure scenario.\n3. **Rebuild chunks VDB**.\n4. **Rebuild ALL vector storages** — use this after changing the embedding\n   model or embedding dimension.\n\n## Important notes\n\n- **Stop the server first.** The tool drops and rewrites vector storages;\n  concurrent writers (any backend, not just file-based ones) can corrupt\n  data or lose updates.\n- **Embedding model/dimension changes.** Run the tool with the new embedding\n  configuration and rebuild all vector storages. A consistency check can still\n  pass when every vector record exists but was created with the old embedding\n  model or dimension — unless the backend recorded its embedding-space\n  provenance, in which case the check reports the refusal and says the rebuild\n  is mandatory rather than reporting every record as missing.\n- **A refused target is dropped only after you confirm the rebuild.** Choosing\n  the consistency check never destroys anything, so a refusal can be diagnosed\n  before it is acted on. Options 2–4 recover exactly the targets they rebuild:\n  picking option 2 leaves a refused `chunks_vdb` refused until option 3 or 4\n  runs.\n- **Embedding cost.** A rebuild re-embeds every affected record. On large\n  datasets this means real API cost and time. Use the check mode first, and\n  rebuild only the storages that need it.\n- **Idempotent / crash-safe.** Sources (graph, `text_chunks`) are never\n  modified. If the tool crashes between drop and rewrite, just re-run it.\n- **`__created_at__` reset.** Backends that store creation timestamps in\n  vector records (nano, faiss) will show fresh timestamps after a rebuild.\n  No query logic depends on them.\n- **Custom-KG placeholder entities.** `UNKNOWN` placeholder nodes created by\n  `ainsert_custom_kg` are rebuilt faithfully from the graph; they may gain a\n  vector record they previously lacked (improving their retrievability).\n- **Chunk enumeration is backend-specific.** `BaseKVStorage` has no key\n  enumeration API, so the tool scans each KV backend directly (JsonKV,\n  Redis, PostgreSQL, MongoDB, OpenSearch). When a new KV backend is added,\n  `enumerate_kv_keys()` in `rebuild_vdb.py` must be extended.\n\n## Library usage\n\nThe core rebuild/check functions are plain async functions that accept your\nown initialized storage instances:\n\nRebuild targets must persist a queryable vector index. Passing a graph-only\nbackend such as `NoopVectorDBStorage` raises before source records are read or\nthe target is dropped; configure a persistent vector backend first.\n\n```python\nfrom lightrag.tools.rebuild_vdb import (\n    check_vdb_consistency,\n    rebuild_chunks_vdb,\n    rebuild_entities_vdb,\n    rebuild_relationships_vdb,\n)\n\nreport = await check_vdb_consistency(graph, entities_vdb, relationships_vdb)\nif not report[\"consistent\"]:\n    await rebuild_entities_vdb(graph, entities_vdb, global_config)\n    await rebuild_relationships_vdb(graph, relationships_vdb, global_config)\n```\n\n<!-- File: lightrag/tools/README_SOURCE_CONFLICT_REPAIR.md (Tokens: ~1360 | Category: anthropic-skill) -->\n# Source-Conflict Listing / Repair\n\nOffline companion to the typed source resolver (LR2 §5.5).\n\nA canonical source key — a filename with any parser `[hint]` stripped — is\nexpected to belong to exactly one **primary** document. History can break that:\ncustom-ID inserts, legacy ids, and basename collisions predate the strict\nresolver. When two or more primary rows claim one key, the resolver returns a\n*conflict* and `/documents/scan` refuses to act on that file: it is not\nenqueued, no record is deleted and nothing is archived, because picking a winner\nautomatically could silently retire the document you wanted to keep.\n\nThis tool (and the equivalent HTTP endpoints) is how an operator settles such a\nconflict explicitly.\n\n- The winner is never chosen for you: you name the `primary_doc_id` to keep.\n- Every other candidate is marked `metadata.is_duplicate=true` with\n  `original_doc_id=<primary>`. **No content is ever deleted** — the demoted rows\n  keep their status and their `full_docs` entry; they only lose their claim on\n  the canonical source, which is what makes the resolver return a single\n  primary afterwards.\n- Only `doc_status` and `full_docs` are opened. No vector store, graph store,\n  LLM or embedding model is touched, so a doc_status bookkeeping problem never\n  requires a reachable Milvus/Neo4j/LLM endpoint to fix. `full_docs` is read (not\n  written) to verify the primary you named — see *Safety*.\n\n## Usage\n\nCLI (honors `WORKING_DIR` / `WORKSPACE` and the `LIGHTRAG_*` storage variables\nfrom `.env`, exactly like the server):\n\n```bash\n# What is conflicting?\npython -m lightrag.tools.source_conflict_repair list [--limit 50] [--all]\n\n# Plan a repair (dry-run: nothing is modified)\npython -m lightrag.tools.source_conflict_repair repair \\\n    --source report.docx --primary doc-abc123\n\n# Commit it\npython -m lightrag.tools.source_conflict_repair repair \\\n    --source report.docx --primary doc-abc123 --apply\n```\n\n`--workspace` overrides the `WORKSPACE` env var. Exit code is `1` when a repair\nis refused (unknown primary, candidate set changed, backend cannot repair).\n\nLibrary (for a deployment that builds its own `LightRAG`):\n\n```python\nfrom lightrag.tools.source_conflict_repair import (\n    collect_source_conflicts,\n    repair_one_conflict,\n)\n\nconflicts = await collect_source_conflicts(rag.doc_status, limit=50)\nresult = await repair_one_conflict(\n    rag.doc_status,\n    \"report.docx\",\n    \"doc-abc123\",\n    workspace=rag.workspace,\n    # Required for a COMMIT: it is what verifies the primary you named can\n    # actually keep the source. A dry-run needs none.\n    full_docs=rag.full_docs,\n    apply=True,\n)\n```\n\n## Online equivalent\n\n```\nGET  /documents/source_conflicts?limit=50&cursor=<next_cursor>\nPOST /documents/source_conflicts/repair\n     {\"canonical_source_key\": \"...\", \"primary_doc_id\": \"...\",\n      \"expected_candidate_count\": N, \"expected_candidate_fingerprint\": \"...\",\n      \"dry_run\": false}\n```\n\nThe HTTP flow is two explicit calls: a dry-run returns the\n`candidate_count`/`fingerprint` pair, and the commit echoes both back. The CLI\nperforms the same two steps for you — it automates the copy-paste, not the\nguard.\n\n## Safety\n\n- Listing and a repair without `--apply` mutate nothing.\n- A commit is mutually exclusive with every in-deployment writer that can change\n  which documents claim the key: enqueue (the enqueue-serialize lock), the\n  processing stage's duplicate marking (a keyed lock on the canonical source key,\n  which the marking takes too), and clear/delete + scan classification + manual\n  reset (a pending-enqueue reservation). So the two operations are ordered rather\n  than interleaved: if the marking goes first the commit refuses before demoting\n  anything; if the commit goes first it verifies the key is settled and returns,\n  and a later marking is an ordinary state transition. Only the standalone CLI and\n  a second deployment writing the same database fall outside this.\n- A commit re-reads the candidate set under the backend's repair lock and\n  proceeds only when the count and fingerprint still match the dry-run\n  (compare-and-set), so a concurrent enqueue / delete / repair fails the commit\n  instead of being overwritten. A refused commit changes nothing; re-run it.\n- A repair interrupted half-way needs no reconciliation and leaves no marker\n  behind: the rows it finished already express `duplicate`, the rest still\n  resolve as a conflict. Run the command again.\n- Repeating a completed repair is safe: the same request is refused (its token\n  described the old candidate set) and a fresh run is a no-op with one\n  candidate and nothing to demote.\n- A commit refuses a primary that cannot keep the source, because the demotions\n  are irreversible — a repair only demotes, and a key left with no candidate has\n  no conflict left to settle:\n  - no `full_docs` content (an unprocessable stub a scan would delete) → 409;\n  - content that already exists under a **different** source, which the\n    processing stage will mark `FAILED [DUPLICATE:content_hash]` → 409;\n  - the content could not be *verified* at all — read failure, a backend without\n    strict point reads, or no `full_docs` handle → 503, retry later. Unverified\n    is not verified.\n- Not covered, by construction: a primary that has not been parsed yet has no\n  content hash, so nothing can predict whether its content will turn out to\n  duplicate another document. The commit itself still succeeds, and the key\n  really is unique when it returns — no marking can interleave with it. A\n  marking that lands afterwards is an ordinary state transition, not a\n  half-applied repair: the outcome is the ordinary content-dedup steady state\n  (the content lives under that document and this key ends up with no primary),\n  and no content is lost.\n\n<!-- File: prompts/UserCustomizePrompts.md (Tokens: ~22 | Category: generic) -->\n# Folder for User-Defined Prompts\n\nThis folder is designated for storing customized prompt templates created by users.\n</yakaai_skills>"}