{"owner":"LearningCircuit","repo":"local-deep-research","hasSkills":true,"totalSkillsCount":3,"totalTokensCount":9678,"categories":["anthropic-skill","project-spec"],"hasMcp":false,"mcpConfig":null,"found":[".semgrep/rules/check-safe-requests.yaml","docs/architecture.md",".semgrep/rules/README.md"],"skills":{".semgrep/rules/check-safe-requests.yaml":"rules:\n  - id: unsafe-requests-get\n    patterns:\n      - pattern: requests.get(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.get() bypasses SSRF protection.\n      Use safe_get() from security module instead.\n      Import: from ...security import safe_get\n      For localhost services: safe_get(url, allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-post\n    patterns:\n      - pattern: requests.post(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.post() bypasses SSRF protection.\n      Use safe_post() from security module instead.\n      Import: from ...security import safe_post\n      For localhost services: safe_post(url, allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-session\n    patterns:\n      - pattern: requests.Session()\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.Session() bypasses SSRF protection.\n      Use SafeSession() from security module instead.\n      Import: from ...security import SafeSession\n      For localhost services: SafeSession(allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-put\n    patterns:\n      - pattern: requests.put(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.put() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n  - id: unsafe-requests-delete\n    patterns:\n      - pattern: requests.delete(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.delete() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n  - id: unsafe-requests-patch\n    patterns:\n      - pattern: requests.patch(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.patch() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n","docs/architecture.md":"# Architecture Overview\n\nThis document provides detailed technical diagrams of Local Deep Research's architecture.\n\n## System Architecture\n\n```mermaid\nflowchart TB\n    subgraph USER[\"👤 User Interface\"]\n        WEB[Web Browser<br/>localhost:5000]\n    end\n\n    subgraph BACKEND[\"🌐 FastAPI Backend\"]\n        API[REST API + WebSocket]\n        AUTH[Authentication<br/>CSRF Protection]\n        ROUTES[Research Routes]\n    end\n\n    subgraph RESEARCH[\"🔬 Research Engine\"]\n        STRAT[Strategy Selector]\n        QGEN[Question Generator]\n        EXEC[Search Executor]\n        SYNTH[Report Synthesizer]\n    end\n\n    subgraph LLM[\"🤖 LLM Providers\"]\n        direction TB\n        LOCAL_LLM[/\"🏠 Local LLMs\"\\]\n        OLLAMA[Ollama]\n        LMSTUDIO[LM Studio]\n        CLOUD_LLM[/\"☁️ Cloud LLMs\"\\]\n        OPENAI[OpenAI]\n        ANTHROPIC[Anthropic]\n        GEMINI[Google Gemini]\n        OPENROUTER[OpenRouter<br/>100+ models]\n    end\n\n    subgraph SEARCH[\"🔍 Search Engines\"]\n        direction TB\n        LOCAL_SEARCH[/\"🏠 Local Search\"\\]\n        SEARXNG[SearXNG]\n        ELASTIC[Elasticsearch]\n        LIBRARY[Document Library]\n        WEB_SEARCH[/\"🌐 Web Search\"\\]\n        TAVILY[Tavily]\n        BRAVE[Brave]\n        ACADEMIC[/\"📚 Academic\"\\]\n        ARXIV[ArXiv]\n        PUBMED[PubMed]\n        SEMANTIC[Semantic Scholar]\n    end\n\n    subgraph STORAGE[\"💾 Storage Layer\"]\n        SQLCIPHER[(SQLCipher DB<br/>AES-256 Encrypted)]\n        VECTORS[(Vector Store<br/>Embeddings)]\n        FILES[(File Storage<br/>PDFs & Docs)]\n    end\n\n    subgraph OUTPUT[\"📄 Output\"]\n        MD[Markdown]\n        PDF[PDF Export]\n        LATEX[LaTeX]\n        QUARTO[Quarto]\n        RIS[RIS/BibTeX]\n    end\n\n    WEB <--> API\n    API --> AUTH\n    AUTH --> ROUTES\n    ROUTES --> STRAT\n\n    STRAT --> QGEN\n    QGEN --> EXEC\n    EXEC --> SYNTH\n\n    EXEC <--> OLLAMA & LMSTUDIO\n    EXEC <--> OPENAI & ANTHROPIC & GEMINI & OPENROUTER\n\n    EXEC <--> SEARXNG & ELASTIC & LIBRARY\n    EXEC <--> TAVILY & BRAVE\n    EXEC <--> ARXIV & PUBMED & SEMANTIC\n\n    SYNTH --> SQLCIPHER\n    LIBRARY <--> VECTORS\n    LIBRARY <--> FILES\n\n    SYNTH --> MD & PDF & LATEX & QUARTO & RIS\n\n    LOCAL_LLM ~~~ OLLAMA\n    CLOUD_LLM ~~~ OPENAI\n    LOCAL_SEARCH ~~~ SEARXNG\n    WEB_SEARCH ~~~ TAVILY\n    ACADEMIC ~~~ ARXIV\n\n    style USER fill:#e1f5fe\n    style BACKEND fill:#fff3e0\n    style RESEARCH fill:#f3e5f5\n    style LLM fill:#e8f5e9\n    style SEARCH fill:#fce4ec\n    style STORAGE fill:#fff8e1\n    style OUTPUT fill:#e0f2f1\n```\n\n## Research Flow\n\n```mermaid\nflowchart LR\n    subgraph INPUT[\"1️⃣ Input\"]\n        Q[Research Query]\n    end\n\n    subgraph ITERATE[\"2️⃣ Iterative Research\"]\n        direction TB\n        GEN[Generate<br/>Questions]\n        SEARCH[Parallel<br/>Search]\n        ANALYZE[Analyze<br/>Results]\n        GEN --> SEARCH --> ANALYZE\n        ANALYZE -.->|\"Need more info?\"| GEN\n    end\n\n    subgraph SYNTHESIZE[\"3️⃣ Synthesis\"]\n        REPORT[Generate<br/>Report]\n        CITE[Add<br/>Citations]\n        REPORT --> CITE\n    end\n\n    subgraph OUTPUT[\"4️⃣ Output\"]\n        RESULT[Final Report<br/>with Sources]\n    end\n\n    Q --> GEN\n    ANALYZE --> REPORT\n    CITE --> RESULT\n\n    style INPUT fill:#e3f2fd\n    style ITERATE fill:#f3e5f5\n    style SYNTHESIZE fill:#e8f5e9\n    style OUTPUT fill:#fff3e0\n```\n\n## Deployment Options\n\n```mermaid\nflowchart TB\n    subgraph FULL_LOCAL[\"🏠 Self-Hosted (Maximum Privacy)\"]\n        direction LR\n        L_LDR[LDR] <--> L_OLLAMA[Ollama]\n        L_LDR <--> L_SEARX[SearXNG]\n        L_LDR <--> L_DB[(Encrypted DB)]\n    end\n\n    subgraph HYBRID[\"⚡ Hybrid (Balanced)\"]\n        direction LR\n        H_LDR[LDR] <--> H_OLLAMA[Ollama]\n        H_LDR <-->|\"Web Search\"| H_CLOUD[Cloud APIs]\n        H_LDR <--> H_DB[(Encrypted DB)]\n    end\n\n    subgraph CLOUD[\"☁️ Cloud-Powered (Maximum Speed)\"]\n        direction LR\n        C_LDR[LDR] <--> C_OPENAI[OpenAI/Claude]\n        C_LDR <--> C_TAVILY[Tavily/Brave]\n        C_LDR <--> C_DB[(Encrypted DB)]\n    end\n\n    style FULL_LOCAL fill:#e8f5e9\n    style HYBRID fill:#fff3e0\n    style CLOUD fill:#e3f2fd\n```\n\n## Feature Map\n\n```mermaid\nmindmap\n  root((Local Deep<br/>Research))\n    Research\n      Quick Summary\n      Detailed Reports\n      Follow-up Questions\n      Research Strategies\n    Search Sources\n      Web Search\n        SearXNG\n        Tavily\n        Brave\n      Academic\n        ArXiv\n        PubMed\n        Semantic Scholar\n      Local\n        Document Library\n        Collections\n        Elasticsearch\n    LLM Support\n      Local\n        Ollama\n        LM Studio\n      Cloud\n        OpenAI\n        Anthropic\n        Google\n        OpenRouter\n        Requesty\n    Output\n      Markdown\n      PDF\n      LaTeX\n      Quarto\n      RIS/BibTeX\n    Features\n      News Subscriptions\n      Cost Analytics\n      Benchmarking\n      RAG Search\n      Per-User Encryption\n```\n\n## Component Details\n\n### LLM Providers\n\n| Provider | Type | Description |\n|----------|------|-------------|\n| Ollama | Local | Self-hosted open-source models |\n| LM Studio | Local | Desktop app for local models |\n| OpenAI | Cloud | GPT-4, GPT-3.5 |\n| Anthropic | Cloud | Claude 3 family |\n| Google | Cloud | Gemini models |\n| OpenRouter | Cloud | 100+ models via single API |\n| Requesty | Cloud | 100+ models via single API |\n\n### Search Engines\n\n| Engine | Type | Best For |\n|--------|------|----------|\n| SearXNG | Local/Self-hosted | Privacy, aggregated results |\n| Tavily | Cloud | AI-optimized search |\n| ArXiv | Academic | Physics, CS, Math papers |\n| PubMed | Academic | Biomedical research |\n| Semantic Scholar | Academic | Cross-discipline papers |\n| Wikipedia | Knowledge | General knowledge |\n| Your Documents | Local | Private document search |\n\n### Output Formats\n\n| Format | Use Case |\n|--------|----------|\n| Markdown | Default, web display |\n| PDF | Sharing, printing |\n| LaTeX | Academic papers |\n| Quarto | Reproducible documents |\n| RIS/BibTeX | Reference managers |\n\n## Knowledge Loop: Research → Library → Future Research\n\nOne of LDR's powerful features is the ability to build a personal knowledge base that improves future research.\n\n```mermaid\nflowchart TB\n    subgraph RESEARCH[\"1️⃣ Research\"]\n        Q[Your Question] --> Engine[Research Engine]\n        Engine --> Results[Results + Sources]\n    end\n\n    subgraph DOWNLOAD[\"2️⃣ Download\"]\n        Results --> Track[Track Sources]\n        Track --> Queue[Download Queue]\n        Queue --> Extract[Download & Extract Text]\n    end\n\n    subgraph LIBRARY[\"3️⃣ Library\"]\n        Extract --> Store[(Document Storage)]\n        Store --> Organize[Organize into Collections]\n    end\n\n    subgraph INDEX[\"4️⃣ Index\"]\n        Organize --> Chunk[Chunk Documents]\n        Chunk --> Embed[Generate Embeddings]\n        Embed --> FAISS[(Vector Index)]\n    end\n\n    subgraph REUSE[\"5️⃣ Reuse\"]\n        FAISS --> SearchEngine[Collection as Search Engine]\n        SearchEngine --> Q\n    end\n\n    style RESEARCH fill:#e3f2fd\n    style DOWNLOAD fill:#fff3e0\n    style LIBRARY fill:#e8f5e9\n    style INDEX fill:#f3e5f5\n    style REUSE fill:#fce4ec\n```\n\n### How It Works\n\n1. **Research Completes** → Sources are tracked in `ResearchResource` table\n2. **Download Sources** → Click \"Get All Research PDFs\" to queue downloads\n   - Smart downloaders for ArXiv, PubMed, Semantic Scholar, etc.\n   - Automatic text extraction from PDFs\n3. **Build Library** → Documents stored in encrypted database\n   - Deduplication via content hash\n   - Multiple storage modes: database (encrypted), filesystem, text-only\n4. **Create Collections** → Organize documents by topic/project\n   - Each collection can have different embedding settings\n   - Documents can belong to multiple collections\n5. **Index for Search** → Generate vector embeddings\n   - Configurable chunk size and overlap\n   - FAISS index for fast similarity search\n6. **Use in Future Research** → Select collection as search engine\n   - RAG search finds relevant passages\n   - Results cite back to your documents\n\n### Key Components\n\n| Component | Purpose |\n|-----------|---------|\n| `DownloadService` | Manages PDF downloads with source-specific strategies |\n| `LibraryService` | Queries and manages document library |\n| `LibraryRAGService` | Creates vector indices for semantic search |\n| `CollectionSearchEngine` | Searches collections using RAG |\n| `Document` | Stores text content, metadata, file references |\n| `DocumentChunk` | Stores indexed text chunks with embeddings |\n| `Collection` | Groups documents with shared embedding settings |\n\n### Storage Options\n\n| Mode | Security | Use Case |\n|------|----------|----------|\n| Database | AES-256 encrypted | Default, maximum security |\n| Filesystem | Unencrypted | Need external tool access |\n| Text Only | Encrypted text, no PDFs | Minimal storage |\n\n---\n\n## Technical Analysis & Project Health\n\n*Last updated: December 2024*\n\nThis section provides a comprehensive technical analysis of the codebase, including quality metrics, architecture patterns, and project health indicators.\n\n### Project Statistics\n\n| Metric | Count |\n|--------|-------|\n| Test Classes | 809+ |\n| Search Engine Implementations | 25 |\n| LLM Provider Implementations | 9 |\n| Search Strategies | 5 |\n| Abstract Base Classes | 26 |\n| CI/CD Workflows | 57 |\n| Security Scanners in CI | 22+ |\n| Core Dependencies | 63 |\n\n### Architecture Patterns\n\n#### Extensibility Design\n\nThe codebase follows a consistent pattern for extensibility:\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    Abstract Base Classes                     │\n├─────────────────────────────────────────────────────────────┤\n│ BaseSearchEngine      │ Common interface for 25+ engines    │\n│ BaseSearchStrategy    │ Strategy pattern for research       │\n│ BaseCitationHandler   │ Citation processing abstraction     │\n│ BaseQuestionGenerator │ Question generation interface       │\n│ BaseExporter          │ Export format abstraction           │\n└─────────────────────────────────────────────────────────────┘\n```\n\n#### Search Engine Plugin System\n\nNew search engines can be added by:\n1. Creating a class inheriting from `BaseSearchEngine`\n2. Placing it in `web_search_engines/engines/`\n3. Register the engine in `ENGINE_REGISTRY` and add it to the security module allowlist in `security/module_whitelist.py`\n\n```python\n# Example: Adding a new search engine\nclass SearchEngineCustom(BaseSearchEngine):\n    def run(self, query: str) -> List[Dict]:\n        # Implementation\n        pass\n```\n\n#### LLM Provider Integration\n\nSupports 10 LLM providers with auto-discovery:\n- Ollama (local)\n- LM Studio (local)\n- OpenAI\n- Anthropic\n- Google Gemini\n- OpenRouter (100+ models)\n- Requesty (100+ models)\n- DeepSeek\n- Mistral\n- Groq\n\n### Quality Ratings\n\n```mermaid\npie title Project Health Ratings\n    \"Code Quality (86)\" : 86\n    \"Security (93)\" : 93\n    \"Extensibility (93)\" : 93\n    \"Performance (85)\" : 85\n    \"UX/DX (83)\" : 83\n    \"Maintenance (89)\" : 89\n```\n\n#### Detailed Ratings\n\n| Category | Score | Highlights |\n|----------|-------|------------|\n| **Code Quality** | 86/100 | 809+ test classes, ruff/mypy enforcement, comprehensive pre-commit hooks |\n| **Security** | 93/100 | SQLCipher AES-256 encryption, full SHA pinning in CI, 22+ security scanners |\n| **Extensibility** | 93/100 | 26 abstract base classes, plugin architecture, strategy pattern throughout |\n| **Performance** | 85/100 | Adaptive rate limiting, cache stampede protection, parallel search execution |\n| **UX/Developer Experience** | 83/100 | Real-time WebSocket updates, in-tool documentation, comprehensive error handling |\n| **Maintenance** | 89/100 | 57 CI/CD workflows, automated security scanning, structured changelog |\n| **Overall** | 88/100 | Production-ready with excellent security and extensibility |\n\n### Security Architecture\n\n```mermaid\nflowchart TB\n    subgraph SECURITY[\"🔐 Security Layers\"]\n        direction TB\n        AUTH[Signed session cookie<br/>Starlette SessionMiddleware<br/>+ CSRFMiddleware]\n        ENCRYPT[SQLCipher<br/>AES-256 Encryption]\n        SCAN[22+ Security Scanners<br/>CodeQL, Semgrep, Bandit...]\n        PIN[Full SHA Pinning<br/>All 57 CI Workflows]\n    end\n\n    AUTH --> ENCRYPT\n    ENCRYPT --> SCAN\n    SCAN --> PIN\n\n    style SECURITY fill:#e8f5e9\n```\n\n**Security Features:**\n- Per-user encrypted databases (SQLCipher with AES-256)\n- Full GitHub Action SHA pinning (not tag-based)\n- Comprehensive CI security scanning:\n  - CodeQL (Python, JavaScript)\n  - Semgrep (custom rulesets)\n  - Bandit (Python security)\n  - Trivy (container scanning)\n  - Dependency review\n  - Secret scanning\n\n### Performance Optimizations\n\n| Feature | Implementation |\n|---------|----------------|\n| **Parallel Search** | `concurrent.futures.ThreadPoolExecutor` for multi-question search |\n| **Rate Limiting** | Adaptive system with `learning_rate=0.3` |\n| **Cache Protection** | `fetch_events` + `fetch_locks` for stampede prevention |\n| **Progress Streaming** | Socket.IO ASGI sub-app mounted at `/ws` for real-time UI updates |\n| **Cross-Engine Filtering** | LLM-powered relevance scoring and deduplication |\n\n#### Thread & Resource Lifecycle\n\nUnlike typical web apps that share a single database connection pool, this application maintains **separate database engines per user** because each user has their own encrypted SQLite file with a unique SQLCipher key ([SQLCipher](https://www.zetetic.net/sqlcipher/) is an encrypted extension of SQLite). This creates a threading challenge: every background thread needs its own engine for the user it serves.\n\nA single **QueuePool** engine is kept per user (`pool_size=20`,\n`max_overflow=40`, `pool_timeout=10`) in\n`DatabaseManager.connections[username]`. It is created with\n`check_same_thread=False`, so it is shared safely across the AnyIO\nworker threads that serve synchronous route handlers AND background\nthreads (research workers, scheduler jobs, metric writers). All of these acquire sessions from the\nsame pool, which keeps FD usage bounded by `pool_size + max_overflow`\nper user.\n\n> Earlier revisions of the code maintained a second, parallel system of\n> per-`(username, thread_id)` **NullPool** engines for background metric\n> writes. That system was removed because orphaned entries leaked\n> SQLCipher + WAL file handles (3 FDs per active connection) under load,\n> eventually exhausting the 1024 FD soft limit. Routing all work through\n> the single per-user QueuePool made FD usage bounded and let us delete\n> ~200 lines of thread-engine bookkeeping.\n\n##### Resource Cleanup Layers\n\nCleanup is defense-in-depth with multiple layers:\n\n| Layer | Trigger | What it cleans |\n|-------|---------|----------------|\n| `@thread_cleanup` decorator | Thread function exit (normal or exception) | DB session (returned to per-user pool), settings context, search context |\n| `WorkerCleanupAPIRoute` (`web/dependencies/threadpool.py`) | Synchronous FastAPI endpoint exit (normal or exception) | Sessions and ambient thread-local state on the AnyIO worker that served the endpoint |\n| `WorkerCleanupStreamingResponse` (`web/dependencies/threadpool.py`) | Each synchronous streamed-response iterator step (chunk, completion, or exception) | Sessions and ambient thread-local state on whichever AnyIO worker advanced that step |\n| `finally` blocks | Per-research in `run_research_process()` | Search engine HTTP sessions, strategy thread pool executors |\n| `DatabaseMiddleware` `finally` block (`web/fastapi_app.py`) | After each HTTP request | Sweeps dead workers and cleans event-loop-owned state; synchronous endpoint and stream-iterator state are reclaimed on their owning workers by the two wrappers above |\n| Periodic pool dispose | Every 30 min | Calls `engine.dispose()` on per-user QueuePool engines to release SQLCipher+WAL file handles that accumulate from out-of-order connection closes |\n| Logout cascade | User logout | Scheduler unregister (removes password) → DB close (`engine.dispose()`) → session destroy |\n| Stale session cleanup | `clear_session_if_unrecoverable()`, called from `require_auth()` when the DB connection is missing (`web/dependencies/auth.py`) | Clears the signed session cookie for users whose DB connection is gone and who have no recovery credential. Not sampled/throttled — unlike the old `before_request` hook it only runs on an already-failed auth check |\n\n##### Research Thread Lifecycle\n\n```mermaid\nflowchart TD\n    A[\"POST /api/start_research\"] --> B{\"Slots available?\"}\n    B -- \"Yes (direct)\" --> C[\"start_research_process()\"]\n    B -- \"No (queued)\" --> D[\"Database queue\"]\n    D --> E[\"_process_queue_loop<br/>(daemon thread, 10s poll)\"]\n    E --> C\n\n    C --> F[\"Research thread<br/>(daemon, semaphore-gated)\"]\n    F --> G[\"@thread_cleanup wraps<br/>run_research_process()\"]\n\n    G --> H[\"Strategy.find_relevant_information()\"]\n    H --> I[\"ThreadPoolExecutor sub-tasks<br/>(within strategies only)\"]\n    I --> J[\"Worker @thread_cleanup:<br/>close worker DB session\"]\n\n    J --> K[\"finally: search_engine.close()<br/>system.close() → strategy.close()\"]\n    K --> L[\"Main @thread_cleanup:<br/>close DB session, return connection to pool\"]\n\n    subgraph sweep [\"Defense-in-depth (independent timers)\"]\n        M[\"Dead-thread credential sweep<br/>(~60s via processor_v2)\"] -.-> N[\"Remove stale credentials<br/>for dead thread IDs\"]\n        O[\"Pool dispose<br/>(30 min via connection_cleanup)\"] -.-> P[\"engine.dispose() on all<br/>QueuePool engines\"]\n    end\n\n    style sweep fill:#f0f0f0,stroke:#999\n```\n\n##### FD Budget\n\nEach SQLCipher connection in WAL (Write-Ahead Logging) mode uses **2 file descriptors** (main db + WAL file). All connections to the same database within a process share **1 SHM** (shared-memory) file descriptor. The formula per user database is: `connections × 2 + 1`.\n\n| Component | FD Formula | With defaults |\n|-----------|-----------|---------------|\n| QueuePool (steady state) | `logged_in_users × (pool_size × 2 + 1)` | `users × 41` FDs |\n| QueuePool (peak) | `logged_in_users × ((pool_size + max_overflow) × 2 + 1)` | `users × 121` FDs |\n\nDefault Linux ulimit is 1024 soft (bare metal), which is tight for multi-user deployments. Docker's daemon default (typically 1M+) is adequate. QueuePool engines are created at login and disposed at logout, so only active users consume FDs.\n\nFor more on diagnosing FD exhaustion, see [Troubleshooting - Resource Exhaustion](./troubleshooting.md#resource-exhaustion).\n\n##### Key Files\n\n| File | Role |\n|------|------|\n| `src/local_deep_research/database/encrypted_db.py` | `DatabaseManager`, engine lifecycle, pool management |\n| `src/local_deep_research/database/thread_local_session.py` | `@thread_cleanup` decorator, thread-local sessions, credential cleanup |\n| `src/local_deep_research/web/fastapi_app.py` | `DatabaseMiddleware` per-request cleanup (replaces Flask's `teardown_appcontext`), lifespan orchestration |\n| `src/local_deep_research/web/services/research_service.py` | Research thread creation, `run_research_process()` |\n| `src/local_deep_research/web/queue/processor_v2.py` | Queue processing, credential cleanup trigger |\n\n### Areas for Improvement\n\nWhile the project scores highly overall, these areas have room for growth:\n\n1. **Integration Testing** - More end-to-end tests for full research workflows\n2. **API Documentation** - OpenAPI/Swagger spec for REST endpoints\n3. **Metrics Dashboard** - Prometheus/Grafana integration for monitoring\n4. **Resource Observability** - Expose FD count, thread count, and connection pool stats in /api/v1/health; add periodic sweep logging\n5. **Async Architecture** - Migration to async/await for I/O-bound operations\n\n### Key Source Files\n\n| Component | Location | Purpose |\n|-----------|----------|---------|\n| Research Engine | `src/local_deep_research/search_system.py` | Main `AdvancedSearchSystem` class |\n| Strategies | `src/local_deep_research/advanced_search_system/strategies/` | Research strategy implementations |\n| Search Engines | `src/local_deep_research/web_search_engines/engines/` | 25 search engine implementations |\n| Report Generation | `src/local_deep_research/report_generator.py` | `IntegratedReportGenerator` |\n| Web API | `src/local_deep_research/web/routers/` | FastAPI route handlers (`web/routes/` now holds a few shared helpers only, not route handlers) |\n| Database | `src/local_deep_research/database/` | SQLCipher models and migrations |\n| Encrypted DB | `src/local_deep_research/database/encrypted_db.py` | Per-user SQLCipher engine lifecycle |\n| Thread Sessions | `src/local_deep_research/database/thread_local_session.py` | Thread-safe session management and cleanup |\n| Settings | `src/local_deep_research/config/` | Configuration and LLM setup |\n\n### Contributing to Architecture\n\nWhen extending the system:\n\n1. **Adding Search Engines**: Inherit from `BaseSearchEngine`, implement `run()` method\n2. **Adding Strategies**: Inherit from `BaseSearchStrategy`, implement `analyze_topic()` method\n3. **Adding LLM Providers**: Add to `config/llm_config.py` with proper initialization\n4. **Adding Export Formats**: Inherit from base exporter pattern in `utilities/`\n\nSee [CONTRIBUTING.md](../CONTRIBUTING.md) for detailed guidelines.\n",".semgrep/rules/README.md":"# LDR Custom Security Rules\n\nThis directory contains custom Semgrep security rules specific to Local Deep Research (LDR).\n\n## Rules Overview\n\n### ldr-security.yaml\n\nLDR-specific security rules covering:\n\n1. **Hardcoded Secrets**\n   - Detects API keys, passwords, tokens in source code\n   - Severity: ERROR\n   - CWE-798\n\n2. **SQL Injection Prevention**\n   - `sql-string-concatenation`: follows dynamically constructed strings\n     (f-string, `+`, `%`, `.format()`) to database execution and SQLAlchemy\n     raw-SQL constructors, including inline expressions, aliases and stored\n     values. Named SQL builders also prompt review. Bound SQLAlchemy values\n     remain clear, including mapping values, stored strings without\n     reassignment, and reused `bindparams()` statements. Column identifiers\n     built from an inline or unchanged stored f-string remain clear when\n     quoting is enabled. Unpacked arguments, `quoted_name` overrides,\n     reassigned names, explicit or unknown literal flags, and nested raw-SQL\n     constructors remain reportable. Only the name occurrence is cleared;\n     an unsafe table/selectable argument still reaches execution.\n   - `sql-execute-with-string-format`: `$X.execute(f\"...\")` or\n     `$X.execute(\"...\" + ...)`\n   - Enforces parameterized queries via SQLAlchemy\n   - Severity: ERROR\n   - CWE-89\n\n3. **Code Injection**\n   - Detects dangerous use of eval/exec\n   - Prevents arbitrary code execution\n   - Severity: ERROR\n   - CWE-95\n\n4. **Command Injection**\n   - Detects unsafe use of os.system, shell=True\n   - Enforces subprocess with argument lists\n   - Severity: ERROR\n   - CWE-78\n\n5. **Path Traversal**\n   - `path-traversal-risk`: follows dynamic path construction and fetched\n     database values to file operations, directory enumeration and file\n     responses. A stored path served directly is covered without requiring\n     an intermediate `Path(...)`. Construction or normalization alone is\n     not a file operation. Mutation sinks include `Path.touch`, both paths\n     in `rename`/`replace`, and positional/keyword `os.remove`/`os.unlink`.\n     The one-target `Path.replace` signature keeps string and datetime\n     replacements outside the filesystem rule.\n   - `request-path-traversal-risk`: follows route handler parameters and\n     `request.*` to path construction and filesystem sinks. Database fetch\n     results are handled by the separate dynamic-path rule, so filtering a\n     query by a request ID does not classify the whole stored row as request\n     input. A response's download-name metadata is not its filesystem path.\n   - Only returned values from `validate_safe_path`, `validate_data_path`,\n     `validate_model_path`, `safe_join`, `secure_filename` and\n     `sanitize_filename` clear request taint. Exact call and assignment\n     patterns preserve findings for file access inside their arguments and\n     for later use of the original input. Other `PathValidator` methods and\n     `os.path.basename` are not sanitizers (`basename(\"..\")` is still `\"..\"`).\n   - Intraprocedural only: input is not followed into helper bodies. The\n     sanitizer contracts assume a trusted base directory; they do not prove\n     ownership or protect against concurrent local filesystem changes.\n     These rules prompt a boundary review rather than establish safety.\n   - Severity: WARNING\n   - CWE-22\n\n6. **Unsafe Deserialization**\n   - Detects unsafe YAML/pickle loading\n   - Prevents code execution via deserialization\n   - Severity: ERROR\n   - CWE-502\n\n7. **Weak Randomness**\n   - Detects use of random module for security\n   - Enforces secrets module for crypto operations\n   - Severity: WARNING\n   - CWE-338\n\n8. **Debug Mode in Production**\n   - Detects Flask debug=True\n   - Prevents information disclosure\n   - Severity: ERROR\n   - CWE-489\n\n9. **SSRF Prevention**\n   - Detects URL fetching operations\n   - Reminds to validate URLs\n   - Severity: WARNING\n   - CWE-918\n\n10. **XSS Prevention**\n    - Detects user input in HTML context\n    - Enforces proper escaping\n    - Severity: WARNING\n    - CWE-79\n\n11. **CSRF Protection**\n    - Detects POST endpoints\n    - Reminds to enable CSRF protection\n    - Severity: INFO\n    - CWE-352\n\n12. **Credential Logging**\n    - Detects passwords in log statements\n    - Prevents credential disclosure\n    - Severity: ERROR\n    - CWE-532\n\n## Usage\n\nThese rules are automatically run by the Semgrep CI/CD workflow:\n\n```bash\n# Run locally\nsemgrep --config=.semgrep/rules/ src/\n\n# Run with standard rules\nsemgrep --config=p/security-audit --config=.semgrep/rules/ src/\n```\n\n## GitHub upload and source suppressions\n\nThe pinned Semgrep 1.179.0 omits `nosemgrep` findings from its JSON report but retains\nthem in native SARIF with `inSource` suppression metadata. GitHub's SARIF\nimport does not honor that metadata, so uploading the native report would\npublish the ignored findings as open alerts.\n\nThe workflow validates the complete native JSON and SARIF reports, then\ncreates `semgrep-results.github.sarif`. Only results with recognized\n`inSource` suppressions (no status, or `accepted`) are omitted from that\ncopy, and only when the scanned file holds a reviewed annotation naming the\nresult's rule (see below) on the result's first line or on the line above\nit. The SARIF suppression does not record which rule a comment named, so\nthe validator reads the line back; a suppressed result without such an\nannotation fails validation. Unknown suppression kinds or statuses fail\nvalidation too. All other\nfindings and scan metadata are preserved, and the upload copy is validated\nagain. Scanner errors, skipped rules, empty scope and mismatched reports\nstill prevent upload.\n\nThe diagnostic artifact retains both native reports and the GitHub upload\ncopy for 7 days. Tests using native scanner reports and synthetic fixtures check that\nsuppressing one result preserves every unsuppressed result, including the\nsame rule at other locations and other rules at the same location; the\nreal-scanner test runs three rules on one line, annotates one or two of\nthem, and checks that the strict scan stays error-free and the rest are\nuploaded.\n\nReferences: [Semgrep source suppressions](https://docs.semgrep.dev/ignoring-files-folders-code)\nand [GitHub's supported SARIF properties](https://docs.github.com/en/code-security/reference/code-scanning/sarif-files/sarif-support).\n\n## Adding New Rules\n\nTo add new custom rules:\n\n1. Create a new YAML file in `.semgrep/rules/`\n2. Follow [Semgrep rule syntax](https://semgrep.dev/docs/writing-rules/overview/)\n3. Test the rule: `semgrep --config=.semgrep/rules/your-rule.yaml src/`\n4. Document the rule in this README\n\n## Reviewed source annotations\n\nUse a rule-specific reviewed annotation beside a reviewed false positive,\nat the end of the finding's first line or alone on the line above it:\n\n```python\n# nosemgrep: semgrep.rules.weak-random-generation, reason: Non-security jitter\ndelay_seconds = random.randint(1, 30)\n```\n\nWrite it exactly as `nosemgrep: <rule-id>[, <rule-id>]..., reason: <reason>`,\nstarting a comment (`#`, `//`, `{#`, `<!--` or `/*`), with fully qualified\nrule ids (list several when one line has several reviewed false positives)\nand no comma in the reason. Semgrep 1.177.0 honours a case-insensitive\n`nosem` marker after a space anywhere on a line, including string literals\nand words such as \"NoSemantics\", and a marker without a rule id hides every\nrule on that line. Semgrep also accepts any suffix of a rule id and reads\nthe first word of each comma-separated item after the colon as a rule id.\nThe `check-nosemgrep-annotations` pre-commit hook rejects every other marker\nunder `src/`, including one inside a Python string or docstring, and the\nupload validator rejects a suppressed result without an annotation naming\nits exact rule id.\n\nThe `reason:` item is required, not decoration. Semgrep reads it as one\nmore id; `reason:` is not a valid rule id, so it suppresses nothing. But a\ncomment with a single id makes Semgrep 1.177.0 report a warning for every\nother rule that matches the same line (\"found 'nosem' comment with id ...,\nbut no corresponding rule trying ...\"), and under `--strict` that fails the\nwhole scan, and so the release gate. Semgrep skips that warning when a\ncomment holds two or more ids, so with the `reason:` item a new or updated\nrule matching an annotated line produces an ordinary uploaded alert\ninstead. If that alert is another reviewed false positive, add its rule id\nto the annotation.\n\nKeep the rule enabled elsewhere; do not exclude a file or weaken the release\nthreshold to clear an alert. Re-review the annotation when the expression or\nits input boundary changes.\n\nReviews in October 2026 identified these cases:\n\n| Finding | Why the annotated use is safe | Existing verification |\n| --- | --- | --- |\n| Seven script interpolations | Jinja's `tojson` escapes HTML delimiters before embedding JSON in a script. | Template injection and JSON rendering tests in `tests/security/test_injection_and_template_safety.py`. |\n| Eleven unquoted attribute expressions | Both branches emit fixed `selected`/`checked` tokens or the empty string; no input is inserted into an attribute name. | Inspect the literal branches in `embedding_settings.html`; the rest of each attribute remains quoted. |\n| `help_tip`'s `safe` filter | Every current caller passes a source literal, intentionally containing help markup. | Review all call sites. The existing security test catches simple variable arguments, but does not fully parse composed Jinja expressions. |\n| Four `random` calls | They control subscription jitter or rate-limit exploration, not credentials, identifiers or authorization. | Scheduler and rate-limit tests; cryptographic operations still use `secrets`. |\n| Two apparent secrets | One is an empty API-key default; the other is the existing placeholder only used for unencrypted databases. | Configuration and authentication tests retain the surrounding behavior. |\n| Two pickle calls | They raise `UnpicklingError` on rejected globals/extensions rather than loading data. | `tests/vector_stores/test_faiss_safe_load_security.py` exercises hostile payloads, extension caches, persistent IDs and legitimate docstores. |\n| XML import | It imports `xml.sax.saxutils.escape` to encode text, not an XML parser. | Note-AI prompt rendering tests. |\n| Evernote `TreeBuilder` import | ElementTree's `TreeBuilder` only assembles elements. The Evernote loader's XML is read by defusedxml's parser, which refuses entity declarations, external references and any DOCTYPE with an internal subset. | `tests/document_loaders/test_evernote_extraction.py`. |\n| TOML loader's `Path(source)` | The app builds the loader only in `load_from_bytes`, from the temporary file it has just written; an upload supplies that file's bytes, not its path. No route passes a request value to `get_loader_for_path` or to the loader. | Review the callers of `load_from_bytes` and `get_loader_for_path`. `tests/document_loaders/test_toml_loader.py` loads uploads through `load_from_bytes`. |\n\nThe separate DevSkim annotation in `services/ui.js` covers its fragment-based\nanchor renderer, which already adds both `noopener` and `noreferrer`.\n\nThe combined SQL/path rules carry 100 individually reviewed dispositions.\nThey remain local to the reported operation; no source file, directory or\nrule is excluded:\n\n| Findings | Reviewed boundary |\n| --- | --- |\n| 30 SQL operations | Hex-encoded or explicitly checked SQLCipher keys; typed/range-checked or allowlisted PRAGMAs; dialect-quoted identifiers; migration table/column literals; bound row values; a fixed schema-version integer. |\n| Benchmark and indexing APIs | Explicit local Python/CLI output or input paths. The web benchmark worker uses its separate database runner rather than these file-output functions. |\n| Package and startup files | Fixed filenames and globs beneath installed package paths or operator-configured application directories. |\n| Upload and converter temporary files | Server-generated temporary names, allowlisted extensions and validated stems; web upload callers do not provide scratch directories. |\n| Backup retention cleanup | Fixed-pattern glob results filtered by `is_safe_glob_result` under the server-owned, hashed per-user backup directory. Reordering the checked list to preserve the newly finalized backup does not admit another path. |\n| Library/report cleanup | Existing root resolution and symlink checks at the destructive operation; document deletion callers require an allowed root. Operator opt-in alone does not establish ownership. |\n| Index persistence and model cache | Service-selected index output paths and generated temporary names; curated embedding repository names in the operator model cache. |\n| Five legacy-index sidecar operations | All `RAGIndex.index_path` writes originate in `LibraryRAGService`, using an operator cache root and hash-based filenames. Login-time rekey loads those rows from the authenticated user's DB and derives a fixed-suffix sidecar; no request/import endpoint assigns the column. This is a writer/caller boundary, not an assumption that arbitrary stored paths are safe. New writers or callers require another review. |\n| Six index rename/replace operations | Legacy relocation checks the source root and hash filename; quarantine uses service-generated index paths and timestamp suffixes. Atomic replacements move this call's `mkstemp` file to the explicit index output or fixed-suffix migration sidecar. The migration discovers non-symlink pickle files under the operator cache root. Each annotation is confined to its individual mutation; arbitrary stored paths remain reportable. |\n\nThe previously unresolved legacy-library read was fixed separately in\n#7268. Successful report validation is evidence that the scan is complete,\nnot that the release has no remaining alerts.\n\nNative SQL controls run from `.semgrep/tests/ldr-security.py`. The standalone\npath tests invoke the pinned scanner against unsafe and safe cases, including\ndirect request paths, fetched paths, keyword arguments, nested file accesses,\nignored/reassigned validation results, metadata-only response filenames and\nnormalization without file access. Relative-path validation only sanitizes\nthe relative argument: a request-selected base directory remains tainted.\nForty assigned/inline call controls cover trusted and request-selected roots,\nincluding keyword arguments and the default model root. The workflow uses one\nscanner worker to keep taint findings reproducible.\n\n## Rule Template\n\n```yaml\nrules:\n  - id: your-rule-id\n    pattern: |\n      # Your pattern here\n    message: Description of the security issue\n    languages: [python]\n    severity: ERROR  # or WARNING, INFO\n    metadata:\n      category: security\n      cwe: \"CWE-XXX: Description\"\n      owasp: \"AXX:2021 - Category\"\n```\n\n## References\n\n- [Semgrep Documentation](https://semgrep.dev/docs/)\n- [OWASP Top 10 2021](https://owasp.org/Top10/)\n- [CWE Top 25](https://cwe.mitre.org/top25/)\n- [Semgrep Registry](https://semgrep.dev/explore)\n"},"files":{".semgrep/rules/check-safe-requests.yaml":"rules:\n  - id: unsafe-requests-get\n    patterns:\n      - pattern: requests.get(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.get() bypasses SSRF protection.\n      Use safe_get() from security module instead.\n      Import: from ...security import safe_get\n      For localhost services: safe_get(url, allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-post\n    patterns:\n      - pattern: requests.post(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.post() bypasses SSRF protection.\n      Use safe_post() from security module instead.\n      Import: from ...security import safe_post\n      For localhost services: safe_post(url, allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-session\n    patterns:\n      - pattern: requests.Session()\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.Session() bypasses SSRF protection.\n      Use SafeSession() from security module instead.\n      Import: from ...security import SafeSession\n      For localhost services: SafeSession(allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-put\n    patterns:\n      - pattern: requests.put(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.put() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n  - id: unsafe-requests-delete\n    patterns:\n      - pattern: requests.delete(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.delete() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n  - id: unsafe-requests-patch\n    patterns:\n      - pattern: requests.patch(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.patch() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n","docs/architecture.md":"# Architecture Overview\n\nThis document provides detailed technical diagrams of Local Deep Research's architecture.\n\n## System Architecture\n\n```mermaid\nflowchart TB\n    subgraph USER[\"👤 User Interface\"]\n        WEB[Web Browser<br/>localhost:5000]\n    end\n\n    subgraph BACKEND[\"🌐 FastAPI Backend\"]\n        API[REST API + WebSocket]\n        AUTH[Authentication<br/>CSRF Protection]\n        ROUTES[Research Routes]\n    end\n\n    subgraph RESEARCH[\"🔬 Research Engine\"]\n        STRAT[Strategy Selector]\n        QGEN[Question Generator]\n        EXEC[Search Executor]\n        SYNTH[Report Synthesizer]\n    end\n\n    subgraph LLM[\"🤖 LLM Providers\"]\n        direction TB\n        LOCAL_LLM[/\"🏠 Local LLMs\"\\]\n        OLLAMA[Ollama]\n        LMSTUDIO[LM Studio]\n        CLOUD_LLM[/\"☁️ Cloud LLMs\"\\]\n        OPENAI[OpenAI]\n        ANTHROPIC[Anthropic]\n        GEMINI[Google Gemini]\n        OPENROUTER[OpenRouter<br/>100+ models]\n    end\n\n    subgraph SEARCH[\"🔍 Search Engines\"]\n        direction TB\n        LOCAL_SEARCH[/\"🏠 Local Search\"\\]\n        SEARXNG[SearXNG]\n        ELASTIC[Elasticsearch]\n        LIBRARY[Document Library]\n        WEB_SEARCH[/\"🌐 Web Search\"\\]\n        TAVILY[Tavily]\n        BRAVE[Brave]\n        ACADEMIC[/\"📚 Academic\"\\]\n        ARXIV[ArXiv]\n        PUBMED[PubMed]\n        SEMANTIC[Semantic Scholar]\n    end\n\n    subgraph STORAGE[\"💾 Storage Layer\"]\n        SQLCIPHER[(SQLCipher DB<br/>AES-256 Encrypted)]\n        VECTORS[(Vector Store<br/>Embeddings)]\n        FILES[(File Storage<br/>PDFs & Docs)]\n    end\n\n    subgraph OUTPUT[\"📄 Output\"]\n        MD[Markdown]\n        PDF[PDF Export]\n        LATEX[LaTeX]\n        QUARTO[Quarto]\n        RIS[RIS/BibTeX]\n    end\n\n    WEB <--> API\n    API --> AUTH\n    AUTH --> ROUTES\n    ROUTES --> STRAT\n\n    STRAT --> QGEN\n    QGEN --> EXEC\n    EXEC --> SYNTH\n\n    EXEC <--> OLLAMA & LMSTUDIO\n    EXEC <--> OPENAI & ANTHROPIC & GEMINI & OPENROUTER\n\n    EXEC <--> SEARXNG & ELASTIC & LIBRARY\n    EXEC <--> TAVILY & BRAVE\n    EXEC <--> ARXIV & PUBMED & SEMANTIC\n\n    SYNTH --> SQLCIPHER\n    LIBRARY <--> VECTORS\n    LIBRARY <--> FILES\n\n    SYNTH --> MD & PDF & LATEX & QUARTO & RIS\n\n    LOCAL_LLM ~~~ OLLAMA\n    CLOUD_LLM ~~~ OPENAI\n    LOCAL_SEARCH ~~~ SEARXNG\n    WEB_SEARCH ~~~ TAVILY\n    ACADEMIC ~~~ ARXIV\n\n    style USER fill:#e1f5fe\n    style BACKEND fill:#fff3e0\n    style RESEARCH fill:#f3e5f5\n    style LLM fill:#e8f5e9\n    style SEARCH fill:#fce4ec\n    style STORAGE fill:#fff8e1\n    style OUTPUT fill:#e0f2f1\n```\n\n## Research Flow\n\n```mermaid\nflowchart LR\n    subgraph INPUT[\"1️⃣ Input\"]\n        Q[Research Query]\n    end\n\n    subgraph ITERATE[\"2️⃣ Iterative Research\"]\n        direction TB\n        GEN[Generate<br/>Questions]\n        SEARCH[Parallel<br/>Search]\n        ANALYZE[Analyze<br/>Results]\n        GEN --> SEARCH --> ANALYZE\n        ANALYZE -.->|\"Need more info?\"| GEN\n    end\n\n    subgraph SYNTHESIZE[\"3️⃣ Synthesis\"]\n        REPORT[Generate<br/>Report]\n        CITE[Add<br/>Citations]\n        REPORT --> CITE\n    end\n\n    subgraph OUTPUT[\"4️⃣ Output\"]\n        RESULT[Final Report<br/>with Sources]\n    end\n\n    Q --> GEN\n    ANALYZE --> REPORT\n    CITE --> RESULT\n\n    style INPUT fill:#e3f2fd\n    style ITERATE fill:#f3e5f5\n    style SYNTHESIZE fill:#e8f5e9\n    style OUTPUT fill:#fff3e0\n```\n\n## Deployment Options\n\n```mermaid\nflowchart TB\n    subgraph FULL_LOCAL[\"🏠 Self-Hosted (Maximum Privacy)\"]\n        direction LR\n        L_LDR[LDR] <--> L_OLLAMA[Ollama]\n        L_LDR <--> L_SEARX[SearXNG]\n        L_LDR <--> L_DB[(Encrypted DB)]\n    end\n\n    subgraph HYBRID[\"⚡ Hybrid (Balanced)\"]\n        direction LR\n        H_LDR[LDR] <--> H_OLLAMA[Ollama]\n        H_LDR <-->|\"Web Search\"| H_CLOUD[Cloud APIs]\n        H_LDR <--> H_DB[(Encrypted DB)]\n    end\n\n    subgraph CLOUD[\"☁️ Cloud-Powered (Maximum Speed)\"]\n        direction LR\n        C_LDR[LDR] <--> C_OPENAI[OpenAI/Claude]\n        C_LDR <--> C_TAVILY[Tavily/Brave]\n        C_LDR <--> C_DB[(Encrypted DB)]\n    end\n\n    style FULL_LOCAL fill:#e8f5e9\n    style HYBRID fill:#fff3e0\n    style CLOUD fill:#e3f2fd\n```\n\n## Feature Map\n\n```mermaid\nmindmap\n  root((Local Deep<br/>Research))\n    Research\n      Quick Summary\n      Detailed Reports\n      Follow-up Questions\n      Research Strategies\n    Search Sources\n      Web Search\n        SearXNG\n        Tavily\n        Brave\n      Academic\n        ArXiv\n        PubMed\n        Semantic Scholar\n      Local\n        Document Library\n        Collections\n        Elasticsearch\n    LLM Support\n      Local\n        Ollama\n        LM Studio\n      Cloud\n        OpenAI\n        Anthropic\n        Google\n        OpenRouter\n        Requesty\n    Output\n      Markdown\n      PDF\n      LaTeX\n      Quarto\n      RIS/BibTeX\n    Features\n      News Subscriptions\n      Cost Analytics\n      Benchmarking\n      RAG Search\n      Per-User Encryption\n```\n\n## Component Details\n\n### LLM Providers\n\n| Provider | Type | Description |\n|----------|------|-------------|\n| Ollama | Local | Self-hosted open-source models |\n| LM Studio | Local | Desktop app for local models |\n| OpenAI | Cloud | GPT-4, GPT-3.5 |\n| Anthropic | Cloud | Claude 3 family |\n| Google | Cloud | Gemini models |\n| OpenRouter | Cloud | 100+ models via single API |\n| Requesty | Cloud | 100+ models via single API |\n\n### Search Engines\n\n| Engine | Type | Best For |\n|--------|------|----------|\n| SearXNG | Local/Self-hosted | Privacy, aggregated results |\n| Tavily | Cloud | AI-optimized search |\n| ArXiv | Academic | Physics, CS, Math papers |\n| PubMed | Academic | Biomedical research |\n| Semantic Scholar | Academic | Cross-discipline papers |\n| Wikipedia | Knowledge | General knowledge |\n| Your Documents | Local | Private document search |\n\n### Output Formats\n\n| Format | Use Case |\n|--------|----------|\n| Markdown | Default, web display |\n| PDF | Sharing, printing |\n| LaTeX | Academic papers |\n| Quarto | Reproducible documents |\n| RIS/BibTeX | Reference managers |\n\n## Knowledge Loop: Research → Library → Future Research\n\nOne of LDR's powerful features is the ability to build a personal knowledge base that improves future research.\n\n```mermaid\nflowchart TB\n    subgraph RESEARCH[\"1️⃣ Research\"]\n        Q[Your Question] --> Engine[Research Engine]\n        Engine --> Results[Results + Sources]\n    end\n\n    subgraph DOWNLOAD[\"2️⃣ Download\"]\n        Results --> Track[Track Sources]\n        Track --> Queue[Download Queue]\n        Queue --> Extract[Download & Extract Text]\n    end\n\n    subgraph LIBRARY[\"3️⃣ Library\"]\n        Extract --> Store[(Document Storage)]\n        Store --> Organize[Organize into Collections]\n    end\n\n    subgraph INDEX[\"4️⃣ Index\"]\n        Organize --> Chunk[Chunk Documents]\n        Chunk --> Embed[Generate Embeddings]\n        Embed --> FAISS[(Vector Index)]\n    end\n\n    subgraph REUSE[\"5️⃣ Reuse\"]\n        FAISS --> SearchEngine[Collection as Search Engine]\n        SearchEngine --> Q\n    end\n\n    style RESEARCH fill:#e3f2fd\n    style DOWNLOAD fill:#fff3e0\n    style LIBRARY fill:#e8f5e9\n    style INDEX fill:#f3e5f5\n    style REUSE fill:#fce4ec\n```\n\n### How It Works\n\n1. **Research Completes** → Sources are tracked in `ResearchResource` table\n2. **Download Sources** → Click \"Get All Research PDFs\" to queue downloads\n   - Smart downloaders for ArXiv, PubMed, Semantic Scholar, etc.\n   - Automatic text extraction from PDFs\n3. **Build Library** → Documents stored in encrypted database\n   - Deduplication via content hash\n   - Multiple storage modes: database (encrypted), filesystem, text-only\n4. **Create Collections** → Organize documents by topic/project\n   - Each collection can have different embedding settings\n   - Documents can belong to multiple collections\n5. **Index for Search** → Generate vector embeddings\n   - Configurable chunk size and overlap\n   - FAISS index for fast similarity search\n6. **Use in Future Research** → Select collection as search engine\n   - RAG search finds relevant passages\n   - Results cite back to your documents\n\n### Key Components\n\n| Component | Purpose |\n|-----------|---------|\n| `DownloadService` | Manages PDF downloads with source-specific strategies |\n| `LibraryService` | Queries and manages document library |\n| `LibraryRAGService` | Creates vector indices for semantic search |\n| `CollectionSearchEngine` | Searches collections using RAG |\n| `Document` | Stores text content, metadata, file references |\n| `DocumentChunk` | Stores indexed text chunks with embeddings |\n| `Collection` | Groups documents with shared embedding settings |\n\n### Storage Options\n\n| Mode | Security | Use Case |\n|------|----------|----------|\n| Database | AES-256 encrypted | Default, maximum security |\n| Filesystem | Unencrypted | Need external tool access |\n| Text Only | Encrypted text, no PDFs | Minimal storage |\n\n---\n\n## Technical Analysis & Project Health\n\n*Last updated: December 2024*\n\nThis section provides a comprehensive technical analysis of the codebase, including quality metrics, architecture patterns, and project health indicators.\n\n### Project Statistics\n\n| Metric | Count |\n|--------|-------|\n| Test Classes | 809+ |\n| Search Engine Implementations | 25 |\n| LLM Provider Implementations | 9 |\n| Search Strategies | 5 |\n| Abstract Base Classes | 26 |\n| CI/CD Workflows | 57 |\n| Security Scanners in CI | 22+ |\n| Core Dependencies | 63 |\n\n### Architecture Patterns\n\n#### Extensibility Design\n\nThe codebase follows a consistent pattern for extensibility:\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    Abstract Base Classes                     │\n├─────────────────────────────────────────────────────────────┤\n│ BaseSearchEngine      │ Common interface for 25+ engines    │\n│ BaseSearchStrategy    │ Strategy pattern for research       │\n│ BaseCitationHandler   │ Citation processing abstraction     │\n│ BaseQuestionGenerator │ Question generation interface       │\n│ BaseExporter          │ Export format abstraction           │\n└─────────────────────────────────────────────────────────────┘\n```\n\n#### Search Engine Plugin System\n\nNew search engines can be added by:\n1. Creating a class inheriting from `BaseSearchEngine`\n2. Placing it in `web_search_engines/engines/`\n3. Register the engine in `ENGINE_REGISTRY` and add it to the security module allowlist in `security/module_whitelist.py`\n\n```python\n# Example: Adding a new search engine\nclass SearchEngineCustom(BaseSearchEngine):\n    def run(self, query: str) -> List[Dict]:\n        # Implementation\n        pass\n```\n\n#### LLM Provider Integration\n\nSupports 10 LLM providers with auto-discovery:\n- Ollama (local)\n- LM Studio (local)\n- OpenAI\n- Anthropic\n- Google Gemini\n- OpenRouter (100+ models)\n- Requesty (100+ models)\n- DeepSeek\n- Mistral\n- Groq\n\n### Quality Ratings\n\n```mermaid\npie title Project Health Ratings\n    \"Code Quality (86)\" : 86\n    \"Security (93)\" : 93\n    \"Extensibility (93)\" : 93\n    \"Performance (85)\" : 85\n    \"UX/DX (83)\" : 83\n    \"Maintenance (89)\" : 89\n```\n\n#### Detailed Ratings\n\n| Category | Score | Highlights |\n|----------|-------|------------|\n| **Code Quality** | 86/100 | 809+ test classes, ruff/mypy enforcement, comprehensive pre-commit hooks |\n| **Security** | 93/100 | SQLCipher AES-256 encryption, full SHA pinning in CI, 22+ security scanners |\n| **Extensibility** | 93/100 | 26 abstract base classes, plugin architecture, strategy pattern throughout |\n| **Performance** | 85/100 | Adaptive rate limiting, cache stampede protection, parallel search execution |\n| **UX/Developer Experience** | 83/100 | Real-time WebSocket updates, in-tool documentation, comprehensive error handling |\n| **Maintenance** | 89/100 | 57 CI/CD workflows, automated security scanning, structured changelog |\n| **Overall** | 88/100 | Production-ready with excellent security and extensibility |\n\n### Security Architecture\n\n```mermaid\nflowchart TB\n    subgraph SECURITY[\"🔐 Security Layers\"]\n        direction TB\n        AUTH[Signed session cookie<br/>Starlette SessionMiddleware<br/>+ CSRFMiddleware]\n        ENCRYPT[SQLCipher<br/>AES-256 Encryption]\n        SCAN[22+ Security Scanners<br/>CodeQL, Semgrep, Bandit...]\n        PIN[Full SHA Pinning<br/>All 57 CI Workflows]\n    end\n\n    AUTH --> ENCRYPT\n    ENCRYPT --> SCAN\n    SCAN --> PIN\n\n    style SECURITY fill:#e8f5e9\n```\n\n**Security Features:**\n- Per-user encrypted databases (SQLCipher with AES-256)\n- Full GitHub Action SHA pinning (not tag-based)\n- Comprehensive CI security scanning:\n  - CodeQL (Python, JavaScript)\n  - Semgrep (custom rulesets)\n  - Bandit (Python security)\n  - Trivy (container scanning)\n  - Dependency review\n  - Secret scanning\n\n### Performance Optimizations\n\n| Feature | Implementation |\n|---------|----------------|\n| **Parallel Search** | `concurrent.futures.ThreadPoolExecutor` for multi-question search |\n| **Rate Limiting** | Adaptive system with `learning_rate=0.3` |\n| **Cache Protection** | `fetch_events` + `fetch_locks` for stampede prevention |\n| **Progress Streaming** | Socket.IO ASGI sub-app mounted at `/ws` for real-time UI updates |\n| **Cross-Engine Filtering** | LLM-powered relevance scoring and deduplication |\n\n#### Thread & Resource Lifecycle\n\nUnlike typical web apps that share a single database connection pool, this application maintains **separate database engines per user** because each user has their own encrypted SQLite file with a unique SQLCipher key ([SQLCipher](https://www.zetetic.net/sqlcipher/) is an encrypted extension of SQLite). This creates a threading challenge: every background thread needs its own engine for the user it serves.\n\nA single **QueuePool** engine is kept per user (`pool_size=20`,\n`max_overflow=40`, `pool_timeout=10`) in\n`DatabaseManager.connections[username]`. It is created with\n`check_same_thread=False`, so it is shared safely across the AnyIO\nworker threads that serve synchronous route handlers AND background\nthreads (research workers, scheduler jobs, metric writers). All of these acquire sessions from the\nsame pool, which keeps FD usage bounded by `pool_size + max_overflow`\nper user.\n\n> Earlier revisions of the code maintained a second, parallel system of\n> per-`(username, thread_id)` **NullPool** engines for background metric\n> writes. That system was removed because orphaned entries leaked\n> SQLCipher + WAL file handles (3 FDs per active connection) under load,\n> eventually exhausting the 1024 FD soft limit. Routing all work through\n> the single per-user QueuePool made FD usage bounded and let us delete\n> ~200 lines of thread-engine bookkeeping.\n\n##### Resource Cleanup Layers\n\nCleanup is defense-in-depth with multiple layers:\n\n| Layer | Trigger | What it cleans |\n|-------|---------|----------------|\n| `@thread_cleanup` decorator | Thread function exit (normal or exception) | DB session (returned to per-user pool), settings context, search context |\n| `WorkerCleanupAPIRoute` (`web/dependencies/threadpool.py`) | Synchronous FastAPI endpoint exit (normal or exception) | Sessions and ambient thread-local state on the AnyIO worker that served the endpoint |\n| `WorkerCleanupStreamingResponse` (`web/dependencies/threadpool.py`) | Each synchronous streamed-response iterator step (chunk, completion, or exception) | Sessions and ambient thread-local state on whichever AnyIO worker advanced that step |\n| `finally` blocks | Per-research in `run_research_process()` | Search engine HTTP sessions, strategy thread pool executors |\n| `DatabaseMiddleware` `finally` block (`web/fastapi_app.py`) | After each HTTP request | Sweeps dead workers and cleans event-loop-owned state; synchronous endpoint and stream-iterator state are reclaimed on their owning workers by the two wrappers above |\n| Periodic pool dispose | Every 30 min | Calls `engine.dispose()` on per-user QueuePool engines to release SQLCipher+WAL file handles that accumulate from out-of-order connection closes |\n| Logout cascade | User logout | Scheduler unregister (removes password) → DB close (`engine.dispose()`) → session destroy |\n| Stale session cleanup | `clear_session_if_unrecoverable()`, called from `require_auth()` when the DB connection is missing (`web/dependencies/auth.py`) | Clears the signed session cookie for users whose DB connection is gone and who have no recovery credential. Not sampled/throttled — unlike the old `before_request` hook it only runs on an already-failed auth check |\n\n##### Research Thread Lifecycle\n\n```mermaid\nflowchart TD\n    A[\"POST /api/start_research\"] --> B{\"Slots available?\"}\n    B -- \"Yes (direct)\" --> C[\"start_research_process()\"]\n    B -- \"No (queued)\" --> D[\"Database queue\"]\n    D --> E[\"_process_queue_loop<br/>(daemon thread, 10s poll)\"]\n    E --> C\n\n    C --> F[\"Research thread<br/>(daemon, semaphore-gated)\"]\n    F --> G[\"@thread_cleanup wraps<br/>run_research_process()\"]\n\n    G --> H[\"Strategy.find_relevant_information()\"]\n    H --> I[\"ThreadPoolExecutor sub-tasks<br/>(within strategies only)\"]\n    I --> J[\"Worker @thread_cleanup:<br/>close worker DB session\"]\n\n    J --> K[\"finally: search_engine.close()<br/>system.close() → strategy.close()\"]\n    K --> L[\"Main @thread_cleanup:<br/>close DB session, return connection to pool\"]\n\n    subgraph sweep [\"Defense-in-depth (independent timers)\"]\n        M[\"Dead-thread credential sweep<br/>(~60s via processor_v2)\"] -.-> N[\"Remove stale credentials<br/>for dead thread IDs\"]\n        O[\"Pool dispose<br/>(30 min via connection_cleanup)\"] -.-> P[\"engine.dispose() on all<br/>QueuePool engines\"]\n    end\n\n    style sweep fill:#f0f0f0,stroke:#999\n```\n\n##### FD Budget\n\nEach SQLCipher connection in WAL (Write-Ahead Logging) mode uses **2 file descriptors** (main db + WAL file). All connections to the same database within a process share **1 SHM** (shared-memory) file descriptor. The formula per user database is: `connections × 2 + 1`.\n\n| Component | FD Formula | With defaults |\n|-----------|-----------|---------------|\n| QueuePool (steady state) | `logged_in_users × (pool_size × 2 + 1)` | `users × 41` FDs |\n| QueuePool (peak) | `logged_in_users × ((pool_size + max_overflow) × 2 + 1)` | `users × 121` FDs |\n\nDefault Linux ulimit is 1024 soft (bare metal), which is tight for multi-user deployments. Docker's daemon default (typically 1M+) is adequate. QueuePool engines are created at login and disposed at logout, so only active users consume FDs.\n\nFor more on diagnosing FD exhaustion, see [Troubleshooting - Resource Exhaustion](./troubleshooting.md#resource-exhaustion).\n\n##### Key Files\n\n| File | Role |\n|------|------|\n| `src/local_deep_research/database/encrypted_db.py` | `DatabaseManager`, engine lifecycle, pool management |\n| `src/local_deep_research/database/thread_local_session.py` | `@thread_cleanup` decorator, thread-local sessions, credential cleanup |\n| `src/local_deep_research/web/fastapi_app.py` | `DatabaseMiddleware` per-request cleanup (replaces Flask's `teardown_appcontext`), lifespan orchestration |\n| `src/local_deep_research/web/services/research_service.py` | Research thread creation, `run_research_process()` |\n| `src/local_deep_research/web/queue/processor_v2.py` | Queue processing, credential cleanup trigger |\n\n### Areas for Improvement\n\nWhile the project scores highly overall, these areas have room for growth:\n\n1. **Integration Testing** - More end-to-end tests for full research workflows\n2. **API Documentation** - OpenAPI/Swagger spec for REST endpoints\n3. **Metrics Dashboard** - Prometheus/Grafana integration for monitoring\n4. **Resource Observability** - Expose FD count, thread count, and connection pool stats in /api/v1/health; add periodic sweep logging\n5. **Async Architecture** - Migration to async/await for I/O-bound operations\n\n### Key Source Files\n\n| Component | Location | Purpose |\n|-----------|----------|---------|\n| Research Engine | `src/local_deep_research/search_system.py` | Main `AdvancedSearchSystem` class |\n| Strategies | `src/local_deep_research/advanced_search_system/strategies/` | Research strategy implementations |\n| Search Engines | `src/local_deep_research/web_search_engines/engines/` | 25 search engine implementations |\n| Report Generation | `src/local_deep_research/report_generator.py` | `IntegratedReportGenerator` |\n| Web API | `src/local_deep_research/web/routers/` | FastAPI route handlers (`web/routes/` now holds a few shared helpers only, not route handlers) |\n| Database | `src/local_deep_research/database/` | SQLCipher models and migrations |\n| Encrypted DB | `src/local_deep_research/database/encrypted_db.py` | Per-user SQLCipher engine lifecycle |\n| Thread Sessions | `src/local_deep_research/database/thread_local_session.py` | Thread-safe session management and cleanup |\n| Settings | `src/local_deep_research/config/` | Configuration and LLM setup |\n\n### Contributing to Architecture\n\nWhen extending the system:\n\n1. **Adding Search Engines**: Inherit from `BaseSearchEngine`, implement `run()` method\n2. **Adding Strategies**: Inherit from `BaseSearchStrategy`, implement `analyze_topic()` method\n3. **Adding LLM Providers**: Add to `config/llm_config.py` with proper initialization\n4. **Adding Export Formats**: Inherit from base exporter pattern in `utilities/`\n\nSee [CONTRIBUTING.md](../CONTRIBUTING.md) for detailed guidelines.\n",".semgrep/rules/README.md":"# LDR Custom Security Rules\n\nThis directory contains custom Semgrep security rules specific to Local Deep Research (LDR).\n\n## Rules Overview\n\n### ldr-security.yaml\n\nLDR-specific security rules covering:\n\n1. **Hardcoded Secrets**\n   - Detects API keys, passwords, tokens in source code\n   - Severity: ERROR\n   - CWE-798\n\n2. **SQL Injection Prevention**\n   - `sql-string-concatenation`: follows dynamically constructed strings\n     (f-string, `+`, `%`, `.format()`) to database execution and SQLAlchemy\n     raw-SQL constructors, including inline expressions, aliases and stored\n     values. Named SQL builders also prompt review. Bound SQLAlchemy values\n     remain clear, including mapping values, stored strings without\n     reassignment, and reused `bindparams()` statements. Column identifiers\n     built from an inline or unchanged stored f-string remain clear when\n     quoting is enabled. Unpacked arguments, `quoted_name` overrides,\n     reassigned names, explicit or unknown literal flags, and nested raw-SQL\n     constructors remain reportable. Only the name occurrence is cleared;\n     an unsafe table/selectable argument still reaches execution.\n   - `sql-execute-with-string-format`: `$X.execute(f\"...\")` or\n     `$X.execute(\"...\" + ...)`\n   - Enforces parameterized queries via SQLAlchemy\n   - Severity: ERROR\n   - CWE-89\n\n3. **Code Injection**\n   - Detects dangerous use of eval/exec\n   - Prevents arbitrary code execution\n   - Severity: ERROR\n   - CWE-95\n\n4. **Command Injection**\n   - Detects unsafe use of os.system, shell=True\n   - Enforces subprocess with argument lists\n   - Severity: ERROR\n   - CWE-78\n\n5. **Path Traversal**\n   - `path-traversal-risk`: follows dynamic path construction and fetched\n     database values to file operations, directory enumeration and file\n     responses. A stored path served directly is covered without requiring\n     an intermediate `Path(...)`. Construction or normalization alone is\n     not a file operation. Mutation sinks include `Path.touch`, both paths\n     in `rename`/`replace`, and positional/keyword `os.remove`/`os.unlink`.\n     The one-target `Path.replace` signature keeps string and datetime\n     replacements outside the filesystem rule.\n   - `request-path-traversal-risk`: follows route handler parameters and\n     `request.*` to path construction and filesystem sinks. Database fetch\n     results are handled by the separate dynamic-path rule, so filtering a\n     query by a request ID does not classify the whole stored row as request\n     input. A response's download-name metadata is not its filesystem path.\n   - Only returned values from `validate_safe_path`, `validate_data_path`,\n     `validate_model_path`, `safe_join`, `secure_filename` and\n     `sanitize_filename` clear request taint. Exact call and assignment\n     patterns preserve findings for file access inside their arguments and\n     for later use of the original input. Other `PathValidator` methods and\n     `os.path.basename` are not sanitizers (`basename(\"..\")` is still `\"..\"`).\n   - Intraprocedural only: input is not followed into helper bodies. The\n     sanitizer contracts assume a trusted base directory; they do not prove\n     ownership or protect against concurrent local filesystem changes.\n     These rules prompt a boundary review rather than establish safety.\n   - Severity: WARNING\n   - CWE-22\n\n6. **Unsafe Deserialization**\n   - Detects unsafe YAML/pickle loading\n   - Prevents code execution via deserialization\n   - Severity: ERROR\n   - CWE-502\n\n7. **Weak Randomness**\n   - Detects use of random module for security\n   - Enforces secrets module for crypto operations\n   - Severity: WARNING\n   - CWE-338\n\n8. **Debug Mode in Production**\n   - Detects Flask debug=True\n   - Prevents information disclosure\n   - Severity: ERROR\n   - CWE-489\n\n9. **SSRF Prevention**\n   - Detects URL fetching operations\n   - Reminds to validate URLs\n   - Severity: WARNING\n   - CWE-918\n\n10. **XSS Prevention**\n    - Detects user input in HTML context\n    - Enforces proper escaping\n    - Severity: WARNING\n    - CWE-79\n\n11. **CSRF Protection**\n    - Detects POST endpoints\n    - Reminds to enable CSRF protection\n    - Severity: INFO\n    - CWE-352\n\n12. **Credential Logging**\n    - Detects passwords in log statements\n    - Prevents credential disclosure\n    - Severity: ERROR\n    - CWE-532\n\n## Usage\n\nThese rules are automatically run by the Semgrep CI/CD workflow:\n\n```bash\n# Run locally\nsemgrep --config=.semgrep/rules/ src/\n\n# Run with standard rules\nsemgrep --config=p/security-audit --config=.semgrep/rules/ src/\n```\n\n## GitHub upload and source suppressions\n\nThe pinned Semgrep 1.179.0 omits `nosemgrep` findings from its JSON report but retains\nthem in native SARIF with `inSource` suppression metadata. GitHub's SARIF\nimport does not honor that metadata, so uploading the native report would\npublish the ignored findings as open alerts.\n\nThe workflow validates the complete native JSON and SARIF reports, then\ncreates `semgrep-results.github.sarif`. Only results with recognized\n`inSource` suppressions (no status, or `accepted`) are omitted from that\ncopy, and only when the scanned file holds a reviewed annotation naming the\nresult's rule (see below) on the result's first line or on the line above\nit. The SARIF suppression does not record which rule a comment named, so\nthe validator reads the line back; a suppressed result without such an\nannotation fails validation. Unknown suppression kinds or statuses fail\nvalidation too. All other\nfindings and scan metadata are preserved, and the upload copy is validated\nagain. Scanner errors, skipped rules, empty scope and mismatched reports\nstill prevent upload.\n\nThe diagnostic artifact retains both native reports and the GitHub upload\ncopy for 7 days. Tests using native scanner reports and synthetic fixtures check that\nsuppressing one result preserves every unsuppressed result, including the\nsame rule at other locations and other rules at the same location; the\nreal-scanner test runs three rules on one line, annotates one or two of\nthem, and checks that the strict scan stays error-free and the rest are\nuploaded.\n\nReferences: [Semgrep source suppressions](https://docs.semgrep.dev/ignoring-files-folders-code)\nand [GitHub's supported SARIF properties](https://docs.github.com/en/code-security/reference/code-scanning/sarif-files/sarif-support).\n\n## Adding New Rules\n\nTo add new custom rules:\n\n1. Create a new YAML file in `.semgrep/rules/`\n2. Follow [Semgrep rule syntax](https://semgrep.dev/docs/writing-rules/overview/)\n3. Test the rule: `semgrep --config=.semgrep/rules/your-rule.yaml src/`\n4. Document the rule in this README\n\n## Reviewed source annotations\n\nUse a rule-specific reviewed annotation beside a reviewed false positive,\nat the end of the finding's first line or alone on the line above it:\n\n```python\n# nosemgrep: semgrep.rules.weak-random-generation, reason: Non-security jitter\ndelay_seconds = random.randint(1, 30)\n```\n\nWrite it exactly as `nosemgrep: <rule-id>[, <rule-id>]..., reason: <reason>`,\nstarting a comment (`#`, `//`, `{#`, `<!--` or `/*`), with fully qualified\nrule ids (list several when one line has several reviewed false positives)\nand no comma in the reason. Semgrep 1.177.0 honours a case-insensitive\n`nosem` marker after a space anywhere on a line, including string literals\nand words such as \"NoSemantics\", and a marker without a rule id hides every\nrule on that line. Semgrep also accepts any suffix of a rule id and reads\nthe first word of each comma-separated item after the colon as a rule id.\nThe `check-nosemgrep-annotations` pre-commit hook rejects every other marker\nunder `src/`, including one inside a Python string or docstring, and the\nupload validator rejects a suppressed result without an annotation naming\nits exact rule id.\n\nThe `reason:` item is required, not decoration. Semgrep reads it as one\nmore id; `reason:` is not a valid rule id, so it suppresses nothing. But a\ncomment with a single id makes Semgrep 1.177.0 report a warning for every\nother rule that matches the same line (\"found 'nosem' comment with id ...,\nbut no corresponding rule trying ...\"), and under `--strict` that fails the\nwhole scan, and so the release gate. Semgrep skips that warning when a\ncomment holds two or more ids, so with the `reason:` item a new or updated\nrule matching an annotated line produces an ordinary uploaded alert\ninstead. If that alert is another reviewed false positive, add its rule id\nto the annotation.\n\nKeep the rule enabled elsewhere; do not exclude a file or weaken the release\nthreshold to clear an alert. Re-review the annotation when the expression or\nits input boundary changes.\n\nReviews in October 2026 identified these cases:\n\n| Finding | Why the annotated use is safe | Existing verification |\n| --- | --- | --- |\n| Seven script interpolations | Jinja's `tojson` escapes HTML delimiters before embedding JSON in a script. | Template injection and JSON rendering tests in `tests/security/test_injection_and_template_safety.py`. |\n| Eleven unquoted attribute expressions | Both branches emit fixed `selected`/`checked` tokens or the empty string; no input is inserted into an attribute name. | Inspect the literal branches in `embedding_settings.html`; the rest of each attribute remains quoted. |\n| `help_tip`'s `safe` filter | Every current caller passes a source literal, intentionally containing help markup. | Review all call sites. The existing security test catches simple variable arguments, but does not fully parse composed Jinja expressions. |\n| Four `random` calls | They control subscription jitter or rate-limit exploration, not credentials, identifiers or authorization. | Scheduler and rate-limit tests; cryptographic operations still use `secrets`. |\n| Two apparent secrets | One is an empty API-key default; the other is the existing placeholder only used for unencrypted databases. | Configuration and authentication tests retain the surrounding behavior. |\n| Two pickle calls | They raise `UnpicklingError` on rejected globals/extensions rather than loading data. | `tests/vector_stores/test_faiss_safe_load_security.py` exercises hostile payloads, extension caches, persistent IDs and legitimate docstores. |\n| XML import | It imports `xml.sax.saxutils.escape` to encode text, not an XML parser. | Note-AI prompt rendering tests. |\n| Evernote `TreeBuilder` import | ElementTree's `TreeBuilder` only assembles elements. The Evernote loader's XML is read by defusedxml's parser, which refuses entity declarations, external references and any DOCTYPE with an internal subset. | `tests/document_loaders/test_evernote_extraction.py`. |\n| TOML loader's `Path(source)` | The app builds the loader only in `load_from_bytes`, from the temporary file it has just written; an upload supplies that file's bytes, not its path. No route passes a request value to `get_loader_for_path` or to the loader. | Review the callers of `load_from_bytes` and `get_loader_for_path`. `tests/document_loaders/test_toml_loader.py` loads uploads through `load_from_bytes`. |\n\nThe separate DevSkim annotation in `services/ui.js` covers its fragment-based\nanchor renderer, which already adds both `noopener` and `noreferrer`.\n\nThe combined SQL/path rules carry 100 individually reviewed dispositions.\nThey remain local to the reported operation; no source file, directory or\nrule is excluded:\n\n| Findings | Reviewed boundary |\n| --- | --- |\n| 30 SQL operations | Hex-encoded or explicitly checked SQLCipher keys; typed/range-checked or allowlisted PRAGMAs; dialect-quoted identifiers; migration table/column literals; bound row values; a fixed schema-version integer. |\n| Benchmark and indexing APIs | Explicit local Python/CLI output or input paths. The web benchmark worker uses its separate database runner rather than these file-output functions. |\n| Package and startup files | Fixed filenames and globs beneath installed package paths or operator-configured application directories. |\n| Upload and converter temporary files | Server-generated temporary names, allowlisted extensions and validated stems; web upload callers do not provide scratch directories. |\n| Backup retention cleanup | Fixed-pattern glob results filtered by `is_safe_glob_result` under the server-owned, hashed per-user backup directory. Reordering the checked list to preserve the newly finalized backup does not admit another path. |\n| Library/report cleanup | Existing root resolution and symlink checks at the destructive operation; document deletion callers require an allowed root. Operator opt-in alone does not establish ownership. |\n| Index persistence and model cache | Service-selected index output paths and generated temporary names; curated embedding repository names in the operator model cache. |\n| Five legacy-index sidecar operations | All `RAGIndex.index_path` writes originate in `LibraryRAGService`, using an operator cache root and hash-based filenames. Login-time rekey loads those rows from the authenticated user's DB and derives a fixed-suffix sidecar; no request/import endpoint assigns the column. This is a writer/caller boundary, not an assumption that arbitrary stored paths are safe. New writers or callers require another review. |\n| Six index rename/replace operations | Legacy relocation checks the source root and hash filename; quarantine uses service-generated index paths and timestamp suffixes. Atomic replacements move this call's `mkstemp` file to the explicit index output or fixed-suffix migration sidecar. The migration discovers non-symlink pickle files under the operator cache root. Each annotation is confined to its individual mutation; arbitrary stored paths remain reportable. |\n\nThe previously unresolved legacy-library read was fixed separately in\n#7268. Successful report validation is evidence that the scan is complete,\nnot that the release has no remaining alerts.\n\nNative SQL controls run from `.semgrep/tests/ldr-security.py`. The standalone\npath tests invoke the pinned scanner against unsafe and safe cases, including\ndirect request paths, fetched paths, keyword arguments, nested file accesses,\nignored/reassigned validation results, metadata-only response filenames and\nnormalization without file access. Relative-path validation only sanitizes\nthe relative argument: a request-selected base directory remains tainted.\nForty assigned/inline call controls cover trusted and request-selected roots,\nincluding keyword arguments and the default model root. The workflow uses one\nscanner worker to keep taint findings reproducible.\n\n## Rule Template\n\n```yaml\nrules:\n  - id: your-rule-id\n    pattern: |\n      # Your pattern here\n    message: Description of the security issue\n    languages: [python]\n    severity: ERROR  # or WARNING, INFO\n    metadata:\n      category: security\n      cwe: \"CWE-XXX: Description\"\n      owasp: \"AXX:2021 - Category\"\n```\n\n## References\n\n- [Semgrep Documentation](https://semgrep.dev/docs/)\n- [OWASP Top 10 2021](https://owasp.org/Top10/)\n- [CWE Top 25](https://cwe.mitre.org/top25/)\n- [Semgrep Registry](https://semgrep.dev/explore)\n"},"items":[{"name":"architecture.md","path":"docs/architecture.md","rawUrl":"https://raw.githubusercontent.com/LearningCircuit/local-deep-research/HEAD/docs/architecture.md","title":"docs - System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# Architecture Overview\n\nThis document provides detailed technical diagrams of Local Deep Research's architecture.\n\n## System Architecture\n\n```mermaid\nflowchart TB\n    subgraph USER[\"👤 User Interface\"]\n        WEB[Web Browser<br/>localhost:5000]\n    end\n\n    subgraph BACKEND[\"🌐 FastAPI Backend\"]\n        API[REST API + WebSocket]\n        AUTH[Authentication<br/>CSRF Protection]\n        ROUTES[Research Routes]\n    end\n\n    subgraph RESEARCH[\"🔬 Research Engine\"]\n        STRAT[Strategy Selector]\n        QGEN[Question Generator]\n        EXEC[Search Executor]\n        SYNTH[Report Synthesizer]\n    end\n\n    subgraph LLM[\"🤖 LLM Providers\"]\n        direction TB\n        LOCAL_LLM[/\"🏠 Local LLMs\"\\]\n        OLLAMA[Ollama]\n        LMSTUDIO[LM Studio]\n        CLOUD_LLM[/\"☁️ Cloud LLMs\"\\]\n        OPENAI[OpenAI]\n        ANTHROPIC[Anthropic]\n        GEMINI[Google Gemini]\n        OPENROUTER[OpenRouter<br/>100+ models]\n    end\n\n    subgraph SEARCH[\"🔍 Search Engines\"]\n        direction TB\n        LOCAL_SEARCH[/\"🏠 Local Search\"\\]\n        SEARXNG[SearXNG]\n        ELASTIC[Elasticsearch]\n        LIBRARY[Document Library]\n        WEB_SEARCH[/\"🌐 Web Search\"\\]\n        TAVILY[Tavily]\n        BRAVE[Brave]\n        ACADEMIC[/\"📚 Academic\"\\]\n        ARXIV[ArXiv]\n        PUBMED[PubMed]\n        SEMANTIC[Semantic Scholar]\n    end\n\n    subgraph STORAGE[\"💾 Storage Layer\"]\n        SQLCIPHER[(SQLCipher DB<br/>AES-256 Encrypted)]\n        VECTORS[(Vector Store<br/>Embeddings)]\n        FILES[(File Storage<br/>PDFs & Docs)]\n    end\n\n    subgraph OUTPUT[\"📄 Output\"]\n        MD[Markdown]\n        PDF[PDF Export]\n        LATEX[LaTeX]\n        QUARTO[Quarto]\n        RIS[RIS/BibTeX]\n    end\n\n    WEB <--> API\n    API --> AUTH\n    AUTH --> ROUTES\n    ROUTES --> STRAT\n\n    STRAT --> QGEN\n    QGEN --> EXEC\n    EXEC --> SYNTH\n\n    EXEC <--> OLLAMA & LMSTUDIO\n    EXEC <--> OPENAI & ANTHROPIC & GEMINI & OPENROUTER\n\n    EXEC <--> SEARXNG & ELASTIC & LIBRARY\n    EXEC <--> TAVILY & BRAVE\n    EXEC <--> ARXIV & PUBMED & SEMANTIC\n\n    SYNTH --> SQLCIPHER\n    LIBRARY <--> VECTORS\n    LIBRARY <--> FILES\n\n    SYNTH --> MD & PDF & LATEX & QUARTO & RIS\n\n    LOCAL_LLM ~~~ OLLAMA\n    CLOUD_LLM ~~~ OPENAI\n    LOCAL_SEARCH ~~~ SEARXNG\n    WEB_SEARCH ~~~ TAVILY\n    ACADEMIC ~~~ ARXIV\n\n    style USER fill:#e1f5fe\n    style BACKEND fill:#fff3e0\n    style RESEARCH fill:#f3e5f5\n    style LLM fill:#e8f5e9\n    style SEARCH fill:#fce4ec\n    style STORAGE fill:#fff8e1\n    style OUTPUT fill:#e0f2f1\n```\n\n## Research Flow\n\n```mermaid\nflowchart LR\n    subgraph INPUT[\"1️⃣ Input\"]\n        Q[Research Query]\n    end\n\n    subgraph ITERATE[\"2️⃣ Iterative Research\"]\n        direction TB\n        GEN[Generate<br/>Questions]\n        SEARCH[Parallel<br/>Search]\n        ANALYZE[Analyze<br/>Results]\n        GEN --> SEARCH --> ANALYZE\n        ANALYZE -.->|\"Need more info?\"| GEN\n    end\n\n    subgraph SYNTHESIZE[\"3️⃣ Synthesis\"]\n        REPORT[Generate<br/>Report]\n        CITE[Add<br/>Citations]\n        REPORT --> CITE\n    end\n\n    subgraph OUTPUT[\"4️⃣ Output\"]\n        RESULT[Final Report<br/>with Sources]\n    end\n\n    Q --> GEN\n    ANALYZE --> REPORT\n    CITE --> RESULT\n\n    style INPUT fill:#e3f2fd\n    style ITERATE fill:#f3e5f5\n    style SYNTHESIZE fill:#e8f5e9\n    style OUTPUT fill:#fff3e0\n```\n\n## Deployment Options\n\n```mermaid\nflowchart TB\n    subgraph FULL_LOCAL[\"🏠 Self-Hosted (Maximum Privacy)\"]\n        direction LR\n        L_LDR[LDR] <--> L_OLLAMA[Ollama]\n        L_LDR <--> L_SEARX[SearXNG]\n        L_LDR <--> L_DB[(Encrypted DB)]\n    end\n\n    subgraph HYBRID[\"⚡ Hybrid (Balanced)\"]\n        direction LR\n        H_LDR[LDR] <--> H_OLLAMA[Ollama]\n        H_LDR <-->|\"Web Search\"| H_CLOUD[Cloud APIs]\n        H_LDR <--> H_DB[(Encrypted DB)]\n    end\n\n    subgraph CLOUD[\"☁️ Cloud-Powered (Maximum Speed)\"]\n        direction LR\n        C_LDR[LDR] <--> C_OPENAI[OpenAI/Claude]\n        C_LDR <--> C_TAVILY[Tavily/Brave]\n        C_LDR <--> C_DB[(Encrypted DB)]\n    end\n\n    style FULL_LOCAL fill:#e8f5e9\n    style HYBRID fill:#fff3e0\n    style CLOUD fill:#e3f2fd\n```\n\n## Feature Map\n\n```mermaid\nmindmap\n  root((Local Deep<br/>Research))\n    Research\n      Quick Summary\n      Detailed Reports\n      Follow-up Questions\n      Research Strategies\n    Search Sources\n      Web Search\n        SearXNG\n        Tavily\n        Brave\n      Academic\n        ArXiv\n        PubMed\n        Semantic Scholar\n      Local\n        Document Library\n        Collections\n        Elasticsearch\n    LLM Support\n      Local\n        Ollama\n        LM Studio\n      Cloud\n        OpenAI\n        Anthropic\n        Google\n        OpenRouter\n        Requesty\n    Output\n      Markdown\n      PDF\n      LaTeX\n      Quarto\n      RIS/BibTeX\n    Features\n      News Subscriptions\n      Cost Analytics\n      Benchmarking\n      RAG Search\n      Per-User Encryption\n```\n\n## Component Details\n\n### LLM Providers\n\n| Provider | Type | Description |\n|----------|------|-------------|\n| Ollama | Local | Self-hosted open-source models |\n| LM Studio | Local | Desktop app for local models |\n| OpenAI | Cloud | GPT-4, GPT-3.5 |\n| Anthropic | Cloud | Claude 3 family |\n| Google | Cloud | Gemini models |\n| OpenRouter | Cloud | 100+ models via single API |\n| Requesty | Cloud | 100+ models via single API |\n\n### Search Engines\n\n| Engine | Type | Best For |\n|--------|------|----------|\n| SearXNG | Local/Self-hosted | Privacy, aggregated results |\n| Tavily | Cloud | AI-optimized search |\n| ArXiv | Academic | Physics, CS, Math papers |\n| PubMed | Academic | Biomedical research |\n| Semantic Scholar | Academic | Cross-discipline papers |\n| Wikipedia | Knowledge | General knowledge |\n| Your Documents | Local | Private document search |\n\n### Output Formats\n\n| Format | Use Case |\n|--------|----------|\n| Markdown | Default, web display |\n| PDF | Sharing, printing |\n| LaTeX | Academic papers |\n| Quarto | Reproducible documents |\n| RIS/BibTeX | Reference managers |\n\n## Knowledge Loop: Research → Library → Future Research\n\nOne of LDR's powerful features is the ability to build a personal knowledge base that improves future research.\n\n```mermaid\nflowchart TB\n    subgraph RESEARCH[\"1️⃣ Research\"]\n        Q[Your Question] --> Engine[Research Engine]\n        Engine --> Results[Results + Sources]\n    end\n\n    subgraph DOWNLOAD[\"2️⃣ Download\"]\n        Results --> Track[Track Sources]\n        Track --> Queue[Download Queue]\n        Queue --> Extract[Download & Extract Text]\n    end\n\n    subgraph LIBRARY[\"3️⃣ Library\"]\n        Extract --> Store[(Document Storage)]\n        Store --> Organize[Organize into Collections]\n    end\n\n    subgraph INDEX[\"4️⃣ Index\"]\n        Organize --> Chunk[Chunk Documents]\n        Chunk --> Embed[Generate Embeddings]\n        Embed --> FAISS[(Vector Index)]\n    end\n\n    subgraph REUSE[\"5️⃣ Reuse\"]\n        FAISS --> SearchEngine[Collection as Search Engine]\n        SearchEngine --> Q\n    end\n\n    style RESEARCH fill:#e3f2fd\n    style DOWNLOAD fill:#fff3e0\n    style LIBRARY fill:#e8f5e9\n    style INDEX fill:#f3e5f5\n    style REUSE fill:#fce4ec\n```\n\n### How It Works\n\n1. **Research Completes** → Sources are tracked in `ResearchResource` table\n2. **Download Sources** → Click \"Get All Research PDFs\" to queue downloads\n   - Smart downloaders for ArXiv, PubMed, Semantic Scholar, etc.\n   - Automatic text extraction from PDFs\n3. **Build Library** → Documents stored in encrypted database\n   - Deduplication via content hash\n   - Multiple storage modes: database (encrypted), filesystem, text-only\n4. **Create Collections** → Organize documents by topic/project\n   - Each collection can have different embedding settings\n   - Documents can belong to multiple collections\n5. **Index for Search** → Generate vector embeddings\n   - Configurable chunk size and overlap\n   - FAISS index for fast similarity search\n6. **Use in Future Research** → Select collection as search engine\n   - RAG search finds relevant passages\n   - Results cite back to your documents\n\n### Key Components\n\n| Component | Purpose |\n|-----------|---------|\n| `DownloadService` | Manages PDF downloads with source-specific strategies |\n| `LibraryService` | Queries and manages document library |\n| `LibraryRAGService` | Creates vector indices for semantic search |\n| `CollectionSearchEngine` | Searches collections using RAG |\n| `Document` | Stores text content, metadata, file references |\n| `DocumentChunk` | Stores indexed text chunks with embeddings |\n| `Collection` | Groups documents with shared embedding settings |\n\n### Storage Options\n\n| Mode | Security | Use Case |\n|------|----------|----------|\n| Database | AES-256 encrypted | Default, maximum security |\n| Filesystem | Unencrypted | Need external tool access |\n| Text Only | Encrypted text, no PDFs | Minimal storage |\n\n---\n\n## Technical Analysis & Project Health\n\n*Last updated: December 2024*\n\nThis section provides a comprehensive technical analysis of the codebase, including quality metrics, architecture patterns, and project health indicators.\n\n### Project Statistics\n\n| Metric | Count |\n|--------|-------|\n| Test Classes | 809+ |\n| Search Engine Implementations | 25 |\n| LLM Provider Implementations | 9 |\n| Search Strategies | 5 |\n| Abstract Base Classes | 26 |\n| CI/CD Workflows | 57 |\n| Security Scanners in CI | 22+ |\n| Core Dependencies | 63 |\n\n### Architecture Patterns\n\n#### Extensibility Design\n\nThe codebase follows a consistent pattern for extensibility:\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    Abstract Base Classes                     │\n├─────────────────────────────────────────────────────────────┤\n│ BaseSearchEngine      │ Common interface for 25+ engines    │\n│ BaseSearchStrategy    │ Strategy pattern for research       │\n│ BaseCitationHandler   │ Citation processing abstraction     │\n│ BaseQuestionGenerator │ Question generation interface       │\n│ BaseExporter          │ Export format abstraction           │\n└─────────────────────────────────────────────────────────────┘\n```\n\n#### Search Engine Plugin System\n\nNew search engines can be added by:\n1. Creating a class inheriting from `BaseSearchEngine`\n2. Placing it in `web_search_engines/engines/`\n3. Register the engine in `ENGINE_REGISTRY` and add it to the security module allowlist in `security/module_whitelist.py`\n\n```python\n# Example: Adding a new search engine\nclass SearchEngineCustom(BaseSearchEngine):\n    def run(self, query: str) -> List[Dict]:\n        # Implementation\n        pass\n```\n\n#### LLM Provider Integration\n\nSupports 10 LLM providers with auto-discovery:\n- Ollama (local)\n- LM Studio (local)\n- OpenAI\n- Anthropic\n- Google Gemini\n- OpenRouter (100+ models)\n- Requesty (100+ models)\n- DeepSeek\n- Mistral\n- Groq\n\n### Quality Ratings\n\n```mermaid\npie title Project Health Ratings\n    \"Code Quality (86)\" : 86\n    \"Security (93)\" : 93\n    \"Extensibility (93)\" : 93\n    \"Performance (85)\" : 85\n    \"UX/DX (83)\" : 83\n    \"Maintenance (89)\" : 89\n```\n\n#### Detailed Ratings\n\n| Category | Score | Highlights |\n|----------|-------|------------|\n| **Code Quality** | 86/100 | 809+ test classes, ruff/mypy enforcement, comprehensive pre-commit hooks |\n| **Security** | 93/100 | SQLCipher AES-256 encryption, full SHA pinning in CI, 22+ security scanners |\n| **Extensibility** | 93/100 | 26 abstract base classes, plugin architecture, strategy pattern throughout |\n| **Performance** | 85/100 | Adaptive rate limiting, cache stampede protection, parallel search execution |\n| **UX/Developer Experience** | 83/100 | Real-time WebSocket updates, in-tool documentation, comprehensive error handling |\n| **Maintenance** | 89/100 | 57 CI/CD workflows, automated security scanning, structured changelog |\n| **Overall** | 88/100 | Production-ready with excellent security and extensibility |\n\n### Security Architecture\n\n```mermaid\nflowchart TB\n    subgraph SECURITY[\"🔐 Security Layers\"]\n        direction TB\n        AUTH[Signed session cookie<br/>Starlette SessionMiddleware<br/>+ CSRFMiddleware]\n        ENCRYPT[SQLCipher<br/>AES-256 Encryption]\n        SCAN[22+ Security Scanners<br/>CodeQL, Semgrep, Bandit...]\n        PIN[Full SHA Pinning<br/>All 57 CI Workflows]\n    end\n\n    AUTH --> ENCRYPT\n    ENCRYPT --> SCAN\n    SCAN --> PIN\n\n    style SECURITY fill:#e8f5e9\n```\n\n**Security Features:**\n- Per-user encrypted databases (SQLCipher with AES-256)\n- Full GitHub Action SHA pinning (not tag-based)\n- Comprehensive CI security scanning:\n  - CodeQL (Python, JavaScript)\n  - Semgrep (custom rulesets)\n  - Bandit (Python security)\n  - Trivy (container scanning)\n  - Dependency review\n  - Secret scanning\n\n### Performance Optimizations\n\n| Feature | Implementation |\n|---------|----------------|\n| **Parallel Search** | `concurrent.futures.ThreadPoolExecutor` for multi-question search |\n| **Rate Limiting** | Adaptive system with `learning_rate=0.3` |\n| **Cache Protection** | `fetch_events` + `fetch_locks` for stampede prevention |\n| **Progress Streaming** | Socket.IO ASGI sub-app mounted at `/ws` for real-time UI updates |\n| **Cross-Engine Filtering** | LLM-powered relevance scoring and deduplication |\n\n#### Thread & Resource Lifecycle\n\nUnlike typical web apps that share a single database connection pool, this application maintains **separate database engines per user** because each user has their own encrypted SQLite file with a unique SQLCipher key ([SQLCipher](https://www.zetetic.net/sqlcipher/) is an encrypted extension of SQLite). This creates a threading challenge: every background thread needs its own engine for the user it serves.\n\nA single **QueuePool** engine is kept per user (`pool_size=20`,\n`max_overflow=40`, `pool_timeout=10`) in\n`DatabaseManager.connections[username]`. It is created with\n`check_same_thread=False`, so it is shared safely across the AnyIO\nworker threads that serve synchronous route handlers AND background\nthreads (research workers, scheduler jobs, metric writers). All of these acquire sessions from the\nsame pool, which keeps FD usage bounded by `pool_size + max_overflow`\nper user.\n\n> Earlier revisions of the code maintained a second, parallel system of\n> per-`(username, thread_id)` **NullPool** engines for background metric\n> writes. That system was removed because orphaned entries leaked\n> SQLCipher + WAL file handles (3 FDs per active connection) under load,\n> eventually exhausting the 1024 FD soft limit. Routing all work through\n> the single per-user QueuePool made FD usage bounded and let us delete\n> ~200 lines of thread-engine bookkeeping.\n\n##### Resource Cleanup Layers\n\nCleanup is defense-in-depth with multiple layers:\n\n| Layer | Trigger | What it cleans |\n|-------|---------|----------------|\n| `@thread_cleanup` decorator | Thread function exit (normal or exception) | DB session (returned to per-user pool), settings context, search context |\n| `WorkerCleanupAPIRoute` (`web/dependencies/threadpool.py`) | Synchronous FastAPI endpoint exit (normal or exception) | Sessions and ambient thread-local state on the AnyIO worker that served the endpoint |\n| `WorkerCleanupStreamingResponse` (`web/dependencies/threadpool.py`) | Each synchronous streamed-response iterator step (chunk, completion, or exception) | Sessions and ambient thread-local state on whichever AnyIO worker advanced that step |\n| `finally` blocks | Per-research in `run_research_process()` | Search engine HTTP sessions, strategy thread pool executors |\n| `DatabaseMiddleware` `finally` block (`web/fastapi_app.py`) | After each HTTP request | Sweeps dead workers and cleans event-loop-owned state; synchronous endpoint and stream-iterator state are reclaimed on their owning workers by the two wrappers above |\n| Periodic pool dispose | Every 30 min | Calls `engine.dispose()` on per-user QueuePool engines to release SQLCipher+WAL file handles that accumulate from out-of-order connection closes |\n| Logout cascade | User logout | Scheduler unregister (removes password) → DB close (`engine.dispose()`) → session destroy |\n| Stale session cleanup | `clear_session_if_unrecoverable()`, called from `require_auth()` when the DB connection is missing (`web/dependencies/auth.py`) | Clears the signed session cookie for users whose DB connection is gone and who have no recovery credential. Not sampled/throttled — unlike the old `before_request` hook it only runs on an already-failed auth check |\n\n##### Research Thread Lifecycle\n\n```mermaid\nflowchart TD\n    A[\"POST /api/start_research\"] --> B{\"Slots available?\"}\n    B -- \"Yes (direct)\" --> C[\"start_research_process()\"]\n    B -- \"No (queued)\" --> D[\"Database queue\"]\n    D --> E[\"_process_queue_loop<br/>(daemon thread, 10s poll)\"]\n    E --> C\n\n    C --> F[\"Research thread<br/>(daemon, semaphore-gated)\"]\n    F --> G[\"@thread_cleanup wraps<br/>run_research_process()\"]\n\n    G --> H[\"Strategy.find_relevant_information()\"]\n    H --> I[\"ThreadPoolExecutor sub-tasks<br/>(within strategies only)\"]\n    I --> J[\"Worker @thread_cleanup:<br/>close worker DB session\"]\n\n    J --> K[\"finally: search_engine.close()<br/>system.close() → strategy.close()\"]\n    K --> L[\"Main @thread_cleanup:<br/>close DB session, return connection to pool\"]\n\n    subgraph sweep [\"Defense-in-depth (independent timers)\"]\n        M[\"Dead-thread credential sweep<br/>(~60s via processor_v2)\"] -.-> N[\"Remove stale credentials<br/>for dead thread IDs\"]\n        O[\"Pool dispose<br/>(30 min via connection_cleanup)\"] -.-> P[\"engine.dispose() on all<br/>QueuePool engines\"]\n    end\n\n    style sweep fill:#f0f0f0,stroke:#999\n```\n\n##### FD Budget\n\nEach SQLCipher connection in WAL (Write-Ahead Logging) mode uses **2 file descriptors** (main db + WAL file). All connections to the same database within a process share **1 SHM** (shared-memory) file descriptor. The formula per user database is: `connections × 2 + 1`.\n\n| Component | FD Formula | With defaults |\n|-----------|-----------|---------------|\n| QueuePool (steady state) | `logged_in_users × (pool_size × 2 + 1)` | `users × 41` FDs |\n| QueuePool (peak) | `logged_in_users × ((pool_size + max_overflow) × 2 + 1)` | `users × 121` FDs |\n\nDefault Linux ulimit is 1024 soft (bare metal), which is tight for multi-user deployments. Docker's daemon default (typically 1M+) is adequate. QueuePool engines are created at login and disposed at logout, so only active users consume FDs.\n\nFor more on diagnosing FD exhaustion, see [Troubleshooting - Resource Exhaustion](./troubleshooting.md#resource-exhaustion).\n\n##### Key Files\n\n| File | Role |\n|------|------|\n| `src/local_deep_research/database/encrypted_db.py` | `DatabaseManager`, engine lifecycle, pool management |\n| `src/local_deep_research/database/thread_local_session.py` | `@thread_cleanup` decorator, thread-local sessions, credential cleanup |\n| `src/local_deep_research/web/fastapi_app.py` | `DatabaseMiddleware` per-request cleanup (replaces Flask's `teardown_appcontext`), lifespan orchestration |\n| `src/local_deep_research/web/services/research_service.py` | Research thread creation, `run_research_process()` |\n| `src/local_deep_research/web/queue/processor_v2.py` | Queue processing, credential cleanup trigger |\n\n### Areas for Improvement\n\nWhile the project scores highly overall, these areas have room for growth:\n\n1. **Integration Testing** - More end-to-end tests for full research workflows\n2. **API Documentation** - OpenAPI/Swagger spec for REST endpoints\n3. **Metrics Dashboard** - Prometheus/Grafana integration for monitoring\n4. **Resource Observability** - Expose FD count, thread count, and connection pool stats in /api/v1/health; add periodic sweep logging\n5. **Async Architecture** - Migration to async/await for I/O-bound operations\n\n### Key Source Files\n\n| Component | Location | Purpose |\n|-----------|----------|---------|\n| Research Engine | `src/local_deep_research/search_system.py` | Main `AdvancedSearchSystem` class |\n| Strategies | `src/local_deep_research/advanced_search_system/strategies/` | Research strategy implementations |\n| Search Engines | `src/local_deep_research/web_search_engines/engines/` | 25 search engine implementations |\n| Report Generation | `src/local_deep_research/report_generator.py` | `IntegratedReportGenerator` |\n| Web API | `src/local_deep_research/web/routers/` | FastAPI route handlers (`web/routes/` now holds a few shared helpers only, not route handlers) |\n| Database | `src/local_deep_research/database/` | SQLCipher models and migrations |\n| Encrypted DB | `src/local_deep_research/database/encrypted_db.py` | Per-user SQLCipher engine lifecycle |\n| Thread Sessions | `src/local_deep_research/database/thread_local_session.py` | Thread-safe session management and cleanup |\n| Settings | `src/local_deep_research/config/` | Configuration and LLM setup |\n\n### Contributing to Architecture\n\nWhen extending the system:\n\n1. **Adding Search Engines**: Inherit from `BaseSearchEngine`, implement `run()` method\n2. **Adding Strategies**: Inherit from `BaseSearchStrategy`, implement `analyze_topic()` method\n3. **Adding LLM Providers**: Add to `config/llm_config.py` with proper initialization\n4. **Adding Export Formats**: Inherit from base exporter pattern in `utilities/`\n\nSee [CONTRIBUTING.md](../CONTRIBUTING.md) for detailed guidelines.\n","isInternal":false,"tokens":5260,"sizeBytes":21623},{"name":"check-safe-requests.yaml","path":".semgrep/rules/check-safe-requests.yaml","rawUrl":"https://raw.githubusercontent.com/LearningCircuit/local-deep-research/HEAD/.semgrep/rules/check-safe-requests.yaml","title":"Rules Skill","category":"anthropic-skill","format":"yaml","content":"rules:\n  - id: unsafe-requests-get\n    patterns:\n      - pattern: requests.get(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.get() bypasses SSRF protection.\n      Use safe_get() from security module instead.\n      Import: from ...security import safe_get\n      For localhost services: safe_get(url, allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-post\n    patterns:\n      - pattern: requests.post(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.post() bypasses SSRF protection.\n      Use safe_post() from security module instead.\n      Import: from ...security import safe_post\n      For localhost services: safe_post(url, allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-session\n    patterns:\n      - pattern: requests.Session()\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.Session() bypasses SSRF protection.\n      Use SafeSession() from security module instead.\n      Import: from ...security import SafeSession\n      For localhost services: SafeSession(allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-put\n    patterns:\n      - pattern: requests.put(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.put() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n  - id: unsafe-requests-delete\n    patterns:\n      - pattern: requests.delete(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.delete() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n  - id: unsafe-requests-patch\n    patterns:\n      - pattern: requests.patch(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.patch() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n","isInternal":false,"tokens":1128,"sizeBytes":4351},{"name":"README.md","path":".semgrep/rules/README.md","rawUrl":"https://raw.githubusercontent.com/LearningCircuit/local-deep-research/HEAD/.semgrep/rules/README.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"# LDR Custom Security Rules\n\nThis directory contains custom Semgrep security rules specific to Local Deep Research (LDR).\n\n## Rules Overview\n\n### ldr-security.yaml\n\nLDR-specific security rules covering:\n\n1. **Hardcoded Secrets**\n   - Detects API keys, passwords, tokens in source code\n   - Severity: ERROR\n   - CWE-798\n\n2. **SQL Injection Prevention**\n   - `sql-string-concatenation`: follows dynamically constructed strings\n     (f-string, `+`, `%`, `.format()`) to database execution and SQLAlchemy\n     raw-SQL constructors, including inline expressions, aliases and stored\n     values. Named SQL builders also prompt review. Bound SQLAlchemy values\n     remain clear, including mapping values, stored strings without\n     reassignment, and reused `bindparams()` statements. Column identifiers\n     built from an inline or unchanged stored f-string remain clear when\n     quoting is enabled. Unpacked arguments, `quoted_name` overrides,\n     reassigned names, explicit or unknown literal flags, and nested raw-SQL\n     constructors remain reportable. Only the name occurrence is cleared;\n     an unsafe table/selectable argument still reaches execution.\n   - `sql-execute-with-string-format`: `$X.execute(f\"...\")` or\n     `$X.execute(\"...\" + ...)`\n   - Enforces parameterized queries via SQLAlchemy\n   - Severity: ERROR\n   - CWE-89\n\n3. **Code Injection**\n   - Detects dangerous use of eval/exec\n   - Prevents arbitrary code execution\n   - Severity: ERROR\n   - CWE-95\n\n4. **Command Injection**\n   - Detects unsafe use of os.system, shell=True\n   - Enforces subprocess with argument lists\n   - Severity: ERROR\n   - CWE-78\n\n5. **Path Traversal**\n   - `path-traversal-risk`: follows dynamic path construction and fetched\n     database values to file operations, directory enumeration and file\n     responses. A stored path served directly is covered without requiring\n     an intermediate `Path(...)`. Construction or normalization alone is\n     not a file operation. Mutation sinks include `Path.touch`, both paths\n     in `rename`/`replace`, and positional/keyword `os.remove`/`os.unlink`.\n     The one-target `Path.replace` signature keeps string and datetime\n     replacements outside the filesystem rule.\n   - `request-path-traversal-risk`: follows route handler parameters and\n     `request.*` to path construction and filesystem sinks. Database fetch\n     results are handled by the separate dynamic-path rule, so filtering a\n     query by a request ID does not classify the whole stored row as request\n     input. A response's download-name metadata is not its filesystem path.\n   - Only returned values from `validate_safe_path`, `validate_data_path`,\n     `validate_model_path`, `safe_join`, `secure_filename` and\n     `sanitize_filename` clear request taint. Exact call and assignment\n     patterns preserve findings for file access inside their arguments and\n     for later use of the original input. Other `PathValidator` methods and\n     `os.path.basename` are not sanitizers (`basename(\"..\")` is still `\"..\"`).\n   - Intraprocedural only: input is not followed into helper bodies. The\n     sanitizer contracts assume a trusted base directory; they do not prove\n     ownership or protect against concurrent local filesystem changes.\n     These rules prompt a boundary review rather than establish safety.\n   - Severity: WARNING\n   - CWE-22\n\n6. **Unsafe Deserialization**\n   - Detects unsafe YAML/pickle loading\n   - Prevents code execution via deserialization\n   - Severity: ERROR\n   - CWE-502\n\n7. **Weak Randomness**\n   - Detects use of random module for security\n   - Enforces secrets module for crypto operations\n   - Severity: WARNING\n   - CWE-338\n\n8. **Debug Mode in Production**\n   - Detects Flask debug=True\n   - Prevents information disclosure\n   - Severity: ERROR\n   - CWE-489\n\n9. **SSRF Prevention**\n   - Detects URL fetching operations\n   - Reminds to validate URLs\n   - Severity: WARNING\n   - CWE-918\n\n10. **XSS Prevention**\n    - Detects user input in HTML context\n    - Enforces proper escaping\n    - Severity: WARNING\n    - CWE-79\n\n11. **CSRF Protection**\n    - Detects POST endpoints\n    - Reminds to enable CSRF protection\n    - Severity: INFO\n    - CWE-352\n\n12. **Credential Logging**\n    - Detects passwords in log statements\n    - Prevents credential disclosure\n    - Severity: ERROR\n    - CWE-532\n\n## Usage\n\nThese rules are automatically run by the Semgrep CI/CD workflow:\n\n```bash\n# Run locally\nsemgrep --config=.semgrep/rules/ src/\n\n# Run with standard rules\nsemgrep --config=p/security-audit --config=.semgrep/rules/ src/\n```\n\n## GitHub upload and source suppressions\n\nThe pinned Semgrep 1.179.0 omits `nosemgrep` findings from its JSON report but retains\nthem in native SARIF with `inSource` suppression metadata. GitHub's SARIF\nimport does not honor that metadata, so uploading the native report would\npublish the ignored findings as open alerts.\n\nThe workflow validates the complete native JSON and SARIF reports, then\ncreates `semgrep-results.github.sarif`. Only results with recognized\n`inSource` suppressions (no status, or `accepted`) are omitted from that\ncopy, and only when the scanned file holds a reviewed annotation naming the\nresult's rule (see below) on the result's first line or on the line above\nit. The SARIF suppression does not record which rule a comment named, so\nthe validator reads the line back; a suppressed result without such an\nannotation fails validation. Unknown suppression kinds or statuses fail\nvalidation too. All other\nfindings and scan metadata are preserved, and the upload copy is validated\nagain. Scanner errors, skipped rules, empty scope and mismatched reports\nstill prevent upload.\n\nThe diagnostic artifact retains both native reports and the GitHub upload\ncopy for 7 days. Tests using native scanner reports and synthetic fixtures check that\nsuppressing one result preserves every unsuppressed result, including the\nsame rule at other locations and other rules at the same location; the\nreal-scanner test runs three rules on one line, annotates one or two of\nthem, and checks that the strict scan stays error-free and the rest are\nuploaded.\n\nReferences: [Semgrep source suppressions](https://docs.semgrep.dev/ignoring-files-folders-code)\nand [GitHub's supported SARIF properties](https://docs.github.com/en/code-security/reference/code-scanning/sarif-files/sarif-support).\n\n## Adding New Rules\n\nTo add new custom rules:\n\n1. Create a new YAML file in `.semgrep/rules/`\n2. Follow [Semgrep rule syntax](https://semgrep.dev/docs/writing-rules/overview/)\n3. Test the rule: `semgrep --config=.semgrep/rules/your-rule.yaml src/`\n4. Document the rule in this README\n\n## Reviewed source annotations\n\nUse a rule-specific reviewed annotation beside a reviewed false positive,\nat the end of the finding's first line or alone on the line above it:\n\n```python\n# nosemgrep: semgrep.rules.weak-random-generation, reason: Non-security jitter\ndelay_seconds = random.randint(1, 30)\n```\n\nWrite it exactly as `nosemgrep: <rule-id>[, <rule-id>]..., reason: <reason>`,\nstarting a comment (`#`, `//`, `{#`, `<!--` or `/*`), with fully qualified\nrule ids (list several when one line has several reviewed false positives)\nand no comma in the reason. Semgrep 1.177.0 honours a case-insensitive\n`nosem` marker after a space anywhere on a line, including string literals\nand words such as \"NoSemantics\", and a marker without a rule id hides every\nrule on that line. Semgrep also accepts any suffix of a rule id and reads\nthe first word of each comma-separated item after the colon as a rule id.\nThe `check-nosemgrep-annotations` pre-commit hook rejects every other marker\nunder `src/`, including one inside a Python string or docstring, and the\nupload validator rejects a suppressed result without an annotation naming\nits exact rule id.\n\nThe `reason:` item is required, not decoration. Semgrep reads it as one\nmore id; `reason:` is not a valid rule id, so it suppresses nothing. But a\ncomment with a single id makes Semgrep 1.177.0 report a warning for every\nother rule that matches the same line (\"found 'nosem' comment with id ...,\nbut no corresponding rule trying ...\"), and under `--strict` that fails the\nwhole scan, and so the release gate. Semgrep skips that warning when a\ncomment holds two or more ids, so with the `reason:` item a new or updated\nrule matching an annotated line produces an ordinary uploaded alert\ninstead. If that alert is another reviewed false positive, add its rule id\nto the annotation.\n\nKeep the rule enabled elsewhere; do not exclude a file or weaken the release\nthreshold to clear an alert. Re-review the annotation when the expression or\nits input boundary changes.\n\nReviews in October 2026 identified these cases:\n\n| Finding | Why the annotated use is safe | Existing verification |\n| --- | --- | --- |\n| Seven script interpolations | Jinja's `tojson` escapes HTML delimiters before embedding JSON in a script. | Template injection and JSON rendering tests in `tests/security/test_injection_and_template_safety.py`. |\n| Eleven unquoted attribute expressions | Both branches emit fixed `selected`/`checked` tokens or the empty string; no input is inserted into an attribute name. | Inspect the literal branches in `embedding_settings.html`; the rest of each attribute remains quoted. |\n| `help_tip`'s `safe` filter | Every current caller passes a source literal, intentionally containing help markup. | Review all call sites. The existing security test catches simple variable arguments, but does not fully parse composed Jinja expressions. |\n| Four `random` calls | They control subscription jitter or rate-limit exploration, not credentials, identifiers or authorization. | Scheduler and rate-limit tests; cryptographic operations still use `secrets`. |\n| Two apparent secrets | One is an empty API-key default; the other is the existing placeholder only used for unencrypted databases. | Configuration and authentication tests retain the surrounding behavior. |\n| Two pickle calls | They raise `UnpicklingError` on rejected globals/extensions rather than loading data. | `tests/vector_stores/test_faiss_safe_load_security.py` exercises hostile payloads, extension caches, persistent IDs and legitimate docstores. |\n| XML import | It imports `xml.sax.saxutils.escape` to encode text, not an XML parser. | Note-AI prompt rendering tests. |\n| Evernote `TreeBuilder` import | ElementTree's `TreeBuilder` only assembles elements. The Evernote loader's XML is read by defusedxml's parser, which refuses entity declarations, external references and any DOCTYPE with an internal subset. | `tests/document_loaders/test_evernote_extraction.py`. |\n| TOML loader's `Path(source)` | The app builds the loader only in `load_from_bytes`, from the temporary file it has just written; an upload supplies that file's bytes, not its path. No route passes a request value to `get_loader_for_path` or to the loader. | Review the callers of `load_from_bytes` and `get_loader_for_path`. `tests/document_loaders/test_toml_loader.py` loads uploads through `load_from_bytes`. |\n\nThe separate DevSkim annotation in `services/ui.js` covers its fragment-based\nanchor renderer, which already adds both `noopener` and `noreferrer`.\n\nThe combined SQL/path rules carry 100 individually reviewed dispositions.\nThey remain local to the reported operation; no source file, directory or\nrule is excluded:\n\n| Findings | Reviewed boundary |\n| --- | --- |\n| 30 SQL operations | Hex-encoded or explicitly checked SQLCipher keys; typed/range-checked or allowlisted PRAGMAs; dialect-quoted identifiers; migration table/column literals; bound row values; a fixed schema-version integer. |\n| Benchmark and indexing APIs | Explicit local Python/CLI output or input paths. The web benchmark worker uses its separate database runner rather than these file-output functions. |\n| Package and startup files | Fixed filenames and globs beneath installed package paths or operator-configured application directories. |\n| Upload and converter temporary files | Server-generated temporary names, allowlisted extensions and validated stems; web upload callers do not provide scratch directories. |\n| Backup retention cleanup | Fixed-pattern glob results filtered by `is_safe_glob_result` under the server-owned, hashed per-user backup directory. Reordering the checked list to preserve the newly finalized backup does not admit another path. |\n| Library/report cleanup | Existing root resolution and symlink checks at the destructive operation; document deletion callers require an allowed root. Operator opt-in alone does not establish ownership. |\n| Index persistence and model cache | Service-selected index output paths and generated temporary names; curated embedding repository names in the operator model cache. |\n| Five legacy-index sidecar operations | All `RAGIndex.index_path` writes originate in `LibraryRAGService`, using an operator cache root and hash-based filenames. Login-time rekey loads those rows from the authenticated user's DB and derives a fixed-suffix sidecar; no request/import endpoint assigns the column. This is a writer/caller boundary, not an assumption that arbitrary stored paths are safe. New writers or callers require another review. |\n| Six index rename/replace operations | Legacy relocation checks the source root and hash filename; quarantine uses service-generated index paths and timestamp suffixes. Atomic replacements move this call's `mkstemp` file to the explicit index output or fixed-suffix migration sidecar. The migration discovers non-symlink pickle files under the operator cache root. Each annotation is confined to its individual mutation; arbitrary stored paths remain reportable. |\n\nThe previously unresolved legacy-library read was fixed separately in\n#7268. Successful report validation is evidence that the scan is complete,\nnot that the release has no remaining alerts.\n\nNative SQL controls run from `.semgrep/tests/ldr-security.py`. The standalone\npath tests invoke the pinned scanner against unsafe and safe cases, including\ndirect request paths, fetched paths, keyword arguments, nested file accesses,\nignored/reassigned validation results, metadata-only response filenames and\nnormalization without file access. Relative-path validation only sanitizes\nthe relative argument: a request-selected base directory remains tainted.\nForty assigned/inline call controls cover trusted and request-selected roots,\nincluding keyword arguments and the default model root. The workflow uses one\nscanner worker to keep taint findings reproducible.\n\n## Rule Template\n\n```yaml\nrules:\n  - id: your-rule-id\n    pattern: |\n      # Your pattern here\n    message: Description of the security issue\n    languages: [python]\n    severity: ERROR  # or WARNING, INFO\n    metadata:\n      category: security\n      cwe: \"CWE-XXX: Description\"\n      owasp: \"AXX:2021 - Category\"\n```\n\n## References\n\n- [Semgrep Documentation](https://semgrep.dev/docs/)\n- [OWASP Top 10 2021](https://owasp.org/Top10/)\n- [CWE Top 25](https://cwe.mitre.org/top25/)\n- [Semgrep Registry](https://semgrep.dev/explore)\n","isInternal":false,"tokens":3290,"sizeBytes":15065}],"systemPromptSnippet":"<yakaai_skills repo=\"LearningCircuit/local-deep-research\">\n<!-- File: docs/architecture.md (Tokens: ~5260 | Category: project-spec) -->\n# Architecture Overview\n\nThis document provides detailed technical diagrams of Local Deep Research's architecture.\n\n## System Architecture\n\n```mermaid\nflowchart TB\n    subgraph USER[\"👤 User Interface\"]\n        WEB[Web Browser<br/>localhost:5000]\n    end\n\n    subgraph BACKEND[\"🌐 FastAPI Backend\"]\n        API[REST API + WebSocket]\n        AUTH[Authentication<br/>CSRF Protection]\n        ROUTES[Research Routes]\n    end\n\n    subgraph RESEARCH[\"🔬 Research Engine\"]\n        STRAT[Strategy Selector]\n        QGEN[Question Generator]\n        EXEC[Search Executor]\n        SYNTH[Report Synthesizer]\n    end\n\n    subgraph LLM[\"🤖 LLM Providers\"]\n        direction TB\n        LOCAL_LLM[/\"🏠 Local LLMs\"\\]\n        OLLAMA[Ollama]\n        LMSTUDIO[LM Studio]\n        CLOUD_LLM[/\"☁️ Cloud LLMs\"\\]\n        OPENAI[OpenAI]\n        ANTHROPIC[Anthropic]\n        GEMINI[Google Gemini]\n        OPENROUTER[OpenRouter<br/>100+ models]\n    end\n\n    subgraph SEARCH[\"🔍 Search Engines\"]\n        direction TB\n        LOCAL_SEARCH[/\"🏠 Local Search\"\\]\n        SEARXNG[SearXNG]\n        ELASTIC[Elasticsearch]\n        LIBRARY[Document Library]\n        WEB_SEARCH[/\"🌐 Web Search\"\\]\n        TAVILY[Tavily]\n        BRAVE[Brave]\n        ACADEMIC[/\"📚 Academic\"\\]\n        ARXIV[ArXiv]\n        PUBMED[PubMed]\n        SEMANTIC[Semantic Scholar]\n    end\n\n    subgraph STORAGE[\"💾 Storage Layer\"]\n        SQLCIPHER[(SQLCipher DB<br/>AES-256 Encrypted)]\n        VECTORS[(Vector Store<br/>Embeddings)]\n        FILES[(File Storage<br/>PDFs & Docs)]\n    end\n\n    subgraph OUTPUT[\"📄 Output\"]\n        MD[Markdown]\n        PDF[PDF Export]\n        LATEX[LaTeX]\n        QUARTO[Quarto]\n        RIS[RIS/BibTeX]\n    end\n\n    WEB <--> API\n    API --> AUTH\n    AUTH --> ROUTES\n    ROUTES --> STRAT\n\n    STRAT --> QGEN\n    QGEN --> EXEC\n    EXEC --> SYNTH\n\n    EXEC <--> OLLAMA & LMSTUDIO\n    EXEC <--> OPENAI & ANTHROPIC & GEMINI & OPENROUTER\n\n    EXEC <--> SEARXNG & ELASTIC & LIBRARY\n    EXEC <--> TAVILY & BRAVE\n    EXEC <--> ARXIV & PUBMED & SEMANTIC\n\n    SYNTH --> SQLCIPHER\n    LIBRARY <--> VECTORS\n    LIBRARY <--> FILES\n\n    SYNTH --> MD & PDF & LATEX & QUARTO & RIS\n\n    LOCAL_LLM ~~~ OLLAMA\n    CLOUD_LLM ~~~ OPENAI\n    LOCAL_SEARCH ~~~ SEARXNG\n    WEB_SEARCH ~~~ TAVILY\n    ACADEMIC ~~~ ARXIV\n\n    style USER fill:#e1f5fe\n    style BACKEND fill:#fff3e0\n    style RESEARCH fill:#f3e5f5\n    style LLM fill:#e8f5e9\n    style SEARCH fill:#fce4ec\n    style STORAGE fill:#fff8e1\n    style OUTPUT fill:#e0f2f1\n```\n\n## Research Flow\n\n```mermaid\nflowchart LR\n    subgraph INPUT[\"1️⃣ Input\"]\n        Q[Research Query]\n    end\n\n    subgraph ITERATE[\"2️⃣ Iterative Research\"]\n        direction TB\n        GEN[Generate<br/>Questions]\n        SEARCH[Parallel<br/>Search]\n        ANALYZE[Analyze<br/>Results]\n        GEN --> SEARCH --> ANALYZE\n        ANALYZE -.->|\"Need more info?\"| GEN\n    end\n\n    subgraph SYNTHESIZE[\"3️⃣ Synthesis\"]\n        REPORT[Generate<br/>Report]\n        CITE[Add<br/>Citations]\n        REPORT --> CITE\n    end\n\n    subgraph OUTPUT[\"4️⃣ Output\"]\n        RESULT[Final Report<br/>with Sources]\n    end\n\n    Q --> GEN\n    ANALYZE --> REPORT\n    CITE --> RESULT\n\n    style INPUT fill:#e3f2fd\n    style ITERATE fill:#f3e5f5\n    style SYNTHESIZE fill:#e8f5e9\n    style OUTPUT fill:#fff3e0\n```\n\n## Deployment Options\n\n```mermaid\nflowchart TB\n    subgraph FULL_LOCAL[\"🏠 Self-Hosted (Maximum Privacy)\"]\n        direction LR\n        L_LDR[LDR] <--> L_OLLAMA[Ollama]\n        L_LDR <--> L_SEARX[SearXNG]\n        L_LDR <--> L_DB[(Encrypted DB)]\n    end\n\n    subgraph HYBRID[\"⚡ Hybrid (Balanced)\"]\n        direction LR\n        H_LDR[LDR] <--> H_OLLAMA[Ollama]\n        H_LDR <-->|\"Web Search\"| H_CLOUD[Cloud APIs]\n        H_LDR <--> H_DB[(Encrypted DB)]\n    end\n\n    subgraph CLOUD[\"☁️ Cloud-Powered (Maximum Speed)\"]\n        direction LR\n        C_LDR[LDR] <--> C_OPENAI[OpenAI/Claude]\n        C_LDR <--> C_TAVILY[Tavily/Brave]\n        C_LDR <--> C_DB[(Encrypted DB)]\n    end\n\n    style FULL_LOCAL fill:#e8f5e9\n    style HYBRID fill:#fff3e0\n    style CLOUD fill:#e3f2fd\n```\n\n## Feature Map\n\n```mermaid\nmindmap\n  root((Local Deep<br/>Research))\n    Research\n      Quick Summary\n      Detailed Reports\n      Follow-up Questions\n      Research Strategies\n    Search Sources\n      Web Search\n        SearXNG\n        Tavily\n        Brave\n      Academic\n        ArXiv\n        PubMed\n        Semantic Scholar\n      Local\n        Document Library\n        Collections\n        Elasticsearch\n    LLM Support\n      Local\n        Ollama\n        LM Studio\n      Cloud\n        OpenAI\n        Anthropic\n        Google\n        OpenRouter\n        Requesty\n    Output\n      Markdown\n      PDF\n      LaTeX\n      Quarto\n      RIS/BibTeX\n    Features\n      News Subscriptions\n      Cost Analytics\n      Benchmarking\n      RAG Search\n      Per-User Encryption\n```\n\n## Component Details\n\n### LLM Providers\n\n| Provider | Type | Description |\n|----------|------|-------------|\n| Ollama | Local | Self-hosted open-source models |\n| LM Studio | Local | Desktop app for local models |\n| OpenAI | Cloud | GPT-4, GPT-3.5 |\n| Anthropic | Cloud | Claude 3 family |\n| Google | Cloud | Gemini models |\n| OpenRouter | Cloud | 100+ models via single API |\n| Requesty | Cloud | 100+ models via single API |\n\n### Search Engines\n\n| Engine | Type | Best For |\n|--------|------|----------|\n| SearXNG | Local/Self-hosted | Privacy, aggregated results |\n| Tavily | Cloud | AI-optimized search |\n| ArXiv | Academic | Physics, CS, Math papers |\n| PubMed | Academic | Biomedical research |\n| Semantic Scholar | Academic | Cross-discipline papers |\n| Wikipedia | Knowledge | General knowledge |\n| Your Documents | Local | Private document search |\n\n### Output Formats\n\n| Format | Use Case |\n|--------|----------|\n| Markdown | Default, web display |\n| PDF | Sharing, printing |\n| LaTeX | Academic papers |\n| Quarto | Reproducible documents |\n| RIS/BibTeX | Reference managers |\n\n## Knowledge Loop: Research → Library → Future Research\n\nOne of LDR's powerful features is the ability to build a personal knowledge base that improves future research.\n\n```mermaid\nflowchart TB\n    subgraph RESEARCH[\"1️⃣ Research\"]\n        Q[Your Question] --> Engine[Research Engine]\n        Engine --> Results[Results + Sources]\n    end\n\n    subgraph DOWNLOAD[\"2️⃣ Download\"]\n        Results --> Track[Track Sources]\n        Track --> Queue[Download Queue]\n        Queue --> Extract[Download & Extract Text]\n    end\n\n    subgraph LIBRARY[\"3️⃣ Library\"]\n        Extract --> Store[(Document Storage)]\n        Store --> Organize[Organize into Collections]\n    end\n\n    subgraph INDEX[\"4️⃣ Index\"]\n        Organize --> Chunk[Chunk Documents]\n        Chunk --> Embed[Generate Embeddings]\n        Embed --> FAISS[(Vector Index)]\n    end\n\n    subgraph REUSE[\"5️⃣ Reuse\"]\n        FAISS --> SearchEngine[Collection as Search Engine]\n        SearchEngine --> Q\n    end\n\n    style RESEARCH fill:#e3f2fd\n    style DOWNLOAD fill:#fff3e0\n    style LIBRARY fill:#e8f5e9\n    style INDEX fill:#f3e5f5\n    style REUSE fill:#fce4ec\n```\n\n### How It Works\n\n1. **Research Completes** → Sources are tracked in `ResearchResource` table\n2. **Download Sources** → Click \"Get All Research PDFs\" to queue downloads\n   - Smart downloaders for ArXiv, PubMed, Semantic Scholar, etc.\n   - Automatic text extraction from PDFs\n3. **Build Library** → Documents stored in encrypted database\n   - Deduplication via content hash\n   - Multiple storage modes: database (encrypted), filesystem, text-only\n4. **Create Collections** → Organize documents by topic/project\n   - Each collection can have different embedding settings\n   - Documents can belong to multiple collections\n5. **Index for Search** → Generate vector embeddings\n   - Configurable chunk size and overlap\n   - FAISS index for fast similarity search\n6. **Use in Future Research** → Select collection as search engine\n   - RAG search finds relevant passages\n   - Results cite back to your documents\n\n### Key Components\n\n| Component | Purpose |\n|-----------|---------|\n| `DownloadService` | Manages PDF downloads with source-specific strategies |\n| `LibraryService` | Queries and manages document library |\n| `LibraryRAGService` | Creates vector indices for semantic search |\n| `CollectionSearchEngine` | Searches collections using RAG |\n| `Document` | Stores text content, metadata, file references |\n| `DocumentChunk` | Stores indexed text chunks with embeddings |\n| `Collection` | Groups documents with shared embedding settings |\n\n### Storage Options\n\n| Mode | Security | Use Case |\n|------|----------|----------|\n| Database | AES-256 encrypted | Default, maximum security |\n| Filesystem | Unencrypted | Need external tool access |\n| Text Only | Encrypted text, no PDFs | Minimal storage |\n\n---\n\n## Technical Analysis & Project Health\n\n*Last updated: December 2024*\n\nThis section provides a comprehensive technical analysis of the codebase, including quality metrics, architecture patterns, and project health indicators.\n\n### Project Statistics\n\n| Metric | Count |\n|--------|-------|\n| Test Classes | 809+ |\n| Search Engine Implementations | 25 |\n| LLM Provider Implementations | 9 |\n| Search Strategies | 5 |\n| Abstract Base Classes | 26 |\n| CI/CD Workflows | 57 |\n| Security Scanners in CI | 22+ |\n| Core Dependencies | 63 |\n\n### Architecture Patterns\n\n#### Extensibility Design\n\nThe codebase follows a consistent pattern for extensibility:\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                    Abstract Base Classes                     │\n├─────────────────────────────────────────────────────────────┤\n│ BaseSearchEngine      │ Common interface for 25+ engines    │\n│ BaseSearchStrategy    │ Strategy pattern for research       │\n│ BaseCitationHandler   │ Citation processing abstraction     │\n│ BaseQuestionGenerator │ Question generation interface       │\n│ BaseExporter          │ Export format abstraction           │\n└─────────────────────────────────────────────────────────────┘\n```\n\n#### Search Engine Plugin System\n\nNew search engines can be added by:\n1. Creating a class inheriting from `BaseSearchEngine`\n2. Placing it in `web_search_engines/engines/`\n3. Register the engine in `ENGINE_REGISTRY` and add it to the security module allowlist in `security/module_whitelist.py`\n\n```python\n# Example: Adding a new search engine\nclass SearchEngineCustom(BaseSearchEngine):\n    def run(self, query: str) -> List[Dict]:\n        # Implementation\n        pass\n```\n\n#### LLM Provider Integration\n\nSupports 10 LLM providers with auto-discovery:\n- Ollama (local)\n- LM Studio (local)\n- OpenAI\n- Anthropic\n- Google Gemini\n- OpenRouter (100+ models)\n- Requesty (100+ models)\n- DeepSeek\n- Mistral\n- Groq\n\n### Quality Ratings\n\n```mermaid\npie title Project Health Ratings\n    \"Code Quality (86)\" : 86\n    \"Security (93)\" : 93\n    \"Extensibility (93)\" : 93\n    \"Performance (85)\" : 85\n    \"UX/DX (83)\" : 83\n    \"Maintenance (89)\" : 89\n```\n\n#### Detailed Ratings\n\n| Category | Score | Highlights |\n|----------|-------|------------|\n| **Code Quality** | 86/100 | 809+ test classes, ruff/mypy enforcement, comprehensive pre-commit hooks |\n| **Security** | 93/100 | SQLCipher AES-256 encryption, full SHA pinning in CI, 22+ security scanners |\n| **Extensibility** | 93/100 | 26 abstract base classes, plugin architecture, strategy pattern throughout |\n| **Performance** | 85/100 | Adaptive rate limiting, cache stampede protection, parallel search execution |\n| **UX/Developer Experience** | 83/100 | Real-time WebSocket updates, in-tool documentation, comprehensive error handling |\n| **Maintenance** | 89/100 | 57 CI/CD workflows, automated security scanning, structured changelog |\n| **Overall** | 88/100 | Production-ready with excellent security and extensibility |\n\n### Security Architecture\n\n```mermaid\nflowchart TB\n    subgraph SECURITY[\"🔐 Security Layers\"]\n        direction TB\n        AUTH[Signed session cookie<br/>Starlette SessionMiddleware<br/>+ CSRFMiddleware]\n        ENCRYPT[SQLCipher<br/>AES-256 Encryption]\n        SCAN[22+ Security Scanners<br/>CodeQL, Semgrep, Bandit...]\n        PIN[Full SHA Pinning<br/>All 57 CI Workflows]\n    end\n\n    AUTH --> ENCRYPT\n    ENCRYPT --> SCAN\n    SCAN --> PIN\n\n    style SECURITY fill:#e8f5e9\n```\n\n**Security Features:**\n- Per-user encrypted databases (SQLCipher with AES-256)\n- Full GitHub Action SHA pinning (not tag-based)\n- Comprehensive CI security scanning:\n  - CodeQL (Python, JavaScript)\n  - Semgrep (custom rulesets)\n  - Bandit (Python security)\n  - Trivy (container scanning)\n  - Dependency review\n  - Secret scanning\n\n### Performance Optimizations\n\n| Feature | Implementation |\n|---------|----------------|\n| **Parallel Search** | `concurrent.futures.ThreadPoolExecutor` for multi-question search |\n| **Rate Limiting** | Adaptive system with `learning_rate=0.3` |\n| **Cache Protection** | `fetch_events` + `fetch_locks` for stampede prevention |\n| **Progress Streaming** | Socket.IO ASGI sub-app mounted at `/ws` for real-time UI updates |\n| **Cross-Engine Filtering** | LLM-powered relevance scoring and deduplication |\n\n#### Thread & Resource Lifecycle\n\nUnlike typical web apps that share a single database connection pool, this application maintains **separate database engines per user** because each user has their own encrypted SQLite file with a unique SQLCipher key ([SQLCipher](https://www.zetetic.net/sqlcipher/) is an encrypted extension of SQLite). This creates a threading challenge: every background thread needs its own engine for the user it serves.\n\nA single **QueuePool** engine is kept per user (`pool_size=20`,\n`max_overflow=40`, `pool_timeout=10`) in\n`DatabaseManager.connections[username]`. It is created with\n`check_same_thread=False`, so it is shared safely across the AnyIO\nworker threads that serve synchronous route handlers AND background\nthreads (research workers, scheduler jobs, metric writers). All of these acquire sessions from the\nsame pool, which keeps FD usage bounded by `pool_size + max_overflow`\nper user.\n\n> Earlier revisions of the code maintained a second, parallel system of\n> per-`(username, thread_id)` **NullPool** engines for background metric\n> writes. That system was removed because orphaned entries leaked\n> SQLCipher + WAL file handles (3 FDs per active connection) under load,\n> eventually exhausting the 1024 FD soft limit. Routing all work through\n> the single per-user QueuePool made FD usage bounded and let us delete\n> ~200 lines of thread-engine bookkeeping.\n\n##### Resource Cleanup Layers\n\nCleanup is defense-in-depth with multiple layers:\n\n| Layer | Trigger | What it cleans |\n|-------|---------|----------------|\n| `@thread_cleanup` decorator | Thread function exit (normal or exception) | DB session (returned to per-user pool), settings context, search context |\n| `WorkerCleanupAPIRoute` (`web/dependencies/threadpool.py`) | Synchronous FastAPI endpoint exit (normal or exception) | Sessions and ambient thread-local state on the AnyIO worker that served the endpoint |\n| `WorkerCleanupStreamingResponse` (`web/dependencies/threadpool.py`) | Each synchronous streamed-response iterator step (chunk, completion, or exception) | Sessions and ambient thread-local state on whichever AnyIO worker advanced that step |\n| `finally` blocks | Per-research in `run_research_process()` | Search engine HTTP sessions, strategy thread pool executors |\n| `DatabaseMiddleware` `finally` block (`web/fastapi_app.py`) | After each HTTP request | Sweeps dead workers and cleans event-loop-owned state; synchronous endpoint and stream-iterator state are reclaimed on their owning workers by the two wrappers above |\n| Periodic pool dispose | Every 30 min | Calls `engine.dispose()` on per-user QueuePool engines to release SQLCipher+WAL file handles that accumulate from out-of-order connection closes |\n| Logout cascade | User logout | Scheduler unregister (removes password) → DB close (`engine.dispose()`) → session destroy |\n| Stale session cleanup | `clear_session_if_unrecoverable()`, called from `require_auth()` when the DB connection is missing (`web/dependencies/auth.py`) | Clears the signed session cookie for users whose DB connection is gone and who have no recovery credential. Not sampled/throttled — unlike the old `before_request` hook it only runs on an already-failed auth check |\n\n##### Research Thread Lifecycle\n\n```mermaid\nflowchart TD\n    A[\"POST /api/start_research\"] --> B{\"Slots available?\"}\n    B -- \"Yes (direct)\" --> C[\"start_research_process()\"]\n    B -- \"No (queued)\" --> D[\"Database queue\"]\n    D --> E[\"_process_queue_loop<br/>(daemon thread, 10s poll)\"]\n    E --> C\n\n    C --> F[\"Research thread<br/>(daemon, semaphore-gated)\"]\n    F --> G[\"@thread_cleanup wraps<br/>run_research_process()\"]\n\n    G --> H[\"Strategy.find_relevant_information()\"]\n    H --> I[\"ThreadPoolExecutor sub-tasks<br/>(within strategies only)\"]\n    I --> J[\"Worker @thread_cleanup:<br/>close worker DB session\"]\n\n    J --> K[\"finally: search_engine.close()<br/>system.close() → strategy.close()\"]\n    K --> L[\"Main @thread_cleanup:<br/>close DB session, return connection to pool\"]\n\n    subgraph sweep [\"Defense-in-depth (independent timers)\"]\n        M[\"Dead-thread credential sweep<br/>(~60s via processor_v2)\"] -.-> N[\"Remove stale credentials<br/>for dead thread IDs\"]\n        O[\"Pool dispose<br/>(30 min via connection_cleanup)\"] -.-> P[\"engine.dispose() on all<br/>QueuePool engines\"]\n    end\n\n    style sweep fill:#f0f0f0,stroke:#999\n```\n\n##### FD Budget\n\nEach SQLCipher connection in WAL (Write-Ahead Logging) mode uses **2 file descriptors** (main db + WAL file). All connections to the same database within a process share **1 SHM** (shared-memory) file descriptor. The formula per user database is: `connections × 2 + 1`.\n\n| Component | FD Formula | With defaults |\n|-----------|-----------|---------------|\n| QueuePool (steady state) | `logged_in_users × (pool_size × 2 + 1)` | `users × 41` FDs |\n| QueuePool (peak) | `logged_in_users × ((pool_size + max_overflow) × 2 + 1)` | `users × 121` FDs |\n\nDefault Linux ulimit is 1024 soft (bare metal), which is tight for multi-user deployments. Docker's daemon default (typically 1M+) is adequate. QueuePool engines are created at login and disposed at logout, so only active users consume FDs.\n\nFor more on diagnosing FD exhaustion, see [Troubleshooting - Resource Exhaustion](./troubleshooting.md#resource-exhaustion).\n\n##### Key Files\n\n| File | Role |\n|------|------|\n| `src/local_deep_research/database/encrypted_db.py` | `DatabaseManager`, engine lifecycle, pool management |\n| `src/local_deep_research/database/thread_local_session.py` | `@thread_cleanup` decorator, thread-local sessions, credential cleanup |\n| `src/local_deep_research/web/fastapi_app.py` | `DatabaseMiddleware` per-request cleanup (replaces Flask's `teardown_appcontext`), lifespan orchestration |\n| `src/local_deep_research/web/services/research_service.py` | Research thread creation, `run_research_process()` |\n| `src/local_deep_research/web/queue/processor_v2.py` | Queue processing, credential cleanup trigger |\n\n### Areas for Improvement\n\nWhile the project scores highly overall, these areas have room for growth:\n\n1. **Integration Testing** - More end-to-end tests for full research workflows\n2. **API Documentation** - OpenAPI/Swagger spec for REST endpoints\n3. **Metrics Dashboard** - Prometheus/Grafana integration for monitoring\n4. **Resource Observability** - Expose FD count, thread count, and connection pool stats in /api/v1/health; add periodic sweep logging\n5. **Async Architecture** - Migration to async/await for I/O-bound operations\n\n### Key Source Files\n\n| Component | Location | Purpose |\n|-----------|----------|---------|\n| Research Engine | `src/local_deep_research/search_system.py` | Main `AdvancedSearchSystem` class |\n| Strategies | `src/local_deep_research/advanced_search_system/strategies/` | Research strategy implementations |\n| Search Engines | `src/local_deep_research/web_search_engines/engines/` | 25 search engine implementations |\n| Report Generation | `src/local_deep_research/report_generator.py` | `IntegratedReportGenerator` |\n| Web API | `src/local_deep_research/web/routers/` | FastAPI route handlers (`web/routes/` now holds a few shared helpers only, not route handlers) |\n| Database | `src/local_deep_research/database/` | SQLCipher models and migrations |\n| Encrypted DB | `src/local_deep_research/database/encrypted_db.py` | Per-user SQLCipher engine lifecycle |\n| Thread Sessions | `src/local_deep_research/database/thread_local_session.py` | Thread-safe session management and cleanup |\n| Settings | `src/local_deep_research/config/` | Configuration and LLM setup |\n\n### Contributing to Architecture\n\nWhen extending the system:\n\n1. **Adding Search Engines**: Inherit from `BaseSearchEngine`, implement `run()` method\n2. **Adding Strategies**: Inherit from `BaseSearchStrategy`, implement `analyze_topic()` method\n3. **Adding LLM Providers**: Add to `config/llm_config.py` with proper initialization\n4. **Adding Export Formats**: Inherit from base exporter pattern in `utilities/`\n\nSee [CONTRIBUTING.md](../CONTRIBUTING.md) for detailed guidelines.\n\n<!-- File: .semgrep/rules/check-safe-requests.yaml (Tokens: ~1128 | Category: anthropic-skill) -->\nrules:\n  - id: unsafe-requests-get\n    patterns:\n      - pattern: requests.get(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.get() bypasses SSRF protection.\n      Use safe_get() from security module instead.\n      Import: from ...security import safe_get\n      For localhost services: safe_get(url, allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-post\n    patterns:\n      - pattern: requests.post(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.post() bypasses SSRF protection.\n      Use safe_post() from security module instead.\n      Import: from ...security import safe_post\n      For localhost services: safe_post(url, allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-session\n    patterns:\n      - pattern: requests.Session()\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.Session() bypasses SSRF protection.\n      Use SafeSession() from security module instead.\n      Import: from ...security import SafeSession\n      For localhost services: SafeSession(allow_localhost=True)\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n      references:\n        - https://cwe.mitre.org/data/definitions/918.html\n        - https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html\n\n  - id: unsafe-requests-put\n    patterns:\n      - pattern: requests.put(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.put() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n  - id: unsafe-requests-delete\n    patterns:\n      - pattern: requests.delete(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.delete() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n  - id: unsafe-requests-patch\n    patterns:\n      - pattern: requests.patch(...)\n    paths:\n      exclude:\n        - \"**/security/safe_requests.py\"\n        - \"**/tests/**\"\n        - \"**/*_test.py\"\n        - \"**/test_*.py\"\n        - \"**/examples/**\"\n    message: >-\n      Direct requests.patch() bypasses SSRF protection.\n      Use SafeSession() from security module for HTTP requests.\n    languages: [python]\n    severity: ERROR\n    metadata:\n      category: security\n      cwe: \"CWE-918: Server-Side Request Forgery (SSRF)\"\n      owasp: \"A10:2021 - Server-Side Request Forgery\"\n\n<!-- File: .semgrep/rules/README.md (Tokens: ~3290 | Category: anthropic-skill) -->\n# LDR Custom Security Rules\n\nThis directory contains custom Semgrep security rules specific to Local Deep Research (LDR).\n\n## Rules Overview\n\n### ldr-security.yaml\n\nLDR-specific security rules covering:\n\n1. **Hardcoded Secrets**\n   - Detects API keys, passwords, tokens in source code\n   - Severity: ERROR\n   - CWE-798\n\n2. **SQL Injection Prevention**\n   - `sql-string-concatenation`: follows dynamically constructed strings\n     (f-string, `+`, `%`, `.format()`) to database execution and SQLAlchemy\n     raw-SQL constructors, including inline expressions, aliases and stored\n     values. Named SQL builders also prompt review. Bound SQLAlchemy values\n     remain clear, including mapping values, stored strings without\n     reassignment, and reused `bindparams()` statements. Column identifiers\n     built from an inline or unchanged stored f-string remain clear when\n     quoting is enabled. Unpacked arguments, `quoted_name` overrides,\n     reassigned names, explicit or unknown literal flags, and nested raw-SQL\n     constructors remain reportable. Only the name occurrence is cleared;\n     an unsafe table/selectable argument still reaches execution.\n   - `sql-execute-with-string-format`: `$X.execute(f\"...\")` or\n     `$X.execute(\"...\" + ...)`\n   - Enforces parameterized queries via SQLAlchemy\n   - Severity: ERROR\n   - CWE-89\n\n3. **Code Injection**\n   - Detects dangerous use of eval/exec\n   - Prevents arbitrary code execution\n   - Severity: ERROR\n   - CWE-95\n\n4. **Command Injection**\n   - Detects unsafe use of os.system, shell=True\n   - Enforces subprocess with argument lists\n   - Severity: ERROR\n   - CWE-78\n\n5. **Path Traversal**\n   - `path-traversal-risk`: follows dynamic path construction and fetched\n     database values to file operations, directory enumeration and file\n     responses. A stored path served directly is covered without requiring\n     an intermediate `Path(...)`. Construction or normalization alone is\n     not a file operation. Mutation sinks include `Path.touch`, both paths\n     in `rename`/`replace`, and positional/keyword `os.remove`/`os.unlink`.\n     The one-target `Path.replace` signature keeps string and datetime\n     replacements outside the filesystem rule.\n   - `request-path-traversal-risk`: follows route handler parameters and\n     `request.*` to path construction and filesystem sinks. Database fetch\n     results are handled by the separate dynamic-path rule, so filtering a\n     query by a request ID does not classify the whole stored row as request\n     input. A response's download-name metadata is not its filesystem path.\n   - Only returned values from `validate_safe_path`, `validate_data_path`,\n     `validate_model_path`, `safe_join`, `secure_filename` and\n     `sanitize_filename` clear request taint. Exact call and assignment\n     patterns preserve findings for file access inside their arguments and\n     for later use of the original input. Other `PathValidator` methods and\n     `os.path.basename` are not sanitizers (`basename(\"..\")` is still `\"..\"`).\n   - Intraprocedural only: input is not followed into helper bodies. The\n     sanitizer contracts assume a trusted base directory; they do not prove\n     ownership or protect against concurrent local filesystem changes.\n     These rules prompt a boundary review rather than establish safety.\n   - Severity: WARNING\n   - CWE-22\n\n6. **Unsafe Deserialization**\n   - Detects unsafe YAML/pickle loading\n   - Prevents code execution via deserialization\n   - Severity: ERROR\n   - CWE-502\n\n7. **Weak Randomness**\n   - Detects use of random module for security\n   - Enforces secrets module for crypto operations\n   - Severity: WARNING\n   - CWE-338\n\n8. **Debug Mode in Production**\n   - Detects Flask debug=True\n   - Prevents information disclosure\n   - Severity: ERROR\n   - CWE-489\n\n9. **SSRF Prevention**\n   - Detects URL fetching operations\n   - Reminds to validate URLs\n   - Severity: WARNING\n   - CWE-918\n\n10. **XSS Prevention**\n    - Detects user input in HTML context\n    - Enforces proper escaping\n    - Severity: WARNING\n    - CWE-79\n\n11. **CSRF Protection**\n    - Detects POST endpoints\n    - Reminds to enable CSRF protection\n    - Severity: INFO\n    - CWE-352\n\n12. **Credential Logging**\n    - Detects passwords in log statements\n    - Prevents credential disclosure\n    - Severity: ERROR\n    - CWE-532\n\n## Usage\n\nThese rules are automatically run by the Semgrep CI/CD workflow:\n\n```bash\n# Run locally\nsemgrep --config=.semgrep/rules/ src/\n\n# Run with standard rules\nsemgrep --config=p/security-audit --config=.semgrep/rules/ src/\n```\n\n## GitHub upload and source suppressions\n\nThe pinned Semgrep 1.179.0 omits `nosemgrep` findings from its JSON report but retains\nthem in native SARIF with `inSource` suppression metadata. GitHub's SARIF\nimport does not honor that metadata, so uploading the native report would\npublish the ignored findings as open alerts.\n\nThe workflow validates the complete native JSON and SARIF reports, then\ncreates `semgrep-results.github.sarif`. Only results with recognized\n`inSource` suppressions (no status, or `accepted`) are omitted from that\ncopy, and only when the scanned file holds a reviewed annotation naming the\nresult's rule (see below) on the result's first line or on the line above\nit. The SARIF suppression does not record which rule a comment named, so\nthe validator reads the line back; a suppressed result without such an\nannotation fails validation. Unknown suppression kinds or statuses fail\nvalidation too. All other\nfindings and scan metadata are preserved, and the upload copy is validated\nagain. Scanner errors, skipped rules, empty scope and mismatched reports\nstill prevent upload.\n\nThe diagnostic artifact retains both native reports and the GitHub upload\ncopy for 7 days. Tests using native scanner reports and synthetic fixtures check that\nsuppressing one result preserves every unsuppressed result, including the\nsame rule at other locations and other rules at the same location; the\nreal-scanner test runs three rules on one line, annotates one or two of\nthem, and checks that the strict scan stays error-free and the rest are\nuploaded.\n\nReferences: [Semgrep source suppressions](https://docs.semgrep.dev/ignoring-files-folders-code)\nand [GitHub's supported SARIF properties](https://docs.github.com/en/code-security/reference/code-scanning/sarif-files/sarif-support).\n\n## Adding New Rules\n\nTo add new custom rules:\n\n1. Create a new YAML file in `.semgrep/rules/`\n2. Follow [Semgrep rule syntax](https://semgrep.dev/docs/writing-rules/overview/)\n3. Test the rule: `semgrep --config=.semgrep/rules/your-rule.yaml src/`\n4. Document the rule in this README\n\n## Reviewed source annotations\n\nUse a rule-specific reviewed annotation beside a reviewed false positive,\nat the end of the finding's first line or alone on the line above it:\n\n```python\n# nosemgrep: semgrep.rules.weak-random-generation, reason: Non-security jitter\ndelay_seconds = random.randint(1, 30)\n```\n\nWrite it exactly as `nosemgrep: <rule-id>[, <rule-id>]..., reason: <reason>`,\nstarting a comment (`#`, `//`, `{#`, `<!--` or `/*`), with fully qualified\nrule ids (list several when one line has several reviewed false positives)\nand no comma in the reason. Semgrep 1.177.0 honours a case-insensitive\n`nosem` marker after a space anywhere on a line, including string literals\nand words such as \"NoSemantics\", and a marker without a rule id hides every\nrule on that line. Semgrep also accepts any suffix of a rule id and reads\nthe first word of each comma-separated item after the colon as a rule id.\nThe `check-nosemgrep-annotations` pre-commit hook rejects every other marker\nunder `src/`, including one inside a Python string or docstring, and the\nupload validator rejects a suppressed result without an annotation naming\nits exact rule id.\n\nThe `reason:` item is required, not decoration. Semgrep reads it as one\nmore id; `reason:` is not a valid rule id, so it suppresses nothing. But a\ncomment with a single id makes Semgrep 1.177.0 report a warning for every\nother rule that matches the same line (\"found 'nosem' comment with id ...,\nbut no corresponding rule trying ...\"), and under `--strict` that fails the\nwhole scan, and so the release gate. Semgrep skips that warning when a\ncomment holds two or more ids, so with the `reason:` item a new or updated\nrule matching an annotated line produces an ordinary uploaded alert\ninstead. If that alert is another reviewed false positive, add its rule id\nto the annotation.\n\nKeep the rule enabled elsewhere; do not exclude a file or weaken the release\nthreshold to clear an alert. Re-review the annotation when the expression or\nits input boundary changes.\n\nReviews in October 2026 identified these cases:\n\n| Finding | Why the annotated use is safe | Existing verification |\n| --- | --- | --- |\n| Seven script interpolations | Jinja's `tojson` escapes HTML delimiters before embedding JSON in a script. | Template injection and JSON rendering tests in `tests/security/test_injection_and_template_safety.py`. |\n| Eleven unquoted attribute expressions | Both branches emit fixed `selected`/`checked` tokens or the empty string; no input is inserted into an attribute name. | Inspect the literal branches in `embedding_settings.html`; the rest of each attribute remains quoted. |\n| `help_tip`'s `safe` filter | Every current caller passes a source literal, intentionally containing help markup. | Review all call sites. The existing security test catches simple variable arguments, but does not fully parse composed Jinja expressions. |\n| Four `random` calls | They control subscription jitter or rate-limit exploration, not credentials, identifiers or authorization. | Scheduler and rate-limit tests; cryptographic operations still use `secrets`. |\n| Two apparent secrets | One is an empty API-key default; the other is the existing placeholder only used for unencrypted databases. | Configuration and authentication tests retain the surrounding behavior. |\n| Two pickle calls | They raise `UnpicklingError` on rejected globals/extensions rather than loading data. | `tests/vector_stores/test_faiss_safe_load_security.py` exercises hostile payloads, extension caches, persistent IDs and legitimate docstores. |\n| XML import | It imports `xml.sax.saxutils.escape` to encode text, not an XML parser. | Note-AI prompt rendering tests. |\n| Evernote `TreeBuilder` import | ElementTree's `TreeBuilder` only assembles elements. The Evernote loader's XML is read by defusedxml's parser, which refuses entity declarations, external references and any DOCTYPE with an internal subset. | `tests/document_loaders/test_evernote_extraction.py`. |\n| TOML loader's `Path(source)` | The app builds the loader only in `load_from_bytes`, from the temporary file it has just written; an upload supplies that file's bytes, not its path. No route passes a request value to `get_loader_for_path` or to the loader. | Review the callers of `load_from_bytes` and `get_loader_for_path`. `tests/document_loaders/test_toml_loader.py` loads uploads through `load_from_bytes`. |\n\nThe separate DevSkim annotation in `services/ui.js` covers its fragment-based\nanchor renderer, which already adds both `noopener` and `noreferrer`.\n\nThe combined SQL/path rules carry 100 individually reviewed dispositions.\nThey remain local to the reported operation; no source file, directory or\nrule is excluded:\n\n| Findings | Reviewed boundary |\n| --- | --- |\n| 30 SQL operations | Hex-encoded or explicitly checked SQLCipher keys; typed/range-checked or allowlisted PRAGMAs; dialect-quoted identifiers; migration table/column literals; bound row values; a fixed schema-version integer. |\n| Benchmark and indexing APIs | Explicit local Python/CLI output or input paths. The web benchmark worker uses its separate database runner rather than these file-output functions. |\n| Package and startup files | Fixed filenames and globs beneath installed package paths or operator-configured application directories. |\n| Upload and converter temporary files | Server-generated temporary names, allowlisted extensions and validated stems; web upload callers do not provide scratch directories. |\n| Backup retention cleanup | Fixed-pattern glob results filtered by `is_safe_glob_result` under the server-owned, hashed per-user backup directory. Reordering the checked list to preserve the newly finalized backup does not admit another path. |\n| Library/report cleanup | Existing root resolution and symlink checks at the destructive operation; document deletion callers require an allowed root. Operator opt-in alone does not establish ownership. |\n| Index persistence and model cache | Service-selected index output paths and generated temporary names; curated embedding repository names in the operator model cache. |\n| Five legacy-index sidecar operations | All `RAGIndex.index_path` writes originate in `LibraryRAGService`, using an operator cache root and hash-based filenames. Login-time rekey loads those rows from the authenticated user's DB and derives a fixed-suffix sidecar; no request/import endpoint assigns the column. This is a writer/caller boundary, not an assumption that arbitrary stored paths are safe. New writers or callers require another review. |\n| Six index rename/replace operations | Legacy relocation checks the source root and hash filename; quarantine uses service-generated index paths and timestamp suffixes. Atomic replacements move this call's `mkstemp` file to the explicit index output or fixed-suffix migration sidecar. The migration discovers non-symlink pickle files under the operator cache root. Each annotation is confined to its individual mutation; arbitrary stored paths remain reportable. |\n\nThe previously unresolved legacy-library read was fixed separately in\n#7268. Successful report validation is evidence that the scan is complete,\nnot that the release has no remaining alerts.\n\nNative SQL controls run from `.semgrep/tests/ldr-security.py`. The standalone\npath tests invoke the pinned scanner against unsafe and safe cases, including\ndirect request paths, fetched paths, keyword arguments, nested file accesses,\nignored/reassigned validation results, metadata-only response filenames and\nnormalization without file access. Relative-path validation only sanitizes\nthe relative argument: a request-selected base directory remains tainted.\nForty assigned/inline call controls cover trusted and request-selected roots,\nincluding keyword arguments and the default model root. The workflow uses one\nscanner worker to keep taint findings reproducible.\n\n## Rule Template\n\n```yaml\nrules:\n  - id: your-rule-id\n    pattern: |\n      # Your pattern here\n    message: Description of the security issue\n    languages: [python]\n    severity: ERROR  # or WARNING, INFO\n    metadata:\n      category: security\n      cwe: \"CWE-XXX: Description\"\n      owasp: \"AXX:2021 - Category\"\n```\n\n## References\n\n- [Semgrep Documentation](https://semgrep.dev/docs/)\n- [OWASP Top 10 2021](https://owasp.org/Top10/)\n- [CWE Top 25](https://cwe.mitre.org/top25/)\n- [Semgrep Registry](https://semgrep.dev/explore)\n</yakaai_skills>"}