## File: README.md

# Local Deep Research



## 🚀 What is Local Deep Research?

AI research assistant you control. Run locally for privacy, use any LLM and build your own searchable knowledge base. You own your data and see exactly how it works.

## ⚡ Quick Start

**Option 1: Docker Run (Linux)**
```bash
# Step 1: Pull and run Ollama
docker run -d -p 11434:11434 --name ollama ollama/ollama
docker exec ollama ollama pull gpt-oss:20b

# Step 2: Pull and run SearXNG for optimal search results
docker run -d -p 8080:8080 --name searxng searxng/searxng

# Step 3: Pull and run Local Deep Research
# (the URL line pins SearXNG's address AND marks it operator-approved —
#  private/localhost engine URLs are otherwise blocked by default since
#  v1.10.3. The URL becomes read-only in the web UI; docs/SearXNG-Setup.md
#  lists the alternatives, e.g. an origin allowlist.)
docker run -d --network host \
  --name local-deep-research \
  --volume "deep-research:/data" \
  -e LDR_DATA_DIR=/data \
  -e LDR_SEARCH_ENGINE_WEB_SEARXNG_DEFAULT_PARAMS_INSTANCE_URL=http://localhost:8080 \
  localdeepresearch/local-deep-research
```

> **Mac / Windows / WSL2 users:** `--network host` only works on native Linux. On Docker Desktop it silently fails to publish port 5000 *and* leaves `localhost` pointing at the LDR container itself (so it can't reach Ollama/SearXNG). Use **Option 2** below, or see the [Windows/WSL2 FAQ entry](docs/faq.md#port-5000-not-accessible-on-windows) for a working `docker run` recipe.

**Option 2: Docker Compose**

CPU-only (all platforms):
```bash
curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml && docker compose up -d
```

With NVIDIA GPU (Linux):
```bash
curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.yml && \
curl -O https://raw.githubusercontent.com/LearningCircuit/local-deep-research/main/docker-compose.gpu.override.yml && \
docker compose -f docker-compose.yml -f docker-compose.gpu.override.yml up -d
```

Open http://localhost:5000 after ~30 seconds. For GPU setup, environment variables, and more, see the [Docker Compose Guide](docs/docker-compose-guide.md).

**Option 3: pip install**
```bash
pip install local-deep-research
python -m local_deep_research.web.app   # starts the web UI on http://localhost:5000
```
> You'll also need Ollama (or any OpenAI-compatible LLM endpoint) and SearXNG running — see the [pip install guide](docs/install-pip.md) for the full recipe.
> Works on Windows, macOS, and Linux. SQLCipher encryption is included via pre-built wheels — no compilation needed.
> PDF export on Windows requires Pango ([setup guide](https://doc.courtbouillon.org/weasyprint/stable/first_steps.html)).
> If you encounter issues with encryption, set `export LDR_BOOTSTRAP_ALLOW_UNENCRYPTED=true` to use standard SQLite instead.

**Detailed install guides:** [Docker](docs/installation.md#docker) · [Docker Compose](docs/docker-compose-guide.md) · [pip](docs/install-pip.md) · [Unraid](docs/deployment/unraid.md) · [full install reference](docs/installation.md)

> **Older CPU (x86-64)?** LDR needs an AVX-capable CPU — Intel Sandy Bridge / AMD Bulldozer (2011) or newer. Several scientific Python dependencies (pandas, scikit-learn) ship wheels that crash with `Illegal instruction` on older CPUs. ARM64 (aarch64) is fully supported. Every release is smoke-tested against this floor, including AVX-without-AVX2 CPUs ([#4480](https://github.com/LearningCircuit/local-deep-research/issues/4480)).

## 🏗️ How It Works

### Research

You ask a complex question. Local Deep Research (LDR):
- Does the research for you automatically
- Searches across web, academic papers, and your own documents
- Synthesizes everything into a report with proper citations

Choose the research strategy that fits: quick pipeline modes for fast facts, or fully agentic deep research for complex analysis and academic work.

**LangGraph Agent Strategy** — An autonomous agentic research mode where the LLM decides what to search, which specialized engines to use (arXiv, PubMed, Semantic Scholar, etc.), and when to synthesize. It adaptively switches between search engines based on what it finds and collects significantly more sources than pipeline-based strategies — this is the strategy behind the ~95% SimpleQA result above. Select `langgraph-agent` in Settings.

### Build Your Knowledge Base

```mermaid
flowchart LR
    R[Research] --> D[Download Sources]
    D --> L[(Library)]
    L --> I[Index & Embed]
    I --> S[Search Your Docs]
    S -.-> R
```

Every research session finds valuable sources. Download them directly into your encrypted library — academic papers from ArXiv, PubMed articles, web pages. LDR extracts text, indexes everything, and makes it searchable. Next time you research, ask questions across your own documents and the live web together. Your knowledge compounds over time.

## 🛡️ Security



```mermaid
flowchart LR
    U1[User A] --> D1[(Encrypted DB)]
    U2[User B] --> D2[(Encrypted DB)]
```

Your data stays yours. Each user gets their own isolated SQLCipher database encrypted with AES-256, with the key derived from your password. Your password is never stored — login works by attempting to decrypt your database, so the database files on their own are unusable to anyone who obtains them. Per-user LLM API keys live encrypted inside the same personal database rather than in a shared server-level store.

The [Docker setup](docker-compose.yml) ships with `cap_drop: ALL`, `no-new-privileges`, and a non-root runtime, with the bundled Ollama and SearXNG images pinned by digest. Or run fully local with Ollama + SearXNG and nothing ever leaves your machine.

**In-memory credentials**: Like any application that uses secrets at runtime, credentials are held in process memory during active sessions — mitigated with session-scoped credential lifetimes and core dump exclusion. See the [Security Policy](SECURITY.md#in-memory-credentials) for the full threat model.

**Supply Chain Security**: Docker images are signed with [Cosign](https://github.com/sigstore/cosign) using GitHub's keyless OIDC flow, include SLSA provenance attestations, and ship with attested SPDX SBOMs. See [Verifying images and SBOMs](SECURITY.md#verifying-images-and-sboms) for the step-by-step verification commands.

**Security Transparency**: Scanner suppressions are documented with justifications in [Security Alerts Assessment](.github/SECURITY_ALERTS.md), [Scorecard Compliance](.github/SECURITY_SCORECARD.md), [Container CVE Suppressions](.trivyignore), and [SAST Rule Rationale](bearer.yml). Some alerts (Dependabot, code scanning) can only be dismissed or are very difficult to suppress outside the [GitHub Security tab](https://docs.github.com/en/code-security/dependabot/dependabot-alerts/viewing-and-updating-dependabot-alerts), so the files above do not cover every dismissed finding.

[Detailed Architecture →](docs/architecture.md) | [Security Policy →](SECURITY.md) | [Security Review Process →](docs/processes/security-review-process/)

### 🔒 Privacy & Data

Local Deep Research contains **no telemetry, no analytics, and no tracking**. We do not collect, transmit, or store any data about you or your usage. No analytics SDKs, no phone-home calls, no crash reporting, no external scripts. Usage metrics stay in your local encrypted database.

The only network calls LDR makes are ones **you** initiate: search queries (to engines you configure), LLM API calls (to your chosen provider), and notifications (only if you set up Apprise).

Since we don't collect any usage data, we rely on you to tell us what works, what's broken, and what you'd like to see next — [bug reports](https://github.com/LearningCircuit/local-deep-research/issues), feature ideas, and even which features you love or never use all help us improve LDR.

## 📊 Benchmarks

Headline results from the [community benchmarks](https://huggingface.co/datasets/local-deep-research/ldr-benchmarks) using the `langgraph-agent` strategy with Serper search, fully local via Ollama:

| Model | SimpleQA | xbench-DeepSearch |
|---|---|---|
| Qwen3.6-27B | 95.7% (287/300) | 77.0% (77/100) |
| Qwen3.5-9B  | 91.2% (182/200) | 59.0% (59/100) |
| gpt-oss-20B | 85.4% (295/346) | – |

Caveats: small samples, LLM-grader noise, and SimpleQA contamination risk on newer base models.

Picking a local model? The same community-maintained dataset tracks accuracy across models, search engines, and research strategies — the fastest way to see which Ollama / LM Studio / llama.cpp models actually work well for deep research before you download multi-GB weights. **[Browse the full leaderboard on Hugging Face →](https://huggingface.co/datasets/local-deep-research/ldr-benchmarks)**

[Submit your own results →](https://github.com/LearningCircuit/ldr-benchmarks) (contributors are listed in [CONTRIBUTORS.md](https://github.com/LearningCircuit/ldr-benchmarks/blob/main/CONTRIBUTORS.md)), or [run benchmarks locally →](docs/BENCHMARKING.md).

## ✨ Key Features

### 🔍 Research Modes
- **Quick Summary** - Get answers in 30 seconds to 3 minutes with citations
- **Detailed Research** - Comprehensive analysis with structured findings
- **Report Generation** - Professional reports with sections and table of contents
- **Document Analysis** - Search your private documents with AI

### 🛠️ Advanced Capabilities
- **[LangChain Integration](docs/LANGCHAIN_RETRIEVER_INTEGRATION.md)** - Use any vector store as a search engine
- **[REST API](docs/api-quickstart.md)** - Authenticated HTTP access with per-user databases
- **[Benchmarking](docs/BENCHMARKING.md)** - Test and optimize your configuration
- **[Analytics Dashboard](docs/analytics-dashboard.md)** - Track costs, performance, and usage metrics
- **[Journal Quality System](docs/journal-quality.md)** - Automatic journal reputation scoring with 212K+ indexed sources, predatory detection, and quality dashboard. Powered by [OpenAlex](https://openalex.org) (CC0), [DOAJ](https://doaj.org) (CC0), and [Stop Predatory Journals](https://predatoryjournals.org) (MIT). See the [v1.6.0 announcement](https://www.reddit.com/r/LocalDeepResearch/comments/1svndjb/v160_highly_anticipated_journal_quality_filter/).
- **Real-time Updates** - WebSocket support for live research progress
- **[Chat Mode](docs/features.md#chat-mode)** - Multi-turn research conversations with streaming progress and accumulated context across turns
- **Export Options** - Download results as PDF or Markdown
- **Research History** - Save, search, and revisit past research
- **Adaptive Rate Limiting** - Intelligent retry system that learns optimal wait times
- **Keyboard Shortcuts** - Navigate efficiently (ESC, Ctrl+Shift+1-4)

### 📰 News & Research Subscriptions
- **Automated Research Digests** - Subscribe to topics or specific queries; AI filters and summarizes only the most relevant developments
- **Customizable Delivery** - Daily, weekly, or custom schedules, as markdown reports or structured summaries

### 🌐 Search Sources

#### Free Search Engines
- **Academic**: arXiv, PubMed, Semantic Scholar
- **General**: Wikipedia, SearXNG
- **Technical**: GitHub, Elasticsearch
- **Historical**: Wayback Machine
- **News**: The Guardian, Wikinews

#### Premium Search Engines
- **Tavily** - AI-powered search
- **Google** - Via SerpAPI or Programmable Search Engine
- **Brave Search** - Privacy-focused web search

#### Custom Sources
- **Local Documents** - Search your files with AI
- **LangChain Retrievers** - Any vector store or database
- **Meta Search** - Combine multiple engines intelligently

LDR respects `robots.txt` and identifies itself honestly when fetching web pages — no stealth or anti-detection techniques. In rare cases this means a page that blocks automated access won't be fetched, which we consider the right trade-off.

[Full Search Engines Guide →](docs/search-engines.md)

## 💻 Usage Examples

### Python API
```python
from local_deep_research.api import LDRClient, quick_query

# Option 1: Simplest - one line research
summary = quick_query("username", "password", "What is quantum computing?")
print(summary)

# Option 2: Client for multiple operations
client = LDRClient()
client.login("username", "password")
result = client.quick_research("What are the latest advances in quantum computing?")
print(result["summary"])
```

### HTTP API

*The code example below shows the basic API structure - for working examples, see the link below*

```python
import requests
from bs4 import BeautifulSoup

# Create session and authenticate
session = requests.Session()
login_page = session.get("http://localhost:5000/auth/login")
soup = BeautifulSoup(login_page.text, "html.parser")
login_csrf = soup.find("input", {"name": "csrf_token"}).get("value")

# Login and get API CSRF token
session.post("http://localhost:5000/auth/login",
            data={"username": "user", "password": "pass", "csrf_token": login_csrf})
csrf = session.get("http://localhost:5000/auth/csrf-token").json()["csrf_token"]

# Make API request
response = session.post("http://localhost:5000/api/start_research",
                       json={"query": "Your research question"},
                       headers={"X-CSRF-Token": csrf})
```

🚀 **[Ready-to-use HTTP API Examples → examples/api_usage/http/](examples/api_usage/http/)**
- ✅ **Automatic user creation** - works out of the box
- ✅ **Complete authentication** with CSRF handling
- ✅ **Result retry logic** - waits until research completes
- ✅ **Progress monitoring** and error handling

### Command Line Tools

```bash
# Run benchmarks from CLI
python -m local_deep_research.benchmarks.cli.benchmark_commands simpleqa --examples 50
```

See the [Command Line Tools guide](docs/cli-tools.md) for the full reference.

## 🔗 Bring Your Own Knowledge Base

Connect LDR to your existing knowledge base. Unlike the HTTP client above, `quick_summary()` runs LDR in-process — no server needed — so you can pass it live Python objects such as LangChain retrievers:

```python
from local_deep_research.api import quick_summary

# Use your existing LangChain retriever
result = quick_summary(
    query="What are our deployment procedures?",
    retrievers={"company_kb": your_retriever},
    search_tool="company_kb"
)
```

Works with: FAISS, Chroma, Pinecone, Weaviate, Elasticsearch, and any LangChain-compatible retriever.

[Integration Guide →](docs/LANGCHAIN_RETRIEVER_INTEGRATION.md)

## 🔌 MCP Server (Claude Integration)

LDR provides an MCP (Model Context Protocol) server that allows AI assistants like Claude Desktop and Claude Code to perform deep research. Full setup details in the [MCP Server guide](docs/mcp-server.md).

> ⚠️ **Security Note**: This MCP server is designed for **local use only** via STDIO transport (e.g., Claude Desktop). It has no built-in authentication or rate limiting. Do not expose over a network without implementing proper security controls. See the [MCP Security Best Practices](https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices) for network deployment requirements.

### Installation

```bash
# Install with MCP extras
pip install "local-deep-research[mcp]"
```

### Claude Desktop Configuration

Add to your `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "local-deep-research": {
      "command": "ldr-mcp",
      "env": {
        "LDR_LLM_PROVIDER": "openai",
        "LDR_LLM_OPENAI_API_KEY": "sk-..."
      }
    }
  }
}
```

### Claude Code Configuration

Add to your `.mcp.json` (project-level) or `~/.claude/mcp.json` (global):

```json
{
  "mcpServers": {
    "local-deep-research": {
      "command": "ldr-mcp",
      "env": {
        "LDR_LLM_PROVIDER": "ollama",
        "LDR_LLM_OLLAMA_URL": "http://localhost:11434"
      }
    }
  }
}
```

Any LDR setting can be passed in `env` as an `LDR_*` variable — see the auto-generated [Full Configuration Reference](docs/CONFIGURATION.md) for the complete list.

### Available Tools

| Tool | Description | Duration | LLM Cost |
|------|-------------|----------|----------|
| `search` | Raw results from a specific engine (arxiv, pubmed, wikipedia, ...) | 5-30s | None |
| `quick_research` | Fast research summary | 1-5 min | Yes |
| `detailed_research` | Comprehensive analysis | 5-15 min | Yes |
| `generate_report` | Full markdown report | 10-30 min | Yes |
| `analyze_documents` | Search local collections | 30s-2 min | Yes |
| `list_search_engines` | List available search engines | instant | None |
| `list_strategies` | List research strategies | instant | None |
| `get_configuration` | Get current config | instant | None |

### Individual Search Engines

The `search` tool lets you query specific search engines directly and get raw results (title, link, snippet) — no LLM processing, no cost, fast. This is especially useful for **monitoring and subscriptions** where you want to check for new content regularly without burning LLM tokens.

```
# Search arXiv for recent papers
search(query="transformer architecture improvements", engine="arxiv")

# Search PubMed for medical literature
search(query="CRISPR clinical trials 2024", engine="pubmed")

# Search Wikipedia for quick facts
search(query="quantum error correction", engine="wikipedia")

# Search GitHub for code and repositories
search(query="agentic research frameworks", engine="github")

# Use list_search_engines() to see all available engines
```

### Example Usage

```
"Use quick_research to find information about quantum computing applications"
"Search arxiv for recent papers on diffusion models"
"Generate a detailed research report on renewable energy trends"
```


## 🤖 Supported LLMs

### Local Models
- **Ollama** — connect to its native API (default `http://localhost:11434`)
- **LM Studio** — connect to its OpenAI-compatible server (default `http://localhost:1234/v1`)
- **llama.cpp** — connect to `llama-server`'s OpenAI-compatible endpoint (default `http://localhost:8080/v1`); start with `llama-server -m <model.gguf>`
- Common models: Llama, Mistral, Gemma, DeepSeek, Qwen
- LLM processing stays local (search queries still go to web). No API costs.

> 💡 **Which local model should I pick?** See the [community benchmarks](#-benchmarks) — community-submitted accuracy numbers across local and cloud models, so you can compare before downloading.

### Cloud Models
- OpenAI
- Anthropic Claude
- Google Gemini
- 100+ models via OpenRouter
- 100+ models via Requesty

### Custom Endpoints
- **OpenAI-Compatible Endpoint** — any service speaking the OpenAI chat-completions API (vLLM, llama.cpp, gateways)
- **Anthropic-Compatible Endpoint** — a self-hosted service speaking the Anthropic Messages API (`/v1/messages`); set `llm.anthropic_endpoint.url`

[Model Setup →](docs/env_configuration.md)

## 🔄 Upgrading from Earlier Versions

- **`llm.model` no longer has a default.** Pre-1.6.3 installs auto-filled `gemma3:12b` (Ollama) when no model was configured, which silently downloaded a multi-GB binary. The field is now empty by default — pick a model in Settings → LLM, or research will fail loudly with a clear error.
- **The `auto` and `parallel` meta search engines were removed.** The default langgraph-agent strategy selects engines dynamically per query, which replaces them. Stored settings are migrated automatically (removed values become `searxng`); update any explicit `search_tool="auto"` API calls or `LDR_SEARCH_TOOL=auto` env overrides to a concrete engine such as `searxng`.
- **The `llamacpp` provider now uses HTTP instead of in-process loading.** If you previously set `llm.llamacpp_model_path` to a local `.gguf` file, that setting is no longer read. Instead, run `llama-server -m <your-model.gguf>` (it ships with every modern llama.cpp build) and the default `llm.llamacpp.url` of `http://localhost:8080/v1` will pick it up. Optional API key support is available via `llm.llamacpp.api_key` if you put `llama-server` behind an auth proxy.

## 📚 Documentation

### Getting Started
- [Installation Guide](docs/installation.md)
- [Frequently Asked Questions](docs/faq.md)
- [API Quickstart](docs/api-quickstart.md)
- [Configuration Guide](docs/env_configuration.md)
- [Full Configuration Reference](docs/CONFIGURATION.md)

### Core Features
- [All Features Guide](docs/features.md)
- [Search Engines Guide](docs/search-engines.md)
- [Analytics Dashboard](docs/analytics-dashboard.md)

### Advanced Features
- [LangChain Integration](docs/LANGCHAIN_RETRIEVER_INTEGRATION.md)
- [Benchmarking System](docs/BENCHMARKING.md)
- [Elasticsearch Setup](docs/elasticsearch_search_engine.md)
- [SearXNG Setup](docs/SearXNG-Setup.md)

### Development
- [Docker Compose Guide](docs/docker-compose-guide.md)
- [Development Guide](docs/developing.md)
- [CodeQL Guide](docs/security/CODEQL_GUIDE.md)
- [Release Guide](docs/RELEASE_GUIDE.md)
- [CI/CD Infrastructure](docs/CI_CD_INFRASTRUCTURE.md) — pre-commit hooks, workflows, security scanning
- [Workflow Status Dashboard](docs/ci/workflow-status.md) — live health of every GitHub Actions workflow
- [Test Coverage Report](https://learningcircuit.github.io/local-deep-research/) — pytest coverage published to GitHub Pages

### Examples & Tutorials
- [API Examples](examples/api_usage/)
- [Benchmark Examples](examples/benchmarks/)
- [Optimization Examples](examples/optimization/)

## 📰 Featured In

> "Local Deep Research **deserves special mention** for those who prioritize privacy... **tuned to use open-source LLMs** that can run on consumer GPUs or even CPUs. Journalists, researchers, or companies with sensitive topics can investigate information **without queries ever hitting an external server**."
>
> — [Medium: Open-Source Deep Research AI Assistants](https://medium.com/@leucopsis/open-source-deep-research-ai-assistants-157462a59c14)

### News & Articles
- [Korben.info](https://korben.info/local-deep-research-alternative-gratuite-recherche-ia-sourcee.html) - French tech blog ("Sherlock Holmes numérique")
- [Roboto.fr](https://www.roboto.fr/blog/local-deep-research-l-alternative-open-source-gratuite-deep-research-d-openai) - "L'alternative open-source gratuite à Deep Research d'OpenAI"
- [KDJingPai AI Tools](https://www.kdjingpai.com/en/local-deep-research/) - AI productivity tools coverage
- [AI Sharing Circle](https://aisharenet.com/en/local-deep-research/) - AI resources coverage

### Community Discussions
- [r/LocalLLaMA: 95.7% SimpleQA on a single 3090, fully local](https://www.reddit.com/r/LocalLLaMA/comments/1t1n6o8/we_are_finally_there_qwen3627b_agentic_search_957/) - Qwen3.6-27B benchmark announcement (161K views)
- [r/WebAfterAI: LDR as the agentic layer on top of Ollama / LM Studio / LocalAI](https://www.reddit.com/r/WebAfterAI/comments/1t18wr6/local_deep_research_opensource_ai_research/) - Stack-positioning explainer
- [Hacker News](https://news.ycombinator.com/item?id=43330164) - 190+ points, community discussion
- [LangChain Twitter/X](https://x.com/LangChainAI/status/1901347759757902038) - Official LangChain promotion
- [LangChain LinkedIn](https://www.linkedin.com/posts/langchain_local-deep-research-an-ai-research-activity-7307113456095137792-cXRH) - 400+ likes

### International Coverage

#### 🇨🇳 Chinese
- [Juejin (掘金)](https://juejin.cn/post/7481604667589885991) - Developer community
- [Cnblogs (博客园)](https://www.cnblogs.com/qife122/p/18955032) - Developer blogs
- [GitHubDaily (Twitter/X)](https://x.com/GitHub_Daily/status/1900169979313741846) - Influential tech account
- [Zhihu (知乎)](https://zhuanlan.zhihu.com/p/30886269290) - Tech community
- [A姐分享](https://www.ahhhhfs.com/68713/) - AI resources
- [CSDN](https://blog.csdn.net/gitblog_01198/article/details/147061415) - Installation guide
- [NetEase (网易)](https://www.163.com/dy/article/JQKAS50205567BLV.html) - Tech news portal

#### 🇯🇵 Japanese
- [note.com: 調査革命：Local Deep Research徹底活用法](https://note.com/r7038xx/n/nb3b74debbb30) - Comprehensive tutorial
- [Qiita: Local Deep Researchを試す](https://qiita.com/orca13/items/635f943287c45388d48f) - Docker setup guide
- [LangChainJP (Twitter/X)](https://x.com/LangChainJP/status/1902918110073807073) - Japanese LangChain community

#### 🇰🇷 Korean
- [PyTorch Korea Forum](https://discuss.pytorch.kr/t/local-deep-research/6476) - Korean ML community
- [GeekNews (Hada.io)](https://news.hada.io/topic?id=19707) - Korean tech news

### Reviews & Analysis
- [BSAIL Lab: How useful is Deep Research in Academia?](https://uflbsail.net/uncategorized/how-useful-is-deep-research-in-academia/) - Academic review by contributor [@djpetti](https://github.com/djpetti)
- [The Art Of The Terminal: Use Local LLMs Already!](https://youtu.be/pfxgLX-MxMY?t=1999) - Comprehensive review of local AI tools, featuring LDR's research capabilities
- [Fahd Mirza: Local Deep Research + Ollama — Free AI Research Assistant You Control](https://youtu.be/Q6kygd04sFI) - Installation walkthrough (349K-subscriber channel). The SearXNG startup issue shown in the video has since been fixed in [#3881](https://github.com/LearningCircuit/local-deep-research/pull/3881) — clean Docker installs now start out of the box.
- [BC Adventure Tech: Run ChatGPT Deep Research Locally (Unraid AI Agent Setup)](https://youtu.be/bhy5TqmoLYo) - Unraid setup and showcase

### Related Projects
- [SearXNG LDR-Academic](https://github.com/porespellar/searxng-LDR-academic) - Academic-focused SearXNG fork with 12 research engines (arXiv, Google Scholar, PubMed, etc.) designed for LDR
- [DeepWiki Documentation](https://deepwiki.com/LearningCircuit/local-deep-research) - Third-party documentation and guides

> **Note:** Third-party projects and articles are independently maintained. We link to them as useful resources but cannot guarantee their code quality or security.

## 🤝 Community & Support

- [Discord](https://discord.gg/ttcqQeFcJ3) - Get help and share research techniques
- [Reddit](https://www.reddit.com/r/LocalDeepResearch/) - Updates and showcases
- [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues) - Bug reports

## 🧑‍💻 Contributing

We welcome contributions of all sizes — from typo fixes to new features. The key rule: **keep PRs small and atomic** (one change per PR). For larger changes, please open an issue or start a discussion first — we want to protect your time and make sure your effort leads to a successful merge rather than a misaligned PR. See our [Contributing Guide](CONTRIBUTING.md) to get started.

## Acknowledgements

Local Deep Research is built on the work of many open-access initiatives, academic databases, and open-source projects. We are grateful to:

### Academic & Research Data

| Source | What It Provides | License |
|--------|-----------------|---------|
| [OpenAlex](https://openalex.org) | Academic metadata for ~280K sources and ~120K institutions, including DOAJ status | CC0 |
| [DOAJ](https://doaj.org) | Directory of Open Access Journals — open-access verification (via OpenAlex) | CC0 |
| [arXiv](https://arxiv.org) | Preprints in physics, mathematics, CS, and more | Various (see arXiv license) |
| [PubMed / NCBI](https://pubmed.ncbi.nlm.nih.gov) | Biomedical and life sciences literature | Public domain (US Gov) |
| [Semantic Scholar](https://www.semanticscholar.org) | Cross-discipline academic search with citation data | [Terms](https://www.semanticscholar.org/product/api/license) |
| [NASA ADS](https://ui.adsabs.harvard.edu) | Astrophysics, physics, and astronomy papers | [Terms](https://ui.adsabs.harvard.edu/help/terms/) |
| [Zenodo](https://zenodo.org) | Open research data, datasets, and software | Various per record |
| [PubChem](https://pubchem.ncbi.nlm.nih.gov) | Chemistry and biochemistry database | Public domain (US Gov) |
| [Stop Predatory Journals](https://predatoryjournals.org) | Predatory journal/publisher blacklist | MIT |
| [JabRef](https://github.com/JabRef/abbrv.jabref.org) | Journal abbreviation database | CC0 |

### Knowledge & Content Sources

[Wikipedia](https://www.wikipedia.org) &bull; [OpenLibrary](https://openlibrary.org) &bull; [Project Gutenberg](https://www.gutenberg.org) &bull; [GitHub](https://github.com) &bull; [Stack Exchange](https://stackexchange.com) &bull; [The Guardian](https://www.theguardian.com) &bull; [Wayback Machine](https://web.archive.org)

### Infrastructure & Frameworks

[LangChain](https://github.com/hwchase17/langchain) &bull; [Ollama](https://ollama.ai) &bull; [SearXNG](https://searxng.org/) &bull; [FAISS](https://github.com/facebookresearch/faiss)

### Support Open Access

These projects run on donations and grants, not paywalls. If Local Deep Research is useful to you, consider giving back to the open-access ecosystem that makes it possible:

- [arXiv](https://arxiv.org/about/give) — free preprints for physics, math, CS, and more
- [PubMed / NLM](https://www.nlm.nih.gov/pubs/donations/donations.html) — open biomedical literature
- [Wikipedia / Wikimedia](https://donate.wikimedia.org) — the free encyclopedia
- [Internet Archive](https://archive.org/donate) — the Wayback Machine and open digital library
- [DOAJ](https://doaj.org/support) — curating and verifying open-access journals worldwide
- [OpenAlex](https://openalex.org) — open scholarly metadata (sponsored by [OurResearch](https://ourresearch.org))
- [Project Gutenberg](https://www.gutenberg.org/donate/) — free ebooks since 1971

## 📄 License

MIT License - see [LICENSE](LICENSE) file.

**Dependencies:** All third-party packages use permissive licenses (MIT, Apache-2.0, BSD, etc.) - see [allowlist](.github/workflows/dependency-review.yml)

---

## File: community_benchmark_results/README.md

# Local Deep Research Benchmark Results

> ## 📦 This directory is being archived — new results go to a dedicated repo
>
> Community benchmark results have moved to a dedicated GitHub repository
> (source of truth) with auto-synced leaderboard CSVs on Hugging Face:
>
> 👉 **GitHub (submit PRs here): [LearningCircuit/ldr-benchmarks](https://github.com/LearningCircuit/ldr-benchmarks)**
>
> 👉 **Hugging Face (browse leaderboards): [local-deep-research/ldr-benchmarks](https://huggingface.co/datasets/local-deep-research/ldr-benchmarks)**
>
> The new setup offers:
> - CI validation of every submission (schema, sharing-policy, secrets scan)
> - Auto-generated leaderboard CSVs (per-benchmark and combined) synced to HF
> - Dataset Viewer on Hugging Face for browsing
> - One canonical place to compare runs across SimpleQA, BrowseComp, and xbench-DeepSearch
>
> **Where to submit new results:** open a Pull Request against the
> [GitHub repo](https://github.com/LearningCircuit/ldr-benchmarks). The
> same YAML export from the LDR web UI (`/benchmark` → YAML button) works
> unchanged — just drop it under `results/{dataset}/{strategy}/{search_engine}/`.
>
> **What stays here:** the existing `.yaml` result files in this folder
> and `benchmark_template.yaml` are kept as a historical archive and for
> reference. They are not being deleted. New submissions, however, should
> go to the new repo so results stay consolidated in one place.
>
> **Why the move:** keeping benchmark data in the code repo bloats git
> history on every clone, even though the data is static. A dedicated
> repo solves this cleanly and gives us a CI pipeline + viewer + leaderboards
> built for exactly this purpose.

---

## Historical archive (pre-migration)

This directory contains community-contributed benchmark results for various LLMs tested with Local Deep Research.

## Contributing Your Results

### Easy Method (v0.6.0+)
1. Run benchmarks using the LDR web interface at `/benchmark`
2. Go to Benchmark Results page
3. Click the green "YAML" button next to your completed benchmark
4. Review the downloaded file and add any missing info (hardware specs are optional)
5. Submit a PR to add your file to this directory

### Manual Method
1. Run benchmarks using the LDR web interface at `/benchmark`
2. Copy `benchmark_template.yaml` to a new file named: `[model_name]_[date].yaml`
   - Example: `llama3.3-70b-q4_2025-01-23.yaml`
   - Optional: Include your username: `johnsmith_llama3.3-70b-q4_2025-01-23.yaml`
3. Fill in your results manually
4. Submit a PR to add your file to this directory

## Important Guidelines

- **Test both strategies**: focused-iteration and source-based
- **Use consistent settings**: Start with 20-50 SimpleQA questions
- **Include all metadata**: Hardware specs, configuration, and versions are crucial
- **Be honest**: Negative results are as valuable as positive ones
- **Add notes**: Your observations help others understand the results
- **Review for PII**: If you include individual examples in your export, review the file for any personally identifiable information before submitting a PR

## Recommended Test Configuration

### For Large Models (70B+)
- Context Window: 32768+ tokens
- Focused-iteration: 8 iterations, 5 questions each
- Source-based: 5 iterations, 3 questions each

### For Smaller Models (<70B)
- Context Window: 16384+ tokens (adjust based on model)
- Focused-iteration: 5 iterations, 3 questions each
- Source-based: 3 iterations, 3 questions each

## Current Baseline

- **Model**: GPT-4.1-mini
- **Strategy**: focused-iteration (8 iterations, 5 questions)
- **Accuracy**: ~95% on SimpleQA (preliminary results from 20-100 question samples)
- **Search**: SearXNG
- **Verified by**: 2 independent testers

## Understanding the Results

### Accuracy Ranges
- **90%+**: Excellent - matches GPT-4 performance
- **80-90%**: Very good - suitable for most research tasks
- **70-80%**: Good - works well with human oversight
- **<70%**: Limited - may struggle with complex research

### Common Issues
- **Low accuracy**: Often due to insufficient context window
- **Timeouts**: Model too slow for iterative research
- **Memory errors**: Reduce context window or batch size
- **Rate limiting**: SearXNG may throttle excessive requests

## Viewing Results

Browse the YAML files in this directory to see how different models perform. Look for patterns like:
- Which quantization levels maintain accuracy
- Minimum viable model size for research tasks
- Best strategy for different model architectures
- Hardware requirements for acceptable performance

## Questions?

Join our [Discord](https://discord.gg/ttcqQeFcJ3) to discuss results and get help with benchmarking.

---

## File: docs/enhancements/metrics/dashboard-improvements.md

# Metrics Dashboard Enhancement Status

## Current State (Implemented Features)

The metrics dashboard now provides comprehensive analytics with:

### ✅ Core Analytics
- **Summary cards**: Total tokens, total researches, avg response time, success rate
- **Expandable token breakdown**: Click "Total Tokens Used" to view detailed input/output splits
- **Dual filtering system**: Time range (7D, 30D, 3M, 1Y, All) + Research mode (Quick Summary, Detailed, All)
- **Response time tracking**: Converted from milliseconds to seconds for better readability
- **Success rate monitoring**: Percentage of successful LLM calls

### ✅ Advanced Token Analytics
- **Per-research averages**: Input, output, and total tokens per research session
- **Total usage breakdown**: Complete token consumption across all sessions
- **Model usage analysis**: Detailed breakdown by model with token/call counts
- **Time-series visualization**: Token consumption trends over time with interactive charts

### ✅ Enhanced Data Collection
- **Call stack tracking**: Function-level LLM usage tracking with file/function context
- **Research mode filtering**: Separate analytics for quick vs detailed research types
- **Phase-based tracking**: Token usage by research phase and search iteration
- **Search engine tracking**: Monitor which search engines are being used

### ✅ User Experience Improvements
- **Interactive tooltips**: Contextual help throughout the dashboard
- **Responsive filtering**: Real-time dashboard updates when changing filters
- **Clean visual design**: Elegant expandable cards with smooth animations
- **Empty state handling**: Graceful handling when no data is available

## Pending Enhancement Proposals

### 1. Time-Series Analytics 📈 *(Partially Implemented)*

**✅ Completed:**
- Basic time-series token usage visualization
- Date range selectors (7D, 30D, 3M, 1Y, All)
- Timestamp-based queries in token counter

**🔄 Remaining:**
- **Research frequency patterns** - Identify peak usage times and patterns
- **Performance trends** - Monitor efficiency improvements/degradation over time
- **Seasonal analysis** - Understand usage patterns across different time periods
- **Hourly/daily heatmaps** - Visual patterns of usage intensity

### 2. Cost Analysis 💰

Transform token counts into actionable financial insights:
- **Token costs per model** - Calculate actual spending based on model pricing
- **Cost breakdown by research** - Show real $ spent per query
- **Budget tracking** - Set and monitor spending limits with alerts
- **Cost efficiency metrics** - ROI analysis for research investments
- **Projected costs** - Estimate future spending based on usage trends

**Implementation:**
- Add pricing configuration for different models
- Create cost calculation utilities
- Build budget management interface
- Add cost-based alerts and notifications

### 3. Quality & Performance Metrics ⚡ *(Partially Implemented)*

**✅ Completed:**
- **Research success rates** - Monitor completion vs failure rates
- **Average response time tracking** - LLM call response times in seconds
- **Error rate tracking** - Monitor and analyze failure patterns

**🔄 Remaining:**
- **Average research duration** - Time-to-completion analytics for entire research sessions
- **Token efficiency** - Quality of results per token spent
- **Query complexity analysis** - Resource usage for simple vs complex queries
- **Quality scoring mechanisms** - Automated research quality assessment

### 4. Individual Research Deep Dive 🔍 *(Partially Implemented)*

**✅ Completed:**
- **Recent researches list** - Clickable list with token counts and timestamps
- **Phase-based token tracking** - Usage by search, analysis, synthesis phases
- **Model utilization tracking** - Which models used for what tasks

**🔄 Remaining:**
- **Per-Research Page** (`/metrics/research/{id}`):
  - **Token usage timeline** - Visual flow of token consumption during research
  - **Search engine performance** - Effectiveness of different engines for the query
  - **Cost analysis** - Complete financial breakdown
  - **Efficiency scoring** - Performance rating vs similar researches

### 5. Comparative Analytics 📊

Enable data-driven optimization decisions:
- **Model performance comparison** - Efficiency metrics across different models
- **Research type analysis** - Scientific vs general queries patterns
- **A/B testing framework** - Compare different research strategies
- **Benchmarking** - Performance against historical averages
- **Best practices identification** - Highlight most efficient approaches

**Implementation:**
- Build comparison visualization tools
- Add research categorization system
- Create A/B testing infrastructure
- Design benchmarking algorithms

### 6. Resource Optimization Insights 🎯

Automated recommendations for cost optimization:
- **Waste detection** - Identify failed high-cost queries
- **Model recommendations** - Suggest cheaper alternatives for similar tasks
- **Usage anomalies** - Detect unusual patterns requiring attention
- **Optimization suggestions** - Actionable insights for cost reduction
- **Efficiency alerts** - Notify when performance degrades

**Implementation:**
- Create anomaly detection algorithms
- Build recommendation engines
- Add automated alert systems
- Design optimization suggestion framework

### 7. Advanced Visualizations 📉

Enhanced visual analytics:
- **Heatmaps** - Usage patterns by time of day/week
- **Flow diagrams** - Token flow through research stages
- **Comparative charts** - Side-by-side efficiency comparisons
- **Interactive dashboards** - Drill-down capabilities
- **Custom visualization builder** - User-configurable charts

**Implementation:**
- Integrate advanced charting libraries
- Create interactive visualization components
- Build custom chart configuration tools
- Add dashboard personalization features

### 8. Export and Reporting 📋

Data portability and external integration:
- **CSV/PDF exports** - Download metrics data
- **Scheduled reports** - Automated weekly/monthly summaries
- **Custom date range analysis** - Flexible reporting periods
- **API endpoints** - Integration with external analytics tools
- **Email notifications** - Automated report delivery

**Implementation:**
- Add export functionality to existing endpoints
- Create report generation system
- Build scheduling infrastructure
- Design external API interface

### 9. Real-time Monitoring 🔴

Live system monitoring:
- **Live token usage** - Real-time consumption tracking
- **System health metrics** - API response times, error rates
- **Resource alerts** - Threshold-based notifications
- **Active research monitoring** - Track ongoing research progress
- **Performance dashboards** - Real-time system status

**Implementation:**
- Add WebSocket-based real-time updates
- Create system health monitoring
- Build alert management system
- Design real-time dashboard components

### 10. Enhanced Model Analytics 📄

Deep-dive model performance analysis:

**Per-Model Page** (`/metrics/model/{model_name}`):
- **Historical usage trends** - Model adoption over time
- **Cost efficiency analysis** - Price/performance comparisons
- **Task-specific performance** - Effectiveness for different research types
- **Usage recommendations** - When to use this model
- **Comparative benchmarks** - Performance vs other models

**Implementation:**
- Create model-specific analytics pages
- Add task categorization system
- Build performance comparison tools
- Design recommendation algorithms

## Technical Considerations

### Database Enhancements
- Add cost tracking columns to metrics tables
- Implement time-series optimized storage
- Create indexes for performance queries

### Frontend Architecture
- Modular component design for reusability
- Responsive design for mobile accessibility
- Progressive loading for large datasets

### API Design
- RESTful endpoints for all metrics data
- Pagination for large result sets
- Caching for frequently accessed data

### Performance Optimization
- Database query optimization
- Client-side caching strategies
- Lazy loading for complex visualizations

## Recent Achievements Summary

### December 2024 Major Release
The metrics dashboard received a comprehensive overhaul with the following key features:

1. **Dual Filtering System** 🎯
   - Time range filtering: 7D, 30D, 3M, 1Y, All
   - Research mode filtering: Quick Summary, Detailed, All
   - Real-time dashboard updates when filters change

2. **Expandable Token Analytics** 📊
   - Interactive token breakdown card with click-to-expand functionality
   - Detailed input/output token splits
   - Average per research vs total usage comparison
   - Smooth animations and elegant design

3. **Enhanced Data Collection** 📈
   - Comprehensive call stack tracking with file/function context
   - Research phase and iteration tracking
   - Response time monitoring (converted to human-readable seconds)
   - Success rate calculation and display

4. **Improved User Experience** ✨
   - Interactive tooltips throughout the dashboard
   - Clean, responsive design with smooth animations
   - Graceful empty state handling
   - Contextual help and explanations

## Next Priority Features

Based on current gaps, the highest value remaining enhancements are:

1. **Cost Analysis** 💰 - Transform token usage into financial insights
2. **Real-time Monitoring** 🔴 - Live system monitoring and alerts
3. **Individual Research Pages** 🔍 - Detailed per-research analytics
4. **Export and Reporting** 📋 - Data portability and automated reports

## Success Metrics

- **User Engagement**: Time spent on metrics pages
- **Decision Impact**: Changes in model usage patterns
- **Cost Savings**: Reduction in unnecessary token spending
- **Efficiency Gains**: Improvement in tokens-per-successful-research ratio

---

*This document serves as a living specification for metrics dashboard enhancements. Each proposal should be evaluated for feasibility, user value, and implementation complexity before development.*

---

## File: docs/enhancements/metrics/enhanced-token-tracking.md

# Enhanced Token Tracking Data Collection

## Current Implementation Status

### ✅ **Implemented Features**
- **Token Usage Tracking** - Basic token metrics fully operational
- **Model Usage Tracking** - Model performance and usage patterns
- **Star Reviews Analytics** - Comprehensive rating analytics with visualizations
- **Metrics Dashboard Integration** - Full web UI with charts and data tables
- **Database Schema** - Stable schema with fixed foreign key constraints

### Current Token Tracking

The existing system tracks basic token usage:
- **research_id** - Links to specific research sessions
- **model_name** - LLM model used
- **provider** - Model provider (OpenAI, Anthropic, Ollama, etc.)
- **prompt_tokens, completion_tokens, total_tokens** - Token usage counts
- **timestamp** - When the LLM call was made

**Note**: Database foreign key constraints were recently fixed to ensure all token tracking data is properly saved without constraint errors.

## Proposed Enhanced Data Collection

### ✅ **Already Available (Can Add Immediately)**

#### Research Context
- **research_query** - The original research question/query
- **research_mode** - `"quick"` or `"detailed"` research mode
- **research_phase** - Current stage of research:
  - `init`, `setup`, `iteration_1`, `iteration_2`, etc.
  - `question_generation`, `parallel_search`, `search_complete`
  - `output_generation`, `report_generation`, `complete`, `error`

#### Performance Metrics
- **response_time_ms** - Duration of LLM call (wrap with timing)
- **success_status** - `"success"`, `"error"`, `"timeout"`
- **error_type** - Specific error if call failed

#### Search Engine Context
- **search_engines_planned** - Which engines were planned (from progress messages)
- **search_engine_selected** - Which engine was actually used
- **search_iteration** - Which search iteration this relates to

### ⚠️ **Requires Minor Implementation Work**

#### Cost Analysis
- **estimated_cost_usd** - Calculated cost based on model pricing tables
- **model_tier** - Classification: `"free"`, `"paid"`, `"premium"`

#### Enhanced Performance
- **query_length** - Character count of original research query
- **tokens_per_second** - Processing speed metric (calculated from response_time_ms)

#### Session Context
- **session_id** - Unique session identifier for grouping related requests
- **request_source** - Origin of request: `"web_ui"`, `"api"`, `"cli"`, `"benchmark"`

### ❌ **Not Feasible with Current Architecture**

#### Complex Quality Metrics
- **response_quality_score** - Would require LLM evaluation system
- **search_results_used** - Not directly tracked in current system
- **result_relevance** - Would need sophisticated analysis

#### Advanced Context
- **retry_count** - Would require modifying LLM retry logic
- **cache_hit** - No caching system currently implemented
- **concurrent_requests** - Complex to track accurately

## Implementation Details

### Current Database Schema (Implemented)

```sql
-- TokenUsage table (current implementation)
CREATE TABLE token_usage (
    id INTEGER PRIMARY KEY,
    research_id INTEGER,  -- Foreign key constraint removed to fix tracking
    model_name TEXT,
    provider TEXT,
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    total_tokens INTEGER,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- ModelUsage table (current implementation)
CREATE TABLE model_usage (
    id INTEGER PRIMARY KEY,
    research_id INTEGER,  -- Foreign key constraint removed to fix tracking
    model_name TEXT,
    provider TEXT,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- ResearchRating table (fully implemented)
CREATE TABLE research_rating (
    id INTEGER PRIMARY KEY,
    research_id INTEGER NOT NULL,
    rating INTEGER NOT NULL CHECK (rating BETWEEN 1 AND 5),
    feedback TEXT,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
```

### Enhanced Database Schema (Proposed for Future)

```sql
-- Enhanced TokenUsage table
CREATE TABLE enhanced_token_usage (
    id INTEGER PRIMARY KEY,
    research_id INTEGER,
    model_name TEXT,
    provider TEXT,

    -- Existing token metrics
    prompt_tokens INTEGER,
    completion_tokens INTEGER,
    total_tokens INTEGER,

    -- NEW: Research context (available immediately)
    research_query TEXT,
    research_mode TEXT,
    research_phase TEXT,
    search_iteration INTEGER,

    -- NEW: Performance metrics (available immediately)
    response_time_ms INTEGER,
    success_status TEXT DEFAULT 'success',
    error_type TEXT,

    -- NEW: Search engine context (available immediately)
    search_engines_planned TEXT, -- JSON array
    search_engine_selected TEXT,

    -- NEW: Cost analysis (minor work required)
    estimated_cost_usd REAL,
    model_tier TEXT,

    -- NEW: Enhanced metrics (minor work required)
    query_length INTEGER,
    tokens_per_second REAL,
    session_id TEXT,
    request_source TEXT,

    -- Timing
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
```

### Integration Points

#### TokenCountingCallback Enhancement
The callback can be enhanced to capture additional context:

```python
class TokenCountingCallback:
    def __init__(self, research_id=None, research_context=None):
        self.research_id = research_id
        self.research_context = research_context or {}

    def on_llm_start(self, serialized, prompts, **kwargs):
        self.start_time = time.time()
        # Extract research phase from context
        # Extract search engine info from context

    def on_llm_end(self, response, **kwargs):
        self.response_time = (time.time() - self.start_time) * 1000
        # Calculate additional metrics
        # Save enhanced data to database
```

#### Progress Message Integration
Extract search engine information from existing progress messages:

```python
# From research progress system
if "SEARCH_PLAN:" in message:
    engines = message.split("SEARCH_PLAN:")[1].strip()
    research_context["planned_engines"] = engines

if "ENGINE_SELECTED:" in message:
    engine = message.split("ENGINE_SELECTED:")[1].strip()
    research_context["selected_engine"] = engine
```

#### Cost Calculation System
Add model pricing configuration:

```python
MODEL_PRICING = {
    "gpt-4": {"prompt": 0.03, "completion": 0.06},  # per 1K tokens
    "gpt-3.5-turbo": {"prompt": 0.001, "completion": 0.002},
    "claude-3-opus": {"prompt": 0.015, "completion": 0.075},
    # ... other models
}

def calculate_cost(model_name, prompt_tokens, completion_tokens):
    if model_name in MODEL_PRICING:
        pricing = MODEL_PRICING[model_name]
        prompt_cost = (prompt_tokens / 1000) * pricing["prompt"]
        completion_cost = (completion_tokens / 1000) * pricing["completion"]
        return prompt_cost + completion_cost
    return 0.0
```

## Benefits of Enhanced Tracking

### Immediate Value
- **Cost visibility** - See actual spending per research and model
- **Performance monitoring** - Track response times and success rates
- **Phase analysis** - Understand where tokens are consumed in research process
- **Search optimization** - Identify most effective search engines

### Analytics Possibilities
- **Cost optimization** - Identify expensive queries and suggest alternatives
- **Performance trends** - Monitor system performance over time
- **Usage patterns** - Understand how different research modes perform
- **Model comparison** - Compare efficiency across different models

### Future Dashboard Features
- **Cost breakdown charts** - Visual cost analysis per model/research
- **Performance timeline** - Response time trends
- **Phase efficiency** - Token usage by research stage
- **Search engine effectiveness** - Success rates by engine

## Implementation Priority

### ✅ Phase 1: Core Infrastructure (COMPLETED)
1. ✅ Fixed database foreign key constraints to enable token tracking
2. ✅ Implemented star reviews analytics system
3. ✅ Created comprehensive metrics dashboard with visualizations
4. ✅ Added SQLAlchemy ORM-based data access patterns
5. ✅ Integrated Chart.js for data visualization
6. ✅ Added responsive design with dark theme integration

### Phase 2: Enhanced Token Tracking (NEXT)
1. Add research context (query, mode, phase)
2. Add response timing
3. Add success/error tracking
4. Add search engine context from progress messages

### Phase 3: Cost Analysis
1. Implement model pricing configuration
2. Add cost calculation to token tracking
3. Add model tier classification

### Phase 4: Advanced Features
1. Add session tracking
2. Add request source detection
3. Implement calculated metrics (tokens_per_second, etc.)

## Data Sources

### Research Service Context
- Research query, mode, and ID available in research service
- Phase information from progress callback system
- Session context from Flask request

### Progress Message System
- Search engine planning and selection
- Research iteration tracking
- Error states and completion status

### LLM Integration Points
- Token counts from LLM responses
- Model and provider information
- Response timing from callback wrapper

## Recent Implementation Work

### Star Reviews Analytics (December 2024)
- **Complete implementation** of dedicated star reviews page at `/metrics/star-reviews`
- **API endpoints** for real-time data aggregation using SQLAlchemy ORM
- **Interactive visualizations** including:
  - LLM model performance bar charts
  - Search engine effectiveness charts
  - Rating distribution over time
  - Research analytics with clickable links to research details
- **Dark theme integration** with responsive design
- **Comprehensive testing** with Puppeteer UI automation

### Database Fixes (December 2024)
- **Critical fix**: Removed foreign key constraints from TokenUsage and ModelUsage tables
- **Problem**: Foreign keys referenced non-existent "research_history" table causing silent failures
- **Solution**: Modified `src/local_deep_research/metrics/db_models.py` to use simple integer fields
- **Result**: Token tracking now works correctly without constraint errors

### Files Modified
- `src/local_deep_research/web/routes/metrics_routes.py` - Star reviews routes and API
- `src/local_deep_research/web/templates/pages/star_reviews.html` - Complete analytics UI
- `src/local_deep_research/web/templates/pages/metrics.html` - Navigation integration
- `src/local_deep_research/metrics/db_models.py` - Database schema fixes
- `tests/ui_tests/test_star_reviews_debug.js` - Comprehensive UI testing

### Current Status
- ✅ Token tracking fully operational
- ✅ Star reviews analytics complete and deployed
- ✅ Database schema stable and constraint-free
- ✅ Web UI responsive and functional
- ✅ Testing infrastructure in place

---

*This enhanced tracking system provides significantly more insight into research performance, costs, and patterns while remaining feasible to implement with the current architecture. The core infrastructure is now complete and ready for Phase 2 enhancements.*

---

## File: docs/release_notes/0.2.0.md

# Local Deep Research v0.2.0 Release Notes

We're excited to announce Local Deep Research v0.2.0, a major update that brings significant improvements to research capabilities, performance, and user experience.

## Major Enhancements

### New Search Strategies
- **Parallel Search**: Lightning-fast research that processes multiple questions simultaneously
- **Iterative Deep Search**: Enhanced exploration of complex topics with improved follow-up questions
- **Cross-Engine Filtering**: Smart result ranking across multiple search engines for higher quality information

### Improved Search Integrations
- **Enhanced SearxNG Support**: Better integration with self-hosted SearxNG instances
- **Improved GitHub Integration**: More effective search and analysis of code repositories
- **Better Source Selection**: Refined logic for choosing the most appropriate search engines per query

### Technical Improvements
- **Unified Database**: All settings and history now in a single `ldr.db` database
- **Improved Ollama Integration**: Better reliability and error handling with local models
- **Enhanced Error Recovery**: More graceful handling of connectivity issues and API errors

### User Experience
- **Enhanced Logging Panel**: Improved visibility with duplicate detection and better filtering
- **Streamlined Settings UI**: Reorganized settings interface with better organization
- **Research Progress Tracking**: More detailed real-time updates during research

### Development Improvements
- **PDM Support**: Switched to PDM for dependency management
- **Pre-commit Hooks**: Added linting and code quality checks
- **Code Security**: Added CodeQL integration with analysis scripts
- **Improved Documentation**: Better development guides and setup instructions

## API Changes

- Renamed and consolidated some API functions for consistency
- Added support for additional parameters in research configuration
- Improved error handling and response formatting

## Migration Notes

- The application now uses a unified database (`ldr.db`) that will automatically migrate data from older databases
- If upgrading from v0.1.x, your settings and research history will be automatically migrated on first run
- The `llm_config.py` file has been deprecated in favor of direct environment variable configuration

## Bug Fixes

- Fixed issues with settings persistence across sessions
- Resolved UI rendering problems in the history and results pages
- Fixed socket.io event handling and client disconnection issues
- Improved handling of large document collections
- Fixed API endpoint URL inconsistencies

## Contributors
This release represents the combined efforts of multiple contributors :
- @djpetti, @HashedViking, @LearningCircuit (core contributors to this release; sorted alphabetically)
- @dim-tsoukalas, @scottvr (sorted alphabetically)

## Get Involved

- Join our [Discord](https://discord.gg/ttcqQeFcJ3) for support and discussions
- Follow our [Subreddit](https://www.reddit.com/r/LocalDeepResearch/) for announcements and updates
- Report bugs and request features on our [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues)

---

## Installation

Download the [Windows Installer](https://github.com/LearningCircuit/local-deep-research/releases/download/v0.2.0/LocalDeepResearch_Setup.exe) or install via pip:

```bash
pip install local-deep-research
```

Requires Ollama or other LLM provider. See the [README](https://github.com/LearningCircuit/local-deep-research/blob/main/README.md) for complete setup instructions.

---

## File: docs/release_notes/0.4.0.md

# Local Deep Research v0.4.0 Release Notes

We're excited to announce Local Deep Research v0.4.0, bringing significant improvements to search capabilities, model integrations, and overall system performance.

## Major Enhancements

### LLM Improvements
- **Custom OpenAI Endpoint Support**: Added support for custom OpenAI-compatible endpoints
- **Dynamic Model Fetching**: Improved model discovery for both OpenAI and Anthropic using their official packages
- **Increased Context Window**: Enhanced default context window size and maximum limits

### Search Enhancements
- **Journal Quality Assessment**: Added capability to estimate journal reputation and quality for academic sources
- **Enhanced SearXNG Integration**: Fixed API key handling and prioritized SearXNG in auto search
- **Elasticsearch Improvements**: Added English translations to Chinese content in Elasticsearch files

### User Experience
- **Search Engine Visibility**: Added display of selected search engine during research
- **Better API Key Management**: Improved handling of search engine API keys from database settings
- **Custom Context Windows**: Added user-configurable context window size for LLMs

### System Improvements
- **Logging System Upgrade**: Migrated to `loguru` for improved logging capabilities
- **Memory Optimization**: Fixed high memory usage when journal quality filtering is enabled
- **Resumable Benchmarks**: Added support for resuming interrupted benchmark runs

## Bug Fixes
- Fixed broken SearXNG API key setting
- Memory usage optimizations for journal quality filtering
- Cleanup of OpenAI endpoint model loading features
- Various fixes for evaluation scripts
- Improved settings manager reliability

## Development Improvements
- Added test coverage for settings manager
- Cleaner code organization for LLM integration
- Enhanced API key handling from database settings

## New Contributors
- @JayLiu7319 contributed support for Custom OpenAI Endpoint models

## Full Changelog
For complete details of all changes, see the [full changelog](https://github.com/LearningCircuit/local-deep-research/compare/v0.3.12...v0.4.0).

---

## Installation

Download the [Windows Installer](https://github.com/LearningCircuit/local-deep-research/releases/download/v0.4.0/LocalDeepResearch_Setup.exe) or install via pip:

```bash
pip install local-deep-research
```

Requires Ollama or other LLM provider. See the [README](https://github.com/LearningCircuit/local-deep-research/blob/main/README.md) for complete setup instructions.

---

## File: docs/security/CODEQL_GUIDE.md

# Local CodeQL Analysis Guide

This guide explains how to set up and use CodeQL for local security analysis of the Local Deep Research project, covering both Python backend and JavaScript frontend code.

## Prerequisites

1. **CodeQL CLI**
   - Download the latest CodeQL bundle from [GitHub](https://github.com/github/codeql-action/releases)
   - Extract it to a permanent location (e.g., `C:\codeql` on Windows or `/opt/codeql` on Linux)
   - Add the CodeQL executable to your system PATH

2. **Ollama**
   - Install Ollama from [ollama.ai](https://ollama.ai)
   - Pull the required model: `ollama pull deepseek-r1:32b`

3. **Analysis Scripts**
   - Copy the appropriate analysis script from `docs/security/` to your project root:
     - Windows: `analyze_sarif.ps1`
     - Linux/Mac: `analyze_sarif.sh`

## Setup Instructions

### Windows

1. **Install CodeQL**
   ```powershell
   # Download and extract CodeQL bundle
   Invoke-WebRequest -Uri "https://github.com/github/codeql-action/releases/latest/download/codeql-bundle-win64.tar.gz" -OutFile "codeql.zip"
   tar -xvzf codeql.zip -C C:\codeql

   # Add to PATH (run as administrator)
   [Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\codeql\codeql", "Machine")
   ```

2. **Create CodeQL Databases**
   ```powershell
   # Navigate to project root/src (src is recommended for codeql analysis, to avoid analyzing venv folder)
   cd path/to/local-deep-research/src

   # Create Python database
   codeql database create --language=python --source-root . ./python-db

   # Create JavaScript database (for frontend code)
   codeql database create --language=javascript --source-root . ./js-db
   ```

3. **Run Analysis**
   ```powershell
   # Run Python CodeQL analysis
   codeql database analyze ./python-db python-security-and-quality.qls --format=sarif-latest --output=python-results.sarif

   # Run JavaScript CodeQL analysis
   codeql database analyze ./js-db javascript-security-extended.qls --format=sarif-latest --output=js-results.sarif

   # Merge results (optional)
   codeql dataset merge --output=combined-db --source=python-db --source=js-db
   codeql database analyze ./combined-db --format=sarif-latest --output=combined-results.sarif

   # Analyze results with Ollama
   .\analyze_sarif.ps1
   ```

### Linux/Mac

1. **Install CodeQL**
   ```bash
   # Download and extract CodeQL bundle
   wget https://github.com/github/codeql-action/releases/latest/download/codeql-bundle-linux64.tar.gz
   tar -xvzf codeql-bundle-linux64.tar.gz -C /opt/codeql

   # Add to PATH (add to ~/.bashrc or ~/.zshrc)
   export PATH=$PATH:/opt/codeql/codeql
   ```

2. **Create CodeQL Databases**
   ```bash
   # Navigate to project root/src
   cd path/to/local-deep-research/src

   # Create Python database
   codeql database create --language=python --source-root . ./python-db

   # Create JavaScript database
   codeql database create --language=javascript --source-root . ./js-db
   ```

3. **Run Analysis**
   ```bash
   # Make script executable
   chmod +x analyze_sarif.sh

   # Run Python CodeQL analysis
   codeql database analyze ./python-db python-security-and-quality.qls --format=sarif-latest --output=python-results.sarif

   # Run JavaScript CodeQL analysis
   codeql database analyze ./js-db javascript-security-extended.qls --format=sarif-latest --output=js-results.sarif

   # Merge results (optional)
   codeql dataset merge --output=combined-db --source=python-db --source=js-db
   codeql database analyze ./combined-db --format=sarif-latest --output=combined-results.sarif

   # Analyze results with Ollama
   ./analyze_sarif.sh
   ```

## Important Queries

The following CodeQL queries are particularly relevant for our project:

1. **Python Security**
   - `python-security-and-quality.qls`: General security and code quality
   - `python/sql-injection`: SQL injection vulnerabilities
   - `python/hardcoded-credentials`: Hardcoded secrets
   - `python/unsafe-deserialization`: Insecure deserialization

2. **JavaScript Security**
   - `javascript-security-extended.qls`: Comprehensive security checks
   - `javascript/xss`: Cross-site scripting vulnerabilities
   - `javascript/prototype-pollution`: Prototype pollution issues
   - `javascript/express-misconfigured-cors`: CORS misconfigurations
   - `javascript/unsafe-dynamic-import`: Unsafe dynamic imports
   - `javascript/unsafe-eval`: Unsafe eval() usage

3. **Custom Queries**
   - Logging injection vulnerabilities
   - Uninitialized variables
   - API security issues
   - Frontend security best practices

## Analysis Script Features

The analysis scripts (`analyze_sarif.ps1` and `analyze_sarif.sh`) provide:

1. **Input Validation**
   - Checks if Ollama is running
   - Validates SARIF file format
   - Verifies required dependencies

2. **Error Handling**
   - Graceful error messages
   - Color-coded output
   - Detailed error reporting

3. **Output**
   - Human-readable analysis
   - Prioritized findings
   - Recommended fixes
   - Language-specific recommendations

## Troubleshooting

1. **Ollama Connection Issues**
   - Ensure Ollama is running: `ollama serve`
   - Check if the model is pulled: `ollama list`
   - Verify the endpoint in the script matches your setup

2. **CodeQL Database Creation**
   - Clean previous databases: `rm -rf ./python-db ./js-db`
   - Ensure dependencies are installed (Python and Node.js)
   - Check for sufficient disk space
   - For JavaScript: Ensure node_modules is present

3. **Analysis Script Issues**
   - Verify script permissions (Linux/Mac)
   - Check PowerShell execution policy (Windows)
   - Ensure all required tools are installed

## Best Practices

1. **Regular Analysis**
   - Run analysis before major commits
   - Schedule regular security scans
   - Review and address findings promptly
   - Run both frontend and backend scans

2. **Database Management**
   - Clean old databases regularly
   - Keep CodeQL CLI updated
   - Use appropriate query suites
   - Consider using RAM disk for large projects

3. **Results Handling**
   - Document resolved issues
   - Track false positives
   - Share findings with the team
   - Prioritize critical frontend and backend issues

## Additional Resources

- [CodeQL Documentation](https://codeql.github.com/docs/)
- [Python CodeQL Queries](https://github.com/github/codeql/tree/main/python/ql/src)
- [JavaScript CodeQL Queries](https://github.com/github/codeql/tree/main/javascript/ql/src)
- [Ollama Documentation](https://ollama.ai/docs)

---

## File: docs/analytics-dashboard.md

# Metrics Dashboard Guide

The Local Deep Research metrics dashboard provides insights into your research activities, costs, and system performance.

> **Note**: This documentation is maintained by the community and may contain inaccuracies. While we strive to keep it up-to-date, please verify critical information and report any errors via [GitHub Issues](https://github.com/LearningCircuit/local-deep-research/issues).

## Overview

The metrics dashboard tracks:
- Token usage and costs across LLM providers
- Search engine performance and health
- Research session statistics
- User satisfaction ratings
- Rate limiting status

## Accessing Metrics

- **Web Interface**: Navigate to `/metrics` in your browser
- **Direct URL**: `http://localhost:5000/metrics`

## Dashboard Components

### System Overview Cards

The main dashboard displays key metrics:

- **Total Tokens Used**: Shows total token consumption with expandable breakdown by model
- **Total Researches**: Number of research sessions conducted
- **Average Response Time**: System performance metric
- **Success Rate**: Percentage of successful operations
- **User Satisfaction**: Average star rating from user feedback
- **Estimated Costs**: Token-based cost estimation with provider breakdown

### Time-based Filtering

Filter analytics by time period:
- Last 7 days
- Last 30 days
- Last 3 months
- Last year
- All time

Additional filters:
- Research mode (Quick Summary / Detailed / All)

## Detailed Analytics Pages

### Star Reviews Analytics

Access via: `/metrics/star-reviews`

**Features:**
- 5-star rating distribution
- Average ratings by time period
- Rating trends visualization
- Breakdown by model and search engine
- User feedback analysis

### Cost Analytics

Access via: `/metrics/costs`

**Tracked Metrics:**
- Cost breakdown by provider (OpenAI, Anthropic, etc.)
- Token usage details (input/output/total)
- Cost trends over time
- Model-specific cost analysis
- Research type cost comparison

### Rate Limiting Dashboard

**Real-time monitoring of:**
- Search engine rate limit status
- Success/failure rates per engine
- Wait time tracking
- Engine health indicators:
  - 🟢 Healthy: >95% success rate
  - 🟡 Degraded: 70-95% success rate
  - 🔴 Poor: <70% success rate

## Metrics Tracked

### Token Metrics
- Total tokens (input + output)
- Token usage by model
- Average tokens per research
- Token consumption trends

### Search Metrics
- Search engine usage frequency
- Response times per engine
- Success/failure rates
- Results count statistics

### Research Metrics
- Total research sessions
- Research duration
- Completion status
- Strategy usage
- Query complexity

### Performance Metrics
- API response times
- System latency
- Error rates
- Throughput statistics

## Data Export

### Research Reports
Export individual research results as:
- **PDF**: Formatted reports with citations
- **Markdown**: Raw markdown with formatting
- **JSON**: Structured data via API

### Analytics Data
Access analytics data via API:

```bash
# Get overall metrics
curl http://localhost:5000/api/metrics

# Get specific research metrics
curl http://localhost:5000/api/metrics/research/<research_id>

# Get enhanced tracking data
curl http://localhost:5000/api/metrics/enhanced

# Get rating analytics
curl http://localhost:5000/api/star-reviews

# Get cost analytics
curl http://localhost:5000/api/cost-analytics

# Get rate limiting status
curl http://localhost:5000/api/rate-limiting
```

## Visualizations

The dashboard uses Chart.js to display:

- **Line Charts**: Token usage and search activity over time
- **Bar Charts**: Model usage comparison, cost breakdown
- **Pie Charts**: Provider distribution, search engine usage
- **Progress Indicators**: Success rates, health status

## Cost Tracking

### Automatic Cost Calculation

Costs are calculated based on:
- Provider pricing (OpenAI, Anthropic, etc.)
- Actual token usage
- Model-specific rates
- Both input and output tokens

### Supported Providers
- OpenAI (GPT-3.5, GPT-4)
- Anthropic (Claude models)
- Google (Gemini models)
- Local models (shown as $0)

## Rate Limiting Analytics

### Monitoring Features
- Real-time rate limit status
- Historical rate limit events
- Automatic wait time optimization
- Per-engine performance tracking

### Managing Rate Limits

The learned wait times adapt automatically as searches run. To influence
them, adjust the rate limiting profile in Settings (conservative,
balanced, aggressive).

## Privacy and Data Storage

- All analytics data is stored locally
- No external analytics services used
- Data stored in SQLite databases
- Configurable data retention

## Using Analytics for Optimization

### Identify Cost Drivers
1. Review high-token queries in cost analytics
2. Compare model costs vs. quality ratings
3. Optimize model selection for different tasks

### Improve Search Performance
1. Monitor search engine health status
2. Identify frequently rate-limited engines
3. Adjust search strategy based on success rates

### Enhance Research Quality
1. Analyze user ratings by research type
2. Review low-rated sessions for patterns
3. Adjust parameters based on feedback

## Benchmarking Integration

For advanced users, analytics integrates with the benchmarking system:

- Track performance across different configurations
- Visualize optimization results (when matplotlib available)
- Compare quality vs. speed trade-offs
- Export benchmark data for analysis

## API Reference

### Metrics Endpoints

```bash
# General metrics with optional time filter
curl 'http://localhost:5000/api/metrics?days=30&mode=quick'

# Research-specific metrics
curl http://localhost:5000/api/metrics/research/

# Enhanced metrics (detailed tracking)
curl http://localhost:5000/api/metrics/enhanced

# Star ratings data
curl 'http://localhost:5000/api/star-reviews?days=30'

# Cost breakdown
curl 'http://localhost:5000/api/cost-analytics?provider=openai'

# Rate limit status
curl http://localhost:5000/api/rate-limiting
```

## Related Documentation

- [Features Documentation](features.md)
- [Configuration Guide](env_configuration.md)
- [Full Configuration Reference](CONFIGURATION.md)
- [API Documentation](api-quickstart.md)
- [Benchmarking Guide](BENCHMARKING.md)

---

## File: docs/api-quickstart.md

# API Quick Start

## Overview

Local Deep Research provides both HTTP REST API and programmatic Python API access. Since version 1.0, authentication is required for all API endpoints, and the system uses per-user encrypted databases.

## Simplest Usage - Python Client

The easiest way to use the API is with the built-in client that handles all authentication complexity:

```python
from local_deep_research.api import LDRClient, quick_query

# One-liner for quick research
summary = quick_query("username", "password", "What is DNA?")
print(summary)

# Or use the client for multiple operations
client = LDRClient()
client.login("username", "password")
result = client.quick_research("What is machine learning?")
print(result["summary"])
```

No need to worry about CSRF tokens, HTML parsing, or session management!

## Authentication

### Web UI Authentication

The API requires authentication through the web interface first:

1. Start the server:
   ```bash
   python -m local_deep_research.web.app
   ```

2. Open http://localhost:5000 in your browser
3. Register a new account or login
4. Your session cookie will be used for API authentication

### HTTP API Authentication

For HTTP API requests, you need to:

1. First authenticate through the login endpoint
2. Include the session cookie in subsequent requests
3. Include CSRF token for state-changing operations

Example authentication flow:

```python
import requests
from bs4 import BeautifulSoup

# Create a session to persist cookies
session = requests.Session()

# 1. Get login page and extract CSRF token for login
login_page = session.get("http://localhost:5000/auth/login")
soup = BeautifulSoup(login_page.text, 'html.parser')
csrf_input = soup.find('input', {'name': 'csrf_token'})
login_csrf = csrf_input.get('value') if csrf_input else None

# 2. Login with form data (not JSON) including CSRF
login_response = session.post(
    "http://localhost:5000/auth/login",
    data={
        "username": "your_username",
        "password": "your_password",
        "csrf_token": login_csrf
    }
)

if login_response.status_code in [200, 302]:
    print("Login successful")
    # Session cookie is automatically stored
else:
    print(f"Login failed: {login_response.text}")

# 3. Get CSRF token for API requests
csrf_response = session.get("http://localhost:5000/auth/csrf-token")
csrf_token = csrf_response.json()["csrf_token"]

# 4. Make API requests with CSRF header
headers = {"X-CSRF-Token": csrf_token}
api_response = session.post(
    "http://localhost:5000/api/start_research",
    json={
        "query": "What is quantum computing?",
        "model": "gpt-3.5-turbo",
        "search_engines": ["searxng"],
    },
    headers=headers
)
```

## Programmatic API Access

The programmatic API now requires a settings snapshot for proper context:

```python
from local_deep_research.api import quick_summary
from local_deep_research.settings import SettingsManager
from local_deep_research.database.session_context import get_user_db_session

# Get user session and settings
with get_user_db_session(username="your_username", password="your_password") as session:
    settings_manager = SettingsManager(session)
    settings_snapshot = settings_manager.get_all_settings()

    # Use the API with settings snapshot
    result = quick_summary(
        query="What is machine learning?",
        settings_snapshot=settings_snapshot,
        iterations=2,
        questions_per_iteration=3
    )

    print(result["summary"])
```

## API Endpoints

### Research Endpoints

Research endpoints are under `/api/`:

- `POST /api/start_research` - Start new research
- `GET /api/research/{id}/status` - Check research status
- `GET /api/report/{id}` - Get research results
- `POST /api/terminate/{id}` - Stop running research

### Settings Endpoints

Settings endpoints are under `/settings/api/`:

- `GET /settings/api` - Get all settings
- `GET /settings/api/{key}` - Get specific setting
- `PUT /settings/api/{key}` - Update setting
- `GET /settings/api/available-models` - Get available LLM providers
- `GET /settings/api/available-search-engines` - Get search engines

### History Endpoints

- `GET /history/api` - Get research history
- `GET /history/api/{id}` - Get specific research details

## Important Changes from v1.x

1. **Authentication Required**: All API endpoints now require authentication
2. **Settings Snapshot**: Programmatic API calls need settings_snapshot parameter
3. **Per-User Databases**: Each user has their own encrypted database
4. **CSRF Protection**: State-changing requests require CSRF token
5. **New Endpoint Structure**: Research APIs are under `/api/` (e.g., `/api/start_research`)

## Example: Complete Research Flow

```python
import requests
import time

# Setup session and login
session = requests.Session()
session.post(
    "http://localhost:5000/auth/login",
    json={"username": "user", "password": "pass"}
)

# Get CSRF token
csrf = session.get("http://localhost:5000/auth/csrf-token").json()["csrf_token"]
headers = {"X-CSRF-Token": csrf}

# Start research
research = session.post(
    "http://localhost:5000/api/start_research",
    json={
        "query": "Latest advances in quantum computing",
        "model": "gpt-3.5-turbo",
        "search_engines": ["arxiv", "wikipedia"],
        "iterations": 3
    },
    headers=headers
).json()

research_id = research["research_id"]

# Poll for results
while True:
    status = session.get(
        f"http://localhost:5000/api/research/{research_id}/status"
    ).json()

    if status["status"] in ["completed", "failed"]:
        break

    print(f"Progress: {status.get('progress', 'unknown')}")
    time.sleep(5)

# Get final results
results = session.get(
    f"http://localhost:5000/api/report/{research_id}"
).json()

print(f"Summary: {results['summary']}")
print(f"Sources: {len(results['sources'])}")
```

## Rate Limiting

The API includes adaptive rate limiting:
- Default: 60 requests per minute per user
- Automatic retry with exponential backoff
- Rate limits are per-user, not per-IP

## Error Handling

Common error responses:
- `401`: Not authenticated - login required
- `403`: CSRF token missing or invalid
- `404`: Resource not found
- `429`: Rate limit exceeded
- `500`: Server error

Always check response status and handle errors appropriately.

## Next Steps

- See [examples/api_usage](../examples/api_usage/) for complete examples
- Check [docs/CUSTOM_LLM_INTEGRATION.md](CUSTOM_LLM_INTEGRATION.md) for custom LLM setup
- Read [docs/LANGCHAIN_RETRIEVER_INTEGRATION.md](LANGCHAIN_RETRIEVER_INTEGRATION.md) for custom retrievers

---

## File: docs/BENCHMARKING.md

# Benchmarking System

The Local Deep Research benchmarking system evaluates search configurations, models, and strategies using standardized datasets to help you optimize performance.

**Important**: Benchmark results are indicators for configuration testing, not predictors of performance on your specific research topics. What works well on SimpleQA may perform differently on your actual research questions.

## Quick Start

### Web Interface
1. Navigate to **Benchmark** in the web interface
2. Configure your test:
   - Select datasets (SimpleQA recommended)
   - Set number of examples (start with 20-50)
   - Uses your current Settings configuration
3. Click **Start Benchmark** and monitor progress
4. View results in **Benchmark Results** page

## Datasets

### SimpleQA (Recommended)
- Fact-based questions with clear answers
- Best for testing general knowledge retrieval
- Good baseline for comparing configurations

### BrowseComp (Advanced)
- Complex browsing and comparison tasks
- Currently limited performance - use max 20 examples for testing

## Configuration Options

### Search Engines
- **Tavily**: AI-optimized commercial API
- **SearXNG**: Meta-search aggregating multiple engines
- **Brave**: Independent search engine
- **Specialized engines** (ArXiv, PubMed, Wikipedia): Not suitable for general SimpleQA testing

### Strategies
- **Focused Iteration**: Best for SimpleQA fact-based questions
- **Source-Based**: Better for comprehensive research

## Interpreting Results

### Key Metrics
- **Accuracy**: Percentage of correct answers
- **Processing Time**: Time per question (30-60s is typical)
- **Search Results**: Number of search results retrieved per query

### Performance Expectations
- **Focused iteration with SimpleQA**: Around 95% potential with optimal setup
- **Source-based strategy**: Around 70% accuracy, more comprehensive results

## Interpretation

Benchmark numbers are estimates, not exact measurements. This section helps reason about how much to trust a result and when a difference between two runs is meaningful.

### Confidence Intervals for a Single Run

A reported accuracy of "91%" is a point estimate with uncertainty that depends on how many examples you tested. The **Wilson score interval** gives the true range at 95% confidence (it behaves correctly near 0% and 100%, unlike the simpler normal approximation):

```
center    = (p̂ + z²/2n) / (1 + z²/n)
half-width = z × sqrt(p̂(1−p̂)/n + z²/4n²) / (1 + z²/n)

where p̂ = observed accuracy, n = examples run, z = 1.96 for 95% CI
```

**95% confidence margin of error by sample size:**

| Examples (n) | ~70% accuracy | ~85% accuracy | ~91% accuracy | ~95% accuracy |
|:---:|:---:|:---:|:---:|:---:|
| 20  | ±21% | ±17% | ±14% | ±10% |
| 50  | ±13% | ±10% | ±8%  | ±6%  |
| 100 | ±9%  | ±7%  | ±6%  | ±4%  |
| 200 | ±6%  | ±5%  | ±4%  | ±3%  |
| 500 | ±4%  | ±3%  | ±3%  | ±2%  |

> **Key takeaway:** A run of 20 examples has an uncertainty window of ±14–21%. A reported score of "91%" could plausibly be anywhere from 77% to 100%. Use at least 100 examples before drawing any conclusions, and 200+ before comparing configurations.

### Comparing Two Configurations

To tell whether config A is better than config B, you need enough examples that the observed difference is larger than the noise. The table below shows how many examples each configuration needs (run independently, same question set via the same seed) to reliably detect a given absolute accuracy difference at 80% statistical power (α = 0.05, two-sided):

| Difference to detect | Examples needed per config |
|:---|:---:|
| 5 pp (e.g., 85% vs 90%)  | ~680 |
| 10 pp (e.g., 80% vs 90%) | ~200 |
| 15 pp (e.g., 75% vs 90%) | ~90  |

**Rule of thumb:** If the observed difference between two runs is smaller than the margin of error for either run (see the table above), treat the results as a tie.

### When Two Runs Cannot Be Compared

Even with large sample sizes, cross-run comparison is unreliable if any of the following differ between the runs:

- **LDR version** — search logic, prompt templates, and result filtering may have changed between releases
- **Strategy** — `focused_iteration` and `source_based` answer questions differently by design; their scores measure different things
- **Grader model** — changing the evaluation LLM changes what "correct" means; the same system response may grade differently under different graders
- **Random seed / question sample** — some subsets of SimpleQA are inherently easier than others; always use `--seed 42` (or any fixed seed) consistently across compared runs
- **Search engine** — Tavily, SearXNG, and Brave retrieve different content; engine latency also affects what gets retrieved within per-query time limits

Treat each combination of (LDR version, strategy, search engine, grader model, seed) as a distinct experimental condition. Comparisons are only valid within the same condition.

### Evaluator LLM Error

The grader LLM (default: Claude 3.7 Sonnet via OpenRouter) is not perfect. On SimpleQA-style questions it mis-grades approximately **1% of responses** — consistent with calibration results reported in the original SimpleQA paper for similarly capable graders.

What this means in practice:

- **100 examples:** ~1 question is graded wrong. A 1 percentage-point difference (e.g., 91% vs 92%) is indistinguishable from grader noise alone.
- **500 examples:** ~5 questions are graded wrong. A 1% gap is still inside grader noise; differences of 3–4 pp start to be interpretable.
- The grader tends to be **conservative** — it marks ambiguous or partially-correct matches as incorrect — so reported accuracy is a slight underestimate of true accuracy.

**Don't optimize for differences smaller than ~2–3 pp on runs under 500 examples.** The signal is not there.

### Decision Checklist

Before acting on a benchmark result:

```
[ ] n ≥ 100 examples (use ≥ 200 when comparing two configurations)
[ ] Same random seed used across all compared runs
[ ] Same LDR version, strategy, search engine, and grader model
[ ] Observed difference > margin of error for each run (see table above)
[ ] Observed difference > ~2 pp (minimum meaningful above grader noise)
```

---

## Best Practices

### Testing Workflow
1. Start with 20 examples to verify configuration
2. Check that search results are being retrieved
3. Scale to 50-100 examples for reliable metrics
4. Adjust settings based on results

### Troubleshooting
- **Low accuracy**: Verify API keys and search engine connectivity
- **No search results**: Check API credentials and rate limiting
- **Very fast processing**: Usually indicates configuration issues

## Requirements

### API Keys Needed
- **Evaluation**: OpenRouter API key for automatic grading
- **Search**: API key for your chosen search engine
- **LLM**: API key for your language model provider

## Responsible Usage

- Start with small tests to verify configuration
- Use moderate example counts for shared resources
- Monitor API usage in the Metrics page
- Respect rate limits and shared infrastructure

## Important Limitations

Benchmarks test standardized questions and may not reflect performance on:
- Your specific domain or research topics
- Complex, multi-step research questions
- Real-time or recent information queries
- Specialized knowledge areas

Use benchmarks as configuration guidance, then test with your actual research topics to validate performance.

---

The benchmarking system helps you find starting configurations for reliable research results.