# Technical Documentation: PromtEngineer/localGPT

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/PromtEngineer/localGPT)

### Overview (`DOCKER_README.md`)

# 🐳 LocalGPT Docker Deployment Guide

This guide covers running LocalGPT in Docker containers, with Ollama either on the
host (default, best performance) or as a container.

## 🚀 Quick Start

### Complete Setup
```bash
# 1. Install Ollama locally
curl -fsSL https://ollama.ai/install.sh | sh

# 2. Start Ollama server
ollama serve

# 3. Install the models (in another terminal)
ollama pull qwen3.5:9b     # answer generation
ollama pull qwen3.5:4b     # routing, triage, enrichment, verification

# 4. Clone and start LocalGPT
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT
./start-docker.sh

# 5. Access the application
open http://localhost:3000
```

The first `up --build` installs the Python and Node dependencies and then the
`rag-api` container downloads its embedding model from HuggingFace (the reranker follows lazily on the first reranked query),
so allow several minutes before the UI is reachable.

Running this from a script or CI? Add `-y` so the fallback prompt never blocks:

```bash
./start-docker.sh local -y      # or: NONINTERACTIVE=1 ./start-docker.sh
```

## 📋 Prerequisites

- **Docker Desktop** (or Docker Engine 24+ with the Compose plugin), running
- **Ollama** on the host (recommended) or the containerized fallback below
- **8GB+ RAM** (16GB recommended)
- **20GB+ free disk space** — images, plus ~10GB of HuggingFace weights inside
  `rag-api`, plus the Ollama models

## 🏗️ Architecture

### Default Setup (Local Ollama + Docker Containers)
```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Frontend      │────│    Backend      │────│    RAG API      │
│  (Container)    │    │  (Container)    │    │  (Container)    │
│   Port: 3000    │    │   Port: 8000    │    │   Port: 8001    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
        │                                              │
        │  browser streams directly to :8001           │
        └──────────────────────────────────────────────┤
                                                       │ API calls
                                                       ▼
                                              ┌─────────────────┐
                                              │     Ollama      │
                                              │ (Host, default) │
                                              │   Port: 11434   │
                                              └─────────────────┘
```

**Why host Ollama by default?**
- ✅ Better performance (direct GPU access)
- ✅ Simpler setup (one less container)
- ✅ Easier model management
- ✅ Models survive `docker system prune`

### Containerized Ollama

`docker-compose.yml` also defines an `ollama` service behind the `with-ollama`
profile, for machines where you would rather not install Ollama:

```bash
./start-docker.sh container
# equivalently:
#   OLLAMA_HOST=http://ollama:11434 \
#   docker compose --env-file docker.env --profile with-ollama up --build -d

# Pull the models inside the container the first time
docker compose --profile with-ollama exec ollama ollama pull qwen3.5:9b
docker compose --profile with-ollama exec ollama ollama pull qwen3.5:4b
```

Its models persist in the named volume `ollama_data`.

### Startup Order

```
rag-api (healthy: GET /health)  →  backend (healthy: GET /health)  →  frontend
```

`backend` declares `depends_on: rag-api: condition: service_healthy`, and `rag-api`
does not answer `/health` until the agent and the embedding model are loaded (the
reranker is fetched lazily on the first query that reranks).
Seeing `backend` sit in `created` for a few minutes on a cold start is normal —
follow `docker compose logs -f rag-api`.

## 🛠️ Container Details

### Frontend Container (rag-frontend)
- **Image**: `node:20-alpine`, built by `Dockerfile.frontend`
- **Port**: 3000
- **Purpose**: Next.js web interface
- **Health Check**: busybox `wget -qO- http://localhost:3000` (the alpine image has no curl)
- **Build args**: `NEXT_PUBLIC_API_URL`, `NEXT_PUBLIC_RAG_API_URL` — inlined by `next build`

### Backend Container (rag-backend)
- **Image**: `python:3.11-slim`, built by `Dockerfile.backend`
- **Port**: 8000
- **Purpose**: sessions, indexes, uploads, chat history, gateway to the RAG API
- **Health Check**: `curl -f http://localhost:8000/health`
- **Working directory**: `/app`, started with `python backend/server.py`
- **Extras**: `sqlite3` CLI installed for database troubleshooting

### RAG API Container (rag-api)
- **Image**: `python:3.11-slim`, built by `Dockerfile.rag-api`
- **Port**: 8001
- **Purpose**: document indexing, retrieval, the agent loop
- **Health Check**: `curl -f http://localhost:8001/health`
- **Memory**: dominated by the resident embedding and reranker models
- **Concurrency**: single-threaded — one chat or indexing run at a time

## 📂 Volume Mounts & Data

### Persistent Data (bind mounts to host directories)
- `./lancedb/` → `/app/lancedb` — vectors and the native full-text index
- `./index_store/` → `/app/index_store` — document overviews
- `./shared_uploads/` → `/app/shared_uploads` — uploaded documents
- `./backend/` → `/app/backend` — `chat_data.db`

### Named volumes
- `ollama_data` — only used by the optional containerized Ollama

### Shared Between Containers
`backend` and `rag-api` mount the same four directories and both set
`DB_PATH=/app/backend/chat_data.db`, so they share one SQLite file and one vector
store. HuggingFace model weights are **not** mounted — they live in the container's
writable layer and are re-downloaded whenever `rag-api` is recreated.

## 🔧 Configuration

### Environment Variables (docker.env)
```bash
# Ollama on the host. Every service declares
# extra_hosts: ["host.docker.internal:host-gateway"], so this resolves on Linux too.
OLLAMA_HOST=http://host.docker.internal:11434
# Containerized alternative: OLLAMA_HOST=http://ollama:11434

# Service wiring
NODE_ENV=production
RAG_API_URL=http://rag-api:8001

# Browser-facing URLs, inlined into the frontend at build time
NEXT_PUBLIC_API_URL=http://localhost:8000
NEXT_PUBLIC_RAG_API_URL=http://localhost:8001

# Shared SQLite database and vector store
DB_PATH=/app/backend/chat_data.db
LANCEDB_PATH=/app/lancedb

# Models
GENERATION_MODEL=qwen3.5:9b
ENRICHMENT_MODEL=qwen3.5:4b
EMBEDDING_MODEL=microsoft/harrier-oss-v1-0.6b
RERANKER_MODEL=Qwen/Qwen3-Reranker-4B
```

Pass it explicitly (`start-docker.sh` does this for you):

```bash
docker compose --env-file docker.env up --build -d
```

Variables already exported in your shell take precedence over `--env-file`.

`NEXT_PUBLIC_*` are **build-time** for Next.js. `docker-compose.yml` forwards them
to `Dockerfile.frontend` as build args; changing them at runtime does nothing to an
already-built image, so rebuild:

```bash
NEXT_PUBLIC_API_URL=https://gpt.example.com/api \
NEXT_PUBLIC_RAG_API_URL=https://gpt.example.com/rag \
docker compose --env-file docker.env up -d --build frontend
```

### Model Configuration

| Role | Default | Documented options |
|------|---------|--------------------|
| Generation | `qwen3.5:9b` (Ollama) | `qwen3.6:27b` (high-end, ~17GB), `qwen3.5:4b` (light) |
| Enrichment / utility | `qwen3.5:4b` (Ollama) | `qwen3.5:2b` (light) |
| Embedding | `microsoft/harrier-oss-v1-0.6b` — HuggingFace, MIT, 1024 dims | `Qwen/Qwen3-Embedding-4B` (2560 dims, 32K context), `Qwen/Qwen3-Embedding-0.6B` (1024 dims) |
| Reranking (**on by default**) | `Qwen/Qwen3-Reranker-4B` — HuggingFace, loaded lazily on the first reranked query | `BAAI/bge-reranker-v2-m3`, `answerdotai/answerai-colbert-small-v1`, `Qwen/Qwen3-Reranker-0.6B` |

Only the two Ollama models need `ollama pull`. Embedding dimensions are read from
the loaded model, never hardcoded — **changing `EMBEDDING_MODEL` requires rebuilding
existing indexes**, and appending mismatched vectors to a LanceDB table raises an
explicit error. If the reranker cannot be loaded, the pipeline logs a warning and
continues without reranking.

## 🎯 Management Commands

### Start/Stop Services
```bash
# Start all services (local Ollama)
./start-docker.sh

# Start with containerized Ollama
./start-docker.sh container

# Stop all services
./start-docker.sh stop

# Restart services
./start-docker.sh stop && ./start-docker.sh

# Non-interactive (CI)
./start-docker.sh local -y
```

### Monitor Services
```bash
# Check container status
./start-docker.sh status
docker compose ps

# View live logs
./start-docker.sh logs
docker compose logs -f

# View specific service logs
docker compose logs -f rag-api
docker compose logs -f backend
docker compose logs -f frontend
```

### Manual Docker Compose
```bash
# Start manually
docker compose --env-file docker.env up --build -d

# Stop manually
docker compose down

# Rebuild a specific service (code is COPY-ed in, not mounted -
# a restart alone will not pick up source changes)
docker compose build --no-cache rag-api
docker compose up -d rag-api
```

### Health Checks
```bash
curl -f http://localhost:3000 && echo "✅ Frontend OK"
curl -f http://localhost:8000/health && echo "✅ Backend OK"
curl -f http://localhost:8001/health && echo "✅ RAG API OK"
curl -f http://localhost:11434/api/tags && echo "✅ Ollama OK"
```

`test_docker_build.sh` builds and smoke-tests each image individually against the
same endpoints.

## 🐞 Debugging

### Access Container Shells
```bash
# RAG API container (most debugging happens here)
docker compose exec rag-api bash

# Backend container
docker compose exec backend bash

# Frontend container (alpine -> sh, not bash)
docker compose exec frontend sh
```

### Common Debug Commands
```bash
# Test RAG system initialization
docker compose exec rag-api python -c "
from rag_system.factory import get_agent
agent = get_agent('default')
print('✅ RAG System OK')
"

# Test Ollama connection from the container
docker compose exec rag-api curl -s http://host.docker.internal:11434/api/tags

# Check the wiring the containers actually got
docker compose exec rag-api env | grep -E "OLLAMA|MODEL|DB_PATH|LANCEDB"
docker compose exec backend env | grep RAG_API_URL

# Verify the backend can reach the RAG API by service name
docker compose exec backend curl -s http://rag-api:8001/health

# View Python packages
docker compose exec rag-api pip list | grep -E "(torch|transformers|lancedb|docling|rerankers)"
```

### Resource Monitoring
```bash
docker stats

docker system df
du -sh lancedb shared_uploads index_store

docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"
```

## 🚨 Troubleshooting

### Common Issues

#### Container Won't Start
```bash
# Check logs for the specific error
docker compose logs [service-name]

# Rebuild from scratch
./start-docker.sh stop
docker system prune -f
./start-docker.sh

# Check for port conflicts
lsof -i :3000 -i :8000 -i :8001
```

#### `backend` never starts
It is waiting for `rag-api` to report healthy. Check:
```bash
docker compose ps
docker compose logs -f rag-api
docker compose exec rag-api curl -s http://localhost:8001/health
```

#### Can't Connect to Ollama
```bash
# Verify Ollama is running on the host
curl http://localhost:11434/api/tags

# Restart Ollama
pkill ollama
ollama serve

# Test from the container
docker compose exec rag-api curl -s http://host.docker.internal:11434/api/tags
```

#### Every chat answers "Could not connect to the RAG API server"
The backend builds its URLs from `RAG_API_URL`; inside compose that must be
`http://rag-api:8001`, since `localhost` there is the backend container itself.
```bash
docker compose exec backend env | grep RAG_API_URL
docker compose exec backend curl -s http://rag-api:8001/health
```

#### Memory Issues
```bash
docker stats --no-stream
free -h  # on the host

# Docker Desktop → Settings → Resources → Memory → 8GB+

# Lighter configuration
GENERATION_MODEL=qwen3.5:4b \
EMBEDDING_MODEL=Qwen/Qwen3-Embedding-0.6B \
docker compose --env-file docker.env up -d rag-api backend
# (changing the embedding model requires rebuilding your indexes)
```

#### Frontend Build Errors
```bash
docker compose build --no-cache frontend
docker compose up -d frontend
docker compose logs frontend
```

#### Frontend talks to the wrong host
`NEXT_PUBLIC_API_URL` / `NEXT_PUBLIC_RAG_API_URL` are baked in at build time.
Rebuild the frontend image after changing them.

#### Database/Storage Issues
```bash
# Check file permissions
ls -la backend/chat_data.db
ls -la lancedb/

# Reset permissions
chmod 664 backend/chat_data.db
chmod -R 755 lancedb/ shared_uploads/

# Inspect the database (sqlite3 is installed in the image)
docker compose exec backend sqlite3 /app/backend/chat_data.db ".tables"
```

### Performance Notes

- The RAG API is single-threaded: a chat request queued behind an indexing run
  looks like a hang. Check `docker compose logs -f rag-api`.
- Contextual enrichment makes one LLM call per chunk; it dominates indexing time.
  Turn it off in the index build options for the fastest ingest.
- `RAG_CONFIG_MODE=fast` switches the RAG API to vector-only retrieval with no
  reranking, decomposition or verification.

### Complete Reset
```bash
# Nuclear option - resets everything, including your documents and chat history
./start-docker.sh stop
docker compose down
rm -rf lancedb/* index_store/* shared_uploads/* backend/chat_data.db
docker system prune -a --volumes
./start-docker.sh
```

`docker compose down -v` alone only drops the `ollama_data` volume — application
data is in host directories and must be deleted explicitly.

## 🏆 Success Criteria

Your Docker deployment is successful when:

- ✅ `docker compose ps` shows all services healthy
- ✅ All health checks pass (see commands above)
- ✅ You can access http://localhost:3000
- ✅ You can upload documents and create indexes
- ✅ You can chat with your documents
- ✅ No errors in container logs

### What to Expect

- **First build**: installs torch/transformers/docling and builds the Next.js
  bundle; then `rag-api` downloads the ~1.2GB embedding model before it reports
  healthy (the ~7.5GB reranker downloads lazily, on the first reranked query)
- **Restarts**: fast. **Recreating** `rag-api` re-downloads the model weights,
  because no volume is mounted for the HuggingFace cache
- **Concurrency**: one RAG request at a time

## 📚 Additional Resources

- **Detailed Troubleshooting**: See `DOCKER_TROUBLESHOOTING.md`
- **Complete Docker Guide**: See `Documentation/docker_usage.md`
- **Deployment Guide**: See `Documentation/deployment_guide.md`
- **System Architecture**: See `Documentation/architecture_overview.md`
- **Direct Development**: See the main `README.md`

---

**Happy Dockerizing! 🐳** Need help? Check the troubleshooting guide or open an issue.

---

### Overview (`README.md`)

# LocalGPT - Private Document Intelligence Platform

<div align="center">

<p align="center">
<a href="https://trendshift.io/repositories/2947" target="_blank"><img src="https://trendshift.io/api/badge/repositories/2947" alt="PromtEngineer%2FlocalGPT | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
</p>

[](https://github.com/PromtEngineer/localGPT/stargazers)
[](https://github.com/PromtEngineer/localGPT/network/members)
[](https://github.com/PromtEngineer/localGPT/issues)
[](https://github.com/PromtEngineer/localGPT/pulls)
[](https://www.python.org/downloads/)
[](LICENSE)
[](https://www.docker.com/)

<p align="center">
    <a href="https://x.com/engineerrprompt">
      <img src="https://img.shields.io/badge/Follow%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white" alt="Follow on X" />
    </a>
    <a href="https://discord.gg/tUDWAFGc">
      <img src="https://img.shields.io/badge/Join%20our%20Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join our Discord" />
    </a>
  </p>
</div>

## 🚀 What is LocalGPT?

LocalGPT is a **fully private, on-premise Document Intelligence platform**. Ask questions, summarise, and uncover insights from your files with state-of-the-art AI—no data ever leaves your machine.

More than a traditional RAG (Retrieval-Augmented Generation) tool, LocalGPT features a **hybrid search engine** that fuses dense vector search with LanceDB's native full-text search, arbitrated by a **calibrated cross-encoder reranker**. A **smart router** picks between RAG and direct LLM answering for every query, while **contextual enrichment** and sentence-level [Context Pruning](https://huggingface.co/naver/provence-reranker-debertav3-v1) surface only the most relevant content. Optional passes — [Late Chunking](https://jina.ai/news/late-chunking-in-long-context-embedding-models/), an independent answer **verification** step, and experimental multi-vector (late-interaction) retrieval — can be switched on per config; the defaults ship with exactly the components that earned their place in measured evaluations (see [`eval/decisions/`](eval/decisions/)).

The architecture is **modular and lightweight**—enable only the components you need. The RAG core is plain Python built on the standard library's HTTP server, with no web framework and no agent framework in the way.

## ▶️ Video
Watch this [video](https://youtu.be/JTbtGH3secI) to get started with LocalGPT.

| Home | Create Index | Chat |
|------|--------------|------|
|  |  |  |

## ✨ Features

- **Utmost Privacy**: Your data remains on your computer, ensuring 100% security.
- **Versatile Model Support**: Swap generation models freely via Ollama.
- **Diverse Embeddings**: HuggingFace embedding models (harrier-oss-v1, the Qwen3-Embedding family) or any Ollama embedding tag.
- **Reuse Your LLM**: Once downloaded, reuse your LLM without the need for repeated downloads.
- **API**: A REST gateway on port 8000 and the RAG API on port 8001 for building your own applications.
- **CUDA, MPS & CPU**: Embedding and reranking pick CUDA, then Apple MPS, then CPU automatically.

### 📖 Document Processing
- **Formats**: PDF, DOCX, HTML/HTM, Markdown, and TXT, parsed by [Docling](https://github.com/docling-project/docling)
- **OCR fallback**: PDFs with no text layer are re-run through Docling's OCR pipeline; the engine is chosen from whatever is installed (OcrMac on macOS, then EasyOCR, RapidOCR, tesserocr, or the `tesseract` CLI)
- **Contextual Enrichment**: Chunk-level context generated by a small LLM, inspired by [Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval)
- **Late Chunking** (off by default): A second, document-level embedding pass stored in a companion `<table>_lc` table. The 2026-08-18 component ablation measured its removal at the noise floor on single-turn quality while it doubles the vectors written per index, so it now ships disabled; one flag (`retrieval.latechunk.enabled`) re-enables both the index-time build and the query-time leg — multi-turn conversations with drifting phrasing are where it earns its cost
- **Document Overviews**: A short per-document summary written to `index_store/overviews/<id>.jsonl` and used by the router

### 🤖 AI-Powered Chat
- **Natural Language Queries**: Ask questions in plain English
- **Source Attribution**: Answers come back with the chunks they were grounded in
- **Smart Routing**: Chooses between RAG and a direct LLM answer per query
- **Query Decomposition**: Splits complex questions into sub-questions, retrieves per sub-question, then pools the candidates for one rerank and one synthesis pass (per-sub-answer composition is available as an option)
- **Reciprocal Rank Fusion**: Vector and full-text hits are fused with RRF — no weights to tune
- **Reranking**: A cross-encoder pass over the fused candidate set, on by default with calibrated score-based selection ([`eval/DECISIONS.md`](eval/DECISIONS.md))
- **Sentence Pruning**: Optional Provence pruning drops irrelevant sentences from each chunk
- **Semantic Caching**: TTL cache with a 0.98 similarity threshold, scoped to the session
- **Answer Verification** (off by default): A second pass that appends `[Confidence: N%]` to the answer. Ablation measured zero verdict flips from disabling it — it annotates rather than changes answers — so it ships disabled; re-enable with `verification.enabled`

### 🛠️ Developer-Friendly
- **RESTful APIs**: Every UI action is a documented HTTP call
- **Streaming phases**: Server-Sent Events expose each pipeline stage as it runs
- **Flexible Configuration**: Models, chunk size, retrieval mode and toggles per request
- **One master config**: `rag_system/main.py` holds every default, overridable by environment variable

### 🎨 Modern Interface
- **Intuitive Web UI**: Clean, responsive design
- **Session Management**: Organize conversations by topic
- **Index Management**: Easy document collection management
- **Live Progress**: Retrieval, reranking and synthesis stages stream into the chat as they happen

---

## 🚀 Quick Start

### Prerequisites
- Python 3.10+ (3.11 recommended — the Docker images use `python:3.11-slim`)
- Node.js 20+ and npm
- Docker (optional, for containerized deployment)
- 8GB+ RAM (16GB+ recommended)
- Ollama (required for both deployment approaches)

### Option 1: Docker Deployment

```bash
# Clone the repository
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Install Ollama locally (recommended even for Docker)
curl -fsSL https://ollama.ai/install.sh | sh
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b

# Start Ollama
ollama serve

# Start with Docker (in a new terminal)
./start-docker.sh

# Access the application
open http://localhost:3000
```

If you would rather not install Ollama on the host, run it as a container instead:

```bash
./start-docker.sh container
# then pull the models inside the container
docker compose --profile with-ollama exec ollama ollama pull qwen3.5:9b
docker compose --profile with-ollama exec ollama ollama pull qwen3.5:4b
```

`./start-docker.sh` (with no argument) uses local Ollama. If nothing is listening on
port 11434 it offers to switch to the containerized Ollama; add `-y` (or set
`NONINTERACTIVE=1`) to take that fallback without a prompt in scripts and CI.

**Docker Management Commands:**
```bash
# Check container status
docker compose ps

# View logs
docker compose logs -f

# Stop containers
./start-docker.sh stop
```

### Option 2: Direct Development (Recommended for Development)

```bash
# Clone the repository
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Install Python dependencies
pip install -r requirements.txt

# Key dependencies installed:
# - torch==2.4.1, transformers==4.51.0 (embedding + reranker models)
# - lancedb (vector store and full-text search)
# - rerankers (cross-encoder reranking)
# - docling (document parsing)

# Install Node.js dependencies
npm install

# Install and start Ollama
curl -fsSL https://ollama.ai/install.sh | sh
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
ollama serve

# Start the system (in a new terminal)
python run_system.py

# Access the application
open http://localhost:3000
```

**System Management:**
```bash
# Check system health (loads the models and runs a sample query)
python system_health_check.py

# Real HTTP health checks against each service; exits non-zero if one is unhealthy
python run_system.py --health

# Start in production mode (runs `npm run build` before `next start`)
python run_system.py --mode prod

# Skip frontend (Ollama + RAG API + backend only)
python run_system.py --no-frontend

# Tail logs/*.log from another shell
python run_system.py --logs-only

# Stop everything recorded in logs/run_system.pid
python run_system.py --stop
# Or press Ctrl+C in the terminal running python run_system.py
```

**Service Architecture:**
The `run_system.py` launcher manages four services and writes their PIDs to `logs/run_system.pid`:
- **Ollama Server** (port 11434): model serving — reused if already running
- **RAG API Server** (port 8001): indexing, retrieval and the agent loop
- **Backend Server** (port 8000): sessions, indexes, uploads, chat history
- **Frontend Server** (port 3000): Next.js web interface (optional — skipped if `npm` is missing)

On startup the launcher checks that `qwen3.5:9b` and `qwen3.5:4b` are present and
runs `ollama pull` for anything missing.

### Option 3: Manual Component Startup

```bash
# Terminal 1: Start Ollama
ollama serve

# Terminal 2: Start RAG API
python -m rag_system.api_server
# equivalently: python -m rag_system.main api --port 8001

# Terminal 3: Start Backend
python backend/server.py

# Terminal 4: Start Frontend
npm run dev

# Access at http://localhost:3000
```

> Run every command from the repository root. Relative paths (`backend/chat_data.db`,
> `lancedb/`, `index_store/`, `shared_uploads/`) resolve against the current working
> directory, so `cd backend && python server.py` would create a second database at
> `backend/backend/chat_data.db`.

---

### Detailed Installation

#### 1. Install System Dependencies

**Ubuntu/Debian:**
```bash
sudo apt update
sudo apt install python3.11 python3-pip nodejs npm docker.io docker-compose-plugin
```

**macOS:**
```bash
brew install python@3.11 node docker
```

**Windows:**
```bash
# Install Python 3.10+, Node.js 20+, and Docker Desktop
# Then use PowerShell or WSL2
```

#### 2. Install AI Models

Only the two Ollama models need an explicit pull. The embedding model
(`microsoft/harrier-oss-v1-0.6b`, 1.2 GB) is downloaded from HuggingFace the
first time it is used; the reranker (~7.5 GB) is loaded lazily — downloaded
on the first reranked query.

```bash
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Pull the default models
ollama pull qwen3.5:9b          # answer generation
ollama pull qwen3.5:4b          # routing, triage, enrichment, verification
```

#### 3. Configure Environment (optional)

Every setting has a working default, so LocalGPT runs with no `.env` at all.
To override one, create a `.env` in the repository root (`rag_system/main.py`
calls `load_dotenv()` at import, before its config constants are evaluated; the
factory calls it again defensively). `.env.example` lists the same variables
with their code defaults.

| Variable | Default | Read by |
|----------|---------|---------|
| `OLLAMA_HOST` | `http://localhost:11434` | `rag_system/main.py`, `backend/ollama_client.py` |
| `RAG_API_URL` | `http://localhost:8001` | `backend/server.py` (builds `/chat` and `/index`) |
| `NEXT_PUBLIC_API_URL` | `http://localhost:8000` | `src/lib/api.ts` — inlined at `npm run build` |
| `NEXT_PUBLIC_RAG_API_URL` | `http://localhost:8001` | `src/lib/api.ts` — inlined at `npm run build` |
| `DB_PATH` | `backend/chat_data.db` | `backend/database.py` |
| `LANCEDB_PATH` | `storage.lancedb_uri` (`./lancedb`) | `rag_system/main.py` (pipeline profiles), `backend/database.py`, `system_health_check.py` |
| `GENERATION_MODEL` | `qwen3.5:9b` | `rag_system/main.py`, `backend/server.py`, `run_system.py` |
| `ENRICHMENT_MODEL` | `qwen3.5:4b` | `rag_system/main.py`, `backend/server.py`, `run_system.py` |
| `EMBEDDING_MODEL` | `microsoft/harrier-oss-v1-0.6b` | `rag_system/main.py` |
| `RERANKER_MODEL` | `Qwen/Qwen3-Reranker-4B` (loaded lazily on the first reranked query) | `rag_system/main.py` |
| `RAG_CONFIG_MODE` | `default` | `rag_system/api_server.py` (`default` or `fast`) |
| `RAG_API_TIMEOUT` | `600` | `backend/server.py` (seconds to wait for a chat answer) |
| `RAG_API_INDEX_TIMEOUT` | `3600` | `backend/server.py` (seconds to wait for an indexing run) |
| `LLM_BACKEND` | `ollama` | `rag_system/main.py` (`ollama` or `watsonx`) |
| `HF_TOKEN` | unset | HuggingFace, for gated model downloads |

`NEXT_PUBLIC_*` values are baked into the frontend bundle by `next build`.
Changing them requires a rebuild (`npm run build`, or `docker compose build frontend`).

> **Changing `EMBEDDING_MODEL` invalidates existing indexes.** Vector width is read
> from the loaded model, and appending vectors of a different width to an existing
> LanceDB table raises an error telling you to rebuild. Re-create your indexes after
> switching embedding models.

#### 4. Initialize the System

```bash
# Run system health check
python system_health_check.py

# Initialize the SQLite database
python -c "from backend.database import ChatDatabase; ChatDatabase().init_database()"

# Test the RAG imports
python -c "from rag_system.factory import get_agent; print('✅ Installation successful!')"

# Validate the running services
python run_system.py --health
```

---

## 🎯 Getting Started

### 1. Create Your First Index

An **index** is a collection of processed documents that you can chat with.

#### Using the Web Interface:
1. Open http://localhost:3000
2. Click "Create New Index"
3. Upload your documents (PDF, DOCX, TXT, MD, HTML)
4. Configure processing options
5. Click "Build Index"

#### Using the CLI:
```bash
# Index a single file or a whole directory with the 'default' profile
python -m rag_system.main index ./my_documents

# Use the speed-optimised profile instead
python -m rag_system.main index ./my_documents --mode fast

# Ask one question and print the JSON result
python -m rag_system.main chat "What are the key findings?"
```

`index` walks a directory for `.pdf`, `.docx`, `.html`, `.htm`, `.md` and `.txt`
files. It writes into the profile's `storage.text_table_name` (`text_pages_v4`),
which is *not* the per-index table the web UI creates.

#### Using the interactive script (creates a UI-visible index):
```bash
# Guided prompts: name, documents, chunk size, models
python create_index_script.py

# Non-interactive, from a JSON file
python create_index_script.py --create-sample    # writes index_config.sample.json
python create_index_script.py --batch index_config.sample.json
```

#### Using the HTTP API:
```bash
# Create index
curl -X POST http://localhost:8000/indexes \
  -H "Content-Type: application/json" \
  -d '{"name": "My Index", "description": "My documents"}'

# Upload documents (form field name must be "files")
curl -X POST http://localhost:8000/indexes/INDEX_ID/upload \
  -F "files=@document.pdf"

# Build index
curl -X POST http://localhost:8000/indexes/INDEX_ID/build \
  -H "Content-Type: application/json" \
  -d '{"chunk_size": 512, "enable_enrich": true, "enable_latechunk": true}'
```

### 2. Start Chatting

Once your index is built:

1. **Create a Chat Session**: Click "New Chat" or use an existing session
2. **Select Your Index**: Choose which document collection to query
3. **Ask Questions**: Type natural language questions about your documents
4. **Get Answers**: Receive AI-generated responses with source citations

### 3. Advanced Features

#### Per-session and per-request model choice
```bash
# The session's default generation model
curl -X POST http://localhost:8000/sessions \
  -H "Content-Type: application/json" \
  -d '{"title": "High Quality Session", "model": "qwen3.6:27b"}'

# Override it for one message
curl -X POST http://localhost:8000/sessions/SESSION_ID/messages \
  -H "Content-Type: application/json" \
  -d '{"message": "Summarise section 3", "model": "qwen3.5:4b"}'
```

The embedding model is a property of the index, not the session — choose it when
you build the index.

#### API Integration
```python
import requests

# Talk to the RAG API directly
response = requests.post('http://localhost:8001/chat', json={
    'query': 'What are the key findings in the research papers?',
    'session_id': 'your-session-id',
    'retrieval_mode': 'hybrid',
    'retrieval_k': 20,
})

print(response.json()['answer'])
```

---

## 🔧 Configuration

All defaults live in `rag_system/main.py`. Every model name there can be
overridden with the environment variables listed above.

### Model Configuration

| Role | Default | Documented options |
|------|---------|--------------------|
| Generation (answers) | `qwen3.5:9b` | `qwen3.6:27b` (high-end, ~17GB), `qwen3.5:4b` (light) |
| Enrichment / utility (routing, triage, decomposition, verification) | `qwen3.5:4b` | `qwen3.5:2b` (light) |
| Embedding | `microsoft/harrier-oss-v1-0.6b` (MIT, 1024 dims) | `Qwen/Qwen3-Embedding-4B` (2560 dims, 32K context, for multilingual / long-context corpora), `Qwen/Qwen3-Embedding-0.6B` (1024 dims) |
| Reranker (on by default) | `Qwen/Qwen3-Reranker-4B` | `BAAI/bge-reranker-v2-m3` (low latency), `answerdotai/answerai-colbert-small-v1`, `Qwen/Qwen3-Reranker-0.6B` |

```python
# rag_system/main.py
OLLAMA_CONFIG = {
    "host": os.getenv("OLLAMA_HOST", "http://localhost:11434"),
    "generation_model": os.getenv("GENERATION_MODEL", "qwen3.5:9b"),
    "enrichment_model": os.getenv("ENRICHMENT_MODEL", "qwen3.5:4b"),
}

EXTERNAL_MODELS = {
    "embedding_model": os.getenv("EMBEDDING_MODEL", "microsoft/harrier-oss-v1-0.6b"),
    "reranker_model": os.getenv("RERANKER_MODEL", "Qwen/Qwen3-Reranker-4B"),
}
```

Embedding dimensions are never hardcoded — they are measured from the vectors the
loaded model produces. If the reranker fails to load, the pipeline logs a warning
and continues **without** reranking rather than falling back to another model.

Vision / multimodal models are not part of the pipeline. PDF parsing and OCR are
handled by Docling. Models such as GLM-OCR or Qwen3-VL could be added as a
pre-processing step, but **they are not integrated today**.

### Pipeline Configuration

`PIPELINE_CONFIGS` has exactly two profiles. Select one with `RAG_CONFIG_MODE`
(RAG API) or `--mode` (CLI).

#### Default Pipeline (Production-Ready)
```
/* Detailed source-code truncated for AI context efficiency. */
```

#### Fast Pipeline (Speed-Optimized)
```python
"fast": {
    "description": "Speed-optimized pipeline with minimal overhead",
    "retrieval": {
        "search_type": "vector_only",
        "latechunk": {"enabled": False},
        "dense": {"enabled": True}
    },
    "reranker": {"enabled": False},
    "query_decomposition": {"enabled": False},
    "verification": {"enabled": False},
    "retrieval_k": 10,
    "contextual_enricher": {"enabled": False},
    "indexing": {
        "embedding_batch_size": 100,
        "enrichment_batch_size": 50
    }
}
```

### Retrieval Modes

`retrieval_mode` (wire name; `search_type` inside the pipeline config) accepts:

| Value | Behaviour |
|-------|-----------|
| `hybrid` *(default)* | Vector and LanceDB full-text legs run in parallel and are fused with Reciprocal Rank Fusion |
| `vector_only` | Dense vector search only |
| `fts_only` | LanceDB full-text search only |

Anything else is rejected with HTTP 400 by the RAG API. There is no
`dense_weight` / `denseWeight` knob — RRF needs no weights.

### Experimental: multi-vector (late-interaction) retrieval

The repo carries env-gated hooks for ColBERT-style multi-vector retrieval,
served by an out-of-process sidecar (SentenceTransformers v6 needs a newer
torch/transformers stack than the pinned in-repo one). All measured, none
default — the full study is in `eval/decisions/multivector-retrieval-2026-08-19.md`,
`paraphrase-robustness-2026-08-20.md` and `union-fusion-2026-08-20.md`:

| Env | Behaviour | Measured verdict |
|-----|-----------|------------------|
| `MV_RETRIEVAL_ENDPOINT` | Multi-vector MaxSim replaces the dense leg | Loses on both document-phrased and paraphrased queries |
| + `MV_RRF_LEG=1` | Multi-vector runs as a third RRF leg | Break-even; small gain only on paraphrased queries |
| + `MV_UNION=1` | All three legs' candidates are unioned (no RRF cut) and the reranker arbitrates | Best config for paraphrase-heavy / conversational queries (+4/120 real); costs −3/120 on document-phrased queries |

Rule of thumb: if your users quote the documents' own vocabulary, keep the
default 2-leg hybrid; if they ask in their own words, `MV_UNION=1` is the
measured winner (at ~30–40% extra query latency plus the sidecar process).

## 🔬 Evaluation

Every retrieval component in the default profile earned its place in a measured
A/B — and several plausible features are off because they measurably didn't
(late chunking, verification, cross-ref hops, document escalation, multi-vector
retrieval). The harness lives in `eval/`:

- `eval/goldset/` — five 24-question corpora (technical RFCs, M&A documents,
  a service manual, HR policy, this project's docs), a 12-conversation
  multi-turn set (`multiturn.jsonl`), and `paraphrases.jsonl` — verified
  same-meaning rewrites of all 120 questions with ~0.21 content-word overlap,
  for measuring robustness to users who don't phrase queries like the documents.
- `eval/judge.py` — the groundedness judge (deterministic local model or a
  stronger LLM via `JUDGE_MODEL`); judged comparisons use blind multi-voter
  panels on every changed row.
- `eval/decisions/` — one dated record per experiment: setup, numbers,
  flip-level panel verdicts, and the decision. If you want to know why a
  default is what it is, the answer is in there.

---

## 🛠️ Troubleshooting

### Common Issues

#### Installation Problems
```bash
# Check Python version
python --version  # 3.10+ required, 3.11 recommended

# Check dependencies
pip list | grep -E "(torch|transformers|lancedb|docling|rerankers)"

# Reinstall dependencies
pip install -r requirements.txt --force-reinstall
```

#### Model Loading Issues
```bash
# Check Ollama status
ollama list
curl http://localhost:11434/api/tags

# Pull missing models
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
```

#### Database Issues
```bash
# Check database connectivity
python -c "from backend.database import ChatDatabase; db = ChatDatabase(); print('✅ Database OK')"

# Reset database (WARNING: This deletes all sessions, messages and index metadata)
rm backend/chat_data.db
python -c "from backend.database import ChatDatabase; ChatDatabase().init_database()"
```

#### Dimension mismatch after changing the embedding model
```
ValueError: ... changing the embedding model requires rebuilding the index
```
Delete the affected index in the UI (or `DELETE /indexes/{id}`) and rebuild it.

#### Performance Issues
```bash
# Check system resources
python system_health_check.py

# Monitor memory usage
htop  # or Task Manager on Windows

# Use lighter models (the default embedder is already the small one at 1.2GB)
export GENERATION_MODEL=qwen3.5:4b
```

### Getting Help

1. **Check Logs**: `run_system.py` writes structured logs to `logs/`:
   - `logs/system.log`: launcher events
   - `logs/ollama.log`, `logs/rag-api.log`, `logs/backend.log`, `logs/frontend.log`: per-service output
   - `logs/run_system.pid`: PIDs used by `--stop`

2. **System Health**: Run diagnostics:
   ```bash
   python system_health_check.py  # loads models, runs a sample query
   python run_system.py --health  # HTTP checks, non-zero exit on failure
   ```

3. **Health Endpoints**:
   - Backend: `http://localhost:8000/health`
   - RAG API: `http://localhost:8001/health`
   - Ollama: `http://localhost:11434/api/tags`

4. **Documentation**: See [Documentation/system_overview.md](Documentation/system_overview.md), and [Documentation/design_rationale.md](Documentation/design_rationale.md) for why each component is built the way it is — with the evidence and the eval numbers behind every default, plus a "deliberately not implemented" list
5. **GitHub Issues**: Report bugs and request features
6. **Community**: Join our Discord

---

## 🔗 API Reference

Two HTTP services. The backend gateway on **:8000** owns sessions, indexes,
uploads and chat history; the RAG API on **:8001** owns retrieval and indexing.
Both accept `snake_case` and `camelCase` spellings of every option and normalise
them to one canonical key.

### Backend gateway — http://localhost:8000

```
/* Detailed source-code truncated for AI context efficiency. */
```

#### Session chat

```http
POST /sessions/{session_id}/messages
Content-Type: application/json

{
  "message": "What are the main topics discussed?",
  "model": "qwen3.5:9b",
  "retrieval_mode": "hybrid",
  "retrieval_k": 20,
  "reranker_top_k": 10,
  "context_window_size": 1,
  "ai_rerank": true,
  "context_expand": true,
  "query_decompose": true,
  "compose_sub_answers": true,
  "verify": true,
  "provence_prune": false,
  "provence_threshold": 0.1,
  "force_rag": false
}
```

Response:
```json
{
  "response": "…",
  "session": { "...": "updated session row" },
  "source_documents": [],
  "used_rag": true
}
```

The backend decides per message whether to answer directly with Ollama or to
forward to the RAG API. `force_rag: true` skips that decision and always calls the
RAG API. Both the user message and the answer are written to SQLite on this path.

### RAG API — http://localhost:8001

```http
GET  /health          # {"status": "ok"}
GET  /models          # {generation_models, embedding_models}
POST /chat            # {answer, source_documents}
POST /chat/stream     # Server-Sent Events, terminated by a "complete" event
POST /index           # run the indexing pipeline over file_paths
```

#### `POST /chat` and `POST /chat/stream`

```json
{
  "query": "Explain the methodology",
  "session_id": "uuid",
  "table_name": "text_pages_<index_id>",
  "model": "qwen3.5:9b",
  "retrieval_mode": "hybrid",
  "retrieval_k": 20,
  "context_window_size": 1,
  "reranker_top_k": 10,
  "ai_rerank": true,
  "context_expand": true,
  "query_decompose": true,
  "compose_sub_answers": true,
  "verify": true,
  "force_rag": false,
  "provence_prune": false,
  "provence_threshold": 0.1
}
```

`/chat` returns `{"answer": "...", "source_documents": [...]}`. There is no
top-level `confidence` field — when verification runs it appends
`[Confidence: N%]` (and a low-confidence warning) to the answer text itself.

`/chat/stream` emits `data: {"type": "<phase>", "data": {...}}` lines and ends
with a `complete` event carrying the same object `/chat` would return.

`force_rag: true` skips the agent's triage step so the query always goes through
retrieval; `verify`, `ai_rerank`, `query_decompose` and `compose_sub_answers`
still apply. An unsupported `retrieval_mode` is rejected with HTTP 400.

#### `POST /index`

```json
{
  "file_paths": ["/abs/path/doc1.pdf", "/abs/path/doc2.pdf"],
  "session_id": "uuid",
  "table_name": "text_pages_<index_id>",
  "chunk_size": 512,
  "window_size": 2,
  "retrieval_mode": "hybrid",
  "enable_enrich": true,
  "enable_latechunk": false,
  "enable_docling_chunk": true,
  "embedding_model": "microsoft/harrier-oss-v1-0.6b",
  "enrich_model": "qwen3.5:4b",
  "overview_model_name": "qwen3.5:4b",
  "batch_size_embed": 50,
  "batch_size_enrich": 25
}
```

`file_paths` is required; the values above are the defaults applied when a field
is omitted. Response:

```json
{
  "message": "Indexing process for 2 file(s) completed successfully.",
  "table_name": "text_pages_<index_id>",
  "latechunk": false,
  "docling_chunk": true,
  "indexing_config": {
    "chunk_size": 512,
    "retrieval_mode": "hybrid",
    "window_size": 2,
    "enable_enrich": true,
    "embedding_model": "microsoft/harrier-oss-v1-0.6b",
    "enrich_model": "qwen3.5:4b",
    "overview_model_name": "qwen3.5:4b",
    "batch_size_embed": 50,
    "batch_size_enrich": 25
  }
}
```

`retrieval_mode` at index time is validated and recorded with the index config;
it takes effect at query time. `enable_docling_chunk` defaults to `true`
(Docling structure-aware chunking); sending `false` selects the legacy chunker.
Indexing is synchronous — the call returns when
the pipeline finishes, which is why the backend allows up to
`RAG_API_INDEX_TIMEOUT` (default 3600s) for it.

For the full route table see [Documentation/api_reference.md](Documentation/api_reference.md).

### Known limitations

- **The RAG API is single-threaded.** Requests are serialised: one chat or
  indexing run at a time. The backend gateway is threaded, so it stays responsive,
  but a long RAG call blocks the next one.
- **Streamed turns are persisted after the fact.** The chat UI streams from
  `:8001/chat/stream` directly and, when the stream completes, saves the finished
  turn through the gateway (`POST /sessions/{id}/messages/save`). If the browser is
  closed mid-stream, that turn is not saved.
- **Index metadata is per index, not per session.** Choosing a different embedding
  model requires rebuilding the index.

---

## 🏗️ Architecture

```mermaid
graph TB
    UI[Next.js UI :3000] --> API[Backend gateway :8000]
    UI -. "SSE /chat/stream" .-> RAGAPI
    API --> RAGAPI[RAG API :8001]
    RAGAPI --> Agent[RAG Agent]
    Agent --> Retrieval[Retrieval Pipeline]
    Agent --> Ollama[Ollama :11434]

    Retrieval --> Vector[Vector search]
    Retrieval --> FTS[LanceDB full-text search]
    Vector --> RRF[Reciprocal Rank Fusion]
    FTS --> RRF
    RRF --> Rerank["Cross-encoder rerank (on by default)"]

    Vector --> LanceDB[(LanceDB)]
    FTS --> LanceDB

    API --> SQLite[(SQLite: sessions, messages, indexes)]
    RAGAPI --> SQLite
```

Overview of the Retrieval Agent

```
/* Detailed source-code truncated for AI context efficiency. */
```

Inside the retrieval pipeline a query runs: embed → hybrid retrieve (vector +
FTS, fused with RRF) → optional late-chunk leg → cross-encoder rerank (on by
default) → context window expansion → optional Provence sentence
pruning → synthesis.

---

## 🤝 Contributing

We welcome contributions from developers of all skill levels! LocalGPT is an open-source project that benefits from community involvement.

### 🚀 Quick Start for Contributors

```bash
# Fork and clone the repository
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Set up development environment
pip install -r requirements.txt
npm install

# Install Ollama and models
curl -fsSL https://ollama.ai/install.sh | sh
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b

# Verify setup
python system_health_check.py
python run_system.py --mode dev
```

### 📋 How to Contribute

1. **🐛 Report Bugs**: Use our [bug report template](.github/ISSUE_TEMPLATE/bug_report.md)
2. **💡 Request Features**: Use our [feature request template](.github/ISSUE_TEMPLATE/feature_request.md)
3. **🔧 Submit Code**: Follow our [development workflow](CONTRIBUTING.md#development-workflow)
4. **📚 Improve Docs**: Help make our documentation better

### 📖 Detailed Guidelines

For comprehensive contributing guidelines, including:
- Development setup and workflow
- Coding standards and best practices
- Testing requirements
- Documentation standards
- Release process

**👉 See our [CONTRIBUTING.md](CONTRIBUTING.md) guide**

---

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. For models, please check their respective licenses.

---

## 📞 Support

- **Documentation**: [Documentation/system_overview.md](Documentation/system_overview.md)
- **Issues**: [GitHub Issues](https://github.com/PromtEngineer/localGPT/issues)
- **Discussions**: [GitHub Discussions](https://github.com/PromtEngineer/localGPT/discussions)
- **Business Deployment and Customization**: [Contact Us](https://tally.so/r/wv6R2d)
---

<div align="center">

## Star History

[](https://star-history.com/#PromtEngineer/localGPT&Date)

</div>

---

### Overview (`WATSONX_README.md`)

# Watson X Integration with Granite Models

localGPT can run its LLM calls against IBM watsonx.ai Granite models instead of a local
Ollama server.

## Overview

`rag_system` supports two LLM backends, selected with the `LLM_BACKEND` environment
variable:

1. **Ollama** (`ollama`, the default) — models run locally.
2. **Watson X** (`watsonx`) — Granite models hosted on IBM watsonx.ai.

The switch is made in `rag_system/factory.py::_build_llm_client()`, which returns either an
`OllamaClient` or a `WatsonXClient` (`rag_system/utils/watsonx_client.py`) together with
the matching config dict (`OLLAMA_CONFIG` or `WATSONX_CONFIG` from `rag_system/main.py`).
Both dicts expose the same `generation_model` / `enrichment_model` keys, so the agent, the
retrieval pipeline and the indexing pipeline are unchanged.

### What the backend switch does and does not cover

Switched to Watson X:

- Answer generation and sub-answer composition (`generation_model`).
- Query routing, triage, query decomposition, contextual enrichment, document overviews and
  answer verification (`enrichment_model`).

**Always local, regardless of `LLM_BACKEND`:**

- **Embeddings.** `rag_system/indexing/representations.py::select_embedder()` returns a
  Hugging Face model when `EMBEDDING_MODEL` contains a `/`, and an Ollama embedder
  otherwise. There is no Watson X embedding path, and `WatsonXClient` exposes no embedding
  method. Point `EMBEDDING_MODEL` at a Hugging Face repo (the default
  `microsoft/harrier-oss-v1-0.6b`) so no Ollama server is needed for indexing.
- **Reranking** (on by default; `Qwen/Qwen3-Reranker-4B`, loaded lazily on the first
  reranked query) and **Provence
  sentence pruning** — both are local `transformers` models.
- **The backend gateway's direct-LLM path.** `backend/server.py` answers non-document
  questions through `backend/ollama_client.py`, which always
  talks to `OLLAMA_HOST`. The routing decision itself is the deterministic, no-LLM
  `should_use_rag` gate. Watson X only serves requests that reach the RAG API on port
  8001.

## Prerequisites

1. IBM Cloud account with watsonx.ai access
2. A watsonx.ai API key
3. A watsonx.ai project ID

### Getting your credentials

1. Go to [IBM Cloud](https://cloud.ibm.com/)
2. Navigate to the watsonx.ai service
3. Create or select a project
4. Get your API key from IBM Cloud IAM
5. Copy your project ID from the project settings

## Installation

The SDK is **not** installed by the root `requirements.txt` (it is listed there as a
commented-out optional extra). Install it explicitly:

```bash
pip install "ibm-watsonx-ai>=1.3.39"
```

`rag_system/requirements.txt` — the RAG-only dependency list — does pin it, so
`pip install -r rag_system/requirements.txt` also gets you the SDK.

Without the package, `WatsonXClient.__init__` raises
`ImportError: ibm-watsonx-ai package is required.` as soon as the agent is constructed.

## Configuration

Copy the example file and fill in your credentials:

```bash
cp env.example.watsonx .env
```

The variables, with the defaults from `rag_system/main.py`:

```bash
# Choose LLM backend (default: ollama)
LLM_BACKEND=watsonx

# Watson X credentials
WATSONX_API_KEY=your_api_key_here
WATSONX_PROJECT_ID=your_project_id_here
WATSONX_URL=https://us-south.ml.cloud.ibm.com

# Model configuration
WATSONX_GENERATION_MODEL=ibm/granite-13b-chat-v2
WATSONX_ENRICHMENT_MODEL=ibm/granite-8b-japanese
```

`WATSONX_API_KEY` and `WATSONX_PROJECT_ID` are mandatory: `_build_llm_client()` raises
`ValueError: Watson X configuration incomplete.` when either is empty.

Use model ids that exist in your watsonx.ai instance — the two above are only the code
defaults, and `ibm/granite-8b-japanese` in particular is unlikely to be the utility model
you want. IBM's
[supported foundation models](https://www.ibm.com/docs/en/watsonx/saas?topic=solutions-supported-foundation-models)
page lists what is currently available.

## Usage

### Running with Watson X

```bash
export LLM_BACKEND=watsonx
python -m rag_system.main api          # RAG API on port 8001
```

`python -m rag_system.api_server` is equivalent. To index or ask a one-off question from
the CLI:

```bash
python -m rag_system.main index ./shared_uploads
python -m rag_system.main chat "What is in these documents?"
```

Or programmatically:

```python
import os
os.environ['LLM_BACKEND'] = 'watsonx'

from rag_system.factory import get_agent

agent = get_agent(mode="default")
result = agent.run("What is artificial intelligence?")
print(result["answer"])
```

### Switching between backends

```bash
# Local Ollama
export LLM_BACKEND=ollama
python -m rag_system.main api

# Watson X
export LLM_BACKEND=watsonx
python -m rag_system.main api
```

### Using Watson X from the web UI

Document-grounded answers do come from Watson X, with two caveats.

**The model dropdown still lists Ollama tags.** The UI populates it from the *backend
gateway's* `GET :8000/models` (`src/lib/api.ts`), which always queries Ollama. Only the
RAG API's own `GET :8001/models` is backend-aware and returns the configured Granite ids
when `LLM_BACKEND=watsonx`.

**That mismatch is harmless.** A per-request `model` is applied only when it is valid for
the active backend — under Watson X the id must contain a `/`, so a stray Ollama tag such
as `qwen3.5:9b` is ignored with a warning and `WATSONX_GENERATION_MODEL` is used instead.
The override is also scoped to that single request and restored afterwards, so one user's
choice cannot leak into the next request.

The backend gateway also keeps using Ollama for its non-document fast path (its routing
decision is the deterministic, no-LLM `should_use_rag` gate — see above), so a fully
Ollama-free setup means talking to the RAG API on
port 8001 directly.

## API compatibility

`WatsonXClient` implements the three methods the RAG system uses from `OllamaClient`:

```python
from rag_system.utils.watsonx_client import WatsonXClient

client = WatsonXClient(
    api_key="your_api_key",
    project_id="your_project_id",
)

# Blocking completion -> {"response": str, "model": str, "done": True}
response = client.generate_completion(
    model="ibm/granite-13b-chat-v2",
    prompt="Explain quantum computing",
)
print(response['response'])

# Streaming completion -> yields text chunks
for chunk in client.stream_completion(
    model="ibm/granite-13b-chat-v2",
    prompt="Write a story about AI",
):
    print(chunk, end='', flush=True)
```

There is also `generate_completion_async()`, which runs the blocking call in an executor
(the IBM SDK has no native async API). The verifier uses it.

## Limitations

1. **No JSON mode.** `OllamaClient.generate_completion()` forwards `format="json"` to
   Ollama; `WatsonXClient.generate_completion()` accepts the argument and ignores it.
   Triage, the overview router, query decomposition and the verifier all rely on the model
   returning parseable JSON. When parsing fails the code falls back to safe defaults
   (route to RAG, do not decompose, no confidence tag), so answers still work but routing
   and verification are less reliable than on Ollama.

2. **Embeddings and rerankers are never Watson X.** See the section above — they load from
   Hugging Face (or Ollama) in-process.

3. **Streaming.** `stream_completion()` uses the SDK's `generate_text_stream()` and falls
   back to yielding the full response as a single chunk if that method is unavailable.

4. **Errors are swallowed into empty responses.** `generate_completion()` catches
   exceptions and returns `{"response": "", "error": ...}`, so an authentication or quota
   failure shows up as an empty answer rather than an HTTP error. Check the server log.

5. **Rate limits and cost.** Watson X is a metered cloud service; local Ollama is not.

## Troubleshooting

### `ImportError: ibm-watsonx-ai package is required`

Run `pip install "ibm-watsonx-ai>=1.3.39"` in the environment that runs the RAG API.

### `ValueError: Watson X configuration incomplete`

`WATSONX_API_KEY` or `WATSONX_PROJECT_ID` is empty. If you put them in `.env`, make sure the
process is started from the repository root — `load_dotenv()` resolves the file relative to
the working directory.

### Authentication errors

- Verify the API key is correct
- Check that the project ID matches an existing watsonx.ai project
- Ensure your IBM Cloud account has watsonx.ai access

### Model not found

- Verify the model id (e.g. `ibm/granite-13b-chat-v2`)
- Check that the model is available in your instance and region
- Some models require additional entitlements

### Connection errors

- Check your internet connection
- Verify `WATSONX_URL` matches your region
- Check the IBM Cloud status page

### Empty answers with no visible error

See limitation 4 — look at the RAG API's stdout for `Error generating completion: …`.

## Reverting to Ollama

```bash
unset LLM_BACKEND   # or set LLM_BACKEND=ollama
python -m rag_system.main api
```

Indexes built while Watson X was active remain valid: the embeddings were produced locally
and never touched Watson X.

## Support

For Watson X issues:
- [IBM watsonx Documentation](https://www.ibm.com/docs/en/watsonx/saas)
- [watsonx Developer Hub](https://www.ibm.com/watsonx/developer/)
- [IBM Cloud Support](https://cloud.ibm.com/docs/get-support)

For localGPT issues:
- [localGPT GitHub Issues](https://github.com/PromtEngineer/localGPT/issues)

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

This integration follows the same license as localGPT (MIT License).

---

### Overview (`backend/README.md`)

# localGPT Backend

The gateway between the Next.js frontend and the rest of the system. It owns chat sessions,
uploaded documents and index bookkeeping in SQLite, answers simple queries directly with
Ollama, and forwards document-grounded queries and indexing jobs to the RAG API.

```
frontend :3000  ──►  backend/server.py :8000  ──►  rag_system/api_server.py :8001  ──►  Ollama :11434
                              │
                              └──► Ollama :11434 (direct, non-RAG answers)
```

Built on the standard library's `http.server` (`ThreadingTCPServer`, so requests are
handled concurrently). No web framework.

## Prerequisites

1. **Python 3.10+** (3.11 recommended).

2. **Ollama** running locally:
   ```bash
   # https://ollama.com/download, or:
   curl -fsSL https://ollama.ai/install.sh | sh
   ollama serve
   ```

3. **Models** — the defaults the backend resolves to:
   ```bash
   ollama pull qwen3.5:9b   # generation
   ollama pull qwen3.5:4b   # enrichment / utility (used by the RAG API, not by the gateway)
   ```

4. **The RAG API** (`python -m rag_system.api_server`) if you want document-grounded
   answers or indexing. Without it, `/sessions/<id>/messages` still works for non-document
   queries, and RAG queries return a "could not connect" message.

## Setup

```bash
# From the repository root
pip install -r backend/requirements.txt   # requests, python-dotenv

# Smoke-test the Ollama connection (note: this pulls GENERATION_MODEL if it is missing)
python backend/ollama_client.py

# Start the server
python backend/server.py
```

The server listens on `http://localhost:8000`. Run it from the repository root so the
relative paths it uses (`shared_uploads/`, `index_store/`, `lancedb/`, `backend/chat_data.db`)
resolve the same way they do in the Docker image.

Usually you do not start it by hand — `python run_system.py` launches Ollama, the RAG API,
the backend and the frontend together.

## Configuration

All environment variables are optional; the value shown is the code default.

| Variable | Default | Used for |
| --- | --- | --- |
| `OLLAMA_HOST` | `http://localhost:11434` | Direct Ollama calls (`backend/ollama_client.py`). |
| `RAG_API_URL` | `http://localhost:8001` | Base URL for the RAG API; `/chat` and `/index` are built from it. |
| `RAG_API_TIMEOUT` | `600` | Seconds to wait for a RAG chat response before returning 504. |
| `RAG_API_INDEX_TIMEOUT` | `3600` | Seconds to wait for a RAG indexing run before returning 504. |
| `GENERATION_MODEL` | `qwen3.5:9b` | Default answer model. Resolved as env var → `rag_system.main.OLLAMA_CONFIG` → literal default. |
| `ENRICHMENT_MODEL` | `qwen3.5:4b` | Recorded in index metadata as the enrich/overview model. The gateway no longer calls it (routing is local since Phase 2.3). |
| `EMBEDDING_MODEL` | `microsoft/harrier-oss-v1-0.6b` | Recorded in the default index metadata. |
| `DB_PATH` | `backend/chat_data.db` (`/app/backend/chat_data.db` in Docker) | SQLite file for sessions, messages, indexes and documents. |
| `LANCEDB_PATH` | `storage.lancedb_uri` from the default pipeline profile (`./lancedb`) | Vector store the backend drops tables from when an index is deleted. |

The backend imports `PIPELINE_CONFIGS` and `OLLAMA_CONFIG` from `rag_system.main` when the
package is importable, and degrades gracefully (printing a warning) when it is not.

## Request routing

`POST /sessions/<id>/messages` decides per message whether to use RAG. The decision is
made by `should_use_rag(message, idx_ids, force_rag)` — a module-level function in
`server.py` with no LLM call, no file reads and no network I/O:

1. `force_rag` (or `forceRag`) in the body → RAG, unconditionally.
2. No indexes linked to the session → direct Ollama answer.
3. The whole message is smalltalk (`hello`, `thanks!`, `bye`, `ok` — a whole-message
   allowlist regex, max six words) or a question about the assistant itself
   (`who are you`, `what model are you`) → direct Ollama answer.
4. Everything else → RAG.

RAG queries are forwarded to `POST {RAG_API_URL}/chat`; the answer and `source_documents`
come back from there. Direct answers call Ollama with thinking disabled.

The gate is intentionally biased toward RAG ("escalate, don't pre-decide"). The RAG API's
agent triage runs on every forwarded request and can still answer directly, so the gateway
only has to keep greetings out of the retrieval pipeline — it does not have to be right
about which questions the documents can answer. The pre-retrieval enrichment-model router
and its keyword/length fallback (which misrouted any question containing the word "test")
were removed in Phase 2.3; `Documentation/research/` has the evidence.

`backend/test_gateway_routing.py` covers the gate: `.venv/bin/python backend/test_gateway_routing.py`.

**Option casing.** Chat and index options are accepted in both `camelCase` and
`snake_case` and normalised to one canonical `snake_case` key before being forwarded
(`rerankerTopK` → `reranker_top_k`, `enableLatechunk` → `enable_latechunk`, and so on).
Options the caller omits are left out of the payload entirely so the RAG pipeline's own
defaults apply.

**Chat history.** This server is the only writer of chat message rows. The frontend's
streaming path talks to `:8001/chat/stream` directly and then persists the completed
turn here via `POST /sessions/<id>/messages/save`.

## API Endpoints

`Documentation/api_reference.md` has the request/response detail; this is the route table.

### GET

| Route | Returns |
| --- | --- |
| `/health` | `{status, ollama_running, available_models, database_stats}` |
| `/sessions` | `{sessions, total}` |
| `/sessions/cleanup` | `{message, cleanup_count}` — deletes sessions with no messages |
| `/sessions/<session_id>` | `{session, messages}` |
| `/sessions/<session_id>/documents` | `{session, files, file_count}` |
| `/sessions/<session_id>/indexes` | `{indexes, total}` |
| `/models` | `{generation_models, embedding_models}` |
| `/indexes` | `{indexes, total}` |
| `/indexes/<index_id>` | the index record, or 404 |

### POST

| Route | Body | Returns |
| --- | --- | --- |
| `/chat` | `{message, model?, conversation_history?}` | `{response, model, message_count}` — legacy sessionless path, always direct Ollama |
| `/sessions` | `{title?, model?}` | `{session, session_id}` (201) |
| `/sessions/<id>/messages` | `{message, model?, force_rag?, …chat options}` | `{response, session, source_documents, used_rag}` |
| `/sessions/<id>/upload` | `multipart/form-data`, field `files` | `{message, uploaded_files}` |
| `/sessions/<id>/index` | optional JSON index options | the RAG API's `/index` response, or `{message: "No documents to index for this session."}` |
| `/sessions/<id>/rename` | `{title}` | `{message, session}` |
| `/sessions/<sid>/indexes/<iid>` | — | `{message}` — links an index to a session |
| `/indexes` | `{name, description?, metadata?}` | `{index_id}` (201) |
| `/indexes/<id>/upload` | `multipart/form-data`, field `files` | `{message, uploaded_files}` |
| `/indexes/<id>/build` | optional JSON index options | `{response, …applied options}` |

### DELETE

| Route | Returns |
| --- | --- |
| `/sessions/<session_id>` | `{deleted: true}`, or 404 |
| `/indexes/<index_id>` | `{message, index_id}`, or 404 — also drops the index's LanceDB table |

`OPTIONS` on any path returns the CORS preflight headers (`GET, POST, DELETE, OPTIONS`).

## Testing

```bash
curl http://localhost:8000/health

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Hello!"}'
```

For an end-to-end check of the whole stack use `python run_system.py --health` or
`python system_health_check.py`. There is no automated test suite in this directory.

## Frontend integration

The frontend reads `NEXT_PUBLIC_API_URL` (default `http://localhost:8000`) for this
server and `NEXT_PUBLIC_RAG_API_URL` (default `http://localhost:8001`) for the streaming
endpoint. Both are inlined at `next build` time, so they must be set before the frontend
is built.

## Implemented today

- Session-scoped chat with SQLite-persisted history and auto-generated titles.
- Document upload (per session and per index) into `shared_uploads/`.
- Indexing delegated to the RAG API, with the resulting configuration stored as index
  metadata.
- Vector-store bookkeeping: index records point at their LanceDB table, and deleting an
  index drops that table (the late-chunk `_lc` sibling is left behind).
- Retrieval-augmented answers with source documents, plus a direct-LLM fast path for
  general questions.

Streaming is **not** served by this process — the frontend streams from the RAG API's
`/chat/stream` endpoint directly.

---

### Overview (`eval/README.md`)

# `eval/` — Phase 0 evaluation harness

The gate for [`Documentation/research_roadmap.md`](../Documentation/research_roadmap.md).
Nothing in Phase 1 or Phase 2 lands without a measured delta against these numbers.
The numbers themselves live in [`BASELINE.md`](BASELINE.md).

Everything here is read-only with respect to the product: no file under
`rag_system/`, `backend/` or `src/` is modified or imported-and-monkeypatched.
The harness calls the shipped pipeline objects directly.

---

## Layout

| Path | What it is |
|------|------------|
| `corpora/atlas7_service_manual.pdf` | 2-page planted-fact PDF (fictional espresso machine). 21 facts. |
| `corpora/northwind_leave_policy.pdf` | 3-page synthetic HR handbook, generated by `make_hr_handbook.py`. 24 facts. |
| `corpora/*.facts.json` | Sidecars: `{id, topic, expected, summary}` per planted fact. `expected` is the verbatim answer-bearing substring. |
| `corpora/repo_docs.facts.json` | 31 prose anchors into `Documentation/*.md` — the real, heterogeneous, third corpus. Referenced in place, never copied. |
| `corpora/acquisition/*.pdf` | 10 interlinked synthetic M&A documents (2 pages each) — the **cross-reference** corpus, added for roadmap Phase 4. Reused verbatim from [PromtEngineer/agentic-file-search](https://github.com/PromtEngineer/agentic-file-search) `data/test_acquisition/`. |
| `corpora/acquisition.facts.json` | 100 planted facts **plus** a `cross_references` block: the 54 `Document:` / `Exhibit` / `Schedule` pointers between those PDFs, `to: null` for the 2 deliberately dangling ones. |
| `corpora/rfc/*.txt` | 23 interlinked IETF RFCs (QUIC / HTTP-3 family), byte-for-byte from rfc-editor.org — the only corpus whose naming and referencing conventions this project did not author. `corpora/rfc/download.py` re-fetches them and checks the link graph; `corpora/rfc/MANIFEST.md` is the rationale. Sidecar: `corpora/rfc/rfc.facts.json` (26 facts). |
| `corpora/verify_facts.py` | Gate 1: asserts every `expected` string exists in its source document, and every cross-reference cue in its `from` document. Recurses into subdirectories, so the `rfc` sidecar is covered too. |
| `corpora/make_hr_handbook.py` | Regenerates the HR PDF and re-checks its sidecar. |
| `build_goldset.py` | Reverse-generates one query per dimension tuple with `qwen3.5:4b`. One-shot. Covers the three Phase 0 corpora only — `acquisition.jsonl` and `rfc.jsonl` are hand-authored. |
| `finalize_goldset.py` | Applies the recorded human verification pass; writes the committed gold set. |
| `verify_crossref_goldset.py` | Row-level gate for `goldset/acquisition.jsonl`: answer in its named document, no verbatim leak into the query, each answer string unique to one document, `requires_crossref` / `multi_document` consistent with `anchor_doc`, and `anchor_doc` itself a corpus document. |
| `verify_rfc_goldset.py` | The same row-level gate ported to `goldset/rfc.jsonl` (normalisation handles the RFCs' 72-column hard wrapping). |
| `goldset/<corpus>.jsonl` | **The gold set of record.** 120 rows: 24 each for `atlas7`, `hr`, `docs`, `acquisition` and `rfc`. Plus `multiturn.jsonl` (12 hand-authored multi-turn rows) — no runner is wired for it yet. |
| `goldset/_generated/*.raw.jsonl` | Raw model output, kept for audit. |
| `run_eval.py` | Retrieval metrics: recall@5/10/20 first stage, nDCG@10 before and after rerank, the `*_final` family over the list the answer stage actually sees (post-rerank, post-crossref-hop), the `requires_crossref` slices, per-query latency, and the `final == first_stage` invariant check whenever nothing is allowed to reorder or append. |
| `judge.py` | Binary groundedness judge + its own validation harness. |
| `judge_validation.jsonl` | 20 hand-built cases, 10 grounded / 10 subtly ungrounded. |
| `judge_hard_cases.jsonl` | 18 real system answers, hand-adjudicated — the judge screen that decides judge swaps. |
| `validate_judge_hard.py` | Scores a candidate judge against `judge_hard_cases.jsonl`; majority vote over an odd number of runs, parse errors reported as errors rather than coerced to votes. |
| `smoke_e2e.py` | Starts both services on temp storage, indexes the Atlas-7 PDF over HTTP, asserts answers/citations/persistence, tears down. |
| `.eval_indexes/` | Cached LanceDB indexes, keyed by embedder. Git-ignored, safe to delete. |
| `results/` | Run outputs (JSON + log). Git-ignored. |

---

## Running it

All commands are from the repo root, with the project venv.

```bash
# gate 1 — planted facts really are in the source documents
.venv/bin/python eval/corpora/verify_facts.py

# gate 2 + retrieval metrics on the shipped defaults (reranking ON with
# threshold selection since arm G — that is what ships, so that is what a bare
# run measures)
.venv/bin/python eval/run_eval.py --corpus all \
  --json-out eval/results/shipped_defaults.json

# first-stage-only control arm
.venv/bin/python eval/run_eval.py --corpus all --no-rerank \
  --json-out eval/results/shipped_defaults_norerank.json

# with a different reranker, for the reranker A/B — naming one swaps the model
.venv/bin/python eval/run_eval.py --corpus all \
  --reranker Qwen/Qwen3-Reranker-4B \
  --json-out eval/results/rerank_ab.json

# judge validation (TPR / TNR / confusion matrix)
.venv/bin/python eval/judge.py --validate

# end-to-end, starts :8000 and :8001 as children on a temp DB, then kills them
.venv/bin/python eval/smoke_e2e.py
```

`run_eval.py` flags: `--embedder`, `--reranker`,
`--corpus {atlas7,hr,docs,mixed,acq,acq+docs,rfc,all}`, `--k`, `--chunk-size`,
`--no-rerank`, `--retry {profile,on,off}`, `--crossref-hop {profile,on,off}`,
`--overview-prefilter {profile,off,boost,restrict}`, `--overviews {off,on}`,
`--decompose`, `--aggregate {max,mean}`,
`--coverage-only`, `--force-reindex`, `--json-out`, `--verbose`.

`acq` and `acq+docs` are the Phase 4 corpora (see below); `rfc` is the
real-document shakedown corpus (`decisions/rfc-shakedown-2026-08-13.md`).
`mixed` deliberately does **not** include any of them: it is the corpus every
Phase 0/1/2 number is quoted against and it has to keep meaning the same thing.

Since roadmap Phase 2 the harness drives `RetrievalPipeline.retrieve_candidates()`
rather than calling `MultiVectorRetriever.retrieve()` itself, so first stage,
reranking **and the evidence-sufficiency retry** are the shipped code path. Two
consequences worth knowing:

* `--retry` defaults to `profile`, i.e. **on**, matching what ships. It is the one
  stage in the list below that the harness does *not* disable, because it is
  conditional: it only fires on queries whose first pass found weak evidence. Use
  `--retry off` for the control arm of an A/B. When it fires it makes one
  enrichment-model call, so a bare run is no longer fully deterministic — the
  *firing set* is deterministic, the rewrites are not.
* `first_stage_ms` now covers first stage + rerank + any retry on that query, and
  `rerank_ms` is reported as 0 on this path. Per-query timings are no longer
  split by stage.

`--decompose` runs `QueryDecomposer` once per gold row (cached under
`eval/.eval_indexes/_subqueries/`, keyed by corpus + decomposer model + prompt
version, so the on and off arms differ only in whether the sub-queries are
*used*) and hands them to the **rerank** stage. The first stage always uses the
full original query — that is the whole of roadmap item 2.2. With the rerank
stage off there is no consumer for sub-queries, so `--decompose` is skipped
entirely — literally a no-op, not even the LLM calls.

`--embedder` defaults to whatever `EMBEDDING_MODEL` resolves to in
`rag_system/main.py`, i.e. the shipped default `microsoft/harrier-oss-v1-0.6b`
unless the env var overrides it. Each embedder gets its own index directory
(`eval/.eval_indexes/<slug>/`) and the embedder is part of the cache
fingerprint, so two embedders can neither share nor inherit an index.

The rerank stage follows the shipped profile, which has it **on since arm G
(2026-08-14)**: `Qwen/Qwen3-Reranker-4B` with `min_score: 0.5` / `min_keep: 3` /
`top_k: 10` threshold selection ([`DECISIONS.md`](DECISIONS.md) records the
earlier off-by-default call it supersedes). When the stage is on the harness
keeps that selection block and overrides only the model name, so the `final`
metrics describe the list the answer stage actually sees — not the
reorder-without-selection stack a bare block would measure. `--reranker
<model>` swaps the model; `--no-rerank` forces the stage off.

Regenerating the gold set (only when the corpora or the dimension table change):

```bash
.venv/bin/python eval/build_goldset.py --corpus all   # LLM writes the questions
.venv/bin/python eval/finalize_goldset.py             # applies the recorded verification pass
```

---

## How the gold set is built

Reverse-generated the structured way, not "give me some questions":

1. **Dimension tuples** are hand-authored in `build_goldset.py`: (anchor fact(s))
   × (question type: `factoid` / `procedural` / `comparative` / `negative`)
   × (difficulty: `easy` / `hard`). 24 tuples per corpus.
   * `negative` means a question about a restriction, exclusion, limit or
     invalidating condition **that the document does state** — not a question the
     document cannot answer. Every gold row is answerable.
   * `easy` reuses the document's vocabulary; `hard` paraphrases away from it, so
     the lexical leg cannot carry the query.
2. **`qwen3.5:4b` phrases the question** for each tuple, through the repo's own
   `OllamaClient`. The label — the answer-bearing substring — is fixed by the
   tuple, never by the model.
3. **Every pair is verified by hand** (see `finalize_goldset.py`, which records
   the verdict and the reason for each row, and `goldset/*.jsonl`, which carries
   them per row under `verification`).
4. **Gate 1** (`verify_facts.py`): the expected string is in the source document.
5. **Gate 2** (`run_eval.py`, run automatically before scoring): the expected
   string survives conversion + chunking into at least one indexed chunk. Rows
   that fail are printed and land in `coverage_failures` in the results JSON —
   they are never silently dropped.

Gold relevance is **answer-bearing text**, not chunk ids, so the set survives
re-chunking, a different chunk size, and an embedder swap. That is the whole
reason it can gate Phase 1's A/B tests.

### The `acquisition` gold set is different

`goldset/acquisition.jsonl` is **hand-authored**, not model-generated, because
its whole purpose is a property no dimension tuple can express: *the answer is
in a different document from the one the question points at.* Eight of its 24
rows adapt questions from the source repo's `TEST_QUESTIONS.md`; the rest are
new. Its rows carry two extra fields on top of the shared schema:

| Field | Meaning |
|---|---|
| `dimensions.requires_crossref` | **bool.** `true` when the query's premise names or paraphrases document A while at least one `expected` string lives in document B, reachable from A only through an explicit reference. This slice is roadmap item 4.2's metric. |
| `expected_sources`, `anchor_doc`, `multi_document` | The document holding each `expected` string, the document the query's premise points at (`null` when the query names none), and whether the row's answer spans more than one document. `requires_crossref` is exactly `anchor_doc is not None and any(source != anchor_doc)`. |

Every row was verified mechanically by `verify_crossref_goldset.py`, and the
counts are recorded in [`BASELINE.md`](BASELINE.md) § *Phase 4 baseline*: the
`expected` string is in its named document, the query does not contain it
verbatim, and the string occurs in **exactly one** of the ten documents —
without that last check "the answer is in another document" would not be a
claim you could measure. Re-run it any time:

```bash
.venv/bin/python eval/verify_crossref_goldset.py
```

`by_dimension` therefore gains `requires_crossref=true` / `=false` buckets, and
`summary.<corpus>.crossref` / `.crossref_control` carry the same slice per
corpus (pooling across corpora would double-count rows that appear in both
`acq` and `acq+docs`). Rows without the key — every row outside `acquisition` —
are skipped, so no pre-existing slice moves.

## Metric definitions

* `recall@k` — over the first-stage (pre-rerank) ranking. `match: "any"` rows hit
  when one of the top-k chunks contains an expected string; `match: "all"` rows
  (the comparatives) hit only when the top-k *union* covers every expected
  string. Mean over queries.
* `nDCG@10` — binary per-chunk relevance (chunk contains any expected string),
  DCG over the top 10 divided by the IDCG of the same candidate set sorted
  ideally. A query whose candidates contain nothing relevant scores 0, so a
  first-stage miss is never hidden by the ranking metric.
* Consequence worth knowing: a `match: "all"` query that retrieved only one of
  its two anchors scores `recall = 0` but can still score `nDCG@10 = 1.0` —
  ranking was perfect, coverage was not. Read the two columns together.
* Latency is wall-clock per query, split into first stage and rerank, on the
  machine in `BASELINE.md`. It is not a benchmark of anything but this laptop.

## What the harness turns off, and why

`run_eval.py` starts from `PIPELINE_CONFIGS["default"]` and disables:

| Stage | Why |
|-------|-----|
| Contextual enrichment | One LLM round-trip per chunk. Nondeterministic, and it changes the indexed text, which would make the substring labels ambiguous. |
| Document overviews | One LLM round-trip per document; only feeds triage, which the harness does not exercise. |
| Late chunking | Doubles the vectors and merges sibling text into hits, which would smear the substring labels across chunk boundaries. |
| Context expansion | Same smearing problem; and when the reranker runs it is a no-op anyway (see `retrieval_pipeline.md`). |
| Query decomposition, verification, synthesis | Downstream of the two metrics being measured, and each adds LLM calls. Decomposition can be switched back on for the item-2.2 A/B with `--decompose`. |

The evidence-sufficiency retry is deliberately **not** in that list — see the
`--retry` note above.

`chunk_size` is 512, matching what the HTTP path sends (`api_server.py`), not the
1500-token CLI default.

## Caveats that the numbers do not show on their own

* **`atlas7`, `hr` and `acq` in isolation are saturated.** They are 1, 2 and 13
  chunks, so `k=20` sweeps the entire corpus and recall@k is 1.0 by construction.
  Their isolated rows are a smoke check on the plumbing, not a retrieval
  measurement. `mixed` is the row to track for Phase 0–2, `acq+docs` for Phase 4.
* **`acq+docs` is a weak stress test, honestly labelled.** Its 360 distractor
  chunks are localGPT documentation — topically disjoint from an M&A deal room,
  so they compete far less than same-domain distractors would. Read the crossref
  slice with the caveats in `BASELINE.md` § *The crossref slice is not weak*.
* **The `docs` corpus is live repo content.** Editing `Documentation/*.md`
  changes the corpus, invalidates the cached index, and moves the baseline.
  `improvement_plan.md` and `research_roadmap.md` are excluded for exactly this
  reason (`DOCS_EXCLUDE` in `run_eval.py`) — they are the files this harness's
  own bookkeeping edits.
* **Anchors are reused across dimensions.** The two PDFs have ~20 facts each and
  carry 24 queries each, so a few facts back two differently-typed questions.
* **The judge is an LLM.** Its TPR/TNR are measured on 20 hand-built cases, which
  is a small sample; treat the interval, not the point estimate, as the truth.

---

## Benchmarking roadmap Phase 4

Phase 4 (`Documentation/research_roadmap.md` § *Ideas adopted from
agentic-file-search*) is **implemented behind profile flags that ship OFF** —
`retrieval.crossref_hop`, `retrieval.overview_prefilter` and
`retrieval.document_escalation` are all `enabled: False` in the `default`
profile. The harness drives them with `--crossref-hop`, `--overview-prefilter`
and `--overviews`; this section is the protocol the A/Bs follow, written before
the code so the comparison cannot be retro-fitted to a result. The
pre-implementation numbers are recorded in [`BASELINE.md`](BASELINE.md) §
*Phase 4 baseline (pre-implementation)*; every "off" arm below should reproduce
them.

Ground rules, the same three as every other gate in this harness:

1. **Only `acq` and `acq+docs` move.** If a Phase 4 flag changes `mixed`, that
   is a regression to explain, not a result to report. Re-run `--corpus mixed`
   on the same tree for both arms — the `docs` corpus is live repo content, so
   comparing against a number measured on a different tree proves nothing.
2. **One flag at a time**, both arms in the same session, same embedder, same
   `k`, same `chunk_size`. `--force-reindex` whenever the flag touches index
   content or chunk metadata (4.2 and 4.3 do).
3. **Report the `requires_crossref=true` slice next to its control**, never
   alone: the whole claim of item 4.2 is a *gap* between the two, and both move
   when the retriever changes.

### 4.2 — `retrieval.crossref_hop`

The headline Phase 4 A/B, and the one the `acquisition` corpus exists for.

The switch is `--crossref-hop {profile,on,off}`: `apply_phase4_settings()`
writes `retrieval.crossref_hop.enabled` into the config exactly the way
`apply_retry_setting()` writes the retry block — in `run_eval.py`, not as an
env var, and not by editing the shipped profile. Then:

```bash
# off (must reproduce BASELINE.md § Phase 4)
.venv/bin/python eval/run_eval.py --corpus acq+docs --crossref-hop off --force-reindex \
  --json-out eval/results/p4_crossref_off.json
# on
.venv/bin/python eval/run_eval.py --corpus acq+docs --crossref-hop on --force-reindex \
  --json-out eval/results/p4_crossref_on.json
```

Read, in this order: **`summary["acq+docs"]["crossref"]["ndcg@10_first_stage"]`
against `["crossref_control"]`** (0.748 vs 0.796 today), then the crossref
recall vector, then `mixed` for collateral damage. Because the hop *adds*
candidates, also check that `candidates` per query has not silently grown past
`--k` — a recall win bought by retrieving more chunks is not a ranking win.
The `k = 3` column in the baseline table is the sharpest available comparison:
at `k = 3` the crossref slice is at 1.000 recall today, so a hop that pays for
itself has to show up as nDCG, not recall.

### 4.3 — `retrieval.overview_prefilter`

The only Phase 4 item that **cannot reuse the cached indexes**: the harness
disables document overviews unless `--overviews on` is given, and the prefilter
needs them. Both arms must therefore be run with overviews on, which costs one
LLM call per document at index time and makes the index build nondeterministic.
`--overviews on` builds into a separate `_ov` index directory and changes the
fingerprint, so it cannot clobber the tracked indexes.

```bash
# both arms need overviews ON; --overview-prefilter picks the arm
.venv/bin/python eval/run_eval.py --corpus acq+docs --overviews on --force-reindex \
  --overview-prefilter off --json-out eval/results/p4_overview_off.json
.venv/bin/python eval/run_eval.py --corpus acq+docs --overviews on --force-reindex \
  --overview-prefilter boost --json-out eval/results/p4_overview_on.json
```

Do not compare an overview-prefilter arm against the numbers in `BASELINE.md`:
those indexes have no overviews in them. The control arm must be re-measured.
`acq+docs` is the corpus that can show anything here — `acq` alone is 10
documents and 13 chunks, so "restrict to the top documents" has almost nothing
to restrict.

### 4.1 — `retrieval.document_escalation`

**Not a first-stage retrieval metric.** Escalation reassembles a document for
*synthesis* after the 2.1 retry still lands weak, so `run_eval.py` cannot see
it: the harness stops at the ranked chunk list and never synthesises. Two
things it *can* contribute, and they are worth logging:

* **The trigger set.** The retry already reports per query
  (`summary.retry_fired` / `retry_fire_rate` / `retry_kept`; today 1/24 on
  `acq`, 8/48 on `acq+docs`, and 32/48 at `--k 5`). Whatever condition 4.1
  escalates on is a subset of that, so `--retry on` vs `--retry off` bounds how
  often escalation can fire before anyone writes it.
* **Everything else belongs in `smoke_e2e.py` and `judge.py`** — answer contains
  the fact, citation names the right document, groundedness verdict — driven
  over HTTP against a session indexed on `eval/corpora/acquisition/`. That is
  the arm that measures 4.1, and it is answer quality, not recall.

### 4.4 — filter DSL (`filters` on /chat and /chat/stream)

Measurable here only once `run_eval.py` can pass a `filters` argument through
`retrieve_candidates()`. The natural A/B on this corpus is a per-row
`filters` field in the gold set (e.g. `document = "07_nda.pdf"`) on the rows
whose answer document is unambiguous, then:

```bash
.venv/bin/python eval/run_eval.py --corpus acq+docs \
  --json-out eval/results/p4_filters_off.json
.venv/bin/python eval/run_eval.py --corpus acq+docs --filters \
  --json-out eval/results/p4_filters_on.json
```

The number that matters is nDCG@10 on the **control** slice, not the crossref
slice: a document filter derived from the query's anchor document is exactly
the wrong thing to apply to a crossref row, and a filter A/B that improves the
control while wrecking `requires_crossref=true` is the expected — and
reportable — outcome.

### 4.5 — per-query token/cost tracking

Nothing to A/B: it adds fields to the SSE `complete` event, it does not change
retrieval. Assert the fields exist and are non-zero in `smoke_e2e.py`; do not
give it a row in a retrieval table.

### 4.6 — `ask <folder>` ephemeral mode

Also not a `run_eval.py` job — it is a CLI entry point that builds a throwaway
index. The check it needs is an equivalence one: `python -m rag_system.main ask
eval/corpora/acquisition "<query>"` should answer the `acq` gold rows the same
way the persistent index does. Add it to `smoke_e2e.py` as a subprocess
assertion over a handful of gold rows (start with the four `requires_crossref`
rows that already score 1.000, so a failure is unambiguous), and record the
wall clock — the claim being tested is "an ephemeral index beats an ephemeral
agent", which is a latency claim as much as a quality one.

---

### Overview (`rag_system/README.md`)

# RAG System

This directory contains the retrieval-augmented generation engine behind localGPT: the
master configuration, the indexing and retrieval pipelines, the agent loop, and the HTTP
API that the backend gateway and the frontend talk to.

For a deeper, key-by-key reference (config tables, HTTP request/response shapes, SSE
events) see [`DOCUMENTATION.md`](DOCUMENTATION.md) next to this file.

## 1. Overview

The system is **text-only**. Documents are converted to Markdown with
[Docling](https://github.com/docling-project/docling), chunked, optionally enriched with
LLM-generated context, embedded with a local Hugging Face model, and written to
**LanceDB**. Queries are answered by combining LanceDB's native full-text search with
vector search, reranking the merged candidates, and synthesising an answer with a local
Ollama model.

Core capabilities:

-   **Docling ingestion** — PDF, DOCX, HTML/HTM and MD go through Docling; TXT is read
    directly and wrapped in a fenced block. Scanned PDFs (no text layer) take an OCR
    pipeline when an OCR backend is installed.
-   **Hybrid retrieval** — LanceDB full-text (BM25-scored, native to LanceDB) and vector
    search run in parallel and are fused with reciprocal rank fusion. `vector_only` and
    `fts_only` run a single leg.
-   **Cross-encoder reranking** — the merged candidates are reordered by a reranker loaded
    through the [`rerankers`](https://github.com/AnswerDotAI/rerankers) library.
-   **Agentic query handling** — triage (documents vs. general knowledge), optional query
    decomposition with parallel sub-query retrieval, optional answer verification.
-   **Late chunking** — an optional second embedding pass that encodes the whole document
    and mean-pools per-chunk spans, stored in a sibling `<table>_lc` table.

There is **no multimodal path**. Nothing produces image embeddings and no
vision-language model is invoked; PDF understanding is Docling's layout parsing plus
OCR. If you want page-image understanding you have to build it; GLM-OCR or Qwen3-VL
would be reasonable starting points, but neither is integrated today.

## 2. Architecture

### Key modules

-   `main.py` — the MASTER configuration (`OLLAMA_CONFIG`, `WATSONX_CONFIG`,
    `EXTERNAL_MODELS`, `PIPELINE_CONFIGS`) plus a thin argparse CLI. It builds nothing
    itself; the CLI delegates to the factory.
-   `factory.py` — the single factory: `get_agent(mode)`, `get_indexing_pipeline(mode)`
    and `get_pipeline_config(mode)` (which returns a deep copy so per-request overrides
    cannot mutate the master config).
-   `api_server.py` — the HTTP API on port 8001 (`/health`, `/models`, `/chat`,
    `/chat/stream`, `/index`), built on the standard library's `http.server`.
-   `ingestion/` — everything that turns a file into chunks.
    -   `document_converter.py`: Docling converters (no-OCR, OCR, general) and OCR-engine
        selection.
    -   `docling_chunker.py`: token-budgeted chunker that walks the DoclingDocument tree,
        keeping tables atomic and recording the heading path on every chunk.
    -   `chunking.py`: `MarkdownRecursiveChunker`, the heading/paragraph recursive
        splitter used as the `legacy` chunker and as the Docling chunker's fallback.
-   `indexing/` — everything that turns chunks into an index.
    -   `representations.py`: `QwenEmbedder` (local Hugging Face), `OllamaEmbedder`,
        `EmbeddingGenerator` and the `select_embedder()` dispatcher.
    -   `embedders.py`: `LanceDBManager` and `VectorIndexer` (schema, incremental append,
        vector-width guard).
    -   `contextualizer.py`: `ContextualEnricher`, which prepends an LLM summary of the
        surrounding window to each chunk before embedding.
    -   `latechunk.py`: `LateChunkEncoder`.
    -   `overview_builder.py`: per-document overviews used by the triage router.
-   `retrieval/` — `retrievers.py` (`MultiVectorRetriever`) and
    `query_transformer.py` (`QueryDecomposer`).
-   `rerankers/` — `reranker.py` (`CrossEncoderReranker`, the non-`rerankers`-library
    fallback) and `sentence_pruner.py` (Provence sentence-level pruning).
-   `pipelines/` — `indexing_pipeline.py` and `retrieval_pipeline.py`.
-   `agent/` — `loop.py` (`Agent`: triage, decomposition, orchestration, semantic cache,
    per-session history) and `verifier.py` (`Verifier`).
-   `utils/` — `ollama_client.py`, `watsonx_client.py`, `batch_processor.py`,
    `logging_utils.py`.

### Indexing data flow

1.  `DocumentConverter.convert_to_markdown()` converts the file to Markdown. PDFs are
    probed with PyMuPDF for an existing text layer; only text-layer-less PDFs take the
    OCR converter.
2.  `DoclingChunker` (default) or `MarkdownRecursiveChunker` splits the document into
    chunks with a token budget of `chunking.chunk_size`.
3.  `OverviewBuilder` writes a one-paragraph overview per document to
    `index_store/overviews/*.jsonl` (used later for query routing).
4.  If `contextual_enricher.enabled`, `ContextualEnricher` asks the enrichment model for a
    short summary of each chunk's neighbourhood and prepends it to the chunk text. The
    untouched text is kept in `metadata.original_text`.
5.  `EmbeddingGenerator` embeds the chunks in batches and `VectorIndexer` writes them to
    the LanceDB table, then creates the native FTS index on the `text` column.
6.  If late chunking is enabled, `LateChunkEncoder` re-encodes each document as a whole and
    writes per-chunk vectors to `<table>_lc`.

There is no knowledge-graph step: `graph_extractor.py`, `GraphRetriever`,
`GraphQueryTranslator` and the `retrieval.graph` / `graph_strategy` config blocks were
removed on 2026-08-09 (roadmap item 2.5). The path was unreachable, and the evidence is
against it — GraphRAG loses on single-hop retrieval, its multi-hop gains are contested,
and it costs 41–57x at indexing and up to ~377x in query tokens
(`Documentation/research/academic-evidence-2026.md` §6).

### Retrieval data flow

1.  `Agent.run()` routes the query: the overview router first, then a
    "history exists → treat as follow-up" short circuit, then an LLM triage fallback that
    picks `rag_query` or `direct_answer`. `force_rag=True` skips triage
    entirely and pins `rag_query`.
2.  Non-`direct_answer` queries are checked against the in-process semantic cache
    (cosine similarity ≥ `semantic_cache_threshold`, scoped per session by default).
3.  If query decomposition is on, `QueryDecomposer` splits the query. With the shipped
    default (`compose_from_sub_answers: false`, `pooled_first_stage: true`) the first
    stage runs per sub-query, the candidates are pooled and de-duplicated, then get one
    rerank pass and one synthesis; with `compose_from_sub_answers: true` each sub-query
    gets its own full retrieval in parallel, up to 3 workers, and the answers are composed.
4.  `RetrievalPipeline.run()` retrieves via `MultiVectorRetriever.retrieve()` in the
    configured mode, optionally also querying the late-chunk table and merging neighbours.
4b. If `retrieval.retry.enabled` and the first pass found weak evidence — measured as the
    *contrast* between the top candidate and the background of the rest, not the raw top
    similarity — the query is reformulated once on the enrichment model and steps 4-5 run
    again, keeping whichever result set scores better (roadmap 2.1).
5.  The reranker (if enabled) reorders the candidates and keeps `reranker.top_k`. When
    sub-queries are present each candidate is scored against all of them and the scores
    combined with `query_decomposition.rerank_aggregate`.
6.  Context expansion pulls ±`context_window_size` neighbouring chunks from LanceDB.
7.  Provence pruning (off unless requested) drops irrelevant sentences from each chunk.
8.  The generation model synthesises the answer from the surviving chunks, streaming
    tokens to the caller when an event callback is supplied.
9.  If verification is enabled and there are source documents, `Verifier` grades the
    answer and appends ` [Confidence: N%]` (plus a low-confidence warning) to the answer
    string.

## 3. Models

The defaults live in `main.py` and every one of them except the Provence pruner is
overridable with an environment variable.

| Role | Default | Runtime | Env override |
| ---- | ------- | ------- | ------------ |
| **Generation** (answers, sub-answer composition) | `qwen3.5:9b` | Ollama | `GENERATION_MODEL` |
| **Enrichment / utility** (routing, triage, decomposition, contextual enrichment, overviews, verification) | `qwen3.5:4b` | Ollama | `ENRICHMENT_MODEL` |
| **Embedding** | `microsoft/harrier-oss-v1-0.6b` (MIT, 1024-dim) | `transformers`, in-process | `EMBEDDING_MODEL` |
| **Reranker** (on by default) | `Qwen/Qwen3-Reranker-4B` | own yes/no-logit scorer, loaded lazily | `RERANKER_MODEL` |
| **Sentence pruning** (optional) | `naver/provence-reranker-debertav3-v1` | `transformers`, in-process | — (hardcoded in `rerankers/sentence_pruner.py`) |

Documented alternatives:

-   Generation: `qwen3.6:27b` (high-end, ~17 GB) or `qwen3.5:4b` (light).
-   Enrichment: `qwen3.5:2b` (light).
-   Embedding: `Qwen/Qwen3-Embedding-4B` (2560-dim, 32K context; for multilingual or
    long-context corpora), `Qwen/Qwen3-Embedding-0.6B` (1024-dim).
-   Reranker: `BAAI/bge-reranker-v2-m3` (low latency; only pays off with a weaker
    embedder than the default — see [`../eval/DECISIONS.md`](../eval/DECISIONS.md)),
    `answerdotai/answerai-colbert-small-v1` (late interaction — also set
    `reranker.model_type` to `colbert`), `Qwen/Qwen3-Reranker-0.6B`.

Notes:

-   **Embedding dimensions are never hardcoded.** `VectorIndexer.index()` reads the width
    from the vectors the loaded model actually produced and builds the LanceDB schema from
    it. Appending vectors of a different width to an existing table raises an explicit
    error — **changing the embedding model requires re-indexing every existing index.**
-   **The width check is not enough on its own**, because two different models can share
    it (harrier-oss-v1-0.6b and Qwen3-Embedding-0.6B are both 1024-dim). Every table
    therefore records the embedding model that wrote it and whether its vectors are
    L2-normalized; indexing into or querying a table with a different embedder raises
    `EmbedderMismatchError` and names the model to rebuild with.
-   **Vectors are L2-normalized at write and query time**, so LanceDB's default L2
    ordering is the cosine ordering both model cards specify. Tables written before this
    existed carry no marker: they keep working with unnormalized vectors and log a
    warning recommending a rebuild.
-   `QwenEmbedder` truncates inputs at `min(tokenizer.model_max_length, 8192)` tokens.
-   `select_embedder()` treats a name containing `/` as a Hugging Face repo id and anything
    else as an Ollama tag served through `/api/embeddings`.
-   If the reranker cannot be loaded, the pipeline logs a warning and continues **without**
    reranking rather than failing the query.

## 4. Configuration

All configuration lives in `main.py`.

-   **`LLM_BACKEND`** — `ollama` (default) or `watsonx`. See
    [`../WATSONX_README.md`](../WATSONX_README.md).
-   **`OLLAMA_CONFIG`** — `host`, `generation_model`, `enrichment_model`. The generation
    model writes user-facing answers; the enrichment model does all the utility work
    (routing, triage, decomposition, contextual enrichment, document overviews,
    verification).
-   **`WATSONX_CONFIG`** — credentials, URL and the two granite model ids. It has the same
    `generation_model` / `enrichment_model` keys so it is a drop-in replacement.
-   **`EXTERNAL_MODELS`** — `embedding_model` and `reranker_model`, the two Hugging Face
    models loaded in-process.
-   **`PIPELINE_CONFIGS`** — exactly two profiles, `default` and `fast`. Selected with
    `--mode` on the CLI or the `RAG_CONFIG_MODE` environment variable for the API server.

| | `default` | `fast` |
| --- | --- | --- |
| `retrieval.search_type` | `hybrid` | `vector_only` |
| `retrieval.latechunk.enabled` | `true` | `false` |
| `reranker.enabled` | `false` (top 10 when switched on) | `false` |
| `query_decomposition.enabled` | `true` | `false` |
| `verification.enabled` | `true` | `false` |
| `contextual_enricher.enabled` | `true` (window 1) | `false` |
| `retrieval_k` | 20 | 10 |
| `indexing` batch sizes | 50 embed / 10 enrich | 100 embed / 50 enrich |

Both profiles store vectors under `./lancedb` in the table `text_pages_v4`, use
`semantic_cache_threshold: 0.98` and `cache_scope: "session"`.

A per-key table (including which keys the API can override per request) is in
[`DOCUMENTATION.md`](DOCUMENTATION.md#4-configuration-reference).

## 5. Usage

### Prerequisites

1.  **Python 3.10+** (3.11 recommended) and dependencies, installed from the repository
    root:
    ```bash
    pip install -r requirements.txt
    ```
    The root `requirements.txt` is the complete local-run set. `rag_system/requirements.txt`
    is a partial list of the heavy ML dependencies that additionally pins
    `ibm-watsonx-ai` and the macOS-only `ocrmac`. The Docker images install
    `requirements-docker.txt` instead.

2.  **Ollama models**:
    ```bash
    ollama pull qwen3.5:9b
    ollama pull qwen3.5:4b
    ```

3.  **Hugging Face models** — the embedder, reranker and (if used) Provence weights are
    downloaded on first use by `transformers`. Set `HF_TOKEN` if you point at a gated
    repository.

### Command line

Run as a module from the repository root — `python rag_system/main.py` will not work
because the package needs to be importable:

```bash
# Index one file or a whole directory (walks *.pdf, *.docx, *.html, *.htm, *.md, *.txt)
python -m rag_system.main index ./shared_uploads --mode default

# Ask a single question and print the JSON result
python -m rag_system.main chat "What was the revenue growth in Q3?"

# Start the HTTP API (equivalent to python -m rag_system.api_server)
python -m rag_system.main api --port 8001
```

`--mode` accepts `default` or `fast`. There is no interactive REPL.

### Programmatic

```python
from rag_system.factory import get_agent, get_indexing_pipeline

get_indexing_pipeline("default").run(["/abs/path/to/document.pdf"])

agent = get_agent("default")
result = agent.run("What does the contract say about termination?")
print(result["answer"])
print(len(result["source_documents"]))
```

`Agent.run()` returns `{"answer": str, "source_documents": list[dict]}` — there is no
top-level confidence field; the verifier appends its score to the answer string.

### HTTP

```bash
python -m rag_system.api_server            # port 8001
curl http://localhost:8001/health          # {"status": "ok"}
curl -X POST http://localhost:8001/chat \
  -H 'Content-Type: application/json' \
  -d '{"query": "What is in these documents?"}'
```

Endpoints, accepted fields and defaults are documented in
[`DOCUMENTATION.md`](DOCUMENTATION.md#5-http-api).

## 6. Where this fits

The RAG API is one of four processes. The frontend (`:3000`) talks to the backend gateway
(`:8000`), which forwards chat and indexing to this API (`:8001`), which in turn calls
Ollama (`:11434`). The frontend also streams directly from `:8001/chat/stream`. See the
repository [`README.md`](../README.md) and `Documentation/` for the system-level view.

---

### Overview (`Documentation/research/README.md`)

# Research Evidence (August 2026)

Three independent research sweeps on the state of the art in agentic retrieval,
compiled 2026-08-08 by LLM research agents from primary sources only (first-party
engineering blogs, arXiv/ACL/ICLR/SIGIR papers, official model cards and
leaderboards). Anonymous blog posts and SEO content were explicitly rejected;
each report carries its own rejection log and a list of claims that could NOT
be verified.

| File | Scope |
|------|-------|
| [industry-evidence-2026.md](industry-evidence-2026.md) | What Anthropic, OpenAI, Google, Microsoft, LlamaIndex, LangChain, Weaviate, Qdrant, Vespa, Pinecone, Elastic, Exa, Perplexity, Glean et al. have published about production agentic retrieval — architectures, numbers, and where vendors directly contradict each other |
| [academic-evidence-2026.md](academic-evidence-2026.md) | The 2025–26 literature: RL-trained search agents, reasoning-aware retrieval (BRIGHT lineage), late interaction, hybrid fusion, late chunking, context compression, verification/attribution, GraphRAG, memory — with the controlled replications that deflated the 2024 headline claims |
| [component-map-2026.md](component-map-2026.md) | Component-by-component SOTA map for local/self-hosted stacks: parsing, chunking, embeddings, sparse+fusion, rerankers, query planning, routing, loop patterns, verification, compression, memory, evaluation — each claim graded established / emerging / contested |

## How to read these

- These are **evidence documents, not descriptions of localGPT**. Nothing in
  them implies a feature exists in this repo. What localGPT actually does is
  documented in the rest of `Documentation/`.
- Every claim carries its source and date. Claims graded *established* are
  peer-reviewed or independently replicated; *vendor* numbers are self-reported
  and directionally credible at best.
- The actionable distillation lives in
  [../research_roadmap.md](../research_roadmap.md), which maps this evidence to
  concrete, staged changes with acceptance criteria.

Provenance: generated with web access on 2026-08-08; model knowledge cutoffs
predate several cited releases, so everything recent is web-sourced. Each
report's own "could not verify / do not cite" appendix applies.

---

### Documentation Architecture Overview (`Documentation/architecture_overview.md`)

# 🏗️ System Architecture Overview

_Last updated: 2026-08-08_

This document explains how data and control flow through **localGPT** — from a user's browser to model inference and back. It is the ground-truth reference for engineers and integrators; every port, path and endpoint below was read out of the current source.

---

## 1. Process Topology

The system is four separate OS processes. Nothing in `rag_system` runs inside the backend gateway — the gateway talks to the RAG API over HTTP.

```mermaid
flowchart LR
    subgraph Client
        U["👤 User (Browser)"]
        FE["Next.js frontend<br/>:3000"]
        U --> FE
    end

    subgraph Services
        BE["Backend gateway<br/>backend/server.py<br/>:8000"]
        RAG["RAG API<br/>rag_system/api_server.py<br/>:8001"]
        OL["Ollama<br/>:11434"]
    end

    subgraph Storage
        SQL[("SQLite<br/>backend/chat_data.db")]
        LDB[("LanceDB<br/>./lancedb")]
        FS["File system<br/>shared_uploads/ · index_store/"]
    end

    FE -->|"REST: sessions, indexes, uploads, non-streaming chat"| BE
    FE -->|"POST /chat/stream (SSE) — default chat path"| RAG
    BE -->|"POST /chat, POST /index"| RAG
    BE -->|"direct answers only (routing is local)"| OL
    RAG -->|"generation, enrichment, verification"| OL

    BE -->|"sessions, messages, indexes"| SQL
    RAG -->|"index metadata only"| SQL
    RAG --> LDB
    RAG --> FS
```

| Process | Entry point | Port | Server type |
|---------|-------------|------|-------------|
| Frontend | `npm run dev` (or `npm run build && npm run start`) | 3000 | Next.js 15 / React 19 |
| Backend gateway | `python backend/server.py` | 8000 | `socketserver.ThreadingTCPServer` (concurrent) |
| RAG API | `python -m rag_system.api_server` or `python -m rag_system.main api --port 8001` | 8001 | `socketserver.TCPServer` (**requests are serialized**) |
| Ollama | `ollama serve` | 11434 | external |

`python run_system.py` starts all four and aggregates their logs.

The backend port is the module constant `PORT = 8000` in `backend/server.py` and the RAG API port defaults to `8001` in `start_server()`; neither is read from an environment variable. What *is* configurable is where each side looks for the others — see [§6](#6-configuration-entry-points).

### The browser talks to two origins

The chat UI defaults to streaming (`enableStream` is `true` in `src/components/ui/session-chat.tsx`), and streaming goes **directly** from the browser to `:8001/chat/stream`, bypassing the gateway. Session CRUD, uploads, index management and the non-streaming chat fallback go to `:8000`. Any deployment must therefore expose **both** ports to the browser, and the frontend build must know both URLs (`NEXT_PUBLIC_API_URL`, `NEXT_PUBLIC_RAG_API_URL`). Both servers send `Access-Control-Allow-Origin: *`.

---

## 2. Request Paths

### 2.1 Streaming chat (the default UI path)

```mermaid
sequenceDiagram
    participant B as Browser
    participant R as RAG API :8001
    participant O as Ollama :11434
    B->>R: POST /chat/stream {query, session_id, table_name, options}
    R->>R: Agent.run(...) — triage → retrieve → rerank → prune → synthesize
    R->>O: enrichment model (routing, decomposition, verification)
    R->>O: generation model (answer, streamed)
    R-->>B: SSE: analyze, decomposition, retrieval_*, rerank_*, token, complete
```

`Agent.run()` executes the whole pipeline and the handler forwards each phase as an SSE event. See [§2.4](#24-in-process-pipeline-inside-the-rag-api).

### 2.2 Non-streaming chat

```mermaid
sequenceDiagram
    participant B as Browser
    participant G as Backend :8000
    participant R as RAG API :8001
    participant O as Ollama :11434
    B->>G: POST /sessions/{id}/messages {message, options}
    G->>G: persist user message (SQLite)
    G->>G: should_use_rag() — deterministic gate, no model call
    alt route = RAG
        G->>R: POST /chat {query, session_id, table_name, options}
        R-->>G: {answer, source_documents}
    else route = direct
        G->>O: generation model, thinking disabled
    end
    G->>G: persist assistant message (SQLite)
    G-->>B: {response, session, source_documents, used_rag}
```

**The gateway gate.** `should_use_rag(message, idx_ids, force_rag)` in `backend/server.py` is a pure function: `force_rag` → RAG; no linked indexes → direct; a whole-message smalltalk/assistant-meta allowlist regex (`hello`, `thanks!`, `who are you`, capped at six words) → direct; everything else → RAG. No LLM call, no file reads, no network. Measured at ~0.002 ms per message against ~750 ms for the enrichment-model router it replaced (Phase 2.3).

It is deliberately biased toward RAG. The agent-side triage in [§2.4](#24-in-process-pipeline-inside-the-rag-api) still runs on every RAG request and can still return a direct answer, so the gate never has to decide correctly on its own — it only has to avoid sending "hi" through a retrieval pipeline. `used_rag` in the response reports which way the gate went.

### 2.3 Indexing

Uploads land in `shared_uploads/` (backend). Indexing is then triggered either per session (`POST :8000/sessions/{id}/index`) or per named index (`POST :8000/indexes/{id}/build`); both forward to `POST :8001/index`, which runs `IndexingPipeline.run(file_paths)` in the RAG API process.

```mermaid
flowchart LR
    UP["shared_uploads/*"] --> DC["DocumentConverter<br/>(Docling, OCR when available)"]
    DC --> CH["DoclingChunker<br/>(token-budgeted)"]
    CH --> OV["OverviewBuilder<br/>index_store/overviews/&lt;id&gt;.jsonl"]
    CH --> EN["ContextualEnricher<br/>(optional, enrichment model)"]
    EN --> EM["Embeddings<br/>harrier-oss-v1-0.6b"]
    EM --> LT["LanceDB table<br/>+ native FTS index"]
    CH --> LC["Late-chunk encoder<br/>(optional) → &lt;table&gt;_lc"]
```

Contextual enrichment returns copies of the chunks, so the late-chunk leg encodes the **original** chunk text while the main table stores the enriched text (with the original preserved in `metadata.original_text`).

### 2.4 In-process pipeline inside the RAG API

```mermaid
flowchart TD
    Q["Query"] --> T{"Triage"}
    T -->|direct_answer| DA["Generation model, streamed"]
    T -->|rag_query| DEC{"Query decomposition<br/>(enabled in 'default')"}
    DEC -->|n sub-queries| PAR["Parallel per-sub-query retrieval (≤3 threads),<br/>candidates pooled + deduped (arm H default)"]
    DEC -->|single query| RP["RetrievalPipeline.run"]
    PAR --> RR
    RP --> RET["MultiVectorRetriever<br/>FTS + vector, fused with RRF"]
    RET --> LCM["Late-chunk retrieval + ±1 merge (optional)"]
    LCM --> RR["Reranker (on by default)<br/>Qwen3-Reranker-4B"]
    RR --> CE["Context expansion (±window chunks)"]
    CE --> PR["Provence sentence pruning (opt-in)"]
    PR --> SY["Answer synthesis (generation model, streamed)"]
    SY --> VF["Verifier → appends [Confidence: N%]"]
```

With the shipped default (arm H, 2026-08-15) the decomposed path retrieves per sub-query, pools and de-duplicates the candidates, then runs ONE rerank and ONE synthesis over the union context. Sending `compose_sub_answers: true` keeps the older path: answer each sub-query, then compose the final answer from the sub-answers.

Triage order in `Agent._triage_query_async`: the overview router runs first (an LLM call on the enrichment model, grounded in the document overviews loaded for the session); if there is conversation history the query short-circuits to `rag_query`; otherwise a fallback LLM triage picks `rag_query` / `direct_answer`. `force_rag=true` on the RAG API skips triage entirely.

There is no GraphRAG path. `GraphExtractor`, `GraphRetriever`, `GraphQueryTranslator`, the `graph_query` triage outcome and the `retrieval.graph` / `graph_strategy` config blocks were **deleted on 2026-08-09** (roadmap item 2.5): the path was unreachable, GraphRAG loses on single-hop retrieval, its multi-hop gains are contested, and it costs 41–57× at indexing and up to ~377× in query tokens ([`research/academic-evidence-2026.md`](research/academic-evidence-2026.md) §6).

---

## 3. Where State Lives

| Store | Location | Written by | Contents |
|-------|----------|-----------|----------|
| SQLite | `backend/chat_data.db` (`DB_PATH`) | backend (all tables), RAG API (index metadata only) | `sessions`, `messages`, `session_documents`, `indexes`, `index_documents`, `session_indexes` |
| LanceDB | `./lancedb` (`storage.lancedb_uri`, `LANCEDB_PATH`) | RAG API | `text_pages_v4` (default table), `text_pages_<index_id>` (per named index), `<table>_lc` (late-chunk vectors) |
| Overviews | `index_store/overviews/<session_or_index_id>.jsonl`, plus `index_store/overviews/overviews.jsonl` as a global fallback | RAG API (indexing) | one-paragraph document summaries used by the agent's triage router (the gateway gate no longer reads them) |
| Uploads | `shared_uploads/` | backend | original uploaded files, prefixed with a UUID |

**Chat messages are written by `backend/server.py` only** (`handle_session_chat`). The RAG API holds a `ChatDatabase` handle but never touches the `messages` or `sessions` tables.

---

## 4. Threading and Shared State

* The **backend** is threaded (`ThreadingTCPServer`, `daemon_threads = True`). `backend/database.py` opens a fresh SQLite connection per call, so concurrent handlers are safe.
* The **RAG API is single-threaded**. One `/chat`, `/chat/stream` or `/index` request is served at a time; everything else queues. This is deliberate: the process holds one `RAG_AGENT` singleton whose pipeline config is mutated by per-request options.
* Because that config is shared, per-request retrieval options are applied to it — but scoped to the request: the agent snapshots the pipeline config before applying overrides and restores it afterwards, so options no longer leak into later requests. The per-request generation `model` goes through a context manager (`_generation_model_override`) that restores the previous value and rejects model ids that do not match the active `LLM_BACKEND`.
* `factory.get_pipeline_config()` returns a `copy.deepcopy` of the profile, so the module-level `PIPELINE_CONFIGS` in `rag_system/main.py` is never mutated.

---

## 5. Known Architectural Limitations

These are real, current behaviours — not planned work. See [`improvement_plan.md`](improvement_plan.md) for the fixes on the roadmap.

1. **Streamed turns are persisted by a client callback, not by the stream itself.** `POST :8001/chat/stream` writes nothing to SQLite; when the stream completes, the browser posts the finished turn to `POST :8000/sessions/{id}/messages/save`, which stores both messages and derives the session title. A client that consumes the stream directly and skips that call gets no history.
2. **The RAG API serializes requests.** Concurrency is one in-flight RAG request per process.
3. **`enable_docling_chunk` defaults to `true` over HTTP.** `rag_system/api_server.py` maps the flag to `chunker_mode` (`"docling"` when true, `"legacy"` when false), so the Docling chunker runs unless the client sends `false` to select the legacy chunker. (`create_index_script.py` sets `"legacy"` explicitly.)
4. **Service ports are not configurable by environment variable** — only the URLs used to reach them are.

---

## 6. Configuration Entry Points

| Concern | Where |
|---------|-------|
| Model defaults, pipeline profiles | `rag_system/main.py` (`OLLAMA_CONFIG`, `WATSONX_CONFIG`, `EXTERNAL_MODELS`, `PIPELINE_CONFIGS`) |
| Agent / pipeline construction | `rag_system/factory.py` (`get_agent`, `get_indexing_pipeline`, `get_pipeline_config`) |
| Active profile for the RAG API | `RAG_CONFIG_MODE` (default `default`; an unknown value silently falls back to `default`) |
| Service URLs and model overrides | environment variables — see [`.env.example`](../.env.example) and [`system_overview.md`](system_overview.md#7-configuration) |

---

## 7. Component Documents

| Component | Documentation |
|-----------|---------------|
| Whole system, models, configuration | [`system_overview.md`](system_overview.md) |
| Why each component is built this way (evidence + eval numbers; "deliberately not implemented") | [`design_rationale.md`](design_rationale.md) |
| HTTP APIs (backend + RAG API) | [`api_reference.md`](api_reference.md) |
| Indexing pipeline | [`indexing_pipeline.md`](indexing_pipeline.md) |
| Retrieval pipeline | [`retrieval_pipeline.md`](retrieval_pipeline.md) |
| Verifier | [`verifier.md`](verifier.md) |
| Triage / routing | [`triage_system.md`](triage_system.md) |
| Roadmap | [`improvement_plan.md`](improvement_plan.md) |

> **Change management**: when the topology changes (new process, new store, a new browser-facing origin), update this overview first, then the component docs.

---

### Documentation System Overview (`Documentation/system_overview.md`)

# 🏗️ localGPT — Complete System Overview

_Last updated: 2026-08-08_

A comprehensive overview of the localGPT Retrieval-Augmented Generation system: architecture, components, data flow, configuration and operational characteristics. Everything here was verified against the source in this repository; where a feature exists but is not wired up, it says so.

---

## 1. System Architecture

### 1.1 High-level architecture

Four processes. The browser talks to **two** of them.

```mermaid
graph TB
    subgraph "Client Layer"
        Browser["👤 User Browser"]
        UI["Next.js Frontend<br/>React / TypeScript<br/>Port 3000"]
        Browser --> UI
    end

    subgraph "Gateway Layer"
        Backend["Backend Server<br/>backend/server.py<br/>Port 8000"]
    end

    subgraph "Processing Layer"
        RAG["RAG API Server<br/>rag_system/api_server.py<br/>Port 8001"]
    end

    subgraph "LLM Service Layer"
        Ollama["Ollama<br/>Port 11434"]
    end

    subgraph "Storage Layer"
        SQLite[("SQLite<br/>sessions, messages, index metadata")]
        LanceDB[("LanceDB<br/>chunk vectors + native FTS")]
        FileSystem["File system<br/>shared_uploads/ · index_store/"]
    end

    UI -->|"REST"| Backend
    UI -->|"SSE POST /chat/stream (default chat path)"| RAG
    Backend -->|"POST /chat · POST /index"| RAG
    Backend -->|"routing + direct answers"| Ollama
    RAG -->|"generation · enrichment · verification"| Ollama
    Backend --> SQLite
    RAG -->|"index metadata only"| SQLite
    RAG --> LanceDB
    RAG --> FileSystem
```

### 1.2 Component breakdown

| Component | Technology | Port | Purpose |
|-----------|------------|------|---------|
| **Frontend** | Next.js 15, React 19, TypeScript, Tailwind v4 | 3000 | Chat UI, index management, retrieval settings |
| **Backend gateway** | Python 3.10+, `http.server` on `ThreadingTCPServer` | 8000 | Sessions, messages, uploads, index CRUD, first-layer routing |
| **RAG API** | Python 3.10+, `http.server` on `TCPServer` (serialized) | 8001 | Agent, retrieval pipeline, indexing pipeline |
| **Ollama** | External LLM server | 11434 | Generation and enrichment model inference |
| **SQLite** | Embedded | – | Sessions, messages, documents, index metadata |
| **LanceDB** | Embedded vector store | – | Chunk vectors + native full-text (BM25) index |

For the process topology, request sequences and threading model see [`architecture_overview.md`](architecture_overview.md).

---

## 2. Core Functionality

### 2.1 Two-layer routing

Routing happens twice, in two different processes. Layer 1 is a deterministic gate with no model call; layer 2 is the system's single LLM routing layer.

#### Layer 1 — gateway routing (`backend/server.py`, non-streaming path only)

`should_use_rag(message, idx_ids, force_rag)` — a module-level function, no LLM call, no network I/O:

1. `force_rag` → RAG, unconditionally.
2. Session has **no linked indexes** → direct LLM, no RAG.
3. Message is unmistakable smalltalk (`hello`, `thanks!`, `bye`, `ok` — a whole-message allowlist regex capped at six words) or a question about the assistant itself (`who are you`, `what model are you`) → direct LLM.
4. Everything else → RAG.

This is retrieval-first: escalate rather than pre-decide. The gate deliberately over-sends to RAG because layer 2 can still answer directly — the cost of a wrong "send to RAG" is one agent triage call, while the cost of a wrong "answer directly" is an unanswerable question. Pre-retrieval LLM routing was removed here in Phase 2.3: it is the weakest measured routing pattern (`Documentation/research/`), and it duplicated layer 2. The old `_simple_pattern_routing` keyword/length fallback (which misrouted any question containing the word "test") is gone with it.

A `force_rag: true` field on the request skips the gate and always calls the RAG API. This layer does **not** run on the streaming path, because the browser calls the RAG API directly.

#### Layer 2 — agent triage (`rag_system/agent/loop.py`, always)

`_triage_query_async()`:

1. `_route_via_overviews()` — the enrichment model classifies the query against the overviews loaded for the session, returning `direct_answer` or `rag_query`.
2. If conversation history exists for the session, short-circuit to `rag_query`.
3. Otherwise a fallback triage prompt (also on the enrichment model) picks `rag_query` or `direct_answer`.

`force_rag=true` on the RAG API pins `query_type = "rag_query"` and skips all three steps, while still honouring the reranking / decomposition / verification toggles.

Triage is two-way since the graph module was removed on 2026-08-09 (roadmap 2.5). `Agent._normalize_triage()` collapses anything that is not an explicit `direct_answer` — including a stray `graph_query` from a small utility model — to `rag_query`.

### 2.2 Indexing

1. **Upload** — files are stored under `shared_uploads/` with a UUID prefix and recorded in SQLite.
2. **Conversion** — `DocumentConverter` (Docling) produces markdown plus structure. OCR options are chosen by probing which backend is actually installed (OcrMac on macOS, then EasyOCR, RapidOCR, tesserocr, the `tesseract` CLI); when none is available Docling's defaults are used and text-layer PDFs still convert.
3. **Chunking** — `DoclingChunker` packs sentences up to a token budget (`chunk_size`, default 512 over HTTP) using the embedding model's tokenizer. A legacy `MarkdownRecursiveChunker` exists and is selected by `chunker_mode: "legacy"` (used by `create_index_script.py`; over HTTP, `enable_docling_chunk: false` on `POST :8001/index` selects it — the HTTP default is `true`).
4. **Overviews** — the first *n* chunks of each document (default 5) are summarised by the enrichment model into `index_store/overviews/<id>.jsonl`. The agent's triage router reads these; the gateway gate (§2.1) does not.
5. **Contextual enrichment** (optional) — the enrichment model summarises a window of surrounding chunks and prepends it to each chunk. The untouched text is kept in `metadata.original_text`.
6. **Embedding + indexing** — chunks are embedded and written to a LanceDB table; a native FTS index is created on the `text` column.
7. **Late chunking** (optional) — each document is re-encoded in one pass and per-chunk vectors are pooled from it, written to `<table>_lc`. Enrichment produces *copies* of the chunks, so this leg encodes the original text, not the enriched text.

The vector width is taken from the embeddings actually produced. If you point an existing table at a different-dimensional model, `VectorIndexer` raises with a message telling you to rebuild — it will not silently corrupt the table. Because two models can share a width, each table also records the embedding model that wrote it and whether its vectors are L2-normalized; a mismatch raises `EmbedderMismatchError` at index time and at query time, and a table with no marker (built before this existed) keeps working unnormalized with a warning.

### 2.3 Retrieval

1. **Query embedding** — the same embedding model used at index time (an LRU cache holds 256 single-query embeddings). Instruction-tuned families (harrier-oss-v1, Qwen3-Embedding) get the query-side `Instruct: … \nQuery: …` prefix their cards require; documents never do. The query vector is L2-normalized when the table's marker says its vectors are, so LanceDB's L2 ordering is the cosine ordering the cards specify.
2. **Search** — `MultiVectorRetriever.retrieve()` runs LanceDB full-text search and vector search **in parallel** and fuses them with **reciprocal rank fusion** (`1/(60 + rank)` per leg). `search_type` selects `hybrid` (both legs), `vector_only` or `fts_only`; an unrecognised value logs a warning and falls back to `hybrid`. Every returned document carries a finite, higher-is-better `score`.
3. **Late-chunk leg** (optional) — when late chunking is enabled the same query also runs against `<table>_lc` and those hits are appended. Every retrieved chunk (from either leg) then has its text replaced by the concatenation of its ±1 neighbours in the main table, so a hit on one sub-vector still yields readable context. Hits are de-duplicated across the vector/FTS legs on `(document_id, chunk_index)` (shipped 2026-08-14).
4. **Reranking** (**on by default** since arm G, 2026-08-14 — `reranker.enabled: True` with `min_score: 0.5` / `min_keep: 3` threshold-based selection; the measured reason is in [`../eval/DECISIONS.md`](../eval/DECISIONS.md)) — the default `Qwen/Qwen3-Reranker-4B` goes to the in-repo `QwenRerankerScorer`, and any other model is loaded through the `rerankers` library (`reranker.strategy: "rerankers-lib"`, `reranker.model_type: "cross-encoder"`); any other `strategy` value uses the in-repo `CrossEncoderReranker` (`transformers` `AutoModelForSequenceClassification`). The model is loaded lazily on the first reranked query. If the model fails to load, a warning is printed and **reranking is skipped** — there is no second reranker to fall back to.
5. **Context expansion** — each surviving chunk is widened to its neighbours within `context_window_size`.
6. **Sentence pruning** (opt-in, `provence.enabled`) — `naver/provence-reranker-debertav3-v1` drops sentences below `provence.threshold` (default `0.1`); chunks pruned to nothing are removed.
7. **Synthesis** — the generation model writes the answer, streamed token by token.
8. **Verification** (see §2.4).

Retrieval matches against the stored (possibly enriched) text, and chunks coming out of the retriever expose `metadata.original_text` as their `text` when enrichment ran — so enrichment improves recall without pushing its own prefix into the answer's context. Neighbour chunks pulled in by context expansion are returned exactly as stored, so those still carry the enrichment prefix.

### 2.4 Verification

When `verification.enabled` is on (or `verify: true` is sent) **and** the result has source documents, `Verifier.verify_async()` asks the enrichment model for a JSON verdict and the confidence is appended to the answer **string**:

* `" [Confidence: N%]"` for any non-zero score;
* additionally `" [Warning: Low confidence. Groundedness: <bool>]"` when the answer is not grounded or the score is below 50;
* nothing at all when the score parses as 0 (treated as a parser failure).

There is no top-level `confidence` field in the response. Responses are `{answer, source_documents}`.

### 2.5 Query decomposition

Enabled in the `default` profile. The enrichment model splits the raw user query (plus up to 5 recent turns for pronoun resolution) into sub-queries, capped by `query_decomposition.max_sub_queries` (default 10). Sub-queries are retrieved in parallel with at most 3 worker threads. The shipped default (arm H, 2026-08-15) is the **pooled first stage** (`compose_from_sub_answers: false`, `pooled_first_stage: true`): the per-sub-query candidates are pooled and de-duplicated, then get ONE rerank pass and ONE synthesis over the union context. With `compose_from_sub_answers: true` the generation model instead answers each sub-query and composes a final answer from the sub-answers. A decomposition that yields a single sub-query skips the parallel machinery.

### 2.6 Semantic cache and conversation memory

* `TTLCache(maxsize=100, ttl=300)` keyed on the query embedding. A cached answer is reused when cosine similarity ≥ `semantic_cache_threshold` (`0.98`).
* `cache_scope` is `"session"` in both profiles: an entry is only reused inside the session that produced it. Setting it to `"global"` re-enables cross-session reuse — including answers derived from another session's documents.
* Conversation history for triage and query rewriting lives in an in-process `LRUCache(maxsize=100)` on the agent, keyed by `session_id`. It is **not** the SQLite message history and is lost on restart.

---

## 3. Data Architecture

### 3.1 SQLite (`backend/chat_data.db`, override with `DB_PATH`)

```sql
sessions          -- id, title, created_at, updated_at, model_used, message_count
messages          -- id, session_id, content, sender('user'|'assistant'), timestamp, metadata
session_documents -- files uploaded to a session
indexes           -- id, name, description, created_at, updated_at, vector_table_name, metadata
index_documents   -- files belonging to a named index
session_indexes   -- links sessions to indexes
```

Written by `backend/server.py`. The RAG API opens the same database but only reads/writes the `indexes` rows (index metadata); it never writes `messages`.

### 3.2 LanceDB (`./lancedb`, override with `LANCEDB_PATH`)

```
lancedb/
├── text_pages_v4            -- default table (storage.text_table_name)
├── text_pages_<index_id>    -- one table per index created via POST /indexes
└── <table>_lc               -- late-chunk vectors for the table above
```

Each table stores `chunk_id`, `text`, `document_id`, `chunk_index`, `metadata` (JSON) and `vector`, plus a native full-text index on `text`.

### 3.3 File system

```
shared_uploads/                       -- uploaded documents (<uuid>_<original name>)
index_store/overviews/<id>.jsonl      -- per-index / per-session document overviews
index_store/overviews/overviews.jsonl -- global fallback overview file
logs/                                 -- run_system.py service logs + run_system.pid
```

---

## 4. Models

### 4.1 Configured defaults (`rag_system/main.py`)

```python
OLLAMA_CONFIG = {
    "host": os.getenv("OLLAMA_HOST", "http://localhost:11434"),
    "generation_model": os.getenv("GENERATION_MODEL", "qwen3.5:9b"),
    "enrichment_model": os.getenv("ENRICHMENT_MODEL", "qwen3.5:4b"),
}

EXTERNAL_MODELS = {
    "embedding_model": os.getenv("EMBEDDING_MODEL", "microsoft/harrier-oss-v1-0.6b"),
    "reranker_model": os.getenv("RERANKER_MODEL", "Qwen/Qwen3-Reranker-4B"),
}
```

| Role | Default | Used for |
|------|---------|----------|
| Generation | `qwen3.5:9b` (Ollama) | Final answers, sub-answer composition, direct answers |
| Enrichment / utility | `qwen3.5:4b` (Ollama) | Agent triage (the only LLM router), query decomposition, contextual enrichment, document overviews, verification |
| Embedding | `microsoft/harrier-oss-v1-0.6b` (HuggingFace, MIT, 1024 dims) | Index and query embeddings |
| Reranker | `Qwen/Qwen3-Reranker-4B` (HuggingFace, own yes/no-logit scorer) | Reranking retrieved chunks — **on by default** (arm G, 2026-08-14), loaded lazily on the first reranked query ([`../eval/DECISIONS.md`](../eval/DECISIONS.md)) |
| Sentence pruner | `naver/provence-reranker-debertav3-v1` (HuggingFace) | Opt-in sentence-level pruning |

Approximate footprints published by the model authors (not measured here): `qwen3.5:9b` ≈ 6.6 GB at Q4, `qwen3.5:4b` ≈ 3.4 GB, `qwen3.6:27b` ≈ 17 GB, `microsoft/harrier-oss-v1-0.6b` ≈ 1.2 GB, `Qwen/Qwen3-Embedding-4B` ≈ 8 GB in bf16, `Qwen/Qwen3-Embedding-0.6B` ≈ 1.2 GB, `Qwen/Qwen3-Reranker-4B` ≈ 7.5 GB.

### 4.2 Documented alternatives

| Role | Options |
|------|---------|
| Generation | `qwen3.6:27b` (high-end), `qwen3.5:4b` (light) |
| Enrichment | `qwen3.5:2b` (light) |
| Embedding | `Qwen/Qwen3-Embedding-4B` (2560 dims, 32K context — for multilingual / long-context corpora), `Qwen/Qwen3-Embedding-0.6B` (1024 dims, light) |
| Reranker | `BAAI/bge-reranker-v2-m3` (cross-encoder, low latency — only pays off with a weaker embedder than the default), `answerdotai/answerai-colbert-small-v1` (late interaction — also set `reranker.model_type: "colbert"`), `Qwen/Qwen3-Reranker-0.6B` |

Set them with the `GENERATION_MODEL` / `ENRICHMENT_MODEL` / `EMBEDDING_MODEL` / `RERANKER_MODEL` environment variables, or edit `rag_system/main.py`.

> ⚠️ **Changing the embedding model requires re-indexing.** Vector width is derived from the loaded model, and `VectorIndexer` raises rather than appending mismatched vectors to an existing LanceDB table. Width alone is not a sufficient check — `harrier-oss-v1-0.6b` and `Qwen3-Embedding-0.6B` are both 1024 dims — so every table also records the embedding model that wrote it, and indexing into or querying it with a different one raises `EmbedderMismatchError`. Ollama embedding tags are also supported: `select_embedder()` treats a name containing `/` as a HuggingFace repo and anything else as an Ollama tag.

### 4.3 Model selection at runtime

* **Per request** — `model` on `POST :8000/sessions/{id}/messages` and on the RAG API chat endpoints overrides the generation model for that request only. The RAG API rejects ids that do not match the active backend (an Ollama tag will not be forced onto a WatsonX deployment).
* **Per index** — when an index records an `embedding_model` in its metadata, the RAG API switches the retrieval pipeline's embedder to it before querying that index.
* Generation model precedence on the gateway's direct-LLM path: request `model` → the session's `model_used` → `GENERATION_MODEL`.

### 4.4 Vision / multimodal — not integrated

There is **no** vision model in the configuration and no multimodal path in the pipelines: PDF parsing and OCR are handled entirely by Docling. Models such as GLM-OCR or Qwen3-VL could be added as an extension; wiring them up is not done today.

### 4.5 Alternative LLM backend: WatsonX

`LLM_BACKEND=watsonx` swaps the Ollama client for `WatsonXClient` (`WATSONX_CONFIG`: `WATSONX_API_KEY`, `WATSONX_PROJECT_ID`, `WATSONX_URL`, `WATSONX_GENERATION_MODEL`, `WATSONX_ENRICHMENT_MODEL`). It requires `pip install ibm-watsonx-ai` — the root `requirements.txt` lists it as an optional, commented dependency. Embedding and reranking still run locally through HuggingFace. See [`../WATSONX_README.md`](../WATSONX_README.md).

---

## 5. Pipeline Configurations

`PIPELINE_CONFIGS` in `rag_system/main.py` contains exactly two profiles, `default` and `fast`. `RAG_CONFIG_MODE` selects the one the RAG API server uses (default `default`); an unknown value silently falls back to `default`. `factory.get_pipeline_config()` hands out a deep copy, so runtime overrides never mutate the master config.

### 5.1 `default`

```
/* Detailed source-code truncated for AI context efficiency. */
```

### 5.2 `fast`

```python
"fast": {
    "description": "Speed-optimized pipeline with minimal overhead",
    "storage": {"lancedb_uri": "./lancedb", "text_table_name": "text_pages_v4"},
    "retrieval": {
        "search_type": "vector_only",
        "latechunk": {"enabled": False},
        "dense": {"enabled": True}
    },
    "embedding_model_name": EXTERNAL_MODELS["embedding_model"],
    "reranker": {"enabled": False},
    "query_decomposition": {"enabled": False},
    "verification": {"enabled": False},
    "retrieval_k": 10,
    "context_window_size": 0,
    "semantic_cache_threshold": 0.98,
    "cache_scope": "session",
    "contextual_enricher": {"enabled": False, "window_size": 1},
    "indexing": {
        "embedding_batch_size": 100,
        "enrichment_batch_size": 50
    }
}
```

One key in the blocks above currently has no consumer and is inert: the profile's `description`. (`indexing.enable_progress_tracking` used to be listed here; it was assigned to an attribute that is never checked — progress is always tracked — and has since been removed from the profiles.)

### 5.3 Keys read at runtime but absent from the profiles

These have code defaults and can be added to a profile if you want to change them:

| Key | Default | Effect |
|-----|---------|--------|
| `chunking.chunk_size` | `1500` (profile absent) / `512` (HTTP requests) | Token budget per chunk |
| `chunker_mode` | `"docling"` | `"docling"` or `"legacy"` |
| `query_decomposition.max_sub_queries` | `10` | Cap on sub-queries |
| `query_decomposition.rerank_aggregate` | `"mean"` | `mean` or `max`; how per-sub-query rerank scores combine (roadmap 2.2) |
| `retrieval.retry.min_rerank_score` | falls back to `min_top_score` | Retry threshold used when the reranker returns a 0–1 probability |
| `verification.model` / `VERIFIER_MODEL` | unset | HuggingFace NLI/verifier model; unset keeps the LLM-prompt verifier (roadmap 2.4) |
| `verification.threshold` | `0.5` | Grounded/ungrounded cut for the local verifier |
| `reranker.model_type` | `"cross-encoder"` | `rerankers` library model type |
| `reranker.top_percent` | – | Keep a fraction of candidates instead of `top_k` |
| `provence.enabled` / `provence.threshold` | `False` / `0.1` | Sentence-level pruning |
| `overview.enabled` / `overview.model` / `overview.max_chunks` | `True` / enrichment model / `5` | Document overview generation |
| `enrich_model` | enrichment model | Overrides the model used for contextual enrichment |
| `overview_path` | `index_store/overviews/overviews.jsonl` | Where overviews are written |

### 5.4 Where the 20/1/10 request defaults come from

The `retrieval_k: 20`, `context_window_size: 1` and `reranker_top_k: 10` defaults are owned by the **frontend** (`src/components/ui/session-chat.tsx`), not by the RAG API. `rag_system/api_server.py` passes `None` for every option the client omits — `verify`, `ai_rerank`, `query_decompose`, `compose_sub_answers`, `context_expand`, `retrieval_mode` and the three values above — so the profile wins.

Note the practical consequence: for UI clients, context expansion of ±1 chunk is on by default even though both profiles set `context_window_size: 0`. A non-UI HTTP client that omits the field gets the profile value (`0`).

---

## 6. Resource Notes

There are no benchmarks in this repository, so no latency or throughput figures are published here. What determines cost:

* **Memory** is dominated by the models you load: the Ollama generation model, plus the embedding model and (if enabled) the reranker and Provence pruner, which run in the RAG API process via `transformers`.
* **Concurrency** is bounded by the RAG API's single-threaded server: one RAG request at a time per process. The backend gateway is threaded, so session and index CRUD stay responsive while a query runs.
* **Indexing cost** scales with contextual enrichment (one LLM call per chunk — the contextualizer loops chunks inside each batch) and late chunking (a second full encode of every document, plus a second vector table).
* **Query cost** scales with query decomposition (one retrieval per sub-query, then one rerank and one synthesis over the pooled candidates), reranking and verification. The `fast` profile turns all of these off.

Use `python system_health_check.py` to print the resolved configuration, the embedding dimension of the loaded model, and the LanceDB tables that actually exist.

---

## 7. Configuration

### 7.1 Environment variables

Every variable below is read by this repository's code, except `HF_TOKEN` which is consumed by the HuggingFace client libraries. See [`.env.example`](../.env.example) for the annotated file.

| Variable | Default | Read by |
|----------|---------|---------|
| `OLLAMA_HOST` | `http://localhost:11434` | `rag_system/main.py`, `backend/ollama_client.py` |
| `RAG_API_URL` | `http://localhost:8001` | `backend/server.py` (all calls to the RAG API) |
| `NEXT_PUBLIC_API_URL` | `http://localhost:8000` | `src/lib/api.ts` — **inlined at build time** |
| `NEXT_PUBLIC_RAG_API_URL` | `http://localhost:8001` | `src/lib/api.ts` — **inlined at build time** |
| `DB_PATH` | `backend/chat_data.db` (`/app/backend/chat_data.db` in Docker) | `backend/database.py` |
| `LANCEDB_PATH` | `storage.lancedb_uri`, else `./lancedb` | `rag_system/main.py` (pipeline profiles), `backend/database.py`, `system_health_check.py` |
| `GENERATION_MODEL` | `qwen3.5:9b` | `rag_system/main.py`, `backend/server.py`, `run_system.py` |
| `ENRICHMENT_MODEL` | `qwen3.5:4b` | same |
| `EMBEDDING_MODEL` | `microsoft/harrier-oss-v1-0.6b` | `rag_system/main.py` |
| `RERANKER_MODEL` | `Qwen/Qwen3-Reranker-4B` (loaded lazily on the first reranked query) | `rag_system/main.py` |
| `RAG_CONFIG_MODE` | `default` | `rag_system/api_server.py` |
| `LLM_BACKEND` | `ollama` | `rag_system/main.py`, `rag_system/factory.py` |
| `RAG_API_TIMEOUT` | `600` (seconds) | `backend/server.py` — chat calls |
| `RAG_API_INDEX_TIMEOUT` | `3600` (seconds) | `backend/server.py` — indexing calls |
| `HF_TOKEN` | – | `huggingface_hub` (library) — gated model downloads |

`NEXT_PUBLIC_*` values are baked into the JavaScript bundle by `next build`; changing them at runtime has no effect on an already-built frontend. The compose files pass them as build args as well as runtime environment.

Service **ports** are not environment-configurable: `PORT = 8000` in `backend/server.py`, `8001` in `start_server()`, `3000` from Next.js.

### 7.2 Per-request options

Retrieval and indexing behaviour is controlled per request, not by editing a config file. See [`api_reference.md`](api_reference.md) for the full field list. Both casings are accepted end to end: the frontend historically sent camelCase, the gateway sends snake_case, and both the gateway and the RAG API normalise every option to one canonical snake_case key at parse time.

### 7.3 Command-line entry points

```bash
# Index a file or a directory (walks for .pdf .docx .html .htm .md .txt)
python -m rag_system.main index /path/to/docs --mode default

# One-shot query, prints JSON
python -m rag_system.main chat "What does the contract say about termination?" --mode default

# Start the RAG API
python -m rag_system.main api --port 8001
```

`python rag_system/main.py …` does **not** work — the module must be run with `-m` from the project root. Programmatically, `from rag_system.factory import get_agent, get_indexing_pipeline` is the supported entry point; `IndexingPipeline.run(file_paths)` is the public indexing call.

---

## 8. Operations

### 8.0 Prerequisites

* **Python 3.10+** (3.11 recommended — both Docker images are `python:3.11-slim`).
* **Node 20+** for the frontend (`Dockerfile.frontend` is `node:20-alpine`).
* **Ollama** installed and running, with the generation and enrichment models pulled.
* Python dependencies: `pip install -r requirements.txt`. `backend/requirements.txt` is the minimal set for running only the gateway. `pip install ibm-watsonx-ai` is additionally required for `LLM_BACKEND=watsonx`.

### 8.1 Local launcher

```bash
python run_system.py                 # dev mode: all four services
python run_system.py --mode prod     # runs `npm run build` before `npm run start`
python run_system.py --no-frontend   # backend stack only
python run_system.py --health        # HTTP probes each service, exits non-zero if unhealthy
python run_system.py --stop          # terminates the processes recorded in logs/run_system.pid
python run_system.py --logs-only     # tails logs/*.log without starting anything
```

`--health` probes `http://localhost:11434/api/tags`, `:8001/health`, `:8000/health` and `:3000/`. On startup the launcher checks that `GENERATION_MODEL` and `ENRICHMENT_MODEL` are present in Ollama.

### 8.2 Docker

`docker compose --env-file docker.env up -d --build` brings up `rag-api`, `backend` and `frontend`; Ollama runs on the host by default (`OLLAMA_HOST=http://host.docker.internal:11434`, with `extra_hosts: host.docker.internal:host-gateway` so it also resolves on Linux). A containerised Ollama is available behind the `with-ollama` profile. `backend` and `rag-api` bind-mount `./backend`, `./lancedb`, `./index_store` and `./shared_uploads`, so both processes share one SQLite file and one vector store. Health checks use `/health` on both Python services and busybox `wget` for the frontend. See [`docker_usage.md`](docker_usage.md) and [`../DOCKER_README.md`](../DOCKER_README.md).

### 8.3 Health and logging

| Endpoint | Response |
|----------|----------|
| `GET :8000/health` | `{status, ollama_running, available_models, database_stats}` |
| `GET :8001/health` | `{"status": "ok"}` |

`run_system.py` writes per-service logs to `logs/<service>.log` plus `logs/system.log`, with a coloured console formatter. Logging is plain text — there is no JSON formatter and no log rotation (see [`improvement_plan.md`](improvement_plan.md)). The RAG API routes its handler output through the `logging` module; the agent and both pipelines still print progress with `print()`, which is what you see in `logs/rag-api.log`.

---

## 9. Security & Privacy

* **Local by default** — generation, embedding, reranking and pruning all run locally (Ollama + HuggingFace models). Nothing leaves the machine unless you set `LLM_BACKEND=watsonx`, which sends prompts to IBM Cloud.
* **Model downloads** — HuggingFace models are fetched on first use and cached; that is the only outbound traffic in the default setup.
* **No authentication** — neither server implements auth, and both send `Access-Control-Allow-Origin: *`. Ports 8000 and 8001 must not be exposed to an untrusted network.
* **Session isolation** — retrieval is scoped to the tables of the indexes linked to a session, and the semantic cache is session-scoped by default. Setting `cache_scope: "global"` allows one session's document-derived answer to be returned in another.
* **Deletion** — deleting an index removes its rows and drops its LanceDB table. Uploaded files in `shared_uploads/` are not deleted.

---

## 10. Development & Extension

### 10.1 Principles

* Configuration-driven: profiles in `rag_system/main.py`, construction in `rag_system/factory.py`.
* Lazy loading: embedders, rerankers and the pruner are built on first use and cached on the pipeline instance.
* One factory, one RAG API server, one owner per store.

### 10.2 Extension points

| To add… | Do this |
|---------|---------|
| A retriever | Implement the duck-typed contract `retrieve(text_query: str, table_name: str, k: int, search_type: str = "hybrid") -> List[Dict]` and return it from `RetrievalPipeline._get_dense_retriever()`. There is no `BaseRetriever` ABC. |
| A reranker | Plug it into `RetrievalPipeline._get_ai_reranker()`; a `rerankers`-library model only needs `reranker.model_name` + `reranker.model_type`. |
| A chunker | Add a `chunker_mode` branch in `IndexingPipeline.__init__`. |
| An embedding model | Point `EMBEDDING_MODEL` at a HuggingFace repo (contains `/`) or an Ollama tag, then re-index. |
| A pipeline profile | Add an entry to `PIPELINE_CONFIGS` and select it with `RAG_CONFIG_MODE` or `--mode`. |

### 10.3 Validation

There is no automated test suite in this repository. What exists:

* `python system_health_check.py` — imports, configuration dump, LanceDB connectivity, agent construction, embedding dimension, and a sample query against the first available table.
* `python run_system.py --health` — HTTP health probes of all four services.
* `./test_docker_build.sh` — builds the images and probes the container health endpoints.

Building the automated tests is tracked in [`improvement_plan.md`](improvement_plan.md) §8.

---

## 11. Known Limitations

1. **Streamed chat turns are persisted via a follow-up call.** The stream itself (`POST :8001/chat/stream`) writes nothing to SQLite; the UI posts the completed turn to `POST :8000/sessions/{id}/messages/save` when the stream finishes. Direct stream consumers must do the same to get history.
2. **The RAG API serializes requests** — one chat or indexing run at a time per process. Per-request option overrides are scoped to the request: the agent snapshots its config before applying them and restores it afterwards, so they no longer leak into subsequent requests.
3. **`enable_latechunk` defaults to `false` on `POST :8001/index`**, so an HTTP index build without that flag produces no late-chunk table even though the `default` profile enables late chunking. The CLI (`python -m rag_system.main index`) uses the profile value.
4. **A reranker that fails to load is skipped**, not replaced — there is no fallback reranker.
5. **`requirements-docker.txt` has drifted** from `requirements.txt` and still lists packages with no importers.

---

> This overview describes the implementation as of 2026-08-08. When behaviour changes, update [`architecture_overview.md`](architecture_overview.md) and this file together.

---

### Documentation Api Reference (`Documentation/api_reference.md`)

# 📚 API Reference (Backend Gateway & RAG API)

_Last updated: 2026-08-08_

Two HTTP services, both reachable from the browser:

| Service | Base URL | Source |
|---------|----------|--------|
| Backend gateway | `http://localhost:8000` | `backend/server.py` |
| RAG API | `http://localhost:8001` | `rag_system/api_server.py` |

Both send `Access-Control-Allow-Origin: *` on every response (including 404s, which return JSON bodies) and answer `OPTIONS` preflights. Neither sends `Access-Control-Allow-Credentials`, and neither implements authentication.

**Wire format** is snake_case. Both services additionally accept camelCase and normalise it to the same canonical key at parse time, so `rerankerTopK` and `reranker_top_k` are interchangeable. When both spellings are present, the explicit snake_case value wins. The gateway uses an explicit alias table that covers every option it accepts (plus a few legacy names such as `latechunk` and `decompose`); the RAG API converts any camelCase key generically.

---

## 1. Backend Gateway — `http://localhost:8000`

### 1.1 Route table

| Endpoint | Method | Description | Request body | Success response |
|----------|--------|-------------|--------------|------------------|
| `/health` | GET | Health probe with Ollama status and DB stats | – | `{ status, ollama_running, available_models, database_stats }` |
| `/chat` | POST | Stateless chat, no session, no retrieval | `{ message, model?, conversation_history? }` | `{ response, model, message_count }` |
| `/models` | GET | Available generation / embedding models | – | `{ generation_models, embedding_models }` |
| `/sessions` | GET | List sessions | – | `{ sessions, total }` |
| `/sessions` | POST | Create a session | `{ title?, model? }` | 201 `{ session, session_id }` |
| `/sessions/cleanup` | GET | Delete sessions that have no messages | – | `{ message, cleanup_count }` |
| `/sessions/<id>` | GET | Session plus its messages | – | `{ session, messages }` |
| `/sessions/<id>` | DELETE | Delete a session and its messages | – | `{ deleted: true }` |
| `/sessions/<id>/rename` | POST | Rename a session | `{ title }` | `{ message, session }` |
| `/sessions/<id>/messages` | POST | Session chat (persisted) | [Session chat request](#12-session-chat-request) | `{ response, session, source_documents, used_rag }` |
| `/sessions/<id>/messages/save` | POST | Persist a completed streamed turn (the browser calls this after `POST :8001/chat/stream` finishes) | `{ user_message, assistant_message, source_documents?, steps? }` | `{ session, user_message_id, ai_message_id }` — sources and steps are stored in the assistant message's `metadata.source_documents` / `metadata.steps` |
| `/sessions/<id>/documents` | GET | Files uploaded to a session | – | `{ session, files, file_count }` |
| `/sessions/<id>/upload` | POST | Upload files to a session | multipart, field `files` | `{ message, uploaded_files }` |
| `/sessions/<id>/index` | POST | Index the session's documents | [Index options](#14-index-options) (optional) | the RAG API `/index` response |
| `/sessions/<id>/indexes` | GET | Indexes linked to a session | – | `{ indexes, total }` |
| `/sessions/<sid>/indexes/<idx>` | POST | Link an index to a session | – | `{ message }` |
| `/indexes` | GET | List all indexes | – | `{ indexes, total }` |
| `/indexes` | POST | Create a named index | `{ name, description?, metadata? }` | 201 `{ index_id }` |
| `/indexes/<id>` | GET | One index | – | the index object (see below) |
| `/indexes/<id>` | DELETE | Delete an index, its links and its LanceDB table | – | `{ message, index_id }` |
| `/indexes/<id>/upload` | POST | Upload files to an index | multipart, field `files` | `{ message, uploaded_files }` |
| `/indexes/<id>/build` | POST | Build / rebuild the index | [Index options](#14-index-options) (optional) | `{ response, ...echoed options }` |

`uploaded_files` entries are `{ filename, stored_path }`. The index object returned by `GET /indexes/<id>` is `{ id, name, description, created_at, updated_at, vector_table_name, metadata, documents[] }` — it is **not** wrapped in `{ index: … }`.

> **Body required.** `POST /chat`, `POST /sessions`, `POST /sessions/<id>/messages` and `POST /indexes` read `Content-Length` unconditionally; send a JSON body (at minimum `{}`) or the request fails with a 500. `POST /sessions/<id>/rename` returns a clean `400 { "error": "Request body required" }`. `POST /sessions/<id>/index` and `POST /indexes/<id>/build` treat the body as optional.

### 1.2 Session chat request

`POST /sessions/<id>/messages`

```jsonc
{
  "message": "string",            // required
  "model": "qwen3.5:9b",          // optional – generation model for this request
  "force_rag": false,             // optional – skip gateway routing, always call the RAG API

  // Retrieval options, forwarded to the RAG API when the RAG route is taken.
  // Omitted options are not forwarded, so the pipeline profile default applies.
  "compose_sub_answers": true,
  "query_decompose": true,        // alias: "decompose"
  "ai_rerank": true,              // profile default when omitted is ON (arm G, 2026-08-14; eval/DECISIONS.md)
  "context_expand": true,
  "verify": true,
  "retrieval_k": 20,
  "context_window_size": 1,
  "reranker_top_k": 10,
  "retrieval_mode": "hybrid",     // "hybrid" | "vector_only" | "fts_only" (alias: "search_type")
  "provence_prune": false,
  "provence_threshold": 0.1,
  "filters": { }                  // optional metadata filter object, forwarded verbatim and
                                  // validated by the RAG API (§2.1). A present filter also
                                  // forces the RAG route, like force_rag.
}
```

camelCase aliases are accepted for all of them (`composeSubAnswers`, `queryDecompose`, `aiRerank`, `contextExpand`, `retrievalK`, `contextWindowSize`, `rerankerTopK`, `retrievalMode`, `searchType`, `provencePrune`, `provenceThreshold`, `forceRag`).

Response:

```jsonc
{
  "response": "string",           // assistant answer, <think> tags stripped
  "session": { /* ChatSession */ },
  "source_documents": [],         // empty on the direct-LLM route
  "used_rag": true
}
```

This endpoint persists the turn itself: it writes the user message, derives a session title from the first message, then writes the assistant message with its sources in `metadata.source_documents`. (Streamed turns are persisted separately via `/messages/save`.)

`force_rag` decides gateway routing **and** is forwarded to the RAG API, so the agent's own triage is skipped too.

### 1.3 Gateway routing

`POST /sessions/<id>/messages` picks its path with a deterministic gate — no model call, no retrieval, sub-millisecond:

1. `force_rag: true` (or `forceRag`) → RAG, unconditionally.
2. Session has no linked indexes → answered directly by Ollama, no retrieval.
3. The whole message is smalltalk (`hello`, `thanks!`, `bye`, `ok` — an allowlist regex capped at six words) or a question about the assistant itself (`who are you`, `what model are you`) → direct.
4. Anything else → RAG.

`used_rag` in the response tells you which way it went. The gate errs toward RAG on purpose: the RAG API's own agent triage still runs on every forwarded request and can answer directly without retrieving, so a false "use RAG" costs one triage call, not a wrong answer. If you need the answer grounded in documents regardless, send `force_rag`.

The streaming path (`:8001/chat/stream`, the UI default) never touches this gate — the browser calls the RAG API directly and only the agent triage applies.

### 1.4 Index options

Accepted by `POST /sessions/<id>/index` and `POST /indexes/<id>/build`, normalised and forwarded to the RAG API `/index`. **Options you omit are not sent**, so the RAG API's own defaults apply (§2.5).

```jsonc
{
  "chunk_size": 512,
  "window_size": 2,
  "retrieval_mode": "hybrid",       // "hybrid" | "vector_only" | "fts_only"
  "enable_enrich": true,
  "enable_latechunk": false,        // aliases: "latechunk", "enableLatechunk"
  "enable_docling_chunk": true,     // aliases: "doclingChunk", "enableDoclingChunk"; false = legacy chunker — see §2.5
  "embedding_model": "microsoft/harrier-oss-v1-0.6b",
  "enrich_model": "qwen3.5:4b",
  "overview_model_name": "qwen3.5:4b",   // aliases: "overviewModel", "overview_model"
  "batch_size_embed": 50,
  "batch_size_enrich": 25
}
```

`POST /sessions/<id>/index` returns the RAG API's `/index` response unchanged, or `200 { "message": "No documents to index for this session." }` when the session has no uploaded files.

`POST /indexes/<id>/build` echoes the canonical options back alongside the RAG API response, renaming `enable_latechunk` → `latechunk`, `enable_docling_chunk` → `docling_chunk` and `overview_model_name` → `overview_model`, and stores the same values in the index's metadata. If the RAG API reports that the table already exists, the build is treated as idempotent and returns `{ message: "Index already built – skipping rebuild.", note }`.

### 1.5 Error responses

| Status | When |
|--------|------|
| 400 | Missing required field, invalid JSON, no files in a multipart upload |
| 404 | Unknown route, unknown session, unknown index |
| 500 | Unhandled server error |
| 502 | Could not connect to the RAG API (`RAG_API_URL`), or the RAG API returned a non-200 |
| 503 | Ollama is not reachable (`POST /chat` only) |
| 504 | The RAG API did not answer within `RAG_API_TIMEOUT` (chat, default 600 s) or `RAG_API_INDEX_TIMEOUT` (indexing, default 3600 s) |

Error bodies are JSON: `{ "error": "..." }`.

---

## 2. RAG API — `http://localhost:8001`

| Endpoint | Method | Description | Request body | Success response |
|----------|--------|-------------|--------------|------------------|
| `/health` | GET | Liveness probe | – | `{ "status": "ok" }` |
| `/models` | GET | Models available to the active LLM backend | – | `{ generation_models, embedding_models }` |
| `/chat` | POST | Run the full agent pipeline | [Chat request](#21-chat-request) | `{ answer, source_documents }` |
| `/chat/stream` | POST | Same pipeline, streamed as SSE | [Chat request](#21-chat-request) | `text/event-stream` |
| `/index` | POST | Index documents | [Index request](#25-index-request) | see §2.6 |

Unknown routes return 404 `{ "error": "Not Found" }`.

### 2.1 Chat request

```
/* Detailed source-code truncated for AI context efficiency. */
```

Notes:

* Every option, including `retrieval_k`, `context_window_size` and `reranker_top_k`, falls back to the pipeline profile when omitted; only options you actually send override the profile, and the override is scoped to that request — the agent snapshots its config before applying request options and restores it afterwards.
* An unsupported `retrieval_mode` is rejected with `400 { "error": "Unsupported retrieval mode '…'. Supported: hybrid, vector_only, fts_only." }`.
* `model` is applied for the duration of the request and then restored. A model id that does not match the active `LLM_BACKEND` (for example an Ollama tag while `LLM_BACKEND=watsonx`) is ignored with a warning.
* If the table's index metadata records an `embedding_model`, the retrieval pipeline switches to it before searching.
* `filters` is validated by `rag_system/retrieval/filters.py`: unknown fields/operators, wrong types, empty IN-lists, an empty object, and values containing quoting characters (`'`, `"`, `\`, `;`, backtick, control chars — refused, never escaped) all return `400` with the validator's message. Sending `filters` also skips triage, like `force_rag`. Page/date filtering is not supported (the values live inside the metadata JSON column).

### 2.2 Chat response

```jsonc
{
  "answer": "string",
  "source_documents": [
    {
      "chunk_id": "string",
      "text": "string",
      "score": 0.0164,          // higher is better; RRF score in hybrid mode
      "document_id": "report.pdf",
      "chunk_index": 12,
      "metadata": { },
      "rerank_score": 0.87,     // present only when reranking ran
      "bm25": 4.21              // present only when the full-text leg matched this chunk
    }
  ],
  "token_usage": {              // per-query token accounting (roadmap 4.5, always on)
    "by_stage": { "synthesis": { "prompt_tokens": 1192, "output_tokens": 861, "calls": 1 } },
    "total": { "prompt_tokens": 1192, "output_tokens": 861, "calls": 1, "total_tokens": 2053 }
  },
  "document_escalation": [ ]    // present only when full-document escalation fired (flag-gated, off by default)
}
```

An absent key in `by_stage` means that stage made no LLM call, not that it cost
zero tokens. Only Ollama reports real counts; watsonx reports zeros.

There is no top-level `confidence` or `reasoning` field. When verification is enabled the confidence is appended to `answer` as `" [Confidence: N%]"`, plus `" [Warning: Low confidence. Groundedness: <bool>]"` when the answer is judged ungrounded or scores below 50. Nothing is appended when the verifier's score parses as 0.

When nothing is retrieved, `answer` is `"I could not find an answer in the documents."` and `source_documents` is empty.

### 2.3 `POST /chat/stream` (SSE)

Same request body. The response is `text/event-stream`; each event is a single line:

```
data: {"type": "<event>", "data": <payload>}
```

| Event | Payload | Emitted when |
|-------|---------|--------------|
| `analyze` | `{query}` | Start of the run |
| `direct_answer` | `{}` | Triage chose a direct answer |
| `decomposition` | `{sub_queries}` | Query decomposition produced sub-queries |
| `retrieval_started` | `{mode}` or `{count}` | `{mode}` from the retrieval pipeline, `{count}` from the decomposition branch |
| `retrieval_done` | `{count}` | Retrieval finished |
| `retrieval_retry` | `{score_before, score_after, kept, …}` | The evidence-sufficiency retry ran (design_rationale §5) |
| `crossref_hop` | `{targets, chunks_added}` | The cross-reference hop pulled chunks (flag-gated, off by default) |
| `document_escalation` | `{document_id, document_name, chunks_used, chunks_total, approx_tokens, truncated, signal, score, threshold, token_budget}` | Full-document escalation fired (flag-gated, off by default); never contains the document text |
| `rerank_started` / `rerank_done` | `{count}` | Reranking |
| `context_expand_started` / `context_expand_done` | `{count}` | Context expansion |
| `prune_started` / `prune_done` | `{count}` | Provence pruning (only when enabled) |
| `token` | `{text}` | Streamed answer tokens |
| `sub_query_token` | `{index, text, question}` | Tokens from a parallel sub-query |
| `sub_query_result` | `{index, query, answer, source_documents}` | A sub-query finished |
| `single_query_result` | the pipeline result | Decomposition produced exactly one sub-query |
| `final_answer` | `{answer, source_documents}` | Composed answer ready |
| `complete` | `{answer, source_documents, token_usage}` | Final event; clients may close here. `token_usage` has the shape shown in §2.2 |
| `error` | `{error}` | Failure after the stream opened |

> This endpoint does **not** write to SQLite. The browser's default chat path calls it directly and, once the `complete` event arrives, persists the finished turn via `POST :8000/sessions/<id>/messages/save`. Clients that consume the stream directly must do the same if they want the turn in the session history.

### 2.4 `GET /models`

```jsonc
{
  "generation_models": ["qwen3.5:4b", "qwen3.5:9b"],
  "embedding_models": ["Qwen/Qwen3-Embedding-0.6B", "Qwen/Qwen3-Embedding-4B", "Qwen/Qwen3-Embedding-8B", "microsoft/harrier-oss-v1-0.6b"]
}
```

With `LLM_BACKEND=ollama` the generation list comes from `GET {OLLAMA_HOST}/api/tags` (5 s timeout; on failure the list is simply shorter), split by a substring match on `embed` / `bge` / `embedding`. With `LLM_BACKEND=watsonx` it lists the configured WatsonX generation and enrichment models. The embedding list always includes the pipeline's configured embedding model, the default `microsoft/harrier-oss-v1-0.6b` and the three Qwen3-Embedding sizes; it is returned sorted and de-duplicated.

### 2.5 Index request

```jsonc
{
  "file_paths": ["/abs/path/a.pdf", "/abs/path/b.docx"],  // required
  "session_id": "string",              // optional – overview file + metadata target
  "table_name": "string",              // optional – otherwise resolved from session_id

  "chunk_size": 512,                   // default 512  – token budget per chunk
  "window_size": 2,                    // default 2    – contextual-enrichment window
  "enable_enrich": true,               // default true
  "enable_latechunk": false,           // default false
  "enable_docling_chunk": true,        // default true – false selects the legacy fixed-size chunker
  "retrieval_mode": "hybrid",          // optional – validated, recorded on the index config
  "embedding_model": "microsoft/harrier-oss-v1-0.6b",
  "enrich_model": "qwen3.5:4b",
  "overview_model_name": "qwen3.5:4b",
  "batch_size_embed": 50,              // default 50
  "batch_size_enrich": 25              // default 25
}
```

* `session_id` is **optional**. Without it the default LanceDB table is used and overviews go to the global `index_store/overviews/overviews.jsonl` instead of a per-index file.
* `retrieval_mode` cannot change the artifacts written at index time; it is validated (400 on an unsupported value) and stored in the index config as `retrieval.search_type`. The mode that matters is the one you send at query time.
* `embedding_model` is applied to this build **and** recorded in the index metadata when the `session_id` names an actual index (checked by existence); session-scoped builds — whose id is a chat-session UUID with no indexes row — skip the recording. This is what makes queries against that index use the same embedder.
* `enable_latechunk` defaults to `false` here, so an HTTP build without the flag writes no `<table>_lc` table even though the `default` profile enables late chunking.
* `enable_docling_chunk` defaults to `true`; sending `false` selects the legacy fixed-size chunker instead of Docling's structure-aware chunker.
* Unknown fields are ignored silently.

### 2.6 Index response

```jsonc
{
  "message": "Indexing process for 2 file(s) completed successfully.",
  "table_name": "text_pages_<index_id>",
  "latechunk": false,
  "docling_chunk": true,
  "indexing_config": {
    "chunk_size": 512,
    "retrieval_mode": "hybrid",
    "window_size": 2,
    "enable_enrich": true,
    "embedding_model": "microsoft/harrier-oss-v1-0.6b",
    "enrich_model": "qwen3.5:4b",
    "overview_model_name": "qwen3.5:4b",
    "batch_size_embed": 50,
    "batch_size_enrich": 25
  }
}
```

`indexing_config.embedding_model` reports the model actually used for the build, not the raw request value. There is no `indexed_files` field.

Errors: `400 { "error": "A 'file_paths' list is required." }`, `400 { "error": "Invalid JSON" }`, `400` for an unsupported `retrieval_mode`, and `500 { "error": "Failed to start indexing: …" }`.

---

## 3. Frontend Wrapper (`src/lib/api.ts`)

The typed client exported as `chatAPI`. Base URLs come from `NEXT_PUBLIC_API_URL` (default `http://localhost:8000`) and `NEXT_PUBLIC_RAG_API_URL` (default `http://localhost:8001`), both inlined at build time. **The browser talks to both origins**, so a deployment must expose both.

| Method | Target |
|--------|--------|
| `checkHealth()` | `GET :8000/health` |
| `sendMessage({message, model?, conversation_history?})` | `POST :8000/chat` |
| `getSessions()` | `GET :8000/sessions` |
| `createSession(title?, model?)` | `POST :8000/sessions` |
| `getSession(sessionId)` | `GET :8000/sessions/<id>` |
| `sendSessionMessage(sessionId, message, opts)` | `POST :8000/sessions/<id>/messages` |
| `saveStreamedTurn(sessionId, userMessage, assistantMessage, sourceDocuments?)` | `POST :8000/sessions/<id>/messages/save` |
| `deleteSession(sessionId)` | `DELETE :8000/sessions/<id>` |
| `renameSession(sessionId, title)` | `POST :8000/sessions/<id>/rename` |
| `uploadFiles(sessionId, files)` | `POST :8000/sessions/<id>/upload` |
| `indexDocuments(sessionId)` | `POST :8000/sessions/<id>/index` (no options) |
| `getModels()` | `GET :8000/models` |
| `createIndex(name, description?, metadata?)` | `POST :8000/indexes` |
| `uploadFilesToIndex(indexId, files)` | `POST :8000/indexes/<id>/upload` |
| `buildIndex(indexId, opts)` | `POST :8000/indexes/<id>/build` |
| `listIndexes()` | `GET :8000/indexes` |
| `getSessionIndexes(sessionId)` | `GET :8000/sessions/<id>/indexes` |
| `deleteIndex(indexId)` | `DELETE :8000/indexes/<id>` |
| `linkIndexToSession(sessionId, indexId)` | `POST :8000/sessions/<sid>/indexes/<idx>` |
| **`streamSessionMessage(params, onEvent)`** | **`POST :8001/chat/stream`** — the default chat path |

The camelCase argument names on these methods (`retrievalK`, `rerankerTopK`, `doclingChunk`, …) are TypeScript parameter names. `sendSessionMessage` and `streamSessionMessage` serialise them to snake_case JSON; `buildIndex` sends camelCase, which both servers normalise.

Exported model defaults, kept in step with `rag_system/main.py`:

```ts
export const DEFAULT_GENERATION_MODEL = 'qwen3.5:9b';
export const DEFAULT_ENRICHMENT_MODEL = 'qwen3.5:4b';
export const DEFAULT_EMBEDDING_MODEL  = 'microsoft/harrier-oss-v1-0.6b';
```

---

_Derived by reading `backend/server.py`, `rag_system/api_server.py` and `src/lib/api.ts`. Keep it in sync with route, option and response-shape changes._

---

### Documentation Deployment Guide (`Documentation/deployment_guide.md`)

# 🚀 LocalGPT Deployment Guide

_Last updated: 2026-08-08_

This guide covers deploying LocalGPT with Docker or as directly-run processes.

---

## 🎯 Deployment Options

### Option 1: Docker Deployment 🐳
- **Best for**: reproducible environments, keeping dependencies isolated
- **Pros**: one command, pinned base images, health-gated startup order
- **Cons**: slower first build, GPU passthrough is extra work

### Option 2: Direct Processes 💻
- **Best for**: development, debugging, GPU/MPS access
- **Pros**: direct access to code, faster iteration, native acceleration
- **Cons**: more dependencies to manage on the host

---

## 1. Prerequisites

### 1.1 System Requirements

#### **Minimum Requirements**
- **CPU**: 4 cores, 2.5GHz+
- **RAM**: 8GB (16GB recommended)
- **Storage**: 50GB free space
- **OS**: Linux, macOS, or Windows with WSL2

#### **Recommended Requirements**
- **CPU**: 8+ cores, 3.0GHz+
- **RAM**: 32GB+
- **Storage**: 200GB+ SSD
- **GPU**: NVIDIA GPU with 8GB+ VRAM (optional; Apple Silicon uses MPS)

### 1.2 Common Dependencies

```bash
# Ollama (required for both approaches)
curl -fsSL https://ollama.ai/install.sh | sh

# Git for cloning
git 2.30+
```

### 1.3 Docker-Specific Dependencies

```bash
Docker Engine 24.0+
Docker Compose plugin 2.20+
```

### 1.4 Direct Deployment Dependencies

```bash
Python 3.10+   # 3.11 recommended; the images use python:3.11-slim
Node.js 20+
npm 10+
```

---

## 2. 🐳 Docker Deployment

### 2.1 Installation

#### **Step 1: Install Docker**

**Ubuntu/Debian:**
```bash
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
newgrp docker

sudo apt-get update
sudo apt-get install docker-compose-plugin
```

**macOS:**
```bash
brew install --cask docker
# Or download from: https://www.docker.com/products/docker-desktop
```

**Windows:**
```bash
# Install Docker Desktop with the WSL2 backend
# Download from: https://www.docker.com/products/docker-desktop
```

#### **Step 2: Clone Repository**
```bash
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT
```

#### **Step 3: Install Ollama**
```bash
# Runs on the host by default, even with Docker
curl -fsSL https://ollama.ai/install.sh | sh

ollama serve

# In another terminal
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
```

#### **Step 4: Launch**
```bash
# Convenience script (local Ollama)
./start-docker.sh

# Containerized Ollama instead
./start-docker.sh container

# Or manually
docker compose --env-file docker.env up --build -d
```

#### **Step 5: Verify Deployment**
```bash
docker compose ps

curl http://localhost:3000            # Frontend
curl http://localhost:8000/health     # Backend
curl http://localhost:8001/health     # RAG API
curl http://localhost:11434/api/tags  # Ollama
```

### 2.2 Startup Order

`docker-compose.yml` gates the stack on health checks:

```
rag-api (healthy: GET /health)  ->  backend (healthy: GET /health)  ->  frontend
```

`rag-api` loads the embedding model before it answers `/health` (the reranker
loads lazily, on the first reranked query), so
its check uses a 120s start period. Until it passes, `backend` stays in `created`
and `frontend` after it. This is expected on a cold start, not a hang — watch
`docker compose logs -f rag-api`.

### 2.3 Docker Management

```bash
# Convenience script
./start-docker.sh            # start (local Ollama)
./start-docker.sh container  # start (containerized Ollama)
./start-docker.sh stop       # stop
./start-docker.sh logs       # follow logs
./start-docker.sh status     # container status
./start-docker.sh help       # usage

# Compose directly
docker compose ps
docker compose logs -f
docker compose down
docker compose --env-file docker.env up --build -d

# One service
docker compose restart rag-api
docker compose logs -f backend
docker compose exec rag-api python -c "print('hello')"
```

### 2.4 Compose Files

| File | Contents |
|------|----------|
| `docker-compose.yml` | The full stack: `rag-api`, `backend`, `frontend`, and an optional `ollama` service behind the `with-ollama` profile |

`docker-compose.yml` is the only compose file: it defaults to host Ollama and
adds the `ollama` container when you pass `--profile with-ollama`.

---

## 3. 💻 Direct Deployment

### 3.1 Installation

**Python Dependencies:**
```bash
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

pip install -r requirements.txt
```

**Node.js Dependencies:**
```bash
npm install
```

### 3.2 Install and Configure Ollama
```bash
curl -fsSL https://ollama.ai/install.sh | sh
ollama serve

# In another terminal
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
```

### 3.3 Launch

**Option A: Integrated Launcher (Recommended)**
```bash
# Development
python run_system.py

# Production: runs `npm run build`, then `next start`
python run_system.py --mode prod
```

The launcher records the launcher PID plus each child PID in
`logs/run_system.pid`, aggregates every service's stdout into `logs/<service>.log`,
and restarts a required service that exits unexpectedly (checked every 30s).

**Option B: Manual Component Startup — all from the repository root**
```bash
# Terminal 1: RAG API
python -m rag_system.api_server

# Terminal 2: Backend
python backend/server.py

# Terminal 3: Frontend
npm run build && npm run start     # or: npm run dev

# Access at http://localhost:3000
```

> Relative paths (`backend/chat_data.db`, `lancedb/`, `index_store/`,
> `shared_uploads/`) resolve against the working directory. Always start from the
> repository root.

### 3.4 Direct Deployment Management

```bash
python run_system.py --health      # HTTP checks; exit 1 if a required service fails
python run_system.py --logs-only   # tail logs/*.log from another shell
python run_system.py --stop        # terminate everything in logs/run_system.pid
python run_system.py --no-frontend # Ollama + RAG API + backend only
python system_health_check.py      # deep check incl. a real embedding + query
```

`--stop` reads `logs/run_system.pid`, kills the launcher first (so its monitor
cannot restart anything), then each service and its descendants with SIGTERM,
escalating to SIGKILL after 10s. It exits non-zero if there is no pidfile.

---

## 4. Architecture

### 4.1 Docker Architecture

```mermaid
graph TB
    subgraph "Docker Containers"
        Frontend[frontend<br/>Next.js<br/>Port 3000]
        Backend[backend<br/>Gateway + SQLite<br/>Port 8000]
        RAG[rag-api<br/>Indexing + Retrieval<br/>Port 8001]
    end

    subgraph "Host"
        Ollama[Ollama Server<br/>Port 11434]
    end

    Browser --> Frontend
    Browser -. "SSE /chat/stream" .-> RAG
    Frontend --> Backend
    Backend --> RAG
    RAG --> Ollama
    Backend --> Ollama
```

`backend` and `rag-api` both mount `./backend`, `./lancedb` and `./index_store`,
and both point `DB_PATH` at `/app/backend/chat_data.db`, so they share one SQLite
file and one vector store.

### 4.2 Direct Architecture

```mermaid
graph TB
    subgraph "Local Processes"
        Frontend[Next.js<br/>Port 3000]
        Backend[backend/server.py<br/>Port 8000]
        RAG[rag_system.api_server<br/>Port 8001]
        Ollama[Ollama Server<br/>Port 11434]
    end

    Browser --> Frontend
    Browser -. "SSE /chat/stream" .-> RAG
    Frontend --> Backend
    Backend --> RAG
    RAG --> Ollama
    Backend --> Ollama
```

### 4.3 Concurrency

- `backend/server.py` uses a `ThreadingTCPServer`, so it handles requests in
  parallel and stays responsive during a long RAG call.
- `rag_system/api_server.py` uses a plain single-threaded `TCPServer`. **RAG
  requests are serialised** — one chat or indexing run at a time. Plan capacity
  for a single concurrent user of the RAG API, or put a queue in front of it.
- The backend allows `RAG_API_TIMEOUT` (default 600s) for a chat call and
  `RAG_API_INDEX_TIMEOUT` (default 3600s) for indexing, returning 504 on timeout
  and 502 when the RAG API is unreachable — both with a JSON error body.

---

## 5. Configuration

### 5.1 Environment Variables

Every variable below is read by code; the value shown is the default when unset.
`.env.example` carries the same list.

| Variable | Default | Read by |
|----------|---------|---------|
| `OLLAMA_HOST` | `http://localhost:11434` | `rag_system/main.py`, `backend/ollama_client.py` |
| `RAG_API_URL` | `http://localhost:8001` | `backend/server.py` |
| `NEXT_PUBLIC_API_URL` | `http://localhost:8000` | `src/lib/api.ts` (inlined at build time) |
| `NEXT_PUBLIC_RAG_API_URL` | `http://localhost:8001` | `src/lib/api.ts` (inlined at build time) |
| `DB_PATH` | `backend/chat_data.db` | `backend/database.py` |
| `LANCEDB_PATH` | `./lancedb` | `rag_system/main.py` (pipeline profiles), `backend/database.py`, `system_health_check.py` |
| `GENERATION_MODEL` | `qwen3.5:9b` | `rag_system/main.py`, `backend/server.py`, `run_system.py` |
| `ENRICHMENT_MODEL` | `qwen3.5:4b` | same |
| `EMBEDDING_MODEL` | `microsoft/harrier-oss-v1-0.6b` | `rag_system/main.py` |
| `RERANKER_MODEL` | `Qwen/Qwen3-Reranker-4B` (loaded lazily on the first reranked query) | `rag_system/main.py` |
| `RAG_CONFIG_MODE` | `default` | `rag_system/api_server.py` |
| `RAG_API_TIMEOUT` | `600` | `backend/server.py` |
| `RAG_API_INDEX_TIMEOUT` | `3600` | `backend/server.py` |
| `LLM_BACKEND` | `ollama` | `rag_system/main.py` |
| `HF_TOKEN` | unset | HuggingFace downloads |

#### **Docker Configuration (`docker.env`)**
```bash
OLLAMA_HOST=http://host.docker.internal:11434
NODE_ENV=production
RAG_API_URL=http://rag-api:8001
NEXT_PUBLIC_API_URL=http://localhost:8000
NEXT_PUBLIC_RAG_API_URL=http://localhost:8001
DB_PATH=/app/backend/chat_data.db
LANCEDB_PATH=/app/lancedb
GENERATION_MODEL=qwen3.5:9b
ENRICHMENT_MODEL=qwen3.5:4b
EMBEDDING_MODEL=microsoft/harrier-oss-v1-0.6b
RERANKER_MODEL=Qwen/Qwen3-Reranker-4B
```

All compose services declare
`extra_hosts: ["host.docker.internal:host-gateway"]`, so `host.docker.internal`
resolves on Linux as well as macOS and Windows.

`NEXT_PUBLIC_*` are **build-time** values for Next.js. `docker-compose.yml` passes
them as build args to `Dockerfile.frontend`; setting them only at runtime has no
effect on an already-built image. If the browser must reach the services under a
different hostname, set them and rebuild:

```bash
NEXT_PUBLIC_API_URL=https://gpt.example.com/api \
NEXT_PUBLIC_RAG_API_URL=https://gpt.example.com/rag \
docker compose --env-file docker.env up --build -d frontend
```

#### **Direct Deployment Configuration**
```bash
# run_system.py inherits your shell environment unchanged and adds only
# NODE_ENV=production to the Python services in --mode prod.
export OLLAMA_HOST=http://localhost:11434
export RAG_API_URL=http://localhost:8001
python run_system.py
```

### 5.2 Model Configuration

Defaults live in `rag_system/main.py` and are overridable by environment variable.

| Role | Default | Documented options |
|------|---------|--------------------|
| Generation | `qwen3.5:9b` | `qwen3.6:27b` (high-end, ~17GB), `qwen3.5:4b` (light) |
| Enrichment / utility | `qwen3.5:4b` | `qwen3.5:2b` (light) |
| Embedding | `microsoft/harrier-oss-v1-0.6b` (MIT, 1024 dims, ~1.2 GB) | `Qwen/Qwen3-Embedding-4B` (2560 dims, 32K context — multilingual / long-context corpora), `Qwen/Qwen3-Embedding-0.6B` (1024 dims) |
| Reranker (**on by default**) | `Qwen/Qwen3-Reranker-4B` | `BAAI/bge-reranker-v2-m3` (low latency), `answerdotai/answerai-colbert-small-v1`, `Qwen/Qwen3-Reranker-0.6B` |

`GET /models` on either service reports what is actually selectable:
Ollama tags are split into generation and embedding lists by a substring match
(`embed`, `bge`, `embedding` on the RAG API; the backend also matches `text`), and
the HuggingFace embedding models are appended. A tag whose name happens to contain
one of those substrings will be classified as an embedding model.

**Changing the embedding model requires re-indexing.** Vector width is measured
from the loaded model; writing a different width into an existing LanceDB table
fails with an explicit error.

### 5.3 Performance Tuning

There are no `SEARCH_CONFIG` / `CHUNK_OVERLAP` globals. The knobs are pipeline
config keys and per-request fields.

**Indexing throughput** — `PIPELINE_CONFIGS[<mode>]["indexing"]` in
`rag_system/main.py`, or `batch_size_embed` / `batch_size_enrich` on
`POST /index`:

| Key | `default` | `fast` | Request field |
|-----|-----------|--------|---------------|
| `embedding_batch_size` | 50 | 100 | `batch_size_embed` (default 50) |
| `enrichment_batch_size` | 10 | 50 | `batch_size_enrich` (default 25) |

**Chunking** — `chunk_size` on `POST /index` (default 512) feeds the Docling
chunker's `max_tokens`, or the legacy chunker's `max_chunk_size` with
`min_chunk_size = chunk_size // 4`. When no `chunking.chunk_size` is present at
all (for example the `python -m rag_system.main index` CLI path), the pipeline
falls back to 1500. There is no `chunk_overlap` setting.

**Contextual enrichment** is the most expensive part of indexing: one LLM call per
chunk. Disable it with `enable_enrich: false` for the fastest ingest.

**Query cost** — the biggest lever is the profile:

| | `default` | `fast` |
|---|---|---|
| `retrieval.search_type` | `hybrid` | `vector_only` |
| `retrieval_k` | 20 | 10 |
| `reranker.enabled` | true | false |
| `query_decomposition.enabled` | true | false |
| `verification.enabled` | true | false |
| `retrieval.latechunk.enabled` | true | false |

Select the profile with `RAG_CONFIG_MODE=fast` for the RAG API, or `--mode fast`
for the CLI. Individual toggles (`ai_rerank`, `verify`, `query_decompose`,
`context_expand`, `retrieval_k`, `reranker_top_k`) can also be sent per request.

**Memory** — the embedding model stays resident in the RAG API process
(`microsoft/harrier-oss-v1-0.6b`, ~1.2 GB). Reranking is on by default, but the
reranker loads lazily: the first reranked query pulls ~7.5 GB of
`Qwen/Qwen3-Reranker-4B` weights alongside it — see
[`../eval/DECISIONS.md`](../eval/DECISIONS.md) for the quality/latency trade that
decision rests on. Enabling late chunking loads a second copy of the embedding
model.

---

## 6. Operational Procedures

### 6.1 System Monitoring

```bash
# Health
curl -f http://localhost:3000 && echo "✅ Frontend OK"
curl -f http://localhost:8000/health && echo "✅ Backend OK"
curl -f http://localhost:8001/health && echo "✅ RAG API OK"
curl -f http://localhost:11434/api/tags && echo "✅ Ollama OK"

# Or, for a direct deployment
python run_system.py --health
```

`GET /health` on the backend returns Ollama reachability, the model list and
database stats; on the RAG API it returns `{"status": "ok"}` as soon as the agent
has finished loading.

```bash
# Resource usage
docker stats               # Docker
htop                       # host
nvidia-smi                 # GPU, if present
```

### 6.2 Log Management

#### **Docker Logs**
```bash
docker compose logs -f
docker compose logs -f rag-api
docker compose logs > system.log 2>&1
```

#### **Direct Deployment Logs**
```bash
# run_system.py writes per-service files
tail -f logs/system.log logs/rag-api.log logs/backend.log logs/frontend.log

# or use the launcher's own tailer from a second shell
python run_system.py --logs-only
```

### 6.3 Backup and Restore

Everything persistent is a host directory, including under Docker (all mounts are
bind mounts; the only named volume is `ollama_data` for the optional Ollama
container).

#### **Data Backup**
```bash
# Stop first so SQLite and LanceDB are not mid-write
./start-docker.sh stop          # Docker
# or: python run_system.py --stop

mkdir -p backups/$(date +%Y%m%d)
cp backend/chat_data.db backups/$(date +%Y%m%d)/
tar czf backups/$(date +%Y%m%d)/lancedb.tar.gz lancedb/
tar czf backups/$(date +%Y%m%d)/index_store.tar.gz index_store/
tar czf backups/$(date +%Y%m%d)/shared_uploads.tar.gz shared_uploads/

# Only if you use the containerized Ollama, back up its named volume too.
# The project name is the directory name, so the volume is localgpt_ollama_data.
docker run --rm -v localgpt_ollama_data:/data -v $(pwd)/backups:/backup \
  alpine tar czf /backup/ollama_models_$(date +%Y%m%d).tar.gz -C /data .
```

#### **Data Restore**
```bash
./start-docker.sh stop          # or: python run_system.py --stop

cp backups/YYYYMMDD/chat_data.db backend/
tar xzf backups/YYYYMMDD/lancedb.tar.gz
tar xzf backups/YYYYMMDD/index_store.tar.gz

./start-docker.sh               # or: python run_system.py
```

Restore the SQLite file and `lancedb/` together — the database rows point at
LanceDB table names, so a mismatched pair leaves indexes that resolve to nothing.

---

## 7. Troubleshooting

### 7.1 Common Issues

#### **Port Conflicts**
```bash
lsof -i :3000 -i :8000 -i :8001 -i :11434

# Docker
./start-docker.sh stop

# Direct
python run_system.py --stop
```

If a required port (8001 or 8000) is already taken, `run_system.py` logs
`Port … already in use, skipping …` and aborts with "System startup failed". Port
11434 in use is treated as "Ollama already running" and reused; port 3000 in use is
tolerated because the frontend is optional.

#### **Docker Issues**
```bash
docker version                  # daemon reachable?
sudo systemctl restart docker   # Linux
docker system prune -f          # clear build cache
```

#### **Ollama Issues**
```bash
curl http://localhost:11434/api/tags

pkill ollama
ollama serve

ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
```

#### **Backend returns 502/504 for every chat**
The backend could not reach the RAG API (502) or the call timed out (504) — both
come back as JSON error bodies. Check `RAG_API_URL` (must be
`http://rag-api:8001` inside Docker, not `localhost`) and that `rag-api` is
healthy.

### 7.2 Performance Issues

#### **Memory Problems**
```bash
free -h           # Linux
vm_stat           # macOS
docker stats      # containers

# Options:
# 1. Switch to RAG_CONFIG_MODE=fast
# 2. Switch reranking off (ai_rerank=false) — it is on by default and lazily
#    loads ~7.5 GB of reranker weights on the first reranked query
# 3. Use a smaller generation model (GENERATION_MODEL=qwen3.5:4b)
# 4. Disable late chunking so a second embedding model is not loaded
```

#### **Slow Response Times**
```bash
# Where is the time going?
docker compose logs -f rag-api      # stage-by-stage output

time curl -s http://localhost:8001/health   # process responsive?
```

Remember requests to the RAG API are serialised — a query that appears slow may be
queued behind an indexing run.

---

## 8. Production Considerations

### 8.1 Security

LocalGPT ships with **no authentication** and permissive CORS
(`Access-Control-Allow-Origin: *`) on both HTTP services. Before exposing it:

- Put a reverse proxy (nginx, Caddy, Traefik) in front and terminate TLS there
- Add authentication at the proxy
- Publish only port 3000; keep 8000 and 8001 on an internal network
- Note the browser calls the RAG API directly for streaming, so port 8001 must be
  reachable by clients (or proxied) if you leave streaming enabled

### 8.2 Scaling

`docker compose up --scale` does **not** work with the shipped compose file: every
service sets a fixed `container_name` and publishes a fixed host port. Scaling
requires removing those first.

Beyond that, the RAG API is single-threaded and holds mutable state (the resident
agent, its per-session in-memory chat history and the semantic cache), so running
several replicas behind a load balancer needs sticky sessions at minimum. The
realistic path is a single RAG API with a queue in front of it.

---

## 9. Success Criteria

### 9.1 Deployment Verification

- ✅ `docker compose ps` shows all services healthy (or `python run_system.py --health` exits 0)
- ✅ Frontend loads at http://localhost:3000
- ✅ You can create a document index
- ✅ You can chat with uploaded documents
- ✅ No errors in `docker compose logs` / `logs/`

### 9.2 What to Expect

Throughput depends on hardware and model size, so treat these as shape rather than
guarantees:

- Cold start is dominated by model downloads and the RAG API loading the embedding
  model (the reranker loads lazily, on the first reranked query)
- Indexing is dominated by contextual enrichment (one LLM call per chunk)
- Query latency is dominated by generation; `fast` mode removes reranking,
  decomposition and verification
- Concurrency is one RAG request at a time

---

**Happy Deploying! 🚀**

---

### Documentation Design Rationale (`Documentation/design_rationale.md`)

# Design Rationale

_Revision: 2026-08-09. Roadmap item [3.1](research_roadmap.md#phase-3--documentation-make-the-evidence-part-of-the-repos-argument)._

This document answers one question per component: **why does localGPT do it this
way?** It describes only what ships in the current tree. Anything planned,
proposed or merely promising lives in [`research_roadmap.md`](research_roadmap.md)
and [`improvement_plan.md`](improvement_plan.md), not here.

## The method

Three artefacts, in a fixed order:

1. **Evidence** — [`research/`](research/) holds three August-2026 sweeps of
   primary sources (papers, model cards, first-party engineering blogs), each
   claim graded *established* / *emerging* / *contested* and each carrying its
   own "could not verify" appendix. These documents describe the field, **not
   this repo**.
2. **Our own eval** — [`eval/`](../eval/) holds a 72-query gold set over three
   corpora, a recall/nDCG runner, a binary groundedness judge validated against
   hand labels, and an end-to-end smoke test. See [`eval/README.md`](../eval/README.md).
3. **A decision** — nothing changes a default without a measured delta on (2).
   The decisions and their numbers are in [`eval/DECISIONS.md`](../eval/DECISIONS.md)
   and [`eval/decisions/`](../eval/decisions/).

The order matters because **it repeatedly produced the opposite answer from the
literature.** Three times in one week:

| The evidence said | Our eval measured | What shipped |
|---|---|---|
| "Cross-encoder rerank: **YES, unconditionally**, +17.2 pp MRR@3" ([`component-map-2026.md` §5.1](research/component-map-2026.md), §6.5) | `bge-reranker-v2-m3` on top of the new first stage: **−0.022 nDCG@10** on `mixed`, **−0.058** on `docs`, for ~1.6 s/query | Reranking initially **off** by default ([`DECISIONS.md` §1](../eval/DECISIONS.md)); turned **on** with score-threshold selection in arm G, 2026-08-14 |
| Decomposition helps when applied at *reranking* rather than first-stage (2026 MultiConIR/SSRB, [`component-map-2026.md` §6.2](research/component-map-2026.md)) | On the 6 queries that genuinely decompose: **−0.046** (`max`) / **−0.012** (`mean`) nDCG@10 | Shape change shipped; sub-query scoring at rerank now applies whenever the reranker is on (arm G), and the pooled first stage is the default decomposition path (arm H, 2026-08-15) ([`phase2-pipeline.md` §3](../eval/decisions/phase2-pipeline.md)) |
| Bigger instruction-tuned embedders lead the boards; the shipped default was the 8 GB `Qwen3-Embedding-4B` | A 1.2 GB MIT model **dominated** it: mixed nDCG@10 **0.915 vs 0.875**, ~3× lower latency, ~7× less memory | `microsoft/harrier-oss-v1-0.6b` as the default ([`embedder.md` gate section](../eval/decisions/embedder.md)) |

That is the whole method: the literature nominates candidates, our gold set
decides. Where the two disagree, the gold set wins and the disagreement is
written down rather than smoothed over.

Every number below is traceable to a file under `eval/`. Where a claim has no
number, it says so.

---

## 1. Parsing / OCR

**What ships.** Every document goes through [docling](https://github.com/docling-project/docling)
(`rag_system/ingestion/document_converter.py`). PDFs take one of two converters:
a text-layer probe (`_pdf_has_text`, PyMuPDF) decides whether OCR runs at all —
if *any* page has extractable text, the whole document is converted without OCR.
DOCX / HTML / MD go through a third general converter; `.txt` is read directly
and fenced. Conversion returns `(markdown, metadata, DoclingDocument)` so the
chunker can use the element tree rather than re-parsing markdown.

The OCR engine is **probed, not configured**: `build_ocr_options()` walks
`OcrMac → EasyOCR → RapidOCR → tesserocr → tesseract-cli` and picks the first
whose backend is actually importable on this host. There is no VLM parser.

**Why.** The 2026 evidence ([`component-map-2026.md` §1.5](research/component-map-2026.md))
says the local pipeline is "Docling as orchestration + a 0.9–1.2B specialist VLM
as the parsing engine", and that traditional parsers "survive as the fast path,
not the quality path". We ship the orchestration half and not the VLM half, on
purpose:

* The VLM spike ([`glm-ocr-spike.md`](../eval/decisions/glm-ocr-spike.md)) is
  **GO-LATER**, not NO-GO. It demonstrated a real, large win on the one document
  class that matters — a degraded scanned invoice where GLM-OCR read **30/30
  table cells** and the current chain lost every price and 4 of 5 part numbers.
* Three defects block adoption: Ollama's Modelfile ignores prompts (so GLM-OCR's
  table/formula modes are unreachable), some pages are deterministically
  transcribed twice, and docling flattens the model's pipe tables to
  `tables: 0`. None of them is code we should write blind.
* **There is no OCR eval.** `eval/corpora/` contains only digital-born PDFs with
  clean text layers, so nothing in the harness exercises the OCR branch. Adopting
  a parser on leaderboard position alone would violate the gate this repo runs on
  — and the spike showed exactly why: the roadmap's original "#1 on OmniDocBench,
  beats GPT-5.2 by ~10 points" line did not survive source verification (GLM-OCR
  is **third** on v1.6_full, behind PaddleOCR-VL-1.6 and MinerU2.5-Pro).

The probe order was also the cheap win: the RapidOCR probe used to test for the
stale module name `rapidocr_onnxruntime`, so a host with `rapidocr` 3.x installed
silently fell all the way through to the Tesseract CLI. It now accepts either
name (`OCR_BACKENDS`, `document_converter.py`), and on this machine RapidOCR is
what resolves.

**What would change the decision.** Build a scanned/tabular corpus with ground
truth under `eval/corpora/`, then A/B GLM-OCR against the *fixed* classic chain
**and** against docling's other 2026 presets (`lightonocr`, `dots_ocr`,
`nanonets_ocr2`) — one column in a table, not a foregone winner. Independently of
any VLM: `pip install ocrmac` on macOS is a free upgrade over the current chain,
and the text-layer probe should become per-page (today a scanned insert inside a
digital PDF gets no OCR at all).

## 2. Chunking + index-time enrichment

**What ships.** `DoclingChunker` (`rag_system/ingestion/docling_chunker.py`) is
the default (`chunker_mode: "docling"`). It walks the `DoclingDocument` element
tree, emits **tables, code and figures as atomic chunks**, sentence-packs
paragraph nodes up to a token budget, and attaches the heading path and block
type to every chunk's metadata. Token counting uses the *embedding model's own*
tokenizer. Budget is 512 tokens for HTTP index builds
(`rag_system/api_server.py`) and falls back to 1500 for the CLI path, because no
profile sets `chunking.chunk_size`. Overlap is one sentence.

**Index-time contextual enrichment is on** in the `default` profile
(`contextual_enricher: {enabled: True, window_size: 1}`, `rag_system/main.py`):
`ContextualEnricher` asks the enrichment model for a 2–5 sentence situating
summary per chunk from its ±1 neighbours and prepends it to the indexed text,
keeping the original text in metadata (`rag_system/indexing/contextualizer.py`).

**Late chunking** is implemented (`rag_system/indexing/latechunk.py`: embed the
whole document, mean-pool inside chunk spans) and enabled in the `default`
profile, but `POST /index` defaults `enable_latechunk` to `false`, so HTTP builds
and CLI builds differ — tracked as [`improvement_plan.md`](improvement_plan.md)
§3.6, not defended here. When a late-chunk table exists, its hits are merged with
their ±1 siblings before reranking (`retrieval_pipeline.py::_first_stage`).

**Why.**

* **Boring chunking is the right default.** Every controlled study since 2024
  puts the *total* spread across all chunking methods at ~9 points of recall —
  Chroma's 2024 report and a 2026 eight-method / nine-dataset replication both
  land on recursive/fixed splitting as the cost-effective winner, with LLM-driven
  chunkers (DenseX 69.1 Acc@5, 15+ hours) far behind
  ([`component-map-2026.md` §2.1, §2.5](research/component-map-2026.md)). The
  same source's summary is blunt: "chunking is not where your quality is."
* **Structure-awareness is nearly free** when the parser already hands you an
  element tree, which is why tables and code are atomic
  ([`component-map-2026.md` §2.5](research/component-map-2026.md)).
* **Contextual enrichment is the one chunking-adjacent intervention with a large
  measured effect** — 35% top-20 failure-rate reduction from contextual
  embeddings alone, 67% stacked with BM25 and reranking (Anthropic, via
  [`component-map-2026.md` §2.3](research/component-map-2026.md)). It is an
  offline cost, which is the only reason it is affordable here.
* **Late chunking is deliberately conditional.** The evidence grades it
  *contested* — efficient, but losing relevance to contextual retrieval, with the
  sign flipping by corpus ([`component-map-2026.md` §2.2](research/component-map-2026.md)).
  It is a fix for cross-reference breakage, not a general upgrade, which matches
  shipping it as a per-build flag rather than a mandate.

**What would change the decision.** Chunking has never been A/B'd on our gold
set — no `eval/` number defends the 512/1500 budget or the one-sentence overlap.
If chunking is ever revisited, re-run it *after* any local-reader upgrade: the
same literature that says compression benefit shrinks as the reader gets stronger
says the identical thing about chunking ([`component-map-2026.md` §2.4](research/component-map-2026.md)).

## 3. Embeddings

**What ships.** `microsoft/harrier-oss-v1-0.6b` (MIT, 1024-dim, 1.2 GB) is the
default (`rag_system/main.py::EXTERNAL_MODELS`), overridable with
`EMBEDDING_MODEL`. Queries get the `Instruct: {task}\nQuery: {text}` prefix;
**documents do not** (`representations.py::default_query_instruction` +
`apply_query_instruction`, resolved per-pipeline by
`retrieval_pipeline.py::_query_instruction`, which honours
`config["embedding_instruction"]` then `EMBEDDING_INSTRUCTION` then the model
family's default). The prefix applies only to families that were trained with one
(`qwen3-embedding`, `harrier`); everything else gets `""`.

Two index-format guarantees ship alongside it, both in
`rag_system/indexing/embedders.py` and `rag_system/retrieval/retrievers.py`:

* **Per-table embedder identity.** Every table records the model that wrote it
  plus a `normalized` flag in the Arrow schema metadata (with a JSON sidecar
  fallback). Indexing into or querying a table written by a different embedder
  raises `EmbedderMismatchError`.
* **L2 normalization** at write and query time, gated on that marker, so
  LanceDB's default L2 ordering *is* the cosine ordering both model cards
  specify. Legacy unmarked tables keep working unnormalized, with a warning. The
  default table name moved to `text_pages_v4` so the shipped default starts clean.

**Why.**

* **The prefix**: every top-2026 embedder is instruction-tuned, all use the same
  format, all specify no instruction on the document side, and Qwen3-Embedding
  reports 1–5% from the prefix alone
  ([`component-map-2026.md` §3.5](research/component-map-2026.md)). The audit
  found the repo was sending **no prefix at all**
  ([`embedder.md` §1](../eval/decisions/embedder.md)). Our measurement is more
  nuanced than the card: on harrier the prefix helps both metrics
  (mixed nDCG@10 0.908 → 0.915), on Qwen3-0.6B it is a ranking-vs-recall trade
  that washes out after a cross-encoder. Because only the query vector changes,
  turning it on invalidated no index.
* **The model**: not chosen from a leaderboard. Measured head-to-head on our gold
  set, harrier-0.6b beat the then-shipped 8 GB `Qwen3-Embedding-4B` on mixed
  nDCG@10 (**0.915 vs 0.875**), docs nDCG@10 (**0.759 vs 0.638**), recall@5 and
  recall@10, at ~3× lower latency and ~7× less memory
  ([`embedder.md`, gate validation](../eval/decisions/embedder.md)). MIT is also
  strictly more permissive than Apache-2.0.
* **The identity marker exists because the *previous* guard could not have caught
  this swap.** It compared vector *width* only, and harrier-0.6b and
  Qwen3-Embedding-0.6B are both 1024-dim — exactly the swap this adoption makes
  people likely to perform. It would have appended mutually unintelligible
  vectors to a live table, silently ([`DECISIONS.md` §2](../eval/DECISIONS.md)).
* **Normalization was adopted for card-conformance, not for a number, and the
  page says so.** Measured on `mixed`: nDCG@10 0.915 → **0.911**, recall@5 0.917 →
  0.931. A wash. It ships because the previous behaviour was neither cosine nor
  intended, not because it won.

**Post-adoption verification.** `mixed`, 317 chunks, 72 queries, first stage only:
recall@5 **0.944**, recall@10 0.958, recall@20 1.000, nDCG@10 **0.913**
([`DECISIONS.md` §4](../eval/DECISIONS.md)). Note that the `docs` and `mixed`
corpora index live `Documentation/*.md` — **editing this file moves these
numbers.** Only compare runs made against the same tree.

**What would change the decision.** `Qwen/Qwen3-Embedding-4B` stays a documented
option for multilingual or long-context (32K) corpora, which this English,
digital-born gold set does not exercise — keep the prefix on for it, worth +0.059
nDCG@10. Any embedder change forces a re-index **and** re-opens §6: the reranker's
value is a function of first-stage quality, and these two cannot be decided
independently.

## 4. Hybrid retrieval + RRF

**What ships.** `MultiVectorRetriever.retrieve` (`rag_system/retrieval/retrievers.py`)
runs LanceDB's native full-text leg and the dense leg **in parallel** and fuses
them with reciprocal rank fusion at `_RRF_K = 60`. `retrieval_mode` selects
`hybrid` (default), `vector_only` or `fts_only`; an unknown value falls back to
hybrid with a warning. Single-word FTS queries are rewritten to
`"<word>* OR <word>~"` for prefix and fuzzy matching. Rows are de-duplicated on
`chunk_id` (then `_rowid`, then text) across the two legs, and each mode exposes
exactly one "higher is better" `score` field.

**There are no fusion weights, and there is no knob to add them.**

**Why.** BM25 scores are unbounded positives while cosine is bounded, so naive
weighted addition is meaningless; RRF sidesteps normalization entirely by using
ranks ([`component-map-2026.md` §4.3](research/component-map-2026.md)). Hybrid+RRF
measurably beats both legs (T2-RAGBench: Recall@5 0.695 vs BM25 0.644 vs dense
0.587), and — the useful corrective — **BM25 alone beat dense alone there by 5.7
points**, so dropping the sparse leg is not the simplification it looks like.
Qdrant's own published guidance is the most honest vendor position available:
weighted RRF only when you have a tuned eval set with a train/val split, "neither
method dominates universally", and retune whenever retrievers, embeddings or
corpus change. §4.4's bottom line: "Do not spend time tuning fusion before you
have a reranker." This repo's history includes a *broken* weighted blend, which
RRF replaced ([`improvement_plan.md` §0](improvement_plan.md)).

**What would change the decision.** A per-index place to store tuned weights plus
a per-corpus validation split — that is [`improvement_plan.md`](improvement_plan.md)
§1.3, and it is correctly still open. The prerequisite is not the code; it is
having something to tune against that is not the same 72 queries used to evaluate.

## 5. Evidence-sufficiency retry

**What ships.** One conditional second retrieval
(`retrieval_pipeline.py::retrieve_candidates`), on in `default`
(`retrieval.retry: {enabled: True, min_top_score: 0.12, max_attempts: 1}`), off in
`fast`. When the first pass's evidence score falls below the threshold, the
enrichment model rewrites the query once (JSON-formatted so a small model's
"thinking" preamble cannot leak in), retrieval re-runs, and **the better of the
two result sets is kept** — a retry that did not improve the evidence is
discarded, not merged. It is inert in `fts_only` mode and on legacy unnormalized
tables, by design: no signal, no retry. The retry surfaces as a
`retrieval_retry` SSE event.

The signal is **not** the raw top similarity:

```
evidence = (cos_top − cos_background) / (1 − cos_background)
```

with `cos_background` the mean cosine from rank 6 down
(`_dense_evidence_score`). When a reranker returns a calibrated 0–1 probability,
its top score is preferred (`_rerank_evidence_score`); arbitrary logits are
rejected rather than compared to a probability threshold.

**Why.** The evidence says one conditional second iteration captures nearly all
of the deep-loop gain and that stopping criteria should be based on **cumulative
evidence sufficiency, not query count** — across six agents on BrowseComp-Plus,
search volume correlates only weakly with answer quality, and redundant queries
characterise *underperforming* agents ([`component-map-2026.md` §8.6](research/component-map-2026.md)).
The broader 2026 rule is "escalate, don't pre-decide"
([`component-map-2026.md` §6.5, §7.4](research/component-map-2026.md)).

Our own measurement changed the *signal*, which is the part worth recording. The
roadmap said to trigger on the top score. On the gold set, raw top cosine is
**anti-correlated with success**: all three `mixed` first-stage misses scored
*above* the median successful query, and any threshold catching all three fires
on 94% of the successes. Contrast works; absolute similarity mostly encodes how
close a query's phrasing sits to the corpus register
([`phase2-pipeline.md` §2.1](../eval/decisions/phase2-pipeline.md)).

Effect, four runs: fires on **9.7–11.1%** of `mixed`, **+0.008 to +0.017**
nDCG@10 and +0.014 recall@10, **zero per-query regressions**. It repaired one
genuine recall@10 = 0 miss. Mean latency over all queries 96 ms → 186 ms
([`phase2-pipeline.md` §2.3](../eval/decisions/phase2-pipeline.md)).

**Honest limits, carried forward rather than buried.** The threshold was
calibrated on the same 72 queries it is evaluated on, against three first-stage
misses. `0.12` is a starting value, not a tuned constant; it catches **one** of
the three real misses, because catching all three costs a 26% false-fire rate. On
`mixed` the +0.008 is *inside* the one-query noise floor — the reason to ship is
that it is positive on both corpora across four runs with nothing getting worse.

**What would change the decision.** A larger gold set with more first-stage
misses to calibrate against. The firing rate drifted 9.7% → 11.1% as the corpus
grew, so it is a property of the corpus, not a constant — re-check it after any
embedder change.

## 5a. Per-query token accounting

**What ships.** Every Ollama completion the agent makes — streaming or not —
reports `prompt_eval_count` and `eval_count` on its final object. Those are
aggregated per user query, bucketed by pipeline stage (`triage`,
`decomposition`, `synthesis`, `verification`), and returned as `token_usage` on
the `/chat` response body and in the SSE `complete` event. On by default: it
costs one dict update per LLM call and adds no request.

The aggregation point is a `ContextVar` in `rag_system/utils/ollama_client.py`
rather than an argument threaded through every call site, because one
`OllamaClient` is shared by the agent, the retrieval pipeline, the verifier and
the decomposer. `await` and `asyncio.to_thread` propagate it; the agent's
parallel sub-query `ThreadPoolExecutor` copies it explicitly.

Three honest gaps: the retry's reformulation call is billed to `synthesis`,
because it happens inside `RetrievalPipeline.run()` which the agent labels as
one stage; watsonx reports zeros, because the SDK path in use surfaces no
per-call counts; and an absent stage key means "no LLM call in that stage",
not "zero tokens". Evidence: `eval/decisions/phase4-escalation-tokens.md`.

## 5b. Metadata filters and ask-a-folder

**What ships.** `/chat` and `/chat/stream` accept an optional `filters` JSON
object (document id/name, chunk id, chunk-index ranges) compiled to LanceDB
where-clauses that prefilter **both** the vector and FTS legs. There is no
flag: with no `filters` argument the path is byte-identical to not having the
feature (md5-verified against a pre-change tree). Values containing quoting
characters are refused, never escaped; a malformed filter is a 400 from the one
validator in `rag_system/retrieval/filters.py`. Page and date filters are NOT
shipped — they live inside the metadata JSON string column and need real
columns plus a re-index.

`python -m rag_system.main ask <folder> "<question>"` builds an ephemeral index
under a temp directory (fast profile, no enrichment), answers with the standard
pipeline, and removes everything afterwards — including on SIGTERM.
Evidence: `eval/decisions/phase4-filters-askfolder.md`.

## 6. Reranking posture

**What ships.** Since arm G (2026-08-14) the `default` profile ships
`reranker.enabled = True` with threshold selection — `top_k: 10`,
`min_score: 0.5`, `min_keep: 3` (`rag_system/main.py::PIPELINE_CONFIGS`), and
the UI "AI reranker" toggle defaults on to match
(`src/components/ui/session-chat.tsx`). `Qwen/Qwen3-Reranker-4B` is loaded
lazily on the first reranked query through the in-repo
`QwenRerankerScorer` (`rag_system/rerankers/reranker.py`), routed either by
explicit `reranker.model_type: "qwen3"` or by model name
(`retrieval_pipeline.py::_get_ai_reranker`). Any other model still goes through
the `rerankers` library. A reranker that fails to load logs a warning and is
skipped — there is no fallback reranker.

**Why this is the most counter-intuitive decision in the repo.** The evidence is
about as strong as evidence gets: reranking is "the single highest-ROI component
in the stack", +17.2 pp MRR@3 on T2-RAGBench, −1.7 EM when removed from a local
7B ablation, and the 2026 recommendation is a flat "**YES, unconditionally**"
([`component-map-2026.md` §5.1, §6.5](research/component-map-2026.md)).

Our gold set says otherwise, for a specific and explicable reason. The joint
matrix ([`DECISIONS.md` §1](../eval/DECISIONS.md)), `mixed` corpus, same 20
first-stage candidates reordered by each:

| Stack | mixed nDCG@10 | docs nDCG@10 | added latency |
|---|---|---|---|
| **harrier-0.6b, first stage only** | 0.915 | 0.759 | — |
| + `BAAI/bge-reranker-v2-m3` | 0.892 | 0.701 | **+1.6 s — net negative** |
| + `Qwen/Qwen3-Reranker-4B` ← shipped since arm G (2026-08-14) | 0.977 | 0.932 | +12.7 s |

Two findings, both load-bearing:

1. **The cheap cross-encoder now hurts.** bge-reranker-v2-m3's famous +0.232 on
   `docs` was largely a *repair job on a weak first stage*. Improve the first
   stage and the repair becomes damage. This is exactly why 1.1 and 1.2 could not
   be decided independently and were re-measured jointly in one re-index window.
2. **The good reranker is a real win and was initially judged too slow to
   default on.** +0.062
   nDCG@10 on `mixed`, +0.173 on `docs` — the largest single quality win in this
   repo's eval history — for ~12.7 s per query and 7.5 GB of resident weights on
   a single-user, single-threaded server. Arm G (2026-08-14) reversed the "off by
   default" call: that call predated the synthesis context budget, when rank
   order barely mattered because front-truncation fed synthesis the tail of the
   list anyway. Now the budget keeps exactly the top-ranked documents, so
   ordering and selection decide everything the model reads — and `min_score`
   selection sends a small, clean context on easy questions instead of a fixed
   ten.

`Qwen/Qwen3-Reranker-0.6B` was rejected outright: +0.021 nDCG@10 over bge (one to
two queries out of 72, inside the noise band) bought with 1.5–2.8× the latency,
while *losing* recall@10 on both corpora ([`reranker.md` §7](../eval/decisions/reranker.md)).

**One integration finding worth keeping.** `rerankers` 0.10.0 has no
Qwen3-Reranker backend. Loading one through the shipped cross-encoder path builds
a `Qwen3ForSequenceClassification` with a **randomly initialised score head** —
had the batching not thrown, it would have returned untrained noise while
printing "AI reranker initialized successfully". `QwenRerankerScorer` implements
the model card's actual scheme (causal LM, left padding, chat template,
softmax over the `yes`/`no` logits at the final position), and the name-based
route in the loader exists specifically so no configuration can reach the random
head ([`reranker.md` §1](../eval/decisions/reranker.md)).

**What would change the decision.** Three concrete triggers:

* **Re-run the A/B if the embedder changes.** The reranker's headroom is largest
  exactly where the first stage is weakest, so this decision is a function of §3
  and expires with it.
* **Tune the latency knobs.** Batch size (8) and the 2048-token truncation cap
  are both untuned, and reranking only the top 10 candidates instead of 20 is
  unmeasured. Either could move the 12.7 s materially — that is unmeasured work,
  not a promise.
* **A later `rerankers` release** may add a Qwen3 backend, at which point the
  name-based route should be revisited.

Note also what no reranker fixed: `docs_d09` and `docs_d17` degrade under every
model tested. They are a query-understanding problem, and this section is not
where they get solved.

## 7. Query decomposition

**What ships.** Since arm H (2026-08-15) the `default` profile ships
`compose_from_sub_answers: false` with `pooled_first_stage: true`: each
sub-query runs first-stage retrieval, the candidates are pooled and
de-duplicated, and there is ONE rerank pass and ONE synthesis over the union
context (`retrieval_pipeline.py::_pooled_first_stage`). Sub-queries are also
used at the *rerank* stage, where each candidate is scored against every
sub-query and the scores are aggregated by
`query_decomposition.rerank_aggregate` (`"mean"` default, `"max"` available) —
`retrieval_pipeline.py::_rerank_stage`. The compose path — a separate *answer*
per sub-question, composed into the final answer — remains available behind
`compose_from_sub_answers: true` (`rag_system/agent/loop.py`).

Consequence, stated plainly: the earlier posture — **no shipped profile enables
sub-query scoring at rerank** — held while the `default` profile kept the
reranker off and `compose_from_sub_answers: true`. Arm G turned the reranker on
(2026-08-14), so the aggregation path now runs by default, and arm H
(2026-08-15) made the pooled first stage the default decomposition path.

**Why.** The evidence puts decomposition at "conditional — multi-hop, applied at
*rerank*", noting that decomposition at initial retrieval dilutes the query
semantically ([`component-map-2026.md` §6.2, §6.5](research/component-map-2026.md)).

We shipped the shape change and **measured the payload negative.** On `docs` with
`Qwen3-Reranker-4B`, only 6 of 24 queries decompose into more than one sub-query.
On exactly those 6, scoring against sub-queries at rerank is worse under both
aggregates: **0.8862 → 0.8406 (`max`, −0.046)** and **0.8740 (`mean`, −0.012)**.
The whole-corpus "gain" comes entirely from the 18 single-sub-query rows, where
the win is query *rewriting*, not decomposition
([`phase2-pipeline.md` §3](../eval/decisions/phase2-pipeline.md)).

The shape change ships anyway because it is a strict reduction in work — the
aggregate path used to issue N first-stage retrievals and now issues one — and
because the structural check passed: the first-stage number is byte-identical
across all three arms, proving decomposition can no longer touch it. (Arm H's
pooled first stage later re-introduced per-sub-query retrieval; the reduction it
keeps is one rerank pass and one synthesis instead of N.)

**What would change the decision.** n_effective = **6 queries on one corpus**.
That is too small to call the 2026 MultiConIR/SSRB finding wrong; it is big
enough to say it did not reproduce here, which is why nothing was switched on. A
multi-hop corpus with enough genuinely-decomposing queries to move a metric would
re-open it. `mean` stays the default aggregate on the "less bad" argument, not a
positive one.

## 8. Routing / triage

**What ships — two layers, exactly one of which calls an LLM.**

*Layer 1, the gateway* (`backend/server.py::should_use_rag`) is deterministic and
makes no network call: `force_rag` → RAG; no linked indexes → direct LLM;
whole-message smalltalk (≤ 6 words, anchored allowlist, must contain a core
phrase) or assistant-meta ("who are you", "what model are you") → direct LLM;
**everything else → RAG.** Unit-tested at 155/155 in
`backend/test_gateway_routing.py`.

*Layer 2, the agent* (`rag_system/agent/loop.py::_triage_query_async`) is the
system's single LLM routing layer: document overviews + the utility model decide
`rag_query` vs `direct_answer`, with "history exists → `rag_query`" as a shortcut
and an LLM fallback when no overviews are loaded. `_normalize_triage` collapses
anything that is not an explicit `direct_answer` to `rag_query`, so a small model
still emitting the retired `graph_query` label lands on the RAG path.

**Why.** Pre-retrieval LLM routing is the weakest measured pattern of 2026, and
three independent sources agree: four ML approaches to pre-retrieval routing all
failed because "the need for augmentation cannot be determined from the query
alone"; rule-based retriever routing *lost* to fixed hybrid by 1.8 EM; and
TF-IDF+SVM matches or beats neural and LLM routers at ~zero cost
([`component-map-2026.md` §7.1, §7.3, §7.4](research/component-map-2026.md)). The
four-year pattern in §7.3 is "a small discriminative classifier is the right
tool; an LLM router is rarely justified."

The bias toward over-sending to RAG is deliberate and cheap: agent triage runs on
every forwarded request and can still answer directly, so a false "use RAG" costs
one call on a model that would have been called anyway, while a false "answer
directly" costs an unanswerable question. The gateway is a smalltalk filter in
front of the decision-maker, not the decision-maker.

**Measured** ([`phase2-gateway.md` §3](../eval/decisions/phase2-gateway.md)): mean
routing decision **750.613 ms → 0.002 ms** over the same 20 messages, with
**20/20 decision agreement** with the LLM router it replaced. The deleted
keyword fallback was worse than the old docs claimed — it matched greetings by
*substring*, so `'hi'` matched *w**hi**ch*, *t**hi**s* and *mac**hi**ne*, routing
**7 of 8** real Atlas-7 questions to the direct LLM. That is why it was deleted
rather than patched, and why "messages containing test/check route RAG" is now a
regression test.

**What would change the decision.** [`improvement_plan.md`](improvement_plan.md)
§2.1 (embed and cache document overviews for a cosine pre-check) and §2.2
(session-level routing memo) both still stand for the *agent* layer, which is now
the only per-query LLM routing call. The evidence's own carve-out is that routing
still pays for *pipeline depth* selection, implemented as a post-retrieval
cascade — which is what §5's retry is.

## 9. Verification

**What ships.** Verification is on in `default` (`verification: {enabled: True}`).
The shipped backend is an LLM prompt on the utility model
(`rag_system/agent/verifier.py::Verifier.verify_async`), returning a JSON verdict
and a confidence score; a low-confidence or ungrounded verdict appends a warning
to the answer (`agent/loop.py`).

A **seam** exists for a local model: `VERIFIER_MODEL` / `verification.model`
swaps in `LocalNLIVerifier`, which sentence-splits the answer, scores each
sentence against the retrieved evidence as premise, and takes the **minimum** —
one unsupported sentence makes the answer ungrounded, matching the binary
semantics `eval/judge.py` already uses. A model that cannot be loaded **raises**,
printing the availability table, rather than falling back: a verifier that
silently is not the verifier you configured is worse than an error. **The default
is unchanged.**

**Why.** Verification helps as an *external* check, and the 2026 result is that a
4-bit 1B verifier (ThinknCheck, 78.1 BAcc) now beats the 7B 2024 SOTA, which
would make per-answer grounding cheap enough to always run
([`component-map-2026.md` §9.2](research/component-map-2026.md)).

Both of the roadmap's named candidates failed availability checks against the
HuggingFace Hub API: **ThinknCheck has no public weights** (zero models returned;
the paper links no release), and Granite Guardian is either 8B/~16 GB or — in its
38M form — a hate/abuse classifier, the wrong task entirely. Two substitutes were
wired and exercised rather than left as a stub, and both were run against all 20
hand-labelled cases in `eval/judge_validation.jsonl`
([`phase2-pipeline.md` §4](../eval/decisions/phase2-pipeline.md)):

| Verifier | agreement | TPR | TNR |
|---|---|---|---|
| `lytang/MiniCheck-DeBERTa-v3-Large` (1.74 GB) | 19/20 | 10/10 | 9/10 |
| `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` (369 MB) | 18/20 | 8/10 | 10/10 |

They fail in **opposite directions** — MiniCheck missed a swapped-entity error,
the generic NLI model rejected two correct answers — so neither is a drop-in
improvement without its own validation run. Nothing was made the default.

**`[Confidence: N%]` is UX, not a calibrated measurement**, and
[`verifier.md`](verifier.md) says so in a callout. Changing the backend changes
where the number comes from; it does not calibrate it. The evidence supports the
caution independently: faithfulness metrics measure precision and ignore
coverage, so a system gated only on "is every claim supported" learns to abstain
([`component-map-2026.md` §9.4](research/component-map-2026.md)).

**What would change the decision.** Run `eval/judge.py --validate`-style
TPR/TNR discipline on a candidate verifier over more than 20 cases, and beat the
LLM prompt on both directions rather than one. If ThinknCheck ever publishes
weights, it is the first thing to try. Also worth knowing: verifying against the
retrieved chunk alone systematically *under*-detects, because supporting evidence
is frequently outside the truncated passage
([`component-map-2026.md` §9.3](research/component-map-2026.md)).

## 10. Sentence pruning

**What ships.** Provence (`naver/provence-reranker-debertav3-v1`) via
`rag_system/rerankers/sentence_pruner.py`, applied after context expansion and
before synthesis (`retrieval_pipeline.py::run`), with fully-pruned chunks
dropped. It is **opt-in**: `provence.enabled` defaults to `False`, exposed as the
UI toggle "Prune irrelevant sentences" and as `provence_prune` on the API. The
model is loaded lazily, once, behind a lock, and a load failure skips pruning
rather than failing the query.

**Why.** Provence is the one context-reduction variant whose economics survive
2026 scrutiny: it formulates pruning as sequence labelling **unified with
reranking**, so it is folded into a stage you already run, at "negligible to no
drop in performance… at almost no cost"
([`component-map-2026.md` §10.1](research/component-map-2026.md)). §10.5's verdict
is exactly the posture here: "**prune, don't compress.**"

It was kept off by default because localGPT did *not* run a cross-encoder by
default at the time (§6), so Provence's "zero marginal cost" argument — the
entire reason it beats token-level compression — did not hold in the shipped
configuration: an extra DeBERTa forward pass, not a free rider. Since arm G
(2026-08-14) the reranker **is** on by default, so that premise no longer holds;
folding pruning into the rerank stage is now a live candidate, but no `eval/`
number yet measures its effect on our gold set.

**What would change the decision.** Measure it. Reranking returned to the
default profile in arm G (2026-08-14), so re-evaluate folding pruning in with
it. `OpenProvence`
(30M–310M, MIT, ModernBERT) is the lighter candidate the evidence points at.

## 11. Caching and memory

**What ships.** An in-process semantic cache in the agent
(`rag_system/agent/loop.py`): the raw query is embedded, compared by cosine
against cached entries, and a hit at or above `semantic_cache_threshold`
(**0.98**) returns the stored result. `cache_scope` defaults to **`"session"`**,
so an entry from another session is skipped before the similarity is even
computed. Conversation memory is a session transcript — an in-process
`chat_histories` map plus SQLite rows in `backend/chat_data.db` — formatted into
the query as history. There is **no vendor memory layer**, no entity graph, no
consolidation pass.

**Why.**

* **The null baseline wins.** The 2026 guidance is explicit: "plain RAG over a
  session transcript store as the baseline you must beat" — Cloud-RAG beat Mem0
  on LongMemEval-S, a filesystem agent beat Mem0 on LoCoMo, and the benchmark the
  whole category is scored on (LOCOMO) is broadly discredited on structural
  grounds ([`component-map-2026.md` §11.1, §11.4](research/component-map-2026.md)).
  Our session store *is* that null baseline. The same section notes a bare
  embedding-model swap moves accuracy 6.2 pp — **larger than most claimed
  memory-system deltas** — which is where the effort went instead (§3).
* **Session scoping is a privacy fix, not a performance one.** A global cache
  returns one session's answer to another session's user.
  [`improvement_plan.md` §0](improvement_plan.md) records it as closing a
  cross-session answer leak.
* **The 0.98 threshold is deliberately extreme.** Semantic caches are attackable
  by embedding-similarity collision (86% hijacking hit rate), and the consistent
  2026 theme is that hit rate is the wrong headline metric — calibration,
  freshness and collision-resistance decide whether a cache is *safe*
  ([`component-map-2026.md` §11.3](research/component-map-2026.md)). At 0.98
  nothing but a near-identical query hits, which was also verified as a side
  effect of the prefix work: with the query prefix on, mean pairwise cosine over
  the 72 gold queries *falls* (0.361 → 0.216) and max reaches 0.899, so nothing
  crosses the bar either way ([`embedder.md` §8](../eval/decisions/embedder.md)).

**What would change the decision.** Instrument the cache for calibration and
staleness rather than hit rate before loosening the threshold. Graphiti +
FalkorDB Lite is the evidence's pick *if* temporal/entity memory with provenance
is ever actually needed — but the null baseline has to lose on our own eval first,
and there is no memory eval in `eval/` today.

## 12. Streaming and persistence

**What ships.** `POST :8001/chat/stream` emits SSE
(`rag_system/api_server.py::handle_chat_stream`) and every pipeline phase is a
typed event: `retrieval_started`, `retrieval_done`, `retrieval_retry`,
`rerank_started`, `rerank_done`, `context_expand_*`, `prune_*`, `token`,
`sub_query_token`, `complete`. Synthesis streams token-by-token from
`_synthesize_final_answer`. A client disconnect (`BrokenPipeError`) is logged,
not raised.

**The stream writes nothing to SQLite.** When the stream finishes, the UI posts
the completed turn to `POST :8000/sessions/{id}/messages/save`
(`src/lib/api.ts::saveStreamedTurn`, `backend/server.py`), which persists the
user message, the assistant message, the source documents and the pipeline
cascade in `metadata.steps`. Direct stream consumers must do the same to get
history — documented as known limitation #1 in
[`system_overview.md`](system_overview.md#11-known-limitations).

**Why.** This is an architecture choice, not an evidence-backed one, and it is
recorded here so it is not mistaken for the latter. The rationale is
single-writer discipline: the RAG API owns retrieval and the vector store, the
gateway owns SQLite, and no request writes to a store it does not own. The cost
is the extra round-trip and the fact that an interrupted stream persists nothing.

The one thing the evidence *does* motivate is that the retry and the rerank are
surfaced as SSE steps rather than hidden — the 2026 escalation-architecture
literature treats visible, staged escalation as the point of the design
([`component-map-2026.md` §8.6](research/component-map-2026.md), PEA-CAE).

**Verified by** `eval/smoke_e2e.py`: **25/25 assertions**, including planted-fact
answers with non-empty `source_documents`, the `[Confidence: N%]` tag, and the
streamed-turn save/round-trip ([`phase2-gateway.md` §3.4](../eval/decisions/phase2-gateway.md)).
Note the honest gap recorded there: smoke sends `force_rag: True`, so it
exercises the gateway's `force_rag` branch, not the discriminative one.

**What would change the decision.** Making the stream itself durable would need
the RAG API to write session state, which crosses the ownership boundary. The
lower-cost fix is a server-side save on stream completion in the gateway's
proxy path.

---

## 13. Deliberately not implemented

Each of these is **absent on purpose**. If you are about to add one, read the row
first — and bring a gold-set number, not a paper.

| Not implemented | Why (evidence) | Revisit when |
|---|---|---|
| **HyDE** — in any form. `grep -i hyde` over `rag_system/`, `backend/`, `src/` and `eval/` returns **zero hits**; there is no flag, no dead code, no "coming soon". | HyDE underperforms plain dense retrieval on entity/numeric corpora (T2-RAGBench: Recall@5 0.544 vs 0.587, nDCG@10 0.433 vs 0.466) because pseudo-documents fabricate figures; and in a production system only **27.8%** of real queries needed LLM augmentation while synthetic evals implied >90% — "the Coverage Illusion". Verdict: "alive but no longer a default. **Never always-on.**" ([`component-map-2026.md` §6.1](research/component-map-2026.md)) | Only as a *post-retrieval* escalation (retrieval returned nothing → then HyDE), or moved to index time (HyPE). The escalation slot in this codebase is already occupied by §5's retry; a HyDE variant would have to beat it on the gold set. |
| **Multi-query expansion** | The weakest of the three query transformations: on T2-RAGBench multi-query scored Recall@5 0.640 — **worse than plain BM25** (0.644). Prompt-only LLM rewriting measured **−9.0% nDCG@10 (p<0.001)** on FiQA, and an attempt to *gate* the rewriter reached only AUC 0.593 ([`component-map-2026.md` §6.3](research/component-map-2026.md)). | A trained lexical expander with BM25-level cost (the STORM line) that wins on our gold set. Note §5's retry is a *single conditional* rewrite kept only when it scores better — that is the sanctioned shape. |
| **Weighted / tunable fusion knobs** | BM25 and cosine are not on a common scale; "a fixed alpha over raw scores tends to be dominated by whichever retriever has larger raw magnitudes", and no 2026 evidence shows learned fusion beating RRF in the general case ([`component-map-2026.md` §4.3, §4.4](research/component-map-2026.md)). RRF is the scale-free safe default. | A per-index validation split exists to tune against and a place to store per-corpus weights — [`improvement_plan.md`](improvement_plan.md) §1.3. Tuning fusion before a reranker is explicitly the wrong order. |
| **GraphRAG** — removed 2026-08-09 | Loses on single-hop retrieval (64.78 vs 63.01 F1 on NQ); multi-hop gains span +3 to +27 depending entirely on how well the vector baseline is tuned; costs **41–57× at indexing** and up to **~377× in query tokens** ([`academic-evidence-2026.md` §6](research/academic-evidence-2026.md)). It was also unreachable — no shipped profile ever set `graph_strategy`. | A corpus where entity-linking is the task and a tuned vector baseline demonstrably fails on our gold set. Deleted: `graph_extractor.py`, `GraphRetriever`, `GraphQueryTranslator`, the `graph_query` triage outcome, and `networkx`/`fuzzywuzzy`/`python-Levenshtein` ([`phase2-pipeline.md` §1](../eval/decisions/phase2-pipeline.md)). |
| **Vendor memory systems** (Mem0, Zep/Graphiti, Letta, LangMem, GPTCache) | "Plain RAG over a session transcript store [is] the baseline you must beat" — Cloud-RAG beat Mem0 on LongMemEval-S, a filesystem agent beat it on LoCoMo, and LOCOMO itself is discredited (conversations of 16k–26k tokens do not stress memory). GPTCache is dormant since Aug 2024 ([`component-map-2026.md` §11.1, §11.2, §11.4](research/component-map-2026.md)). | The null baseline (§11) loses on a memory eval we actually own. Fix the embedding model before comparing memory systems — a bare swap moves accuracy 6.2 pp, more than most claimed memory deltas. |
| **Deep subagent / parallel fan-out loops** | The pro case (Anthropic's orchestrator-worker, +90.2%) runs at **~15× chat tokens** on frontier models. The 2026 counter-evidence: on repository-level code QA, plain semantic search scored **65.2%** vs deep agentic search **46.2% at >2× cost**, with **41.8% of failures at the planner→subagent hand-off** — "usually silent, ending in a fluent and confident answer that was wrong" ([`component-map-2026.md` §8.6](research/component-map-2026.md)). | Never, for a read-only indexable corpus on a single-user local box. The bounded version of this idea is already shipped: one conditional retry (§5), capped at `max_attempts: 1`. |
| **RL-trained searchers** (Search-R1 lineage) | On BrowseComp-Plus — a fixed, human-verified corpus that disentangles retriever from agent — **Search-R1 + BM25 scores 3.86%** while GPT-5 + BM25 scores 55.9% and GPT-5 + Qwen3-Embedding-8B scores 70.1%, with *fewer* search calls (ACL 2026 Main, [`component-map-2026.md` §8.3](research/component-map-2026.md)). No out-of-distribution transfer; also mis-calibrates confidence, which would corrupt §5's evidence signal. | A local-training story with demonstrated OOD transfer. §8.2 documents the 2026 lineage if that changes; nothing in it is currently a better use of a laptop GPU than a better embedder. |
| **Token-level context compression** (LLMLingua-style) | Across thousands of runs on 30,000 queries and three GPU classes: **at most ~18% end-to-end speedup**, and only when prompt length, ratio and hardware align — outside that window "compression overhead dominates and cancels the decoding gains entirely" (ECIR 2026). Fixed ratios reversed **31%** of pairwise model rankings on LongMemEval-S and obscured **80%** of the gain from a reader upgrade; hard compressors leave the answer path incomplete in 34–60% of multi-hop bridge examples. Upstream quiet since Dec 2024 ([`component-map-2026.md` §10.2–10.5](research/component-map-2026.md)). | Not in this shape. The surviving alternative — extractive sentence pruning — is already shipped as §10. If compression is ever profiled here, do it per model/hardware pair and re-run the ablation on every reader upgrade. |

### 13a. Implemented but measured out of the defaults (Phase 4, 2026-08-09)

These three shipped as flag-gated code, were benchmarked on/off, and stay OFF on
the numbers. The flags exist so the A/Bs can be re-run; the code is not dead, it
is *disabled by measurement*.

| Flag (both profiles) | Verdict | The number that decided it |
|---|---|---|
| `retrieval.document_escalation` (4.1) | **REJECTED as default** (was HOLD) | The 2026-08-09 lift (0/7→2/7 judged on fired queries) was the truncation bug, not document reassembly: the escalated block was appended at the tail and *survived* front-truncation while top-ranked chunks were discarded. The HOLD's condition — re-run after the context-window fix — was executed 2026-08-12 (`eval/decisions/phase4-escalation-rerun.md`): with prompts that fit (249 calls, zero truncation), the escalation-OFF baseline on the identical fire subset went 0/9 → 7/9 and escalation added nothing on top (5/7 → 2/7 mechanically, 5/7 → 5/7 by hand adjudication); both fires under the true product default were regressions on both dates. Flag and code kept — unfalsified for a corpus where document ordering matters within a fitting prompt; this one never presents that case. |
| `retrieval.crossref_hop` (4.2) | **REJECTED as a default** | Fires 0/11 at the shipped k=20; where forced (k=5), 0/11 hopped chunks hit an expected source — target selection is query-blind and lands on hub documents (21/24 hops to 2 documents) — and raising `k` beats the hop at equal context budget in 3 of 4 cells. Index-time `extract_crossrefs` stays ON (free, regex-only, bit-identical text/vectors). `eval/decisions/phase4-retrieval-benchmarks.md`, `phase4-answer-quality.md`. |
| `retrieval.overview_prefilter` (4.3) | **boost: HOLD · restrict: REJECTED** | boost: +0.106 nDCG@10 on the heterogeneous acquisition slice, −0.021 on `mixed` — a per-index opt-in candidate, not a default. restrict: removed the answer document entirely (recall@20 1→0) for 4 queries per corpus. `eval/decisions/phase4-retrieval-benchmarks.md`. |

## 14. How to re-litigate a decision

None of the above is permanent. The process for changing one:

1. **Reproduce the recorded number first.** The `docs` and `mixed` corpora index
   live `Documentation/*.md` content, so **editing documentation moves the
   metric.** Snapshot the pre-change tree and record the chunk count; only
   compare runs made against the same tree.

   ```bash
   .venv/bin/python eval/run_eval.py --corpus all \
     --json-out eval/results/before.json
   ```

2. **Run your change against the same corpus snapshot**, and run it twice if any
   arm makes an LLM call (the retry does), so the spread between runs is visible
   rather than mistaken for signal.

   ```bash
   .venv/bin/python eval/run_eval.py --corpus all \
     --json-out eval/results/after.json
   .venv/bin/python eval/smoke_e2e.py          # 25/25 assertions, exit 0
   ```

3. **Beat the recorded number by more than the noise floor.** On `mixed`, one
   query ≈ **0.014 nDCG@10**. Deltas under that are not findings. A delta that is
   positive on both corpora across multiple runs with zero per-query regressions
   is (that is the standard §5 was held to). Public-leaderboard deltas under ~2
   points are known not to transfer
   ([`component-map-2026.md` §3.6](research/component-map-2026.md)).

4. **Write it down.** Add the outcome and the numbers to
   [`eval/DECISIONS.md`](../eval/DECISIONS.md), with a page under
   [`eval/decisions/`](../eval/decisions/) if the investigation is substantial.
   Record the *rejections* too — the negative results in
   [`reranker.md`](../eval/decisions/reranker.md) and
   [`phase2-pipeline.md`](../eval/decisions/phase2-pipeline.md) are the most
   valuable pages in that directory.

5. **Land code, docs and the eval delta in the same change**, and update the
   relevant section here. Nothing in this file may describe unshipped behaviour —
   that is what [`research_roadmap.md`](research_roadmap.md) is for.

Two standing rules from the harness itself: never quietly repair a gold row to
make your own change look better (`docs_d10` was left failing and reported until
the gate re-anchored it, on the record), and never quote a leaderboard position in
this repo's docs — cite our own eval or nothing.

---

### Documentation Docker Usage (`Documentation/docker_usage.md`)

# 🐳 Docker Usage Guide - LocalGPT

_Last updated: 2026-08-08_

Practical Docker commands and procedures for running LocalGPT in containers.

---

## 📋 Prerequisites

### Required Setup
- Docker Desktop (or Docker Engine 24+ with the Compose plugin) running
- Ollama, either on the host (default) or as a container
- 8GB+ RAM available

### Architecture Overview
```
┌─────────────────────────────────────────────────┐
│                Docker Containers                │
├─────────────────────────────────────────────────┤
│ frontend (3000)  →  backend (8000)  →  rag-api  │
│                                         (8001)  │
└─────────────────────────────────────────────────┘
            │                               │
            │  browser streams              │
            │  directly to :8001            ▼
            └──────────────────►  ┌────────────────────┐
                                  │ Ollama (11434)     │
                                  │ host.docker.internal│
                                  └────────────────────┘
```

`backend` and `rag-api` share `./backend` (SQLite), `./lancedb`, `./index_store`
and `./shared_uploads` as bind mounts.

---

## 1. Quick Start Commands

### Step 1: Clone

```bash
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Verify Docker is running
docker version
```

### Step 2: Ollama

The compose files point the containers at `host.docker.internal:11434` by default,
and declare `extra_hosts: ["host.docker.internal:host-gateway"]` so that also
resolves on Linux.

```bash
# Install Ollama on the host
curl -fsSL https://ollama.ai/install.sh | sh

# Start Ollama (in one terminal)
ollama serve

# Install the models (in another terminal)
ollama pull qwen3.5:9b      # answer generation
ollama pull qwen3.5:4b      # routing, triage, enrichment, verification

ollama list
curl http://localhost:11434/api/tags
```

**Or run Ollama in a container.** `docker-compose.yml` defines an `ollama` service
behind the `with-ollama` profile:

```bash
./start-docker.sh container
# equivalently:
#   OLLAMA_HOST=http://ollama:11434 \
#   docker compose --env-file docker.env --profile with-ollama up --build -d

# Models must be pulled inside the container the first time
docker compose --profile with-ollama exec ollama ollama pull qwen3.5:9b
docker compose --profile with-ollama exec ollama ollama pull qwen3.5:4b
```

The embedding model (`microsoft/harrier-oss-v1-0.6b`) is a HuggingFace download
inside `rag-api`, not an Ollama model — nothing to pull for it. The reranker
(`Qwen/Qwen3-Reranker-4B`, ~7.5 GB) loads lazily: reranking is on by default, so
it is downloaded on the first reranked query, not at startup.

### Step 3: Start Containers

```bash
./start-docker.sh              # local Ollama
./start-docker.sh container    # containerized Ollama
./start-docker.sh stop
./start-docker.sh logs
./start-docker.sh status
./start-docker.sh help
```

`./start-docker.sh` with no argument probes port 11434. If nothing is listening it
offers the containerized fallback; pass `-y`/`--yes` (or set `NONINTERACTIVE=1`) to
take it without a prompt, which is what CI should do. With no TTY and no `-y` it
exits 1 with instructions instead of hanging.

### 1.2 Service Access

- **Frontend**: http://localhost:3000
- **Backend API**: http://localhost:8000
- **RAG API**: http://localhost:8001
- **Ollama**: http://localhost:11434

### 1.3 Startup Order

```
rag-api (healthy)  →  backend (healthy)  →  frontend
```

`rag-api` loads the embedding model before `/health` answers (the reranker loads
lazily, on the first reranked query), so its
check has a 120s start period and `backend` intentionally waits in `created` until
then. `docker compose logs -f rag-api` shows the progress.

---

## 2. Container Management

### 2.1 Using the Convenience Script

```bash
./start-docker.sh
./start-docker.sh stop
./start-docker.sh logs
./start-docker.sh status

# Restart
./start-docker.sh stop && ./start-docker.sh
```

### 2.2 Manual Docker Compose Commands

```bash
# Start all services
docker compose --env-file docker.env up --build -d

# Check status
docker compose ps

# View logs
docker compose logs -f

# Stop all services
docker compose down

# Force rebuild
docker compose build --no-cache
docker compose --env-file docker.env up --build -d
```

Always pass `--env-file docker.env` (as `start-docker.sh` does). Without it the
compose defaults still apply — they mirror `docker.env` — but any value you edit in
`docker.env` is ignored.

### 2.3 Individual Service Management

```bash
docker compose up -d frontend
docker compose up -d backend
docker compose up -d rag-api

docker compose restart rag-api
docker compose stop backend
docker compose logs -f rag-api
```

---

## 3. Development Workflow

### 3.1 Code Changes

No source is bind-mounted — code is `COPY`-ed into the images at build time, so a
restart alone will not pick up an edit. Rebuild the affected service:

```bash
docker compose up -d --build frontend
docker compose up -d --build backend
docker compose up -d --build rag-api

# After a dependency change
docker compose build --no-cache rag-api
docker compose up -d rag-api
```

Editing `NEXT_PUBLIC_API_URL` or `NEXT_PUBLIC_RAG_API_URL` also needs a frontend
**rebuild** — Next.js inlines those at `next build` time and they are passed to
`Dockerfile.frontend` as build args.

### 3.2 Debugging Containers

```bash
# Shells
docker compose exec frontend sh      # node:20-alpine
docker compose exec backend bash     # python:3.11-slim
docker compose exec rag-api bash     # python:3.11-slim

# Run commands in a container
docker compose exec rag-api python -c "from rag_system.factory import get_agent; get_agent('default'); print('✅ RAG System OK')"
docker compose exec backend curl -s http://localhost:8000/health

# Environment
docker compose exec rag-api env | grep -E "OLLAMA|MODEL|DB_PATH|LANCEDB"
```

### 3.3 Compose File Variants

| File | Contents |
|------|----------|
| `docker-compose.yml` | `rag-api`, `backend`, `frontend`, plus an optional `ollama` service behind the `with-ollama` profile |

This is the only compose file (the old `docker-compose.local-ollama.yml` variant
was removed; `--profile with-ollama` covers that flow). There is no
`docker-compose.dev.yml`.

---

## 4. Logging & Monitoring

### 4.1 Log Management

```bash
docker compose logs
docker compose logs frontend
docker compose logs backend
docker compose logs rag-api

docker compose logs -f
docker compose logs --tail=100
docker compose logs -t
docker compose logs > system.log 2>&1
docker compose logs --since=2h
```

### 4.2 System Monitoring

```bash
docker stats
docker stats rag-frontend rag-backend rag-api

docker compose ps
docker inspect rag-api --format='{{.State.Health.Status}}'

docker system info
docker system df
```

Container names are fixed by `container_name`: `rag-frontend`, `rag-backend`,
`rag-api`, and `rag-ollama` for the optional Ollama service.

---

## 5. Ollama Integration

### 5.1 Host Ollama

```bash
curl -fsSL https://ollama.ai/install.sh | sh
ollama serve
curl http://localhost:11434/api/tags

ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
ollama list
```

### 5.2 From Inside a Container

```bash
docker compose exec rag-api curl -s http://host.docker.internal:11434/api/tags

curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.5:4b", "prompt": "Hello", "stream": false}'
```

Ollama logs appear in the terminal running `ollama serve`; for the containerized
variant use `docker compose --profile with-ollama logs -f ollama`.

### 5.3 Model Management

```bash
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
ollama pull qwen3.6:27b     # optional high-end generation model

ollama rm old-model-name
ollama show qwen3.5:9b
```

Point the containers at a different model without editing code:

```bash
GENERATION_MODEL=qwen3.6:27b docker compose --env-file docker.env up -d rag-api backend
```

---

## 6. Data Management

### 6.1 Volumes and Mounts

Every application path is a **bind mount to a host directory**, so ordinary file
tools work. The only named volume is `ollama_data`, used by the optional Ollama
container.

| Host path | Container path | Contents |
|-----------|----------------|----------|
| `./lancedb` | `/app/lancedb` | Vectors and the native full-text index |
| `./index_store` | `/app/index_store` | Document overviews |
| `./shared_uploads` | `/app/shared_uploads` | Uploaded source documents |
| `./backend` | `/app/backend` | `chat_data.db` (shared by backend and rag-api) |

```bash
docker volume ls
docker system df -v

# Back up the host directories directly
tar czf backup/lancedb_backup.tar.gz lancedb/
tar czf backup/index_store_backup.tar.gz index_store/

# Only the containerized Ollama uses a named volume.
# The compose project name is the directory name, so it is localgpt_ollama_data.
docker run --rm -v localgpt_ollama_data:/data -v $(pwd)/backup:/backup \
  alpine tar czf /backup/ollama_models.tar.gz -C /data .

docker volume prune
```

### 6.2 Database Management

```bash
# sqlite3 is installed in both Python images
docker compose exec backend sqlite3 /app/backend/chat_data.db ".tables"

# Back up the database
cp backend/chat_data.db backup/chat_data_$(date +%Y%m%d).db

# Check LanceDB tables from the container
docker compose exec rag-api python -c "
import lancedb
db = lancedb.connect('/app/lancedb')
print('Tables:', db.table_names())
"
```

`backend` and `rag-api` both set `DB_PATH=/app/backend/chat_data.db` and mount the
same host directory, so they read and write one file.

### 6.3 File Management

```bash
docker compose exec rag-api ls -la /app/shared_uploads

docker cp local_file.pdf rag-api:/app/shared_uploads/
docker cp rag-api:/app/shared_uploads/file.pdf ./local_file.pdf

docker compose exec rag-api df -h
```

Because `shared_uploads/` is a bind mount, copying a file into `./shared_uploads`
on the host is equivalent and simpler.

---

## 7. Troubleshooting

### 7.1 Common Issues

#### Container Won't Start
```bash
docker version
lsof -i :3000 -i :8000 -i :8001
docker compose logs [service-name]
```

#### `backend` stays in `created`
That is `depends_on: rag-api: condition: service_healthy` doing its job. Watch
`docker compose logs -f rag-api` — on a cold start it is downloading and loading
the embedding model (the reranker loads lazily later, on the first reranked
query).

#### Ollama Connection Issues
```bash
curl http://localhost:11434/api/tags

pkill ollama
ollama serve

docker compose exec rag-api curl -s http://host.docker.internal:11434/api/tags
```

#### Chats return "Could not connect to the RAG API server"
The backend builds its URLs from `RAG_API_URL`, which must be
`http://rag-api:8001` inside compose (`localhost` there means the backend
container itself).
```bash
docker compose exec backend env | grep RAG_API_URL
docker compose exec backend curl -s http://rag-api:8001/health
```

#### Performance Issues
```bash
docker stats
docker compose ps

# Docker Desktop → Settings → Resources → Memory → 8GB+
```

### 7.2 Reset and Clean

```bash
# Stop everything
./start-docker.sh stop

# Clean containers and images
docker system prune -a

# Complete reset (⚠️ deletes indexes, uploads and chat history)
docker compose down
rm -rf lancedb/* index_store/* shared_uploads/* backend/chat_data.db
docker system prune -a
```

`docker compose down -v` only removes the named `ollama_data` volume — application
data lives in host directories and must be deleted explicitly.

### 7.3 Health Checks

```bash
curl -f http://localhost:3000 && echo "✅ Frontend OK"
curl -f http://localhost:8000/health && echo "✅ Backend OK"
curl -f http://localhost:8001/health && echo "✅ RAG API OK"
curl -f http://localhost:11434/api/tags && echo "✅ Ollama OK"

docker compose ps

# Test model loading inside the container
docker compose exec rag-api python -c "
from rag_system.factory import get_agent
agent = get_agent('default')
print('✅ RAG System initialized successfully')
"
```

These are the same endpoints the container health checks use: `curl -f /health`
for `backend` and `rag-api`, and busybox `wget -qO- http://localhost:3000` for
`frontend` (the alpine image has no curl).

---

## 8. Advanced Usage

### 8.1 Production Deployment

```bash
# docker.env already sets NODE_ENV=production
docker compose --env-file docker.env up --build -d

# All services already declare restart: unless-stopped
docker compose ps
```

There is no authentication and CORS is wide open on both APIs. Put a reverse proxy
in front and publish only what you need. Note the browser streams directly from
port 8001, so that port must be reachable by clients (or proxied) unless you turn
off "Stream phases" in the chat UI.

### 8.2 Scaling

`docker compose up -d --scale backend=2 --scale rag-api=2` **does not work with the
shipped compose file**: each service sets a fixed `container_name`
(`rag-backend`, `rag-api`) and publishes a fixed host port, both of which conflict
on the second replica. Remove `container_name` and the `ports:` mappings (or switch
to a random host port) first.

Even then, the RAG API is single-threaded and holds per-process state — the
resident agent, its in-memory chat history and the semantic cache — so replicas
need sticky sessions at minimum.

### 8.3 Security

```bash
docker scout cves rag-frontend
docker scout cves rag-backend
docker scout cves rag-api

docker compose build --no-cache --pull
```

---

## 9. Configuration

### 9.1 Environment Variables

`docker.env` is passed with `--env-file` and supplies both runtime environment and
compose-level substitution:

```bash
# Ollama on the host; extra_hosts makes this resolve on Linux too
OLLAMA_HOST=http://host.docker.internal:11434
# Containerized alternative: OLLAMA_HOST=http://ollama:11434

NODE_ENV=production
RAG_API_URL=http://rag-api:8001

# Browser-facing; inlined into the frontend bundle at build time
NEXT_PUBLIC_API_URL=http://localhost:8000
NEXT_PUBLIC_RAG_API_URL=http://localhost:8001

# Shared SQLite + vector store
DB_PATH=/app/backend/chat_data.db
LANCEDB_PATH=/app/lancedb

# Models
GENERATION_MODEL=qwen3.5:9b
ENRICHMENT_MODEL=qwen3.5:4b
EMBEDDING_MODEL=microsoft/harrier-oss-v1-0.6b
RERANKER_MODEL=Qwen/Qwen3-Reranker-4B
```

Values already exported in your shell win over `--env-file` — that is how
`start-docker.sh container` overrides `OLLAMA_HOST`.

Changing `EMBEDDING_MODEL` invalidates existing indexes: every table records the
embedding model that wrote it (and its vector width), and writing or querying it
with a different model fails with an explicit error. Rebuild your indexes after
switching.

### 9.2 Custom Configuration

```bash
cp docker.env docker.custom.env
nano docker.custom.env
docker compose --env-file docker.custom.env up -d

# Remember to rebuild if you changed a NEXT_PUBLIC_* value
docker compose --env-file docker.custom.env up -d --build frontend
```

---

## 10. Success Checklist

- ✅ All containers healthy: `docker compose ps`
- ✅ Ollama reachable: `curl http://localhost:11434/api/tags`
- ✅ Frontend loads: `curl http://localhost:3000`
- ✅ Backend responds: `curl http://localhost:8000/health`
- ✅ RAG API responds: `curl http://localhost:8001/health`
- ✅ You can create an index and chat with your documents

### What to Expect

- **First `up --build`** is slow: it installs the Python dependencies (torch,
  transformers, docling) and builds the Next.js bundle, and `rag-api` then
  downloads the ~1.2GB embedding model before it reports healthy (the ~7.5GB
  reranker downloads lazily, on the first reranked query)
- **Restarting** an existing container is fast. The HuggingFace weights are
  downloaded at runtime into the container's writable layer, and no volume is
  mounted for them, so **recreating** `rag-api` (any `up --build`, `down` + `up`, or
  image change) downloads them again. Mount a cache directory and set `HF_HOME` if
  that matters to you.
- **One RAG request at a time** — the RAG API is single-threaded

---

**Happy Containerizing! 🐳**

---

### Documentation Improvement Plan (`Documentation/improvement_plan.md`)

# localGPT — Improvement Road-map

_Revision: 2026-08-09_

Planned work only. Nothing in the **Open** sections is implemented. The **Landed** section records changes that were verified in the current working tree at the revision date — every entry there names the file that proves it, so the list can be re-checked rather than trusted.

> An evidence-based, phased extension of this plan lives in
> [research_roadmap.md](research_roadmap.md), grounded in the August 2026
> research sweeps under [research/](research/). Items graduate from there into
> this file's Landed table as they ship.

---

## 0. Landed (verified in-tree, 2026-08-08; Phase 1 rows added 2026-08-09)

| Area | Change | Verify at |
|------|--------|-----------|
| Architecture | One RAG API server; the parallel `api_server_with_progress.py` is gone | `rag_system/` has a single `api_server.py` |
| Architecture | `factory.py` is the only agent/pipeline factory; `main.py` is config + a thin `index` / `chat` / `api` CLI | `rag_system/factory.py`, `rag_system/main.py` |
| Architecture | Backend gateway is threaded and every RAG API call has a timeout (`RAG_API_TIMEOUT`, `RAG_API_INDEX_TIMEOUT`) | `backend/server.py` |
| Ops | RAG API exposes `GET /health`; `run_system.py --health`, `--stop`, `--logs-only` and `--mode prod` all work | `rag_system/api_server.py`, `run_system.py` |
| Config | Service URLs and model ids come from environment variables, documented in `.env.example` | `.env.example`, `rag_system/main.py`, `backend/server.py`, `src/lib/api.ts` |
| Retrieval | Hybrid search fuses the full-text and vector legs with reciprocal rank fusion instead of a broken weighted blend; `retrieval_mode` (`hybrid` / `vector_only` / `fts_only`) is honoured | `rag_system/retrieval/retrievers.py` |
| Retrieval | **1.1 Late-chunk result merging** — retrieved late-chunks are merged with their ±1 siblings before reranking | `rag_system/pipelines/retrieval_pipeline.py` |
| Retrieval | A reranker that fails to load logs a warning and is skipped instead of throwing | `rag_system/pipelines/retrieval_pipeline.py::_get_ai_reranker` |
| Indexing | **3.3 Auto GPU dtype selection** — CUDA > MPS > CPU with fp16 off CPU, for both the embedder and the late-chunk encoder | `rag_system/indexing/representations.py`, `rag_system/indexing/latechunk.py` |
| Indexing | Vector width is derived from the loaded model, and appending mismatched vectors to an existing table raises with a "rebuild the index" message (part of **3.4**) | `rag_system/indexing/embedders.py::VectorIndexer` |
| Indexing | OCR engine is chosen by probing which backend is installed, so a missing macOS-only engine no longer breaks every conversion | `rag_system/ingestion/document_converter.py` |
| Storage | The LanceDB path is resolved from `LANCEDB_PATH` / the pipeline config instead of three hard-coded literals, so index deletion targets the store indexing writes to | `backend/database.py::resolve_lancedb_path` |
| Privacy | The semantic cache defaults to `cache_scope: "session"`, closing the cross-session answer leak | `rag_system/main.py`, `rag_system/agent/loop.py` |
| Hygiene | DSPy modules and the non-functional ReAct agent are gone from the code; `rank_bm25`, `scikit-learn`, `nltk`, `sentence_transformers`, `colpali-engine` and `matplotlib` were dropped from `requirements.txt` | `grep` returns no code hits for any of them |
| Retrieval | **Phase 1.2 embedder** (`research_roadmap.md` §1.2) — default embedder is `microsoft/harrier-oss-v1-0.6b` (MIT, 1024-dim) with the query-side `Instruct: … Query: …` prefix on for instruction-tuned families; documents stay unprefixed. Measured mixed-corpus first-stage nDCG@10 0.915 vs 0.875 for the previous `Qwen3-Embedding-4B` default, at ~3× lower latency (0.911 for the shipped stack once vectors are normalized — see the row below). `EMBEDDING_MODEL` still overrides | `rag_system/main.py::EXTERNAL_MODELS`, `rag_system/indexing/representations.py::default_query_instruction`, `rag_system/pipelines/retrieval_pipeline.py::_query_instruction`, `eval/DECISIONS.md` |
| Retrieval | **Phase 1.1 reranker** (`research_roadmap.md` §1.1) — the `default` profile shipped `reranker.enabled = False`: with this first stage the cheap cross-encoder measures net-negative (0.915 → 0.892 mixed) and the reranker that wins costs ~12.7 s/query. **Superseded 2026-08-14 (arm G, see §1.9): reranking is now ON by default with `min_score: 0.5` / `min_keep: 3` threshold selection.** `Qwen/Qwen3-Reranker-4B` loads lazily on the first reranked query through the in-repo `QwenRerankerScorer` | `rag_system/main.py::PIPELINE_CONFIGS`, `rag_system/pipelines/retrieval_pipeline.py::_get_ai_reranker`, `src/components/ui/session-chat.tsx`, `eval/DECISIONS.md` |
| Indexing | **Embedder-identity guard** — the width-only check cannot catch a swap between two 1024-dim models, so every table records the embedding model that wrote it (Arrow schema metadata, sidecar fallback). Indexing into or querying a table with a different embedder raises `EmbedderMismatchError` instead of returning nonsense | `rag_system/indexing/embedders.py::read_table_marker`/`assert_embedder_matches`, `rag_system/retrieval/retrievers.py::MultiVectorRetriever._check_table_identity` |
| Indexing | **Cosine normalization** — vectors are L2-normalized at write and query time, so LanceDB's default L2 ordering is the cosine ordering both model cards specify. Gated per table on the `normalized` marker, so legacy tables keep working unnormalized with a warning; the default table moved to `text_pages_v4`. Measured a wash on the gold set (mixed nDCG@10 0.915 → 0.911, recall@5 0.917 → 0.931): adopted for card-conformance, not for a number | `rag_system/indexing/embedders.py::l2_normalize`, `rag_system/retrieval/retrievers.py`, `rag_system/main.py` |
| Evaluation | **Phase 0 harness** (`research_roadmap.md` §0) — three corpora (two planted-fact PDFs + `Documentation/*.md`), a 72-query gold set labelled by answer-bearing text rather than chunk id, an in-process recall@5/10/20 + nDCG@10 runner, a binary groundedness judge validated at TPR 1.00 / TNR 1.00 on 20 hand-labelled cases, and a scripted end-to-end smoke that passes 25/25 assertions. Baseline recorded; every Phase 1/2 item now has something to A/B against | `eval/run_eval.py`, `eval/judge.py`, `eval/smoke_e2e.py`, `eval/goldset/*.jsonl`, `eval/BASELINE.md` |
| Models | **Roadmap 1.1/1.2 — evidence-gated defaults**: embedder `microsoft/harrier-oss-v1-0.6b` + query instruction prefix (mixed nDCG@10 0.915 vs 0.875 for the 8 GB Qwen3-4B); default profile reranker **off** (bge measured net-negative on this first stage; Qwen3-Reranker-4B is the lazy opt-in at +0.06 nDCG for ~12.7 s/query) — **superseded 2026-08-14 (arm G): reranker ON by default with `min_score`/`min_keep` selection**; per-table embedder-identity markers + L2 normalization, `text_pages_v4` | `rag_system/main.py`, `rag_system/indexing/embedders.py`, `eval/DECISIONS.md` |
| Routing | **Roadmap 2.3 — gateway routing is a deterministic gate.** The per-message enrichment-model router and the `_simple_pattern_routing` keyword/length fallback are deleted; `should_use_rag()` routes on `force_rag` → linked indexes → a whole-message smalltalk/assistant-meta allowlist → RAG. ~750 ms/message saved; agent triage is now the only LLM routing layer | `backend/server.py::should_use_rag`, `backend/test_gateway_routing.py` (155/155), `eval/decisions/phase2-gateway.md` |
| Retrieval | **Roadmap 2.5 — graph module removed**: `GraphExtractor`, `GraphRetriever`, `GraphQueryTranslator`, the `graph_query` triage outcome and the `retrieval.graph` / `graph_strategy` config keys are gone; `networkx`, `fuzzywuzzy`, `python-Levenshtein` dropped from all requirements files. Contested gains, 41–57× indexing and up to ~377× query-token cost (`research/academic-evidence-2026.md` §6) | `rag_system/indexing/` has no `graph_extractor.py`; `eval/decisions/phase2-pipeline.md` §1 |
| Retrieval | **Roadmap 2.1 — evidence-sufficiency retry**: one conditional second retrieval on weak evidence (candidate-set contrast signal — raw top similarity measured anti-correlated), on in `default`, off in `fast`. Fires on ~10% of queries, +0.008–0.017 nDCG@10, zero per-query regressions in four runs | `rag_system/pipelines/retrieval_pipeline.py::retrieve_candidates`, `eval/decisions/phase2-pipeline.md` §2 |
| Retrieval | **Roadmap 2.2 — decomposition at rerank**: the first stage always runs once on the full query; sub-queries score candidates at rerank (`query_decomposition.rerank_aggregate`), and first-stage fan-out survives only behind `compose_from_sub_answers`. Measured negative on truly-decomposing queries — no shipped profile enables rerank-decomposition. **Superseded in part: arm G (2026-08-14) turned the reranker on by default, so rerank-decomposition now applies; arm H (2026-08-15) made the pooled first stage the default (`pooled_first_stage: true`, `compose_from_sub_answers: false`) — per-sub-query retrieval, pooled + deduped candidates, ONE rerank + ONE synthesis** | `rag_system/pipelines/retrieval_pipeline.py`, `eval/decisions/phase2-pipeline.md` §3 |
| Verification | **Roadmap 2.4 — verifier model seam**: `VERIFIER_MODEL` / `verification.model` swaps the LLM-prompt verifier for a local NLI model (MiniCheck-DeBERTa 19/20, DeBERTa-MNLI 18/20 on the judge set); default unchanged. ThinknCheck has no public weights | `rag_system/agent/verifier.py::LocalNLIVerifier`, `eval/decisions/phase2-pipeline.md` §4 |
| Eval | `run_eval.py` drives `RetrievalPipeline.retrieve_candidates()` — first stage, retry and rerank are the shipped code path; `--retry`, `--decompose`, `--aggregate` flags added; gold rows `docs_d10` + the triage-model row re-anchored after 2.5/2.3 deleted their source text (recorded per-row) | `eval/run_eval.py`, `eval/goldset/docs.jsonl` |
| Observability | **Roadmap 4.5 — per-query token accounting** (on by default): Ollama `prompt_eval_count`/`eval_count` aggregated per user query, bucketed by stage, returned as `token_usage` on `/chat` and the SSE `complete` event. ContextVar-based; watsonx reports zeros; embedding calls uncounted | `rag_system/utils/ollama_client.py::TokenUsageTracker`, `eval/decisions/phase4-escalation-tokens.md` |
| Retrieval | **Roadmap 4.4 — metadata filter DSL** (no flag; inert without a `filters` argument, md5-verified byte-identical when unused): JSON filters compiled to LanceDB where-clauses prefiltering both search legs; quoting characters refused, malformed filters are the RAG API's 400; gateway forwards `filters` and treats a present filter as force-RAG. Page/date filtering NOT shipped (needs real columns + re-index → §3.7) | `rag_system/retrieval/filters.py`, `rag_system/api_server.py`, `backend/server.py`, `eval/decisions/phase4-filters-askfolder.md` |
| CLI | **Roadmap 4.6 — ask-a-folder**: `python -m rag_system.main ask <folder> "<q>"` builds an ephemeral index (fast profile), answers with the standard pipeline, cleans up everything including on SIGTERM | `rag_system/ask_folder.py`, `eval/decisions/phase4-filters-askfolder.md` |
| Indexing | **Roadmap 4.2 (index half) — cross-reference extraction** (on by default): regex-only extraction of exhibit/section/document-name references into `metadata.crossrefs`, filename resolution incl. numeric-prefix-stripped aliases; verified bit-identical text/vector columns. The **query-time hop is REJECTED as a default** (flag kept, off): 0/11 fires at the shipped k=20, 0/11 target precision where forced, beaten by raising `k` at equal budget | `rag_system/indexing/crossref.py`, `eval/decisions/phase4-retrieval-benchmarks.md`, `eval/decisions/phase4-answer-quality.md` |
| Retrieval | **Roadmap 4.1 escalation — REJECTED as default (2026-08-12 re-run); 4.3 overview prefilter — HELD off**: the HOLD's re-run condition was executed after the num_ctx fix and the lift did not survive — the escalation-off baseline on the identical fire subset went 0/9→7/9 (the 2026-08-09 "lift" was front-truncation favoring the tail-appended document) and both product-default fires were regressions (`eval/decisions/phase4-escalation-rerun.md`); prefilter `boost` wins only on heterogeneous corpora, `restrict` is rejected outright (kills recall@20 on 4 queries/corpus). Both stay flag-gated off | `rag_system/agent/escalation.py`, `rag_system/pipelines/retrieval_pipeline.py`, `eval/decisions/phase4-*.md`, `design_rationale.md` §13a |

The previous revision of this file marked five cleanup items ✅ COMPLETED. Two of those claims (removing unused imports/dependencies, consolidating configuration files) were not true when written and are only partly true now — the surviving work is tracked in §9 below.

---

> **Chunker data loss — FIXED 2026-08-12, REBUILD INDEXES.** `MarkdownRecursiveChunker._split_text`'s separator-reassembly loop dropped the first and every other body segment of any document large enough to split, so long documents entered the index at ~50% of their content (RFC 9000: 49% retained; `design_rationale.md`: 49%). Found by the unseen-corpus RFC shakedown — the authored eval corpora were too small to trigger the path. Fixed and property-tested lossless (30 randomized structured docs, zero non-whitespace loss; post-fix retention 94–99.8%, remainder is whitespace normalization). **Every index built before this fix under-contains its long documents** — product indexes in `index_store/` and the tracked eval indexes should be rebuilt, and pre-fix retrieval baselines on the `docs` corpus are not comparable to post-fix numbers.

## 1. Retrieval accuracy & speed

| ID | Item | Rationale | Notes |
|----|------|-----------|-------|
| 1.2 | Tiered retrieval (ANN pre-filter) | Large tables make LanceDB scans slow. | Narrow to top-N with an in-memory index, then exact search. |
| 1.3 | Corpus-tuned fusion | RRF is weight-free and safe, but a tuned fusion could beat it per corpus. | Would need a validation set and a place to store the setting per index. |
| 1.4 | Query expansion via extracted entities | Richer queries for entity-heavy corpora. | Depends on the Graph-RAG path (§9) being finished or removed. |
| 1.5 | ~~Deduplicate the late-chunk leg~~ **FIXED 2026-08-14** | Late-chunk hits were appended to the base hits with no dedupe, so the same passage occupied two slots — and with the ±1 sibling merge tripling every entry, synthesis context reached ~94k tokens/call while Ollama's slot-split window served only ~16k and silently front-truncated (the model saw the *tail* of the ranking). | `retrieval_pipeline.py` now dedupes across both legs on `(document_id, chunk_index)` and packs rank-ordered docs into an explicit synthesis token budget (default 12k, `synthesis_context_tokens` config) with sibling-span overlap suppression. Measured (arm F, `eval/decisions/synthesis-grounding-ab-2026-08-13.md`): context 335k → ~40k chars, truncation warnings 32 → 0, E2E wall time −32%, judged pass **7/24 → 16/24**. |
| 1.6 | Query-aware crossref-hop targets | The 4.2 hop was rejected as a default partly because target selection is query-blind: `max_hops=1` takes the first unrepresented reference in scan order, which lands on hub documents (21/24 hops → 2 docs, 0/11 expected-source precision). | Score candidate targets against the query (overview embeddings exist) before spending the hop; then re-run `eval/decisions/phase4-retrieval-benchmarks.md`'s A/B. |
| 1.8 | Crossref extraction for real-world naming conventions | The unseen-corpus RFC shakedown measured the extractor at **0/1403 resolved** on 23 real RFCs: filenames never occur in the prose, 355 explicit `[QUIC-TRANSPORT] Section N` cross-document references are discarded by `_SECTION_RE`, and bare `RFC NNNN` mentions have no pattern (`eval/decisions/rfc-shakedown-2026-08-13.md`). The extractor's patterns encode the authored acq corpus's conventions. | Add bracketed-citation and `RFC NNNN`-style patterns + title-based (not just filename-based) resolution; re-measure on the rfc corpus (a rename experiment showed 91 resolutions are reachable). Index-inert either way; the hop stays off. |
| 1.9 | Synthesis grounding on dense unseen technical text | With retrieval fixed (recall@20 0.958 on the rfc corpus), end-to-end answer quality is the measured bottleneck: **5/24** judged pass (Sonnet panel; single-doc 1/14, crossref 4/10). The 9b answers from its prior instead of the supplied snippets and fabricates citations (e.g. inventing a quote from "RFC 9002 §13.4"). The verifier flags these low-confidence but the wrong claim still leads the answer. | A/B run 2026-08-13 (`eval/decisions/synthesis-grounding-ab-2026-08-13.md`): strict snippet-only prompt + temperature-0 decode ADOPTED as default (7/24 vs 5/24 — at the noise floor, adopted for escape-hatch removal + determinism, not claimed as a quality win); **qwen3.6:35b-a3b REJECTED** (equal-or-worse at 4-5/24 under both prompts — grounding is not parameter count at this scale). Strict compose prompt TESTED AND REJECTED same day (arm E: 4/24 vs C's 7/24; all three pass→fail flips were composed rows — strict copy-rules against already-synthesized sub-answer prose drop facts the sub-answers carried; reverted, decision file appendix). **2026-08-14 (arm F): the dominant cause was context overflow, not the model** — synthesis prompts were ~94k tokens against a slot-split ~16k served window, so front-truncation deleted the top-ranked evidence on every call. Cross-leg dedupe + 12k context budget (item 1.5) lifted judged pass to **16/24 (single-doc 10/14, crossref 6/10)** with a near-unanimous panel (1 split/72 votes) and −32% wall time. Item narrows to the crossref/multi-hop residue: next levers are deterministic decomposition (temp 0 — also stabilizes A/B row sets); abstain-on-low-verifier-confidence; passage-level citation forcing. **2026-08-15 (arm G): Qwen3-Reranker-4B enabled by default as final-stage SELECTION** — union-of-max `min_score 0.5` over original+sub-queries keeps only candidates the calibrated scorer marks relevant (mean 8.8, range 3–10 — adaptive context), scored on preserved core chunk text; **18/24 (11/14, 7/10)** vs arm F 16/24 — net +2 within noise floor, adopted as user-directed no-regression feature at +92% wall time (opt-out `reranker.enabled: false`; 0.6B scorer unmeasured latency lever). Supersedes the Phase-1 off-by-default call, which predated the context budget. **2026-08-15 (arms H/H2/G2): decomposition restructured to POOLED** — per-sub-query retrieval pooled+deduped, one source-aware rerank (per-SQ floor guarantees every sub-question keeps evidence), one synthesis with the original query; composer eliminated from the path. Decomposition now decodes at **temperature 0** (deterministic splits — A/B row sets finally stable; all 18 direct rows got identical panel verdicts across the H2/G2 re-run). Clean comparison on identical row sets: pooled 17/24 vs composer 17/24 — true tie; pooled adopted on structure (24 vs 32 synthesis calls, linear scaling in sub-query count, no compose-step fact loss). **2026-08-15 (arm I): source-document labels — 21/24 (single-doc 12/14, crossref 9/10), +4/0 vs H2, first clearly-attributable synthesis quality win.** Diagnosis showed half the crossref failures were attribution failures (answers with every fact correct, failed for not naming "RFC 9001"/"RFC 9000"): the context join carried no source labels while prompt rule 3 forbids doc names not in snippets. Every snippet now opens with `[Source document: <file>]` + rule 7 permits attribution via labels. Remaining residue: rfc_q10/q15 (single-doc), rfc_q17 (nuance), rfc_q20 (gold-nuance split). |
| 1.7 | ~~Sanitize FTS sub-query input~~ **FIXED 2026-08-12** | A decomposer-emitted sub-query containing double quotes made the LanceDB FTS parser raise ("position is not found but required for phrase queries"), killing hybrid retrieval for that query outright (1/24 gold queries during the Phase-4 A/B). | `retrievers.py` now strips double quotes before the FTS leg, and a hybrid search whose FTS leg fails for any other reason degrades to dense-only with a warning instead of returning nothing (`fts_only` mode still propagates). Verified against the exact incident query: 0 docs → 5 docs, answer document ranked first; degradation tested with a simulated FTS failure. |

## 2. Routing / triage

| ID | Item | Rationale |
|----|------|-----------|
| 2.1 | Embed and cache document overviews | The agent router (now the only LLM routing layer) makes an LLM call per query; a cosine pre-check would be far cheaper. |
| 2.2 | Session-level routing memo | Today the only shortcut is "history exists → `rag_query`". Cache the decision instead. |

## 3. Indexing pipeline

| ID | Item | Rationale |
|----|------|-----------|
| 3.1 | Parallel document conversion | Conversion and chunking are serial per file. |
| 3.2 | Incremental indexing | Re-embedding the whole corpus to add one document is wasteful. |
| 3.4 | Post-build health check | The dimension guard exists; a build should also assert the FTS index, the row count and the late-chunk table when requested. |
| 3.5 | Align chunk-size defaults | No profile sets `chunking.chunk_size`, so CLI indexing chunks at 1500 tokens while `POST /index` defaults to 512 (`api_server.py:441`) — same class of split as 3.6. Documented in `system_overview.md` §5.3. |
| 3.6 | Align late-chunk defaults | `POST /index` defaults `enable_latechunk` to `false` while the `default` profile enables late-chunk retrieval, so HTTP builds and CLI builds silently differ (retrieval degrades gracefully when the `_lc` table is absent). |
| 3.7 | Real `page`/`date` columns | Roadmap 4.4 promised page/date filters; they did not ship because both live inside the metadata JSON string column (`metadata LIKE` would false-match `"page": 31` for page 3). Needs first-class columns + a re-index window. |

## 4. Model management

* Registry mapping model tag → dimensions, source and license, validated by the UI and both servers.
* An embedder pool that keeps one copy of each loaded model in memory (a module-level cache exists in `representations.py`; the reranker and pruner have their own ad-hoc singletons).
* Warn — or refuse — when the embedding model configured for a query differs from the one recorded in the index metadata.

## 5. Database & storage

* Garbage-collect orphaned LanceDB tables (an index row deleted outside the API leaves its table behind).
* Delete files from `shared_uploads/` when their session or index is deleted.
* Scheduled SQLite `VACUUM` when fragmentation is high.

## 6. Observability & ops

* JSON structured logging and log rotation in `run_system.setup_logging` (today: plain text, no rotation).
* Move the agent's and pipelines' `print()` progress output onto the `logging` module.
* A `/metrics` endpoint for Prometheus.
* A deep health probe (`/health/deep`) that runs a real end-to-end query.
* Per-request pipeline configuration on the RAG API, so options stop leaking between requests, followed by switching the RAG API to a threading server. *(The leak half has landed: the agent now snapshots its pipeline config before applying per-request overrides and restores it afterwards — `rag_system/agent/loop.py`. The threading-server switch remains.)*
* **Context-window budgeting — FIXED 2026-08-12 (found by the Phase-4 A/B, 2026-08-09):** `ollama_client` never set `num_ctx`, and the served ceiling on this host was **8194 prompt tokens** with silent *front*-truncation — a 373-chunk corpus's synthesis prompt (20–25k tokens) lost its top-ranked evidence without any error. Now both clients (`rag_system/utils/ollama_client.py`, `backend/ollama_client.py`) size `options.num_ctx` per request from the prompt length, bucketed 8k/16k/32k and per-model **monotonic** — the window only ratchets upward, because changing `num_ctx` between calls forces a KV-cache reallocation (measured on the smoke suite: naive per-call bucketing 924s; fixed 32k window 259s; the shipped monotonic ratchet 208s; pre-fix baseline 363s). `OLLAMA_NUM_CTX` pins a value, `OLLAMA_NUM_CTX_MAX` caps the bucket (default 32768), and a filled window (`prompt_eval_count ≥ num_ctx − 16`) logs a loud truncation warning. Verified 2026-08-12: the 100k-char probe that returned `prompt_eval_count: 8194` now evaluates the full prompt (17,351 tokens) and recovers a fact planted at position 0. The 4.1 escalation A/B re-run this unblocked was executed 2026-08-12 (`eval/decisions/phase4-escalation-rerun.md`; verdict: reject). Note the fix also **voids** the 2026-08-09 `acqdocs` baseline in `phase4-answer-quality.md` §4 — that 0/9 was a truncated-context artifact and reads 7/9 on identical inputs post-fix. Still open: prompts >~90k chars hit the 32k default cap — client-side facts-string budgeting remains worthwhile. Related, **FIXED 2026-08-12**: the synthesis stream paths never set `think:false`, so the 9b could burn its window on chain-of-thought and return an empty answer (measured: prompt 9,351 + thinking 7,033 = window exactly, empty `response`). All three streaming synthesis call sites (`retrieval_pipeline._synthesize_final_answer` and both agent-loop compose paths) now pass `enable_thinking=False`; verified at the wire that `think: false` reaches the Ollama payload. The eval harness's `patch_no_think` monkeypatch is now redundant for these paths but harmless.
* Render `token_usage` in the UI (it reaches the browser on the SSE `complete` event and is persisted in the turn's steps snapshot; `conversation-page.tsx` doesn't display it yet), and forward it on the gateway's non-streaming path (`backend/server.py::_handle_rag_query` extracts only `answer` + `source_documents`).
* Judge noise is larger than the Phase-0 validation suggested: 5–7 verdict flips per 24 rows between identical runs were observed in the Phase-4 A/Bs. Direction-deciding cells need k≥5 judge votes (`eval/decisions/phase4-answer-quality.md`). The smoke test shares the underlying cause — its substring assertions run against temperature-1.0 generations, and 1 of 6 runs on 2026-08-09 flaked on one question (a rerun passed 25/25). Consider `temperature 0` for smoke/eval generation calls. Worse, the 2026-08-12 re-run caught the judge returning verdicts its own reasons contradict (an answer containing the gold fact verbatim voted 0/5), and being perturbed by the verifier's `[Confidence: N%] [Warning: …]` suffix, which one judge reason cited as grounds for rejection. Two mitigations landed 2026-08-12: `eval/judge.py` now strips the verifier suffix from both judge slots before scoring, and the judge gained an Anthropic-API backend (eval-only; `JUDGE_MODEL=claude-sonnet-5` routes any `claude-*` model name through the `anthropic` SDK with a server-enforced JSON schema — the product stays fully local). The 18 hand-adjudicated rows from the re-run are preserved as a permanent hard-case benchmark (`eval/judge_hard_cases.jsonl` + `eval/validate_judge_hard.py`; the 4b's k=5 majority scores 13/18 on it — any candidate judge must beat that). **A Sonnet-class judge is now validated (2026-08-13)**: three independent Claude Sonnet subagent voters, each running the exact v1 prompt (suffix-stripped) over all 38 cases, scored **20/20 on the Phase-0 validation set and 18/18 on the hard set, with zero split votes across 114 judgments** — including the fact-present-but-prefaced-with-a-denial row both 4b arms failed (`eval/decisions/judge-sonnet-validation-2026-08-13.json`). Protocol for direction-deciding cells: generate the per-case v1 prompts (as `eval/judge.py` builds them) and fan them out to 3 Sonnet subagent voters, majority decides; the 4b remains the free bulk-pass judge. The `JUDGE_MODEL=claude-*` API backend in `eval/judge.py` runs the identical prompt and is available when API credentials exist.
* **Whole-document context can send the 9b model into verbatim transcription**: with escalation forced on and an untruncated 32k window, one query produced a 35,290-char answer (vs 1,400 baseline) that answered correctly in its first paragraph and then regurgitated `design_rationale.md` text wholesale (`eval/decisions/phase4-escalation-rerun.md` §8.1). Any future feature that injects large verbatim blocks needs an output cap or a "do not transcribe the context" instruction in the synthesis prompt.

## 7. Front-end UX

* SSE-driven progress for indexing (chat already streams phases; indexing is a blocking POST).
* Matched-term highlighting in retrieved snippets.
* Preset buttons (Fast / Balanced / High-Recall) over the retrieval settings.
* Surface the verifier's confidence as a field rather than parsing it out of the answer string.

## 8. Testing & CI

There is no automated test suite. The only checks are `python system_health_check.py`, `python run_system.py --health` and `./test_docker_build.sh`.

* Unit tests for `MultiVectorRetriever.retrieve` across all three modes, including the RRF ordering.
* Integration test: build an index → query → assert at least one source document.
* A GitHub Action that starts Ollama, pulls a small embedding model and runs the smoke test.
* Re-enable the type/lint gates that `next.config.ts` currently disables (`eslint.ignoreDuringBuilds`, `typescript.ignoreBuildErrors`).

## 9. Codebase hygiene

* `docker-compose.local-ollama.yml` now duplicates `docker-compose.yml` (which already defaults to host Ollama). Pick one. *(Resolved: the variant file was removed — `docker-compose.yml` plus `--profile with-ollama` is the only flow.)*
* Run `mypy`, `pylint` and `black` in CI.

---

### Priority matrix (suggested order)

1. **Correctness / data loss**: 3.6
2. **User-visible wins**: 7.1, 7.2, 2.4
3. **Reliability**: 3.4, 5.1, 8.2
4. **Performance**: 1.2, 1.5, 3.1
5. **Long-term maintainability**: 2.1, 4, 9

Rearrange to suit team objectives and available time.

---

### Documentation Indexing Pipeline (`Documentation/indexing_pipeline.md`)

# 🗂️ Indexing Pipeline

_Implementation entry-point: `rag_system/pipelines/indexing_pipeline.py`, with helpers in `rag_system/ingestion/` and `rag_system/indexing/`._

## Overview
Turns documents (PDF, DOCX, HTML/HTM, MD, TXT) into search-ready chunks with embeddings, writes them to LanceDB, builds a full-text index over the same table, and generates a per-document overview used by the triage routers.

Public entry point:

```python
IndexingPipeline.run(file_paths: list[str]) -> None
```

`run()` also accepts a legacy `documents=` keyword as an alias for `file_paths` (`indexing_pipeline.py:157-167`). It returns `None`; progress and results are printed.

## High-level diagram

```mermaid
flowchart TD
    A["Files"] --> B["DocumentConverter (docling)"]
    B --> C["Markdown + DoclingDocument"]
    C --> D{"chunker_mode"}
    D -- docling --> D1["DoclingChunker.chunk_document()"]
    D -- legacy --> D2["MarkdownRecursiveChunker.chunk()"]
    D1 --> OV["OverviewBuilder (per document)"]
    D2 --> OV
    OV --> OVF[["index_store/overviews/&lt;id&gt;.jsonl"]]
    D1 --> E["ContextualEnricher (optional)"]
    D2 --> E
    E --> F["EmbeddingGenerator"]
    F --> G[("LanceDB table")]
    G --> H["create_fts_index('text')"]
    F --> LC["LateChunkEncoder (optional)"]
    LC --> GLC[("LanceDB table &lt;table&gt;_lc")]
```

## Steps in detail

| Step | Module | Key classes | Notes |
|------|--------|-------------|-------|
| Conversion | `ingestion/document_converter.py` | `DocumentConverter` | docling for PDF/DOCX/HTML/MD; plain read for `.txt`. PyMuPDF (`fitz`) is used **only** to test whether a PDF already has a text layer. |
| Chunking | `ingestion/docling_chunker.py`, `ingestion/chunking.py` | `DoclingChunker`, `MarkdownRecursiveChunker` | Docling is the default. Both are token-aware via the embedding model's tokenizer. |
| Overview | `indexing/overview_builder.py` | `OverviewBuilder` | One LLM call per document, from its first N chunks. On by default. |
| Contextual enrichment | `indexing/contextualizer.py` | `ContextualEnricher` | One LLM call per chunk. On in the `default` profile. |
| Embedding | `indexing/representations.py` | `QwenEmbedder`, `OllamaEmbedder`, `EmbeddingGenerator` | `select_embedder()` picks HuggingFace vs Ollama. |
| LanceDB write | `indexing/embedders.py` | `LanceDBManager`, `VectorIndexer` | Appends to an existing table, creates it otherwise. Vector width is taken from the produced embeddings. |
| Full-text index | `pipelines/indexing_pipeline.py:264-284` | – | `tbl.create_fts_index("text", use_tantivy=False)`. This is what makes `hybrid` retrieval work. |
| Late chunking | `indexing/latechunk.py` | `LateChunkEncoder` | Optional second embedding pass into `<table>_lc`. |

Storage location is `storage.lancedb_uri` (also accepted as `storage.db_path` / `storage.lancedb_path`), which both profiles set to `./lancedb`. Per-index tables are named `text_pages_<index_id>` (`backend/database.py:351`); the profile fallback table is `text_pages_v4`.

## Control flow

**Through the UI**

1. `IndexForm` → `chatAPI.buildIndex()` → `POST :8000/indexes/<index_id>/build` (or `POST :8000/sessions/<id>/index` for session uploads).
2. `backend/server.py` normalises the option names (`INDEX_OPTIONS`, `server.py:58-70`), omits anything the caller did not send so pipeline defaults apply, and POSTs to `${RAG_API_URL}/index`.
3. `rag_system/api_server.py:392-473` validates the body, builds a per-request config from a deep copy of the pipeline profile (`_build_index_config_override`, `:197-236`), constructs a fresh `IndexingPipeline` with it, and calls `run(file_paths)`.
4. The chosen embedding model is written into the index's SQLite metadata (`api_server.py:445-449`); retrieval later re-applies it (`api_server.py:116-134`).

**From the command line**

```bash
python -m rag_system.main index /path/to/docs --mode default
```

`_collect_file_paths` (`rag_system/main.py:133-146`) walks a directory for `.pdf`, `.docx`, `.html`, `.htm`, `.md`, `.txt`, or accepts a single file; the `index` branch (`main.py:171-188`) then calls `factory.get_indexing_pipeline(mode).run(file_paths)`. `--mode` accepts `default` or `fast`. `python rag_system/main.py …` as a file path does not work — run it as a module from the project root.

`create_index_script.py` is a separate interactive/batch tool that creates a SQLite index row and points the pipeline at that index's own table (`--batch <config.json>`, `--config <pipeline.json>`, `--create-sample`).

## Request fields (`POST /index`)

Both camelCase and snake_case are accepted; the RAG API normalises to snake_case once at parse time (`api_server.py:51-76`).

| Field | Default | Effect |
|-------|---------|--------|
| `file_paths` | **required** | List of absolute paths. A missing or non-list value returns HTTP 400. |
| `session_id` | – | Sets `overview_path` to `index_store/overviews/<session_id>.jsonl`. |
| `table_name` | resolved from the session's index | Sets `storage.text_table_name` and `retrieval.dense.lancedb_table_name`. |
| `enable_latechunk` | `false` | Writes a second `<table>_lc` table of late-chunked vectors. |
| `enable_docling_chunk` | `true` | Maps to `chunker_mode: "docling"` when true, `"legacy"` when false. See the note below. |
| `chunk_size` | `512` | Token budget per chunk, for both chunkers. |
| `retrieval_mode` (alias `search_type`) | – | Validated against `hybrid` / `vector_only` / `fts_only` (HTTP 400 otherwise) and recorded on the index config as `retrieval.search_type`. It cannot change the artifacts written; it takes effect at query time. |
| `window_size` | `2` | Neighbour window for contextual enrichment. |
| `enable_enrich` | `true` | Contextual enrichment on/off. |
| `embedding_model` | profile value | Overrides `embedding_model_name` and is stamped into the index metadata. |
| `enrich_model` | – | Model for contextual enrichment. |
| `overview_model_name` (aliases `overviewModel`, `overview_model`) | – | Model for document overviews. |
| `batch_size_embed` | `50` | Embedding batch size. |
| `batch_size_enrich` | `25` | Enrichment batch size. |

Response (`api_server.py:451-467`):

```jsonc
{
  "message": "Indexing process for N file(s) completed successfully.",
  "table_name": "text_pages_<id>",
  "latechunk": false,
  "docling_chunk": true,
  "indexing_config": {
    "chunk_size": 512, "retrieval_mode": "hybrid", "window_size": 2,
    "enable_enrich": true, "embedding_model": "microsoft/harrier-oss-v1-0.6b",
    "enrich_model": null, "overview_model_name": null,
    "batch_size_embed": 50, "batch_size_enrich": 25
  }
}
```

`indexing_config.embedding_model` reports the model actually used, not the raw request value. There is no `indexed_files` field.

Unknown fields are ignored rather than rejected.

## Pipeline config keys

Read by `IndexingPipeline.__init__` and by `run()` for the table paths:

| Key | Default in code | `default` profile | `fast` profile |
|-----|-----------------|-------------------|----------------|
| `chunker_mode` | `docling` | not set | not set |
| `chunking.chunk_size` (aliases `chunk_size`, `max_tokens`) | `1500` | not set | not set |
| `overlap_sentences` | `1` | not set | not set |
| `embedding_model_name` | `EXTERNAL_MODELS["embedding_model"]` | `microsoft/harrier-oss-v1-0.6b` | same |
| `storage.lancedb_uri` / `db_path` / `lancedb_path` | – (raises if all absent) | `./lancedb` | `./lancedb` |
| `storage.text_table_name` | falls back to `retrieval.dense.lancedb_table_name`, then `default_text_table` | `text_pages_v4` | `text_pages_v4` |
| `indexing.embedding_batch_size` | `50` | `50` | `100` |
| `indexing.enrichment_batch_size` | `10` | `10` | `50` |
| `retrieval.dense.enabled` | `true` | `true` | `true` |
| `retrieval.latechunk.enabled` (also read as `late_chunking`) | `false` | `true` | `false` |
| `retrieval.latechunk.lancedb_table_name` / `table_suffix` | suffix `_lc` | not set | not set |
| `contextual_enricher.enabled` / `window_size` | `false` / `1` | `true` / `1` | `false` / `1` |
| `enrich_model` (then `enrichment_model_name`, then `OLLAMA_CONFIG["enrichment_model"]`, then `generation_model`) | – | – | – |
| `overview.enabled` | `true` | not set ⇒ on | not set ⇒ on |
| `overview_model_name` / `overview.model` | falls back to enrichment then generation model | not set | not set |
| `overview_first_n_chunks` / `overview.max_chunks` | `5` | not set | not set |
| `overview_path` | `index_store/overviews/overviews.jsonl` | not set | not set |

Note the layering: the CLI/profile path uses the code default `chunk_size` of **1500** tokens because `PIPELINE_CONFIGS` sets no `chunk_size`, while the HTTP path always sends **512**. Likewise `enrichment_batch_size` is 10 from the profile but 25 over HTTP.

## Conversion (`ingestion/document_converter.py`)

Three docling converters are built independently at construction time, so a failure in one does not disable the others (`:67-104`):

* **no-OCR** PDF converter,
* **OCR** PDF converter,
* **general** converter for DOCX/HTML/MD.

`.txt` files bypass docling entirely and are wrapped in a fenced code block (`:152-166`).

For PDFs, `_pdf_has_text()` opens the file with PyMuPDF and checks for any extractable text; if there is none, the OCR converter is used, otherwise the fast no-OCR converter (`:125-150`). When a PDF has no text layer but no OCR converter is available, the pipeline logs and retries without OCR rather than skipping the file.

**OCR engine selection** (`build_ocr_options`, `:22-48`) picks the first engine whose backend is actually installed:

| Order | docling options class | Requires |
|-------|----------------------|----------|
| 1 | `OcrMacOptions` | macOS only (`platform.system() == "Darwin"`) plus the `ocrmac` package |
| 2 | `EasyOcrOptions` | `easyocr` |
| 3 | `RapidOcrOptions` | `rapidocr` (or the older `rapidocr_onnxruntime` package name — either is accepted) |
| 4 | `TesseractOcrOptions` | `tesserocr` |
| 5 | `TesseractCliOcrOptions` | a `tesseract` binary on `PATH` |

If none is available it logs "No OCR engine available; using docling's default OCR settings." On Linux/Docker, install one of the non-macOS backends if you need scanned-PDF support — `ocrmac` is macOS-only and is excluded from `requirements-docker.txt`.

Conversion returns `[(markdown, metadata, DoclingDocument)]` for docling paths and `[(markdown, metadata)]` for `.txt`. A conversion error is caught per document and returns `[]` (`:190-192`), so one bad file does not abort the run.

## Chunking

`chunker_mode` defaults to `"docling"` (`indexing_pipeline.py:27`). `MarkdownRecursiveChunker` is reached when `chunker_mode` is anything else, or when `DoclingChunker` construction raises (`:48-54`).

> **Note:** the RAG API maps `enable_docling_chunk` straight to `chunker_mode` — `"docling"` when true, `"legacy"` when false (`api_server.py::_build_index_config_override`), and the HTTP default is `true`. The legacy chunker is also selectable by setting `chunker_mode: "legacy"` in a pipeline config (which `create_index_script.py` does).

**DoclingChunker** (`ingestion/docling_chunker.py`)

* `chunk_document(doc, …)` walks the `DoclingDocument` with docling's `iterate_items()`, which yields `(item, level)` in true reading order — tables included inline. Items labelled `table` are exported to markdown and emitted as atomic chunks where they appear in the flow; items labelled `section_header` (with their `level`) maintain a `heading_path` and are not emitted as content; page numbers are taken from each item's `prov`; paragraph text is token-packed until `max_tokens`. Anything unexpected in the walk falls back to `split_markdown`.
* A second consolidation pass merges consecutive paragraph chunks that share a page and heading path, up to `max_tokens` (`:199-246`).
* `split_markdown()` is the fallback when only Markdown is available: it runs the legacy chunker with a 10,000-token ceiling, then repacks sentences to `max_tokens`, carrying `overlap` sentences (default 1) into the next window (`:47-83`).
* `chunk_document()` is what runs for documents converted by docling; the pipeline picks it via `hasattr(self.chunker, "chunk_document")` (`indexing_pipeline.py:192-195`).

**MarkdownRecursiveChunker** (`ingestion/chunking.py`)

Recursively splits on `\n## `, `\n### `, `\n#### `, ```` ``` ````, `\n\n`, then on word boundaries, and merges adjacent pieces up to `max_chunk_size` while respecting `min_chunk_size` (`:34-124`). The pipeline passes `min_chunk_size = max(1, chunk_size // 4)`.

Both chunkers count tokens with `AutoTokenizer.from_pretrained(embedding_model_name)`. If the tokenizer cannot be loaded they log a warning and fall back to a 4-characters-per-token approximation.

There is **no chunk-overlap knob**. The docling path carries `overlap_sentences` (default 1) only in `split_markdown`; `chunk_document` emits non-overlapping consolidated blocks; the legacy path has no overlap logic at all.

After chunking, the pipeline stamps a sequential `metadata.chunk_index` on every chunk of the document (`indexing_pipeline.py:202-205`) — this is what context expansion and late-chunk merging use at query time.

## Document overviews

`OverviewBuilder.build_and_store(doc_id, chunks)` (`indexing/overview_builder.py:33-49`) runs once per document, inside the chunking loop and **before** enrichment. It sends the first `first_n_chunks` chunks (default 5), truncated to 5000 characters, to the overview model and appends one JSON line per document:

```jsonc
{"doc_id": "report.pdf", "overview": "…"}
```

Written in **append** mode to `overview_path`, which is one file per index (`index_store/overviews/<index_id>.jsonl` when the API supplies a `session_id`), not one file per document. Failures are caught per document and logged (`indexing_pipeline.py:208-212`). Set `overview.enabled: false` in the pipeline config to skip the stage; there is no HTTP field for it.

## Contextual enrichment

`ContextualEnricher.enrich_chunks(chunks, window_size)` (`indexing/contextualizer.py:82-144`) prepends an LLM-written summary to each chunk's embedded text and preserves the untouched original in `metadata.original_text`:

```
Context: <2-5 sentence summary>

---

<original chunk text>
```

Processing is **sequential**: `BatchProcessor.process_in_batches` is a plain `for` loop over slices with progress reporting and a `gc.collect()` every fifth batch (`utils/batch_processor.py:105-125`). "Batch size" controls reporting granularity and memory, not concurrency. Budget one LLM round-trip per chunk when sizing a run — there is no thread or process pool anywhere in the indexing path.

Chain-of-thought markers (`<think>…</think>`), assistant tags and a leading `Answer:` are stripped from the summary; a summary shorter than 5 characters is discarded and the chunk is indexed unenriched (`contextualizer.py:56-74`).

## Embedding

`select_embedder(model_name, ollama_host)` (`indexing/representations.py:167-173`) is a two-way dispatch:

* the name contains `/` or starts with `http` → `QwenEmbedder` (HuggingFace `AutoModel`, loaded in-process);
* anything else → `OllamaEmbedder`, one HTTP call to `/api/embeddings` per text.

`QwenEmbedder` (`representations.py:15-94`):

* device order CUDA → MPS → CPU; `float16` off CPU;
* weights cached per model name in a module-level dict, so repeated construction reuses them;
* tokenizer is loaded with `padding_side="left"`, and pooling takes the **last real token**, not a mean — with left padding that is simply the final column, and the right-padding case is handled explicitly (`:66-76`);
* `max_length` is `min(tokenizer.model_max_length, 8192)` so `truncation=True` actually truncates;
* NaN/Inf values are replaced with zeros and logged.

`LateChunkEncoder` mean-pools over each span instead (`indexing/latechunk.py:82`) — the two paths deliberately use different pooling and write to different tables.

**Embedding dimensions are never hard-coded.** `VectorIndexer.index()` reads the width from the first produced vector (`indexing/embedders.py:47`) and builds the pyarrow schema from it. If the target table already exists with a different width, indexing raises:

> Table '…' stores N-dim vectors but the current embedding model produced M-dim vectors. Changing the embedding model requires rebuilding the index.

**Changing `EMBEDDING_MODEL` (or the per-request `embedding_model`) requires re-indexing.** `Qwen/Qwen3-Embedding-4B` produces 2560-dim vectors; `microsoft/harrier-oss-v1-0.6b` and `Qwen/Qwen3-Embedding-0.6B` both produce 1024-dim. They are not interchangeable against an existing table — and because a matching width does not make two models compatible, `VectorIndexer` also stamps the embedding model name (and an L2-`normalized` flag) into each table's metadata and refuses to write or read it with a different one.

## LanceDB write and full-text index

`VectorIndexer.index(table_name, chunks, embeddings)` (`indexing/embedders.py:39-135`) writes one row per chunk:

| Column | Content |
|--------|---------|
| `vector` | fixed-size float32 list, width from the model |
| `text` | the text that was embedded (enriched, if enrichment ran) |
| `chunk_id`, `document_id`, `chunk_index` | flattened identifiers used by context expansion |
| `metadata` | the whole chunk dict as a JSON string, including `original_text` |

Chunks whose vector contains NaN or Inf are skipped with a warning. The table is created when it does not exist; when it does, re-indexing a document first deletes that document's existing rows (delete-by-`document_id` before append — `VectorIndexer.index`), so re-running a build **replaces** the previous chunks instead of duplicating them. The `<table>_lc` late-chunk table gets the same treatment. `tbl.add(..., on_bad_vectors='drop')` is retried with a zero-fill strategy on failure.

Immediately after the vector write, the pipeline ensures a Lance native full-text index on the `text` column (`indexing_pipeline.py:264-284`), guarding against both the LanceDB default name `text_idx` and this project's older name `fts_text` so a rebuild does not raise. This index is what the `hybrid` and `fts_only` retrieval modes query — there is no separate BM25 store on disk and no `bm25_path` config key.

No ANN index is created. `create_index` / IVF-PQ appears nowhere in `rag_system/`, so vector search is an exhaustive scan. That is fine for the corpus sizes this project targets and avoids the training-set-size requirements of IVF-PQ.

## Late chunking (optional)

When `retrieval.latechunk.enabled` is true, a second pass runs per document (`indexing_pipeline.py:286-327`):

1. Concatenate the document's chunk texts with newlines and record each chunk's character span.
2. Feed the whole document through `LateChunkEncoder.encode()` — one forward pass, truncated at 8192 tokens — and mean-pool the token hidden states inside each span.
3. Write those vectors to `<table>_lc` (or `latechunk.lancedb_table_name` when set) with the same chunk rows.

Each chunk vector is therefore produced with knowledge of the whole document. A per-document encode failure, or a vector/chunk count mismatch, logs a warning and skips that document. The cost is a second full copy of the embedding model in memory and roughly double the vectors written.

The retrieval side reads `<table>_lc` with the same default suffix — see `retrieval_pipeline.md`.

## Knowledge graph — removed 2026-08-09

There is no knowledge-graph step any more. `indexing/graph_extractor.py`, the
`retrieval.graph.*` config keys and the `.gml` writer were deleted at roadmap
item 2.5. The path had never been armed (both shipped profiles set
`enabled: false`), and the evidence argues against reviving it: GraphRAG *loses*
on single-hop retrieval, its multi-hop gains span +3 to +27 points depending on
how well the vector baseline is tuned, and it costs **41–57× at indexing** and up
to **~377× in query tokens** — see
[`research/academic-evidence-2026.md`](research/academic-evidence-2026.md) §6.
`networkx` was dropped from `requirements.txt` in the same change.

An index built before this change is unaffected: the graph lived in a standalone
`.gml` file that nothing else read. A stale `index_store/graph/` directory can be
deleted by hand.

## Error handling

* **Per file** — conversion or chunking errors are caught, logged as `❌ Error processing <path>`, counted by the progress tracker, and the run continues (`indexing_pipeline.py:219-222`).
* **No chunks at all** — the run raises `RuntimeError` ("No text chunks were generated from the supplied documents…"), which surfaces as a 500 from `POST /index` so a failed conversion is never reported as a successful build (`:226-231`).
* **Duplicate table** — `VectorIndexer` appends instead of recreating (`embedders.py:105-120`); the backend additionally treats an "already exists" error from the RAG API as non-fatal and reports "Index already built – skipping rebuild." (`backend/server.py`, `handle_build_index`).
* **Dimension mismatch** — hard `ValueError`, on purpose. Silently dropping or recreating an index would corrupt it.
* **FTS index** — creation failures are logged, not raised; the vectors are already written and `vector_only` retrieval still works.
* **Overview / late-chunk / enrichment failures** — logged and skipped; the main vector index is still produced.

At the end, `_print_final_statistics` reports files processed, chunks generated, average chunks per file, which components ran, and the batch sizes used (`indexing_pipeline.py:349-372`).

## Not integrated

* **Vision / multimodal.** There is no vision model anywhere in the pipeline: no image embeddings are produced and no image table is written. PDF understanding is docling's layout parsing plus OCR. Models such as GLM-OCR or Qwen3-VL would be reasonable extensions, but no code path exists for them today.
* **Parallel document processing.** Documents are processed one at a time and enrichment is one sequential LLM call per chunk. There is no `ProcessPoolExecutor` or `ThreadPoolExecutor` in the indexing path.

---
_Keep this document updated when stages, config keys, or the `/index` contract change._

---

### Documentation Installation Guide (`Documentation/installation_guide.md`)

# 📦 LocalGPT Installation Guide

_Last updated: 2026-08-08_

This guide provides step-by-step instructions for installing and setting up
LocalGPT using either Docker or a direct development install.

---

## 🎯 Installation Options

### Option 1: Docker Deployment 🐳
- **Best for**: reproducible environments, keeping Python dependencies isolated
- **Requirements**: Docker Desktop + Ollama (host or containerized)
- **Setup time**: ~10 minutes plus image build and model downloads

### Option 2: Direct Development 💻
- **Best for**: development, customization, debugging
- **Requirements**: Python + Node.js + Ollama
- **Setup time**: ~15 minutes plus model downloads

---

## 1. Prerequisites

### 1.1 System Requirements

#### **Minimum Requirements**
- **CPU**: 4 cores, 2.5GHz+
- **RAM**: 8GB (16GB recommended)
- **Storage**: 50GB free space
- **OS**: macOS 10.15+, Ubuntu 20.04+, Windows 10+ (WSL2)

#### **Recommended Requirements**
- **CPU**: 8+ cores, 3.0GHz+
- **RAM**: 32GB+ (for the larger generation models)
- **Storage**: 200GB+ SSD
- **GPU**: NVIDIA GPU with 8GB+ VRAM (optional; Apple Silicon uses MPS)

Disk budget for the defaults: `qwen3.5:9b` and `qwen3.5:4b` in Ollama, plus
`microsoft/harrier-oss-v1-0.6b` (~1.2GB) in the HuggingFace cache
(`~/.cache/huggingface`). Reranking is on by default, but the reranker loads
lazily — `Qwen/Qwen3-Reranker-4B` (~7.5GB) is fetched on the first reranked
query, not at startup.

### 1.2 Common Dependencies

**Required for both approaches:**
- **Ollama**: model runtime (always required)
- **Git**: 2.30+ for cloning the repository

**Docker-specific:**
- **Docker Desktop**: 24.0+ with the Compose plugin

**Direct Development-specific:**
- **Python**: 3.10+ (3.11 recommended — the Docker images use `python:3.11-slim`)
- **Node.js**: 20+ with npm

---

## 2. Ollama Installation (Required for Both)

### 2.1 Install Ollama

#### **macOS/Linux:**
```bash
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Verify installation
ollama --version
```

#### **Windows:**
```bash
# Download from: https://ollama.ai/download
# Run the installer and follow setup wizard
```

### 2.2 Pull the Models

```bash
# Start Ollama server
ollama serve

# In another terminal, install the two default models
ollama pull qwen3.5:9b      # answer generation
ollama pull qwen3.5:4b      # routing, triage, decomposition, enrichment, verification

# Verify models are installed
ollama list

# Test Ollama
ollama run qwen3.5:4b "Hello, how are you?"
```

The embedding and reranker models are **not** Ollama models — they are downloaded
from HuggingFace automatically the first time the pipeline loads them.

**⚠️ Important**: Keep Ollama running (`ollama serve`) for the entire setup process.

---

## 3. 🐳 Docker Installation & Setup

### 3.1 Install Docker

#### **macOS:**
```bash
# Install Docker Desktop via Homebrew
brew install --cask docker

# Or download from: https://www.docker.com/products/docker-desktop/
# Start Docker Desktop from Applications

# Verify installation
docker --version
docker compose version
```

#### **Ubuntu/Debian:**
```bash
# Update system
sudo apt-get update

# Install Docker using convenience script
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Add user to docker group
sudo usermod -aG docker $USER
newgrp docker

# Install Docker Compose V2
sudo apt-get install docker-compose-plugin

# Verify installation
docker --version
docker compose version
```

#### **Windows:**
1. Download Docker Desktop from https://www.docker.com/products/docker-desktop/
2. Run installer and enable WSL 2 integration
3. Restart computer and start Docker Desktop
4. Verify in PowerShell: `docker --version`

### 3.2 Clone and Start

```bash
# Clone repository
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Verify Ollama is running on the host
curl http://localhost:11434/api/tags

# Start Docker containers (uses local Ollama)
./start-docker.sh

# Or, without host Ollama:
#   ./start-docker.sh container
# For scripts and CI, add -y so the script never prompts:
#   ./start-docker.sh local -y     (or: NONINTERACTIVE=1 ./start-docker.sh)

# Verify deployment
./start-docker.sh status
```

The first build compiles the frontend and installs the Python dependencies, and
`rag-api` loads the embedding model before it reports healthy (the reranker loads
lazily, on the first reranked query). The
`backend` service has `depends_on: rag-api: service_healthy`, so it deliberately
waits. Expect several minutes on a cold start.

### 3.3 Test Docker Deployment

```bash
# Test all endpoints
curl -f http://localhost:3000 && echo "✅ Frontend OK"
curl -f http://localhost:8000/health && echo "✅ Backend OK"
curl -f http://localhost:8001/health && echo "✅ RAG API OK"
curl -f http://localhost:11434/api/tags && echo "✅ Ollama OK"

# Access the application
open http://localhost:3000
```

---

## 4. 💻 Direct Development Setup

### 4.1 Install Development Dependencies

#### **Python Setup:**
```bash
# Clone repository
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Create virtual environment (recommended)
python -m venv venv

# Activate virtual environment
source venv/bin/activate  # macOS/Linux
# venv\Scripts\activate   # Windows

# Install Python dependencies
pip install -r requirements.txt

# Verify Python setup
python -c "import torch; print('✅ PyTorch OK')"
python -c "import transformers; print('✅ Transformers OK')"
python -c "import lancedb; print('✅ LanceDB OK')"
python -c "import docling; print('✅ Docling OK')"
```

`requirements.txt` at the repository root is the full install used by
`run_system.py`. Two variants exist for narrower cases:

| File | Purpose |
|------|---------|
| `requirements.txt` | Everything the local stack needs |
| `requirements-docker.txt` | Used by `Dockerfile.backend` and `Dockerfile.rag-api` (no macOS-only packages) |
| `rag_system/requirements.txt` | RAG core only; adds macOS `ocrmac` and `ibm-watsonx-ai` |
| `backend/requirements.txt` | Gateway only (`requests`, `python-dotenv`) |

#### **Node.js Setup:**
```bash
# Install Node.js dependencies
npm install

# Verify Node.js setup
node --version  # Should be 20+
npm --version
```

### 4.2 Start Direct Development

```bash
# Ensure Ollama is running
curl http://localhost:11434/api/tags

# Start all components with one command
python run_system.py

# Or start components manually, each from the repository root:
# Terminal 1: python -m rag_system.api_server
# Terminal 2: python backend/server.py
# Terminal 3: npm run dev
```

> Always run from the repository root. `backend/chat_data.db`, `lancedb/`,
> `index_store/` and `shared_uploads/` are resolved relative to the working
> directory, so `cd backend && python server.py` would create a second database at
> `backend/backend/chat_data.db`.

### 4.3 Test Direct Development

```bash
# Deep check: imports, config, LanceDB, embedding model, sample query
python system_health_check.py

# HTTP health check per service
python run_system.py --health

# Test endpoints
curl -f http://localhost:3000 && echo "✅ Frontend OK"
curl -f http://localhost:8000/health && echo "✅ Backend OK"
curl -f http://localhost:8001/health && echo "✅ RAG API OK"
```

---

## 5. Detailed Installation Steps

### 5.1 Repository Setup

```bash
# Clone repository
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Check repository structure
ls -la

# These are created on demand, but you can pre-create them
mkdir -p lancedb index_store shared_uploads logs backend

# Set permissions
chmod -R 755 lancedb index_store shared_uploads
```

The SQLite file is created automatically the first time `ChatDatabase` is
constructed — you do not need to `touch` it.

### 5.2 Configuration

LocalGPT runs with no configuration file. Every variable below has a default that
matches the code. Override them in the shell, in a `.env` at the repository root
(`load_dotenv()` runs on import of `rag_system/main.py` and
`rag_system/factory.py`), or in `docker.env` for containers. `.env.example` ships
the same list.

#### **Environment Variables**

| Variable | Default | Read by |
|----------|---------|---------|
| `OLLAMA_HOST` | `http://localhost:11434` | `rag_system/main.py`, `backend/ollama_client.py` |
| `RAG_API_URL` | `http://localhost:8001` | `backend/server.py` |
| `NEXT_PUBLIC_API_URL` | `http://localhost:8000` | `src/lib/api.ts` (build-time) |
| `NEXT_PUBLIC_RAG_API_URL` | `http://localhost:8001` | `src/lib/api.ts` (build-time) |
| `DB_PATH` | `backend/chat_data.db` | `backend/database.py` |
| `LANCEDB_PATH` | `./lancedb` | `rag_system/main.py` (pipeline profiles), `backend/database.py`, `system_health_check.py` |
| `GENERATION_MODEL` | `qwen3.5:9b` | `rag_system/main.py`, `backend/server.py`, `run_system.py` |
| `ENRICHMENT_MODEL` | `qwen3.5:4b` | same |
| `EMBEDDING_MODEL` | `microsoft/harrier-oss-v1-0.6b` | `rag_system/main.py` |
| `RERANKER_MODEL` | `Qwen/Qwen3-Reranker-4B` (loaded lazily on the first reranked query) | `rag_system/main.py` |
| `RAG_CONFIG_MODE` | `default` | `rag_system/api_server.py` (`default` or `fast`) |
| `RAG_API_TIMEOUT` | `600` | `backend/server.py` |
| `RAG_API_INDEX_TIMEOUT` | `3600` | `backend/server.py` |
| `LLM_BACKEND` | `ollama` | `rag_system/main.py` (`ollama` or `watsonx`) |
| `HF_TOKEN` | unset | HuggingFace downloads |

For Docker these are set in `docker.env` and passed with
`docker compose --env-file docker.env`:

```bash
OLLAMA_HOST=http://host.docker.internal:11434
NODE_ENV=production
RAG_API_URL=http://rag-api:8001
NEXT_PUBLIC_API_URL=http://localhost:8000
NEXT_PUBLIC_RAG_API_URL=http://localhost:8001
DB_PATH=/app/backend/chat_data.db
LANCEDB_PATH=/app/lancedb
GENERATION_MODEL=qwen3.5:9b
ENRICHMENT_MODEL=qwen3.5:4b
EMBEDDING_MODEL=microsoft/harrier-oss-v1-0.6b
RERANKER_MODEL=Qwen/Qwen3-Reranker-4B
```

`run_system.py` does **not** invent environment variables. It inherits your shell
environment unchanged and only adds `NODE_ENV=production` to the Python services
in `--mode prod`. If you want a non-default `OLLAMA_HOST` or `RAG_API_URL`, export
it before launching.

`NEXT_PUBLIC_*` values are inlined into the frontend bundle by `next build`, so
they are build-time settings. In Docker they are passed as build args in
`docker-compose.yml`; changing them means `docker compose build frontend`.

#### **Model Configuration**

| Role | Default | Documented options |
|------|---------|--------------------|
| Generation | `qwen3.5:9b` | `qwen3.6:27b` (high-end, ~17GB), `qwen3.5:4b` (light) |
| Enrichment / utility | `qwen3.5:4b` | `qwen3.5:2b` (light) |
| Embedding | `microsoft/harrier-oss-v1-0.6b` (MIT, 1024 dims) | `Qwen/Qwen3-Embedding-4B` (2560 dims, 32K context), `Qwen/Qwen3-Embedding-0.6B` (1024 dims) |
| Reranker (**on by default**) | `Qwen/Qwen3-Reranker-4B`, loaded lazily by the in-repo `QwenRerankerScorer` | `BAAI/bge-reranker-v2-m3`, `answerdotai/answerai-colbert-small-v1`, `Qwen/Qwen3-Reranker-0.6B` |

Notes:
- Embedding dimensions are read from the loaded model, never hardcoded.
  **Changing `EMBEDDING_MODEL` requires rebuilding every existing index** —
  appending vectors of a different width to a LanceDB table raises an explicit
  error telling you to re-index.
- If the reranker fails to load, the pipeline logs a warning and continues
  **without reranking**. There is no secondary reranker to fall back to.
- Any name containing `/` is treated as a HuggingFace model; anything else is
  treated as an Ollama tag, so an Ollama embedding model such as
  `nomic-embed-text` also works if you have pulled it.
- Vision / multimodal models are not wired into any pipeline. PDF parsing and OCR
  are handled by Docling.

### 5.3 Database Initialization

```bash
# Initialize SQLite database (also happens automatically at first use)
python -c "
from backend.database import ChatDatabase
db = ChatDatabase()
print('✅ Database initialized at', db.db_path)
"

# Verify database
sqlite3 backend/chat_data.db ".tables"
```

---

## 6. Verification & Testing

### 6.1 System Health Checks

#### **Comprehensive Health Check:**
```bash
# For Docker deployment
./start-docker.sh status
docker compose ps

# For Direct development
python system_health_check.py
python run_system.py --health

# Universal health check
curl -f http://localhost:3000 && echo "✅ Frontend OK"
curl -f http://localhost:8000/health && echo "✅ Backend OK"
curl -f http://localhost:8001/health && echo "✅ RAG API OK"
curl -f http://localhost:11434/api/tags && echo "✅ Ollama OK"
```

#### **RAG System Test:**
```bash
# Test RAG system initialization (factory.py is the single entry point)
python -c "
from rag_system.factory import get_agent
agent = get_agent('default')
print('✅ RAG System initialized successfully')
"

# Test embedding generation and report the real dimension
python -c "
from rag_system.factory import get_agent
agent = get_agent('default')
embedder = agent.retrieval_pipeline._get_text_embedder()
test_emb = embedder.create_embeddings(['Hello world'])
print(f'✅ Embedding generated: {test_emb.shape}')
"
```

### 6.2 Functional Testing

#### **Document Upload Test:**
1. Access http://localhost:3000
2. Click "Create New Index"
3. Upload a PDF document
4. Configure settings and build index
5. Test chat functionality

#### **API Testing:**
```bash
# Test session creation
curl -X POST http://localhost:8000/sessions \
  -H "Content-Type: application/json" \
  -d '{"title": "Test Session"}'

# Test models endpoints
curl http://localhost:8000/models
curl http://localhost:8001/models

# Test health endpoints
curl http://localhost:8000/health
curl http://localhost:8001/health
```

---

## 7. Troubleshooting Installation

### 7.1 Common Issues

#### **Ollama Issues:**
```bash
# Ollama not responding
curl http://localhost:11434/api/tags

# If it fails, restart Ollama
pkill ollama
ollama serve

# Reinstall models if needed
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
```

#### **Docker Issues:**
```bash
# Docker daemon not running
docker version

# Restart Docker Desktop (macOS/Windows)
# Or restart docker service (Linux)
sudo systemctl restart docker

# Clear Docker cache if build fails
docker system prune -f
```

#### **Python Issues:**
```bash
# Check Python version
python --version  # 3.10+ required, 3.11 recommended

# Check virtual environment
which python
pip list | grep torch

# Reinstall dependencies
pip install -r requirements.txt --force-reinstall
```

#### **Node.js Issues:**
```bash
# Check Node version
node --version  # Should be 20+

# Clear and reinstall
rm -rf node_modules package-lock.json
npm install
```

#### **Scanned PDFs produce no text:**
A PDF with no text layer is re-run through Docling's OCR pipeline, but only with an
engine that is actually installed. `rag_system/ingestion/document_converter.py`
probes, in order: `ocrmac` (macOS only), `easyocr`, `rapidocr` (or the older
`rapidocr_onnxruntime` package name — either is accepted),
`tesserocr`, then the `tesseract` binary. Install whichever suits your platform —
`pip install ocrmac` on macOS (it is listed in `rag_system/requirements.txt` but not
in the root `requirements.txt`), or `pip install easyocr` / `apt install tesseract-ocr`
elsewhere. At startup the converter prints the engine it chose (`OCR engine: …`) or
`No OCR engine available; using docling's default OCR settings.`

### 7.2 Performance Issues

#### **Memory Problems:**
```bash
# Check system memory
free -h  # Linux
vm_stat  # macOS

# For Docker: Increase memory allocation
# Docker Desktop → Settings → Resources → Memory → 8GB+

# Use lighter models
export GENERATION_MODEL=qwen3.5:4b
# (the default embedder is already the small one, ~1.2GB)
```

#### **Slow Performance:**
- Use SSD storage for `lancedb/` and `shared_uploads/`
- Switch to `RAG_CONFIG_MODE=fast` (vector-only retrieval, no reranking,
  no decomposition, no verification)
- Lower `indexing.embedding_batch_size` / `enrichment_batch_size` if you are
  swapping rather than compute-bound
- Remember the RAG API serialises requests — one query or indexing run at a time

---

## 8. Post-Installation Setup

### 8.1 Model Experiments

```bash
# Install additional generation models
ollama pull qwen3.6:27b            # highest quality, ~17GB
ollama pull qwen3.5:2b             # lightest utility model

# Try one for a single request
curl -X POST http://localhost:8001/chat \
  -H "Content-Type: application/json" \
  -d '{"query": "Hello", "model": "qwen3.5:4b"}'
```

A per-request `model` is applied only for that request and then restored, and is
rejected with a warning if it is not valid for the active `LLM_BACKEND`.

### 8.2 Security Configuration

Neither server implements authentication, and both send
`Access-Control-Allow-Origin: *`. Do not expose ports 8000/8001 outside a trusted
network.

```bash
# Set proper file permissions
chmod 600 backend/chat_data.db    # Restrict database access
chmod 700 lancedb/                # Restrict vector DB access

# Configure firewall (production)
sudo ufw allow 3000/tcp           # Frontend
sudo ufw deny 8000/tcp            # Backend (internal only)
sudo ufw deny 8001/tcp            # RAG API (internal only)
```

### 8.3 Backup Setup

```bash
# Create backup script
cat > backup_system.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="backups/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# Backup databases and indexes
cp -r backend/chat_data.db "$BACKUP_DIR/"
cp -r lancedb "$BACKUP_DIR/"
cp -r index_store "$BACKUP_DIR/"
cp -r shared_uploads "$BACKUP_DIR/"

echo "Backup completed: $BACKUP_DIR"
EOF

chmod +x backup_system.sh
```

All four paths are plain host directories (bind-mounted into the containers), so
a file copy is a complete backup. Stop the services first so SQLite and LanceDB
are not mid-write.

---

## 9. Success Criteria

### 9.1 Installation Complete When:

- ✅ `python run_system.py --health` exits 0
- ✅ Frontend loads at http://localhost:3000
- ✅ `ollama list` shows `qwen3.5:9b` and `qwen3.5:4b`
- ✅ You can create a document index
- ✅ You can chat with uploaded documents
- ✅ No error messages in `logs/` or the terminal

### 9.2 Performance Expectations

Numbers depend heavily on hardware, model size and document length. On a machine
matching the recommended requirements, expect:

- First startup dominated by model downloads (Ollama tags + the ~1.2GB embedding
  model from HuggingFace); subsequent startups load from cache. The reranker
  (~7.5GB) is downloaded separately, lazily on the first reranked query
- Indexing dominated by contextual enrichment — it runs one LLM call per chunk, so
  turn it off (`enable_enrich: false`) for the fastest ingest
- Query latency dominated by generation; `RAG_CONFIG_MODE=fast` removes reranking,
  decomposition and verification

---

## 10. Next Steps

### 10.1 Getting Started

1. **Upload Documents**: Create your first index
2. **Explore Features**: Try different retrieval modes and models
3. **Customize**: Adjust chunk size, enrichment and verification per request
4. **Scale**: Add more documents and create multiple indexes

### 10.2 Additional Resources

- **Quick Start**: See `Documentation/quick_start.md`
- **Docker Usage**: See `Documentation/docker_usage.md`
- **Deployment**: See `Documentation/deployment_guide.md`
- **System Architecture**: See `Documentation/architecture_overview.md`
- **API Reference**: See `Documentation/api_reference.md`
- **WatsonX backend**: See `WATSONX_README.md`

---

**Congratulations! 🎉** Visit http://localhost:3000 to start chatting with your documents.

---

### Documentation Prompt Inventory (`Documentation/prompt_inventory.md`)

# 📜 Prompt Inventory (Ground-Truth)

_Every prompt hard-coded in the codebase, re-derived from the current source._

> Edit process: if you change a prompt in code, update the line range here in the same commit.
> Line numbers drift as the code moves; the function and variable names are the stable references.

## Which model runs which prompt

There are two model roles (`rag_system/main.py` `OLLAMA_CONFIG`):

| Role | Config key | Default | Env override |
|------|-----------|---------|--------------|
| Generation — user-facing answers | `generation_model` | `qwen3.5:9b` | `GENERATION_MODEL` |
| Utility — routing, triage, decomposition, enrichment, overviews, verification | `enrichment_model` | `qwen3.5:4b` | `ENRICHMENT_MODEL` |

Inside the agent the utility model is resolved by `Agent._utility_model()` (`rag_system/agent/loop.py:56-58`), which returns `ollama_config["enrichment_model"]` and falls back to `generation_model` when that key is absent. No prompt hard-codes a model name.

---

## 1. Indexing / context enrichment

| ID | File & lines | Variable / builder | Model | Purpose |
|----|--------------|--------------------|-------|---------|
| `overview_builder.default` | `rag_system/indexing/overview_builder.py` `13-19` | `OverviewBuilder.DEFAULT_PROMPT` | overview model (resolved at `indexing_pipeline.py:128-133`; utility model by default) | One-paragraph document overview used by the triage routers. Input is the first `first_n_chunks` chunks (default 5), truncated to 5000 characters at `overview_builder.py:37`. |
| `contextualizer.system` | `rag_system/indexing/contextualizer.py` `12` | `SYSTEM_PROMPT` | enrichment model | Role instruction for the summariser. |
| `contextualizer.local_context` | same file `14-16` | `LOCAL_CONTEXT_PROMPT_TEMPLATE` | — | Wraps the neighbouring-chunk window in `<local_context>` tags. |
| `contextualizer.chunk` | same file `18-26` | `CHUNK_PROMPT_TEMPLATE` | — | Shows the target chunk and carries the actual instruction: a 2-5 sentence context summary, "Answer *only* with the succinct context and nothing else." |

The three contextualizer parts are concatenated into a single `/api/generate` completion prompt at `contextualizer.py:42-51` (no chat roles) and sent at `contextualizer.py:53` with `enable_thinking=False`.

## 2. Retrieval / query transformation

| ID | File & lines | Model | Purpose |
|----|--------------|-------|---------|
| `query_transformer.decompose` | `rag_system/retrieval/query_transformer.py` `38-84` (system) + `87-196` (few-shot examples) + `250-265` (assembly) | utility model (`loop.py:30`) | Resolve pronouns/ellipsis against the last 5 turns, then split the query into standalone sub-queries. Returns RFC-8259 JSON `{requires_decomposition, reasoning, resolved_query, sub_queries}`; sent with `format="json"` at `:268`; the list is deduplicated and capped by `max_sub_queries` at `:290` (default 10). |
| `query_transformer.decompose_single_turn` | `rag_system/retrieval/query_transformer.py`, `QueryDecomposer._decompose_single_turn` | utility model | The byte-frozen single-turn decomposer — the default path for history-less queries, and the prompt every single-turn bench number was measured against (arm C..K). It must stay byte-identical: even cosmetic edits (smart quotes, added examples) were measured to shift temp-0 decompositions on the gold set (arm L, 2026-08-16); any change re-triggers the full 5-bench gate. |

| `retrieval.retry.reformulate` | `rag_system/pipelines/retrieval_pipeline.py`, `_reformulate_query` | enrichment/utility model, `format="json"` | Fires **only** when the evidence-sufficiency retry triggers (roadmap 2.1). Asks for one rewrite of a weak-evidence query into the concrete nouns and synonyms a document would use, preserving every entity and constraint. Returns `{"query": "…"}`; `format="json"` is what keeps a small model's thinking preamble out of the rewritten query. |

Two legacy example blocks still live in the file at `199-203` and `205-248`, but they are excluded from the assembled prompt — the two lines that would concatenate them are commented out at `:253-254`.

## 3. Answer synthesis

| ID | File & lines | Model | Purpose |
|----|--------------|-------|---------|
| `retrieval_pipeline.synth_final` | `rag_system/pipelines/retrieval_pipeline.py`, `_synthesize_final_answer` | generation model | Turn the retrieved snippets into the final answer (7 numbered hard rules; instructs the model to reply exactly "I could not find that information in the provided documents." when the snippets do not cover the question; rule 7 covers the `[Source document: …]` line each snippet carries). Streamed token-by-token via `stream_completion`; each token is forwarded to the `event_callback` as a `token` event. |

## 4. Agent loop (`rag_system/agent/loop.py`)

| ID | Lines | Model | Purpose |
|----|-------|-------|---------|
| `agent.loop.history_wrapper` | `163-171` | none — template only | `_format_query_with_history` builds the `contextual_query` string that is embedded into downstream prompts. It makes no LLM call. |
| `agent.loop.overview_router` | `615-630` | utility model (call at `632-634`, `format="json"`) | First routing pass. Interpolates the loaded document overviews (first 40, `:612-613`) under a `DOCUMENT OVERVIEWS:` header and returns `{"category": "direct_answer"}` or `{"category": "rag_query"}`. |
| `agent.loop.triage_fallback` | `agent/loop.py`, `_triage_query_async` | utility model, `format="json"` | Last-resort routing. Reached only when the overview router returns `None` (no overviews loaded) **and** there is no chat history. Two-way vocabulary: `rag_query` / `direct_answer`; `_normalize_triage()` maps anything else to `rag_query`. |
| `agent.loop.direct_answer` | `331-336` | generation model (streamed at `342-344`) | Answers on the `direct_answer` route from conversation history or general knowledge. Caps the reply at 1-2 sentences. |
| `agent.loop.compose_sub` | `490-515` | generation model (streamed at `519-522`) | Compose one final answer from the JSON list of sub-question/sub-answer pairs. Used when `query_decomposition.compose_from_sub_answers` is true and decomposition produced more than one sub-query. |

## 5. Verifier

| ID | File & lines | Model | Purpose |
|----|--------------|-------|---------|
| `verifier.fact_check` | `rag_system/agent/verifier.py` `25-85` | utility model (`loop.py:29`) | Grounding check with three few-shot examples and a `# TASK` block. The prompt is built in four appends: the base f-string ends at `:75`, the context is appended clamped to 4000 characters at `:76`, the answer at `:81`, and the `<OUTPUT>` tag at `:82-85`. Sent asynchronously with `format="json"`. Verdict labels: `SUPPORTED` / `NOT_SUPPORTED` / `NEEDS_CLARIFICATION`. **Skipped entirely when `VERIFIER_MODEL` / `verification.model` names a local NLI verifier** — that backend makes no LLM call at all (roadmap 2.4, `verifier.md`). |

## 6. Backend router (fast path) — removed

The gateway's old "Respond with exactly one word: USE_RAG or DIRECT_LLM" LLM router prompt was deleted in Phase 2.3. Gateway routing is now the deterministic no-LLM gate `should_use_rag` (`backend/server.py`), which makes no model call at all, so there is no backend prompt to inventory.

---

### Notes

* `rag_system/utils/watsonx_client.py:222` contains the string `prompt="What is AI?"`, but it is literal text inside a `print()` usage banner — it is never sent to a model and is therefore not inventoried.
* There is no prompt registry module; every prompt above is an inline literal at the cited location.
* There is no ReAct-style think/act/observe prompt anywhere. The agent's stages are triage → (optional) decomposition → retrieval → rerank → expand → prune → synthesis → verification.
* The eval judge prompts (`eval/judge.py`) exist but are out of scope of this runtime inventory.

---

### Documentation Quick Start (`Documentation/quick_start.md`)

# ⚡ Quick Start Guide - LocalGPT

_Get up and running in about 10 minutes (plus model download time)._

---

## 🚀 Choose Your Deployment Method

### Option 1: Docker Deployment 🐳

Best for: isolated environments, running the same stack everywhere.

### Option 2: Direct Development 💻

Best for: development, customization, debugging, faster iteration.

Both need Ollama. The Docker path runs Ollama on the host by default (better GPU
access), but can also run it as a container.

---

## 🐳 Docker Deployment

### Prerequisites
- Docker Desktop (or Docker Engine 24+ with the Compose plugin) installed and running
- 8GB+ RAM available
- Internet connection (first build downloads Python and Node packages; first query
  downloads the embedding model from HuggingFace; the reranker follows on first use)

### Step 1: Clone

```bash
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Ensure Docker is running
docker version
```

### Step 2: Install Ollama Locally

**By default the containers talk to Ollama on the host:**

```bash
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Start Ollama (in one terminal)
ollama serve

# Install models (in another terminal)
ollama pull qwen3.5:9b     # answer generation
ollama pull qwen3.5:4b     # routing, triage, enrichment, verification
```

Prefer not to install Ollama on the host? Skip this step and use
`./start-docker.sh container` below.

### Step 3: Start Docker Containers

```bash
# Start all containers against local Ollama
./start-docker.sh

# Or manually:
docker compose --env-file docker.env up --build -d
```

Containerized Ollama instead:

```bash
./start-docker.sh container
# Pull the models inside the container the first time
docker compose --profile with-ollama exec ollama ollama pull qwen3.5:9b
docker compose --profile with-ollama exec ollama ollama pull qwen3.5:4b
```

`./start-docker.sh` with no argument checks port 11434. If nothing is listening it
offers to switch to the containerized Ollama; pass `-y` (or set `NONINTERACTIVE=1`)
to accept that without a prompt in scripts.

### Step 4: Verify Deployment

```bash
# Check container status (backend waits for rag-api to report healthy)
docker compose ps

# Test endpoints
curl http://localhost:3000            # Frontend
curl http://localhost:8000/health     # Backend
curl http://localhost:8001/health     # RAG API
```

The `rag-api` container loads the agent and the embedding model at startup, so its
health check has a 120s start period and the first `docker compose up` can take
several minutes before `backend` starts. The reranker (~7.5 GB) is fetched lazily on
the first query that reranks — expect one slow first answer.

### Step 5: Access Application

Open your browser to: **http://localhost:3000**

---

## 💻 Direct Development

### Prerequisites
- Python 3.10+ (3.11 recommended)
- Node.js 20+ and npm
- 8GB+ RAM available

### Step 1: Clone and Install Dependencies

```bash
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT

# Install Python dependencies
pip install -r requirements.txt

# Install Node.js dependencies
npm install
```

### Step 2: Install and Configure Ollama

```bash
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Start Ollama (in one terminal)
ollama serve

# Install models (in another terminal)
ollama pull qwen3.5:9b
ollama pull qwen3.5:4b
```

### Step 3: Start the System

```bash
# Start all components with one command
python run_system.py
```

`run_system.py` reuses an already-running Ollama, pulls any missing model, then
starts the RAG API, the backend and the frontend.

**Or start components manually in separate terminals — all from the repository root:**

```bash
# Terminal 1: RAG API
python -m rag_system.api_server

# Terminal 2: Backend
python backend/server.py

# Terminal 3: Frontend
npm run dev
```

> Do not `cd backend` first. The SQLite path defaults to `backend/chat_data.db`
> relative to the working directory, so running from inside `backend/` creates a
> second database at `backend/backend/chat_data.db`.

### Step 4: Verify Installation

```bash
# Loads the models and runs a sample query against the first LanceDB table
python system_health_check.py

# HTTP health check per service (exits non-zero if a required one is unhealthy)
python run_system.py --health

# Test endpoints
curl http://localhost:3000            # Frontend
curl http://localhost:8000/health     # Backend
curl http://localhost:8001/health     # RAG API
```

### Step 5: Access Application

Open your browser to: **http://localhost:3000**

---

## 🎯 First Use Guide

### 1. Create a Chat Session
- Click "New Chat" in the interface
- Give your session a descriptive name

### 2. Upload Documents
- Click "Create New Index"
- Upload PDF, DOCX, TXT, MD or HTML files
- Configure processing options:
  - **Chunk Size**: 512 (default)
  - **Embedding Model**: `microsoft/harrier-oss-v1-0.6b` (default)
  - **Enable Enrichment**: Yes
- Click "Build Index" and wait for processing

Indexing is synchronous: the request stays open until the pipeline finishes.

### 3. Start Chatting
- Select your built index
- Ask questions about your documents:
  - "What is this document about?"
  - "Summarize the key points"
  - "What are the main findings?"
  - "Compare the arguments in section 3 and 5"

The chat settings panel exposes the same knobs the API does: search type
(`hybrid` / `vector_only` / `fts_only`), retrieval_k, reranker top-k, context
window, decomposition, verification, Provence pruning, and "Stream phases".

> **Stream phases is on by default**, and that path streams straight from the RAG
> API to the browser. When the stream completes, the UI saves the finished turn
> through the gateway (`POST /sessions/{id}/messages/save`), so the conversation
> **is** written to the chat history database. Only direct (non-UI) stream
> consumers must save the turn themselves.

---

## 🔧 Management Commands

### Docker Commands

```bash
# Container management
./start-docker.sh                    # Start (local Ollama)
./start-docker.sh container          # Start (containerized Ollama)
./start-docker.sh stop               # Stop all containers
./start-docker.sh logs               # View logs
./start-docker.sh status             # Check status
./start-docker.sh help               # Usage

# Manual Docker Compose
docker compose ps                    # Check status
docker compose logs -f               # Follow logs
docker compose down                  # Stop containers
docker compose --env-file docker.env up --build -d   # Rebuild and start
```

### Direct Development Commands

```bash
# System management
python run_system.py                # Start all services
python run_system.py --mode prod    # `npm run build` then `next start`
python run_system.py --no-frontend  # Ollama + RAG API + backend only
python run_system.py --health       # HTTP health checks
python run_system.py --logs-only    # Tail logs/*.log from another shell
python run_system.py --stop         # Stop everything in logs/run_system.pid
python system_health_check.py       # Deep check: models, LanceDB, sample query

# Individual components (from the repository root)
python -m rag_system.api_server     # RAG API only
python backend/server.py            # Backend only
npm run dev                          # Frontend only

# Stop: Ctrl+C in the terminal running the services, or `python run_system.py --stop`
```

---

## 🆘 Quick Troubleshooting

### Docker Issues

**Containers not starting?**
```bash
# Check Docker daemon
docker version

# The backend will not start until rag-api reports healthy
docker compose ps
docker compose logs -f rag-api
```

**Port conflicts?**
```bash
# Check what's using ports
lsof -i :3000 -i :8000 -i :8001

# Stop conflicting processes
./start-docker.sh stop
```

### Direct Development Issues

**Import errors?**
```bash
# Check Python installation
python --version  # 3.10+ required, 3.11 recommended

# Reinstall dependencies
pip install -r requirements.txt --force-reinstall
```

**Node.js errors?**
```bash
# Check Node version
node --version    # Should be 20+

# Reinstall dependencies
rm -rf node_modules package-lock.json
npm install
```

### Common Issues

**Ollama not responding?**
```bash
# Check if Ollama is running
curl http://localhost:11434/api/tags

# Restart Ollama
pkill ollama
ollama serve
```

**Out of memory?**
```bash
# Check memory usage
docker stats  # For Docker
htop          # For direct development

# Use lighter models
export GENERATION_MODEL=qwen3.5:4b
# (the default embedder is already the small one, ~1.2GB)
```

**Answers ignore my documents?**
Check that the session is linked to a built index — the backend only routes to the
RAG API when the session has one. Sending `"force_rag": true` on
`POST /sessions/{id}/messages` bypasses the router.

---

## 📊 System Verification

```bash
# Check all endpoints
curl -f http://localhost:3000 && echo "✅ Frontend OK"
curl -f http://localhost:8000/health && echo "✅ Backend OK"
curl -f http://localhost:8001/health && echo "✅ RAG API OK"
curl -f http://localhost:11434/api/tags && echo "✅ Ollama OK"

# For Docker: Check containers
docker compose ps
```

---

## 🎉 Success!

If you see:
- ✅ All services responding
- ✅ Frontend accessible at http://localhost:3000
- ✅ No error messages

You're ready to start using LocalGPT!

### What's Next?

1. **📚 Upload Documents**: Add files to create an index
2. **💬 Start Chatting**: Ask questions about your documents
3. **🔧 Customize**: Try `RAG_CONFIG_MODE=fast`, different models, other retrieval modes
4. **📖 Learn More**: Check the documentation below

### 📁 Key Files

```
localGPT/
├── 🐳 start-docker.sh           # Docker deployment script
├── 🏃 run_system.py             # Direct development launcher
├── 🩺 system_health_check.py    # Deep system verification
├── 🛠️ create_index_script.py    # Interactive / batch index creation
├── 📋 requirements.txt          # Python dependencies
├── 📦 package.json              # Node.js dependencies
├── ⚙️  .env.example              # Every environment variable with its default
├── 📁 Documentation/            # Complete documentation
├── 📁 rag_system/               # RAG API, agent, pipelines (config in main.py)
├── 📁 backend/                  # Gateway server + SQLite database
└── 📁 src/                      # Next.js frontend
```

### 📖 Additional Resources

- **🏗️ Architecture**: See `Documentation/architecture_overview.md`
- **🔧 Configuration**: See `Documentation/system_overview.md`
- **🚀 Deployment**: See `Documentation/deployment_guide.md`
- **🐳 Docker**: See `Documentation/docker_usage.md`
- **🐛 Troubleshooting**: See `DOCKER_TROUBLESHOOTING.md`

---

## 🛠️ Indexing Without the UI

### Built-in CLI

```bash
# Index a file or a whole directory with the 'default' profile
python -m rag_system.main index ./my_documents

# Speed-optimised profile
python -m rag_system.main index ./my_documents --mode fast

# Ask one question and print the JSON result
python -m rag_system.main chat "What are the key findings?" --mode default

# Start the RAG API (same as `python -m rag_system.api_server`)
python -m rag_system.main api --port 8001
```

`index` walks a directory for `.pdf`, `.docx`, `.html`, `.htm`, `.md` and `.txt`.
It writes into the profile's shared table (`text_pages_v4`), not into a per-index
table, so indexes built this way are not listed in the web UI.

### Interactive / Batch Script

For an index the web UI can see, use `create_index_script.py` — it creates the
database row, uploads the document records and writes to `text_pages_<index_id>`:

```bash
# Guided prompts
python create_index_script.py

# Write a template, edit it, then run it
python create_index_script.py --create-sample     # writes index_config.sample.json
python create_index_script.py --batch index_config.sample.json

# Use a custom pipeline config instead of PIPELINE_CONFIGS["default"]
python create_index_script.py --config my_pipeline.json
```

The batch file looks like this — replace the placeholder paths with your own
absolute paths:

```json
{
  "index_name": "Sample Batch Index",
  "index_description": "Example batch index configuration",
  "documents": [
    "/absolute/path/to/first.pdf",
    "/absolute/path/to/second.pdf"
  ],
  "processing": {
    "chunk_size": 512,
    "enable_enrich": true,
    "enable_latechunk": true,
    "enable_docling": true,
    "embedding_model": "microsoft/harrier-oss-v1-0.6b",
    "enrich_model": "qwen3.5:4b",
    "retrieval_mode": "hybrid",
    "window_size": 2
  }
}
```

Both paths:
- ✅ Parse documents with Docling (OCR fallback for scanned PDFs)
- ✅ Chunk, optionally enrich, and embed
- ✅ Write vectors plus a native full-text index into LanceDB
- ✅ Generate a document overview used by the query router

The script exits non-zero on failure and deletes the half-created index row.

---

**Happy RAG-ing! 🚀**

---

### Documentation Research Roadmap (`Documentation/research_roadmap.md`)

# Evidence-Based Roadmap

_Created 2026-08-08. Status of every item here is **PLANNED, not implemented**,
except where a row says otherwise — when an item ships, move its row to
`improvement_plan.md`'s completed table and update the component docs. Do not
describe any of this as current behavior._

**Phase 1 is complete (2026-08-09).** Items 1.1, 1.2 and 1.3 were measured and
decided; the adopted defaults, the joint matrix behind them and what stays
opt-in are in [`../eval/DECISIONS.md`](../eval/DECISIONS.md).

This roadmap turns the findings in [`research/`](research/) into staged,
testable changes. Ordering principle, straight from the evidence: **retriever
and reranker quality dominate everything else** (+14.2 pts from an embedder
swap on BrowseComp-Plus, ACL 2026; +17.2 pp MRR@3 from a cross-encoder), every
clever layer on top is conditional, and no upgrade counts until it is measured
on our own eval set (public-leaderboard deltas under ~2 points do not transfer;
ranking inversions on production corpora are documented).

---

## Phase 0 — Evaluation harness (prerequisite; nothing else lands without it)

The single most consistent practitioner finding: generic metrics create false
confidence; teams need a small, owned eval set and binary judgments
(Hamel/Husain–Shankar FAQ; Anthropic "20–50 tasks from real failures is a
great start"; Chroma's generative-benchmarking ranking inversions).

| # | Item | Detail | Acceptance |
|---|------|--------|------------|
| 0.1 | Gold retrieval set | Reverse-generate ~50–100 queries from indexed chunks across 2–3 real corpora + the planted-fact test PDF, using structured dimension tuples (topic × question-type × difficulty), not freeform "give me questions". Store under `eval/goldset/` with the chunk IDs that must be retrieved. | Set committed; every query hand-checked once |
| 0.2 | Retrieval metrics runner | `eval/run_eval.py` hits the RAG API: **recall@k** at first stage, **nDCG@10 after rerank** — the two metrics the evidence says matter. Runs in minutes, no LLM judge needed. | Baseline numbers for the current stack recorded in `eval/BASELINE.md` |
| 0.3 | Groundedness judge | Binary pass/fail LLM judge for answer faithfulness, validated against ~20 hand-labeled answers (report TPR/TNR, not raw agreement). Binary, not Likert, per the near-unanimous practitioner canon. | Judge agrees with hand labels ≥90% before it gates anything |
| 0.4 | End-to-end smoke | Scripted: index the test PDF → 5 planted-fact questions → assert grounding + citation presence + persistence round-trip. Extends the E2E script already exercised on 2026-08-08. | Runs green on the current tree |

**Effort:** ~1 day. **Risk:** low. **Everything below cites Phase 0 numbers in
its acceptance criteria.**

## Phase 1 — Component upgrades (A/B against Phase 0, adopt only on wins)

| # | Item | Evidence | Plan | Risk |
|---|------|----------|------|------|
| 1.1 ✅ **DONE 2026-08-09** — reranking now ships **off**, with `Qwen/Qwen3-Reranker-4B` as the model the toggle loads. _(Superseded 2026-08-14, arm G: reranking now ships **on** by default with `min_score: 0.5` / `min_keep: 3` threshold selection.)_ Decision + numbers: [`../eval/DECISIONS.md`](../eval/DECISIONS.md), evidence: [`../eval/decisions/reranker.md`](../eval/decisions/reranker.md) | **Reranker A/B: Qwen3-Reranker-4B (and 0.6B) vs bge-reranker-v2-m3** | bge-v2 lineage static for 2 years; near-zero on instruction-following (FollowIR −0.01) and weak on code (41.38 MTEB-Code); Qwen3-Reranker-4B leads permissive options (69.76 MTEB-R, Apache 2.0). Reranker is the highest-ROI slot in the stack. | Config-swap via `RERANKER_MODEL`; verify the `rerankers` library loads Qwen3's causal yes/no-logit scoring (it is NOT a SequenceClassification model — may need a small custom scorer). Measure nDCG@10 delta and per-query latency on MPS. | Integration: medium. Throughput regression is expected; accept only if quality delta justifies it, else keep 0.6B or bge |
| 1.2 ✅ **DONE 2026-08-09** — `microsoft/harrier-oss-v1-0.6b` adopted as the default with the query prefix on, in the same re-index window as 1.1 and the two index-format fixes. Decision + numbers: [`../eval/DECISIONS.md`](../eval/DECISIONS.md), evidence: [`../eval/decisions/embedder.md`](../eval/decisions/embedder.md) | **Embedder audit + A/B: instruction prefix, then harrier-oss-v1 vs Qwen3-Embedding** | All top-2026 embedders are instruction-tuned (+1–5% from the query-side prefix alone); microsoft/harrier-oss-v1 (MIT, Mar 2026) beats Qwen3-Embedding at equal size (0.6B: 69.0 vs 64.3 MMTEB). | First verify our query path sends the `Instruct: … Query: …` prefix (free win if missing). Then A/B harrier-0.6b and Qwen3-4B on the gold set. **Changing the default forces re-indexing** — decide once, alongside 1.1, in a single index-format window. | Low code risk; migration cost is the re-index. harrier is decoder/last-token-pooling like Qwen3, so `embedders.py` should need at most a config entry |
| 1.3 | **GLM-OCR behind Docling for scanned/complex PDFs** — _spike complete 2026-08-09: **GO-LATER**, see `eval/decisions/glm-ocr-spike.md`_ | 0.9B MIT specialist VLM; **95.22 OmniDocBench v1.6_full (third, behind PaddleOCR-VL-1.6 96.34 and MinerU2.5-Pro 95.75)** — an earlier "#1 / beats GPT-5.2 by ~10 pts" framing from the research sweep did not survive source verification and is retracted. Spike proved the serving path on Apple Silicon: official `ollama pull glm-ocr` (2.2GB), and our pinned docling 2.118.1 already ships a `glm_ocr` preset with an Ollama override. Parse quality on a degraded scanned invoice: 30/30 table cells vs the current chain losing every price. | Blocked on three items before adoption: (a) Ollama's Modelfile ignores prompts, so GLM-OCR's table/formula modes are unreachable there; (b) deterministic double-transcription on some pages; (c) docling flattens its pipe tables (`tables: 0`). Prerequisite: add a scanned/tabular corpus with ground truth to `eval/` — nothing in Phase 0 exercises the OCR branch — and A/B GLM-OCR against docling's other 2026 presets (`lightonocr`, `dots_ocr`, `nanonets_ocr2`…), not as a foregone winner. | Cheaper wins shipped first: the RapidOCR probe bug is fixed (stale module name sent every scan to Tesseract CLI), and `pip install ocrmac` gives macOS the best classic chain |

**Effort:** ~2–3 days including eval runs. **Decision gate:** adopt each only
on a measured win; record adopted/rejected + numbers in `eval/DECISIONS.md`.
**Gate cleared 2026-08-09** — see [`../eval/DECISIONS.md`](../eval/DECISIONS.md).
1.1 and 1.2 could not be decided separately (the reranker's value depends on the
first stage), so they were re-measured jointly and shipped in one window
together with the embedder-identity guard and cosine normalization that the 1.2
audit flagged. 1.3 remains GO-LATER: no code was written.

## Phase 2 — Pipeline-shape fixes — _COMPLETE 2026-08-09; measurements in `eval/decisions/phase2-pipeline.md` and `phase2-gateway.md`; Landed rows graduated to improvement_plan.md. Notable: 2.2 measured negative on truly-decomposing queries and ships disabled; 2.1's suggested signal was replaced after measuring anti-correlation. (2.2's "ships disabled" posture was superseded: arm G, 2026-08-14, turned the reranker on by default so rerank-decomposition now applies, and arm H, 2026-08-15, made the pooled first stage the default decomposition path.)_

| # | Item | Evidence | Plan |
|---|------|----------|------|
| 2.1 | **Evidence-sufficiency retry** | One conditional second retrieval iteration captures ~95% of deep-loop gains (+7.1 EM for iteration 1→2 on a local 7B; iterations ≥3 are noise). Terminate on evidence sufficiency, not query count. | If top-k rerank scores fall below a threshold, reformulate once (enrichment model) and retrieve again; hard cap 2 iterations; off in `fast` profile. Surface as a step in the SSE cascade so the UI shows it |
| 2.2 | **Decomposition at rerank, not first-stage** | Decomposition at initial retrieval "frequently harms via semantic dilution"; it helps applied at reranking (2026 finding, MultiConIR/SSRB). | Keep the full query for hybrid retrieval; score candidates against sub-queries during rerank. Touches `retrieval_pipeline.py` only |
| 2.3 | **Cheapen gateway routing** | Pre-retrieval LLM routing is the weakest pattern in the 2026 evidence (four ML approaches failed; TF-IDF+SVM ≥ LLM routers at ~zero cost; fixed-hybrid beat adaptive routing). | Replace the gateway's LLM routing call with a cheap gate (heuristics + optional logit-margin TARG-style check), keep `force_rag`, keep agent-side triage as the single LLM routing layer. Net: one less LLM call per message, simpler failure surface |
| 2.4 | **Dedicated verifier model (optional)** | Verification helps only as an external check; a 4-bit 1B verifier (ThinknCheck, CC0) now beats the 7B 2024 SOTA; Granite Guardian (Apache) is the permissive alternative. Current LLM-prompt verifier works but is slower and uncalibrated. | Add `VERIFIER_MODEL` option: local NLI/verifier model scoring answer-vs-evidence per sentence; keep the LLM prompt as fallback. Present `[Confidence: N%]` as UX, never as a calibrated measurement (document this) |
| 2.5 | **Delete the graph module** | GraphRAG loses single-hop, contested multi-hop gains, 41–57× indexing / up to ~377× query-token cost; unreachable in this repo already (no profile sets `graph_strategy`). | Remove `graph_extractor.py`, `GraphRetriever`, `GraphQueryTranslator` + config remnants; note the decision and evidence in `design_rationale.md` (resolves improvement_plan §9) |

**Effort:** ~2 days. Each item is independently shippable and independently
revertible; each lands with a Phase-0 eval delta.

## Phase 3 — Documentation: make the evidence part of the repo's argument

**3.1 is complete (2026-08-09):** [`design_rationale.md`](design_rationale.md)
ships one section per component with its evidence and its eval number, plus the
"deliberately not implemented" list. 3.2's README/roadmap/docs-index cross-links
landed with it.

| # | Item | Plan |
|---|------|------|
| 3.1 ✅ **DONE 2026-08-09** → [`design_rationale.md`](design_rationale.md) | One section per component: what localGPT does, and the evidence for why (with citations into `research/`). Includes a **"deliberately not implemented"** list — HyDE-by-default, multi-query expansion, weighted fusion knobs, GraphRAG, vendor memory systems, deep subagent fan-out — each with its citation, so future contributors don't re-add deprecated patterns |
| 3.2 | Cross-link | README gets one line pointing at the rationale; `improvement_plan.md` links each open item to its evidence; this roadmap's rows move to improvement_plan as they complete |
| 3.3 | Honesty rule (already in force) | Every phase lands code + docs + eval delta in the same change. Nothing in `design_rationale.md` may describe unshipped behavior — that is what this roadmap file is for |

**Effort:** ~half a day, mostly distillation from the synthesis already written.

## Explicitly out of scope (evidence-negative — revisit only with new evidence)

- **Deep agent loops / parallel subagent fan-out** — pays on open-web breadth-first research at ~15× token budgets; fails silently elsewhere (41.8% of failures at the hand-off).
- **RL-trained searchers** — no out-of-distribution transfer (Search-R1 = its base model, ACL 2026); also mis-calibrates confidence, which would break 2.3's logit gate.
- **Always-on HyDE / multi-query** — measured negative on entity/numeric corpora; multi-query scored below plain BM25.
- **Vendor memory layers** — the null baseline (RAG over the transcript) wins; our session store already is the null baseline.
- **Token-level context compression (LLMLingua-style)** — ≤18% best-case speedup, net-negative outside a narrow window; we already ship the surviving alternative (Provence pruning).

## Sequencing summary

```
Phase 0 (eval harness) ──► Phase 1 (reranker / embedder / parser A/Bs) ──► Phase 2 (pipeline shape) ──► Phase 3 (rationale docs)
     ~1 day                      ~2–3 days                                     ~2 days                      ~0.5 day
     ✅ done                     ✅ done 2026-08-09 (1.3 = GO-LATER)
```

Phase 0 blocks everything. Phases 1 and 2 can interleave per-item, but 1.1/1.2
should conclude before any re-index-requiring release. Phase 3.1 can be drafted
in parallel at any time.

## Phase 4 — Ideas adopted from agentic-file-search (implemented + benchmarked 2026-08-09)

**Phase 4 is complete.** Every item below was implemented flag-gated, benchmarked
on/off, and decided at the adoption gate. Verdicts (evidence in
`eval/decisions/phase4-*.md`):

| # | Verdict | One-line reason |
|---|---------|-----------------|
| 4.1 | **REJECTED as default** — implemented, off (was HOLD) | The HOLD's condition was executed 2026-08-12: with the num_ctx fix in place the lift did not survive — the escalation-off baseline on the identical fire subset went 0/9→7/9 (the "lift" was front-truncation favoring the tail-appended document) and both product-default fires were regressions. Flag and code kept. `eval/decisions/phase4-escalation-rerun.md`. |
| 4.2 | **Extraction ADOPTED (on); hop REJECTED as a default (off, flag kept)** | Extraction is free, regex-only and index-inert. The hop fires 0/11 at the shipped k=20, hits 0/11 expected sources where forced (target selection is query-blind), and raising `k` beats it at equal context budget in 3 of 4 cells. |
| 4.3 | **boost HOLD (off); restrict REJECTED** | boost: nDCG@10 0.773→0.879 on the heterogeneous slice but a loss on `mixed` — per-index opt-in candidate, not a default. restrict removed the answer document entirely for 4 queries per corpus (recall@20 1→0). |
| 4.4 | **ADOPTED** (no flag: inert without a `filters` argument) | Byte-identical behavior when unused (md5-verified); injection refused, both search legs prefiltered. **Correction to the row below: page/date filters did NOT ship** — they live inside the metadata JSON string and need real columns + a re-index. |
| 4.5 | **ADOPTED (on by default)** | Zero-risk observability; `token_usage` per stage on `/chat` and the SSE `complete` event. |
| 4.6 | **ADOPTED** (opt-in CLI) | `python -m rag_system.main ask <folder> "<q>"`; ephemeral index, verified cleanup including on SIGTERM. |

Original planning table follows, kept for the evidence trail:

Source: [PromtEngineer/agentic-file-search](https://github.com/PromtEngineer/agentic-file-search)
(FsExplorer lineage — an agentic filesystem QA agent: three-phase scan/dive/backtrack,
Docling parsing, grep/glob tools, DuckDB+VSS indexed search with a metadata-filter DSL,
exploration traces, per-query token/cost tracking). The evidence lens for every item:
escalate-don't-pre-decide (PEA-CAE), retriever quality dominates agency
(BrowseComp-Plus), and filesystem agents win small corpora but lose to ranked
retrieval at scale with ~39x token cost (BM25-wins-at-scale) — so we adopt its
*escalation mechanisms*, not its loop.

| # | Item | From their design | Our incorporation | Evidence fit |
|---|------|-------------------|-------------------|--------------|
| 4.1 | **Full-document escalation** | `parse_file` / `get_document` deep-read tools | When the evidence-sufficiency retry (2.1) still lands weak, reassemble the top-cited document IN ORDER from its chunks (chunk_index exists in metadata) and hand it to synthesis, token-capped, one document max. New RAG API helper + agent step; SSE event `document_escalation`. | PEA-CAE escalation; DOS-RAG document-order finding; bounded, not a loop |
| 4.2 | **Cross-reference hop** | Phase-3 backtracking on "See Exhibit B" | Index-time regex extraction of intra-corpus references (exhibit/section/filename mentions) into chunk metadata; query-time one-hop pull of the referenced doc's overview + top chunks when a top-ranked chunk carries one. Capped at one hop, no LLM in the hop. | Fixes the real "cross-references are invisible to embeddings" gap without unbounded agency |
| 4.3 | **Overview prefilter ("peripheral vision")** | Phase-1 parallel scan + RELEVANT/MAYBE/SKIP triage | We already build per-doc overviews; embed them once and use overview-vs-query similarity to boost/restrict chunk retrieval to top documents on multi-document indexes. No per-query LLM cost (their scan phase is an LLM call per document — the measured 39x failure mode). | Jason Liu facets/peripheral-vision; avoids their linear-cost scan |
| 4.4 | **Metadata filter DSL / self-query** | `semantic_search(filters="field=value, field in (a,b)")` on DuckDB | Same surface on LanceDB `where` clauses: accept `filters` on /chat + /chat/stream, wire to chunk metadata (document name, page, date). LLM filter-extraction later — small local models measured ~0.999 F1 on easy/medium filter translation. | The one query-planning technique local models already nail (component-map §6.4) |
| 4.5 | **Per-query token/cost tracking** | TokenTracker + cost summary per query | Surface Ollama's prompt_eval_count/eval_count per stage in the SSE `complete` event and UI (local cost = time + watts, still worth showing). | Their nicest UX idea; zero risk |
| 4.6 | **Ephemeral "ask a folder" mode** | Index-free filesystem QA | `python -m rag_system.main ask <folder> "<q>"`: build a temp in-memory/throwaway index (fast profile, no enrichment), answer, delete. Same pipeline, no agent loop. | FS-agents win small corpora — but an ephemeral *index* beats an ephemeral *agent* on our own retriever-dominance evidence |

**Deliberately NOT adopted** (goes in design_rationale §13 when implemented): the
LLM-per-document scan phase (linear cost in corpus size — the exact pattern
BM25-wins-at-scale measured at ~39x tokens), the free-form ReAct exploration loop
(search volume correlates weakly with quality; silent hand-off failures), cloud
Gemini (violates the local/privacy premise), llama-index-workflows + DuckDB
(duplicate our framework-free stack and LanceDB).

Sequencing: 4.5 and 4.4 are independent quick wins; 4.1 depends on 2.1's signal
(shipped); 4.2/4.3 are index-format-adjacent and should share a re-index window;
4.6 is CLI-only. All gated on the Phase 0 harness like everything else — 4.2 and
4.3 need multi-document gold queries with cross-references added to eval/goldset.

---

### Documentation Retrieval Pipeline (`Documentation/retrieval_pipeline.md`)

# 📥 Retrieval Pipeline

_Maps to `rag_system/pipelines/retrieval_pipeline.py`, orchestrated by `rag_system/agent/loop.py`, with helpers in `retrieval/` and `rerankers/`._

## Role
Given a user query and one LanceDB table, retrieve the most relevant chunks and synthesise an answer with its source documents.

Two objects share the work:

* **`Agent`** (`rag_system/agent/loop.py`) owns triage, the semantic cache, query decomposition, sub-answer composition, conversation history and verification.
* **`RetrievalPipeline`** (`rag_system/pipelines/retrieval_pipeline.py`) owns everything from retrieval to synthesis for a *single* query string. `Agent` may call it once, or once per sub-query in parallel.

## Sub-components

| Stage | Module | Key classes / functions | Notes |
|-------|--------|-------------------------|-------|
| Query decomposition | `retrieval/query_transformer.py` | `QueryDecomposer.decompose()` | Optional. Splits a query into standalone sub-queries; runs on the utility model. |
| Retrieval | `retrieval/retrievers.py` | `MultiVectorRetriever.retrieve()` | Runs LanceDB full-text and/or vector search over one table. There is no separate BM25 retriever class — lexical search is LanceDB's native FTS index. |
| Reranking | `pipelines/retrieval_pipeline.py`, `rerankers/reranker.py` | `_get_ai_reranker()`, `QwenRerankerScorer`, `rerankers.Reranker`, `CrossEncoderReranker` | On by default since arm G (2026-08-14); the model loads lazily on the first reranked query. Qwen3-Reranker names go to `QwenRerankerScorer`; otherwise the model is loaded through the `rerankers` library. `CrossEncoderReranker` is the non-library fallback branch. |
| Sentence pruning | `rerankers/sentence_pruner.py` | `SentencePruner.prune_documents()` | Provence (`naver/provence-reranker-debertav3-v1`). Off unless requested. |
| Synthesis | `pipelines/retrieval_pipeline.py` | `_synthesize_final_answer()` | Streams an LLM completion on the generation model. |
| Verification | `agent/verifier.py` | `Verifier.verify_async()` | See `verifier.md`. |

### Removed: the graph path

`GraphQueryTranslator`, `GraphRetriever` and `GraphExtractor` were **deleted on
2026-08-09** (roadmap item 2.5). They were unreachable — no shipped profile ever
set `graph_strategy` — and the evidence is against re-adding them: GraphRAG
*loses* on single-hop retrieval, its multi-hop gains range from +3 points to +27
depending on how well the vector baseline is tuned, and it costs **41–57× at
indexing** and up to **~377× in query tokens**. See
[`research/academic-evidence-2026.md`](research/academic-evidence-2026.md) §6.
`networkx` and `fuzzywuzzy` left `requirements.txt` with them.

## End-to-end flow

```mermaid
flowchart TD
    Q["User query"] --> T["Triage (see triage_system.md)"]
    T -- direct_answer --> DA["Direct LLM answer"]
    T -- rag_query --> C{"Semantic cache hit?"}
    C -- yes --> OUT["answer + source_documents"]
    C -- no --> D{"Decomposition enabled?"}
    D -- yes --> SUB["1..N sub-queries retrieved in parallel,<br/>candidates pooled + deduped (arm H default)"]
    D -- no --> ONE["Single query"]
    SUB --> RP
    ONE --> RP
    subgraph RP [RetrievalPipeline.run]
        R1["Retrieve (hybrid RRF / vector_only / fts_only)"] --> R2["Late-chunk table + sibling merge (optional)"]
        R2 --> R3["AI rerank (on by default)"]
        R3 --> R4["Context expansion (optional)"]
        R4 --> R5["Provence pruning (optional)"]
        R5 --> R6["Synthesis (streamed)"]
    end
    RP --> V["Verification (optional)"]
    RP -. "opt-in compose path" .-> COMP["Compose sub-answers"]
    COMP --> V
    DA -- "no source_documents, skipped" --> V
    V --> OUT
```

## Stage detail

### 1. Retrieval — `MultiVectorRetriever.retrieve()` (`retrievers.py`)

```python
retrieve(text_query: str, table_name: str, k: int, search_type: str = "hybrid",
         *, where: Optional[str] = None) -> List[Dict]
```

`where` is a LanceDB SQL predicate applied as a prefilter to every leg (metadata
filters, roadmap 4.4 — see "Feature surface added 2026-08-14+" below).

`search_type` selects which LanceDB legs run. Unknown values log a warning and degrade to `hybrid` (`retrievers.py:99-102`).

| Mode | Legs | Score returned |
|------|------|----------------|
| `hybrid` (default) | FTS and vector, run concurrently on a 2-worker thread pool (`retrievers.py:139-143`), fused by reciprocal rank fusion | RRF score |
| `vector_only` | vector only | `1 / (1 + distance)` |
| `fts_only` | FTS only | LanceDB's BM25 score |

Details:

* **FTS leg** — `tbl.search(query=..., query_type="fts").limit(k)`. Single-word queries are rewritten to `"<word>* OR <word>~"` to add prefix and fuzzy matching (`retrievers.py:117-118`). This is LanceDB's built-in full-text index (created at index time, see `indexing_pipeline.md`) — no SQLite, no Porter stemming, no configurable stop-word or n-gram handling.
* **Vector leg** — the query is embedded once and memoised in a 256-entry `lru_cache` per retriever instance, then `tbl.search(vector).limit(k)`. Before searching, the retriever reads the table's embedder marker: a table written by a different embedding model raises `EmbedderMismatchError` (which is deliberately *not* swallowed by the catch-all below), and the query vector is L2-normalized only when the table's own vectors are, so LanceDB's default L2 ordering matches the cosine ordering the model cards specify. A table with no marker is searched the legacy, unnormalized way with a one-time warning.
* **Fusion** — each leg contributes `1 / (60 + rank)` (`_RRF_K = 60`, `retrievers.py:20`). Rows are deduplicated on `chunk_id`, falling back to `_rowid` then `text` (`retrievers.py:83-89`), summed, sorted, and truncated to `k`. There is no weighted linear blend and no `dense_weight` knob.
* Each leg fetches `k` rows, so hybrid examines up to `2k` candidates and returns `k`.
* Every returned doc carries a finite, higher-is-better `score`. The raw per-leg values `bm25` and `_distance` are attached **only** when that leg actually hit the row.
* `metadata` is accepted as either a dict or a JSON string; `text` falls back through `metadata.original_text` → `row.text` → `""` (`retrievers.py:179-202`).
* On any exception other than `EmbedderMismatchError` the method logs and returns `[]`, so a missing table degrades to zero results rather than an error — **unless** a `where` filter is set: a filtered search that fails re-raises, because "0 results" and "the restriction did not run" are different answers and only one of them is safe. An embedder mismatch propagates instead — a wrong-model answer is worse than an error.

### 2. Late chunking at query time (`retrieval_pipeline.py:289-341`)

When the merged late-chunk config is enabled, two extra things happen:

1. A second `retrieve()` runs against the late-chunk table and its hits are appended to the candidate list (`:293-305`). The table name is `latechunk.lancedb_table_name` if set, otherwise `<table><table_suffix>` with `table_suffix` defaulting to `_lc` (`:85-92`) — the same name `IndexingPipeline` writes.
2. **Sibling merging** (`:316-341`): for every retrieved chunk, the ±1 neighbouring chunks from the same document are fetched from the main table and their text is concatenated into that chunk's `text`. This runs off the config flag alone, whether or not the `_lc` table exists.

The config block is merged across both container names and both spellings — `retrieval.late_chunking`, `retrieval.latechunk`, `retrievers.late_chunking`, `retrievers.latechunk`, later writes winning (`:70-83`) — so a profile setting and a runtime API override can come from different places and both apply.

The `default` profile enables late chunking at query time (`main.py:57-59`), while the RAG API's `/index` endpoint defaults `enable_latechunk` to `false` (`api_server.py:410`). Unless you explicitly index with late chunking on, the `_lc` table does not exist: the extra `retrieve()` logs "Could not search table …" and returns nothing, while sibling merging still applies.

### 2b. Evidence-sufficiency retry (`RetrievalPipeline.retrieve_candidates`)

_Roadmap item 2.1, shipped 2026-08-09. On in the `default` profile, off in `fast`._

One conditional second retrieval, and only one — the evidence for iterative
retrieval says iteration 1→2 captures ~95% of the gains and iterations ≥3 are
noise. It wraps the first stage **and** the rerank stage, so it sees the ranking
the caller would actually have got.

```python
"retrieval": {"retry": {"enabled": True, "min_top_score": 0.12, "max_attempts": 1}}
```

**The signal.** Not the raw top similarity. That was measured on the gold set and
is *anti*-correlated with success: the three `mixed` first-stage misses each
scored a **higher** top cosine than the median successful query, because absolute
similarity mostly encodes how close a query's phrasing sits to the corpus's
register. What carries signal is contrast — how far the best candidate stands
above everything else the query dragged in:

```
evidence = (cos_top − cos_background) / (1 − cos_background)
```

`cos_background` is the mean cosine of candidates from rank 6 down; the
denominator rescales against this query's reachable headroom, keeping the result
in 0–1 and comparable across queries. Cosine comes from LanceDB's squared-L2
`_distance` on the L2-normalized v4 tables (`cos = 1 − d/2`), so the retry is only
armed on a normalized table — on a legacy table, or in `fts_only` mode, the score
is `None` and nothing fires. **RRF scores are never used**: they encode rank, not
confidence, and are near-identical for every query.

When reranking is on, the top reranker score is preferred instead, but only when
it is a genuine 0–1 probability (`QwenRerankerScorer` returns P("yes")); an
arbitrary logit is rejected rather than compared against a probability threshold.
The threshold for that path is `min_rerank_score`, defaulting to `min_top_score`.

**What happens on a fire.** `_reformulate_query()` makes one `format="json"` call
on the **enrichment** model asking for a rewrite in the vocabulary a document
would use, the whole first stage + rerank runs again on it, and **the better of
the two result sets by the same score is kept** — a retry that does not improve
the evidence is discarded, never merged. A `retrieval_retry` event goes out
through `event_callback`, so the RAG API's SSE stream carries it and the UI
cascade shows a "Rechecking weak evidence" step.

**Measured** (`../eval/decisions/phase2-pipeline.md`): fires on **9.7% of `mixed`
queries** (7/72) and 20.8% of `docs`, and moved `mixed` first-stage nDCG@10 from
0.889 to 0.901/0.906 across two runs with **zero per-query regressions**. It
repaired `docs_d16`, a genuine recall@10 miss.

### 3. AI reranking (`retrieval_pipeline.py`, `_rerank_stage`)

Loaded lazily behind `_ai_reranker_init_lock` so only one thread performs the heavy `from_pretrained()`; the `.rank()` call itself is serialised behind `_rerank_lock` because the `rerankers` backends are not thread-safe.

Config keys read from `reranker`:

| Key | Default in code | `default` profile |
|-----|-----------------|-------------------|
| `enabled` | falsy | **`true`** — reranking is on by default since arm G, 2026-08-14 ([`../eval/DECISIONS.md`](../eval/DECISIONS.md)) |
| `model_name` | none — missing logs a warning and skips reranking | `EXTERNAL_MODELS["reranker_model"]` = `Qwen/Qwen3-Reranker-4B`, loaded lazily on the first reranked query |
| `strategy` | `rerankers-lib` | `rerankers-lib` |
| `model_type` | `cross-encoder` | not set ⇒ `cross-encoder` |
| `top_k` | all retrieved docs | `10` |
| `min_score` | unset (no threshold) | `0.5` — Qwen scorer only: candidates the calibrated scorer marks below this P(relevant) against every query are dropped |
| `min_keep` | unset | `3` — floor on candidates kept regardless of `min_score` |
| `top_percent` | unset | unset — when set (0 < p ≤ 1) it overrides `top_k` with `max(1, len(docs) * p)` |

A model whose `model_type` is `qwen3`, or whose name contains `qwen3-reranker`, is routed to the in-repo `QwenRerankerScorer` — the `rerankers` library builds Qwen3-Reranker with a randomly initialised score head, so this route is what makes the default reranker model correct rather than merely loadable. Otherwise `strategy: "rerankers-lib"` loads `rerankers.Reranker(model_name, model_type=model_type)`, and any other value constructs the local `CrossEncoderReranker` (`rerankers/reranker.py:5`), an `AutoModelForSequenceClassification` cross-encoder with batched scoring and an early-exit heuristic.

**If the reranker fails to load, the pipeline logs `⚠️ Could not load reranker '<name>' (<err>). Continuing without reranking.` and continues with the unranked candidates.** There is no second fallback model.

#### Decomposition: pooled first stage by default, sub-query scoring at rerank

_Roadmap item 2.2, shipped 2026-08-09; pooled first stage added by arm H, 2026-08-15._

Decomposing the **first stage** dilutes it semantically; the 2026 evidence
(MultiConIR/SSRB) puts the win at the reranking stage instead. So:

* The shipped default (arm H, 2026-08-15) is the **pooled first stage**
  (`pooled_first_stage: true`, `compose_from_sub_answers: false`): each
  sub-query runs first-stage retrieval, the candidates are pooled and
  de-duplicated, and there is ONE rerank pass and ONE synthesis over the union
  context (`_pooled_first_stage`).
* When sub-queries are supplied *and* the reranker is on (the default since
  arm G, 2026-08-14), every candidate is
  scored against **every** sub-query and the per-sub-query scores are combined
  with `query_decomposition.rerank_aggregate` — `"mean"` (default) or `"max"`.
  `mean` is the default because it measured better than `max` on both halves of
  the A/B ([`../eval/decisions/phase2-pipeline.md`](../eval/decisions/phase2-pipeline.md) §3).
* When the reranker is off there is no rerank stage, so the sub-queries only
  drive the pooled first-stage fan-out.

The compose path is still available behind its pre-existing flag,
`query_decomposition.compose_from_sub_answers` (the UI "compose sub-answers"
toggle): it needs a separate *answer* per sub-question
to compose from, which a single shared candidate set cannot produce, so it runs a
full `RetrievalPipeline.run()` per sub-query in parallel. Exactly what runs:

| `query_decomposition` | reranker | First stage | Rerank scored against |
|---|---|---|---|
| off | either | once, full query | full query |
| on, `compose_from_sub_answers: true` (opt-in) | either | **once per sub-query**, in parallel | that sub-query |
| on, pooled (profile default) | on | **once per sub-query**, pooled + deduped | **all sub-queries, aggregated** |
| on, pooled (profile default) | off | once per sub-query, pooled + deduped | — (no rerank stage) |
| on, one sub-query after decomposition | either | once, the resolved query | the resolved query |

### 4. Context expansion (`RetrievalPipeline._run_after_candidates`)

When the effective window size is greater than 0, each surviving doc is expanded with its neighbours from the same document via a metadata-only LanceDB filter (`document_id = ... AND chunk_index BETWEEN ...`), run across a thread pool. The union is deduplicated on `chunk_id` and re-sorted by `rerank_score`, then `_distance`, then `score`, then document order.

Two properties worth knowing:

* The rerank-score selection is applied to the **seed set, before expansion**: only chunks carrying a `rerank_score` (plus cross-reference hop chunks, which are appended after reranking by design) seed the expansion, so the freshly added neighbours survive rerank filtering. (The old code filtered the expanded set afterwards, which deleted every neighbour whenever the reranker ran.)
* When the caller passed a metadata filter, the expansion queries run **inside** that filter — it is captured in the submitting thread (the executor's workers do not inherit thread-locals), so expansion can never pull in chunks outside the restricted corpus.

### 5. Provence sentence pruning (`retrieval_pipeline.py:448-462`)

Runs between context expansion and synthesis when `provence.enabled` is set. Loads `naver/provence-reranker-debertav3-v1` once behind `_sentence_pruner_lock`, drops sentences scoring below `provence.threshold` (default `0.1`, `:455`), and then removes any chunk whose text was pruned to nothing (`:460`). If the model cannot be downloaded or loaded, `SentencePruner` logs and `prune_documents()` echoes its input unchanged.

No shipped profile contains a `provence` block, so pruning is off unless a request enables it.

### 6. Synthesis (`retrieval_pipeline.py:223-261, 502-514`)

The surviving chunk texts are joined with blank lines and passed to `_synthesize_final_answer()`, which streams a completion on the **generation** model. Each token is pushed to `event_callback("token", {"text": ...})` — synthesis is push-based, not an iterator.

Before serialisation, `vector` and `_distance` are removed from every doc and NaN/Inf floats are nulled (`:479-500`). The return value is:

```jsonc
{ "answer": "...", "source_documents": [ /* chunk dicts */ ] }
```

with `{"answer": "I could not find an answer in the documents.", "source_documents": []}` when nothing survives (`:476-477`).

Since commit a3f999a, each snippet is prefixed with a `[Source document: …]` line
naming the file it came from, and synthesis rule 7 tells the model to name that
source when the question asks where something is defined or attribution matters.
Sources are also returned as the `source_documents` array and rendered by the UI as a collapsible list.

### 7. Semantic cache (`agent/loop.py:130-154, 305-324, 587-594`)

Owned by `Agent`, not by the pipeline:

* `TTLCache(maxsize=100, ttl=300)` (`loop.py:33`) keyed by raw query text, storing `{embedding, result, session_id}`.
* Looked up by cosine similarity against the freshly embedded query; a hit requires similarity ≥ `semantic_cache_threshold` (`0.98` in both profiles).
* `cache_scope` defaults to `"session"`: entries from a different `session_id` are skipped (`loop.py:141`). Set it to `"global"` to share cached answers across sessions — note that this can return one session's document-derived answer to another.
* Skipped entirely on the `direct_answer` route.
* Per-query embeddings are additionally memoised by the retriever's own 256-entry `lru_cache`.

## Configuration flags

Both camelCase and snake_case are accepted; the RAG API normalises them to snake_case once at parse time (`api_server.py:51-76`).

| Wire field | RAG API default when absent | `default` profile | `fast` profile | Effect |
|------------|-----------------------------|-------------------|----------------|--------|
| `retrieval_mode` (alias `search_type`) | not set ⇒ profile value | `hybrid` | `vector_only` | `hybrid` / `vector_only` / `fts_only`. Anything else is rejected with HTTP 400 (`api_server.py:161-168`). |
| `retrieval_k` | not sent ⇒ profile value | `20` | `10` | Rows fetched per leg, and the size of the fused candidate list. (The `20` UI default comes from the frontend, not the RAG API.) |
| `reranker_top_k` | not sent ⇒ profile value | `10` | reranker off | Docs kept after reranking. |
| `context_window_size` | not sent ⇒ profile value | `0` | `0` | Neighbouring chunks merged around each hit. The frontend sends `1` by default (`session-chat.tsx`), so UI traffic effectively uses `1`; a client that omits the field gets the profile's `0`. |
| `ai_rerank` | not sent ⇒ profile value | reranker **enabled** (arm G) | reranker disabled | Toggles `reranker.enabled`. |
| `query_decompose` | not sent ⇒ profile value | `true` | `false` | Toggles `query_decomposition.enabled`. |
| `compose_sub_answers` | not sent ⇒ profile value | `false` (pooled first stage, arm H) | — | Compose one answer from sub-answers vs. the default pooled candidates with a single synthesis. |
| `filters` | not sent ⇒ unfiltered | — | — | Metadata filter object (roadmap 4.4), compiled to a LanceDB `where` prefilter on both legs; a malformed filter is a 400. A present filter also skips triage at the agent. |
| `context_expand` | not sent | — | — | `false` forces `window_size_override=0` for this request. |
| `verify` | not sent ⇒ profile value | `true` | `false` | See `verifier.md`. |
| `force_rag` | `false` | — | — | Skip triage, force the RAG path; all other toggles still apply. |
| `provence_prune` | not sent ⇒ disabled | absent | absent | Enable Provence sentence pruning. |
| `provence_threshold` | not sent ⇒ `0.1` | absent | absent | Pruning threshold. Not exposed in the UI. |
| `model` | not sent | — | — | Per-request generation model. Applied through a context manager that restores the previous value afterwards, and ignored with a warning when the id does not suit the active backend (`api_server.py:88-113`). |

UI defaults (`src/components/ui/session-chat.tsx`): compose `false`, decompose `true`, aiRerank `true`, contextExpand `true`, stream `true`, verify `true`, forceDocs `false`, provencePrune `false`, retrievalK `20`, contextWindowSize `1`, rerankerTopK `10`, searchType `hybrid`.

There is no `dense_weight` / `denseWeight` knob anywhere in the stack, and no `fusion` config block.

## Entry points

```python
# rag_system/pipelines/retrieval_pipeline.py
RetrievalPipeline.run(query, table_name=None, window_size_override=None, event_callback=None,
                      sub_queries=None, *, filters=None) -> Dict

# rag_system/agent/loop.py
Agent.run(query, table_name=None, session_id=None, compose_sub_answers=None, query_decompose=None,
          ai_rerank=None, context_expand=None, verify=None, retrieval_k=None, context_window_size=None,
          reranker_top_k=None, retrieval_mode=None, force_rag=False, event_callback=None,
          *, filters=None) -> Dict
```

`Agent.run` is a synchronous wrapper around `_run_async`. Both `/chat` and `/chat/stream` go through `Agent.run` — including the `force_rag` path (`api_server.py:354-390`), so no request shape bypasses the toggles. `RetrievalPipeline` has no iterator/`answer_stream` entry point.

Build them with the factory, never by hand:

```python
from rag_system.factory import get_agent
agent = get_agent("default")          # deep copy of PIPELINE_CONFIGS["default"]
result = agent.run("What does the contract say about termination?")
```

## Streaming event protocol

`POST /chat/stream` returns `text/event-stream`; every frame is `data: {"type": <event>, "data": <payload>}\n\n` (`api_server.py:329-333`).

| Event | Payload | Emitted by |
|-------|---------|-----------|
| `analyze` | `{query}` | `loop.py:250-251` |
| `direct_answer` | `{}` | `loop.py:328-329` |
| `decomposition` | `{sub_queries}` | `loop.py:378-379` |
| `retrieval_started` | `{mode}` (pipeline) or `{count}` (decomposition) | `retrieval_pipeline.py:276-277`, `loop.py:384-385` |
| `retrieval_done` | `{count}` | `retrieval_pipeline.py:307-308`, `loop.py:401, 485` |
| `filters_applied` | `{spec, where, candidates}` — only when a metadata filter was sent | `RetrievalPipeline._retrieve_candidates_filtered` |
| `retrieval_retry` | the retry info dict — only when the evidence-sufficiency retry fires | `RetrievalPipeline._post_candidates` |
| `crossref_hop` | `{targets, chunks_added, ...}` — only when `retrieval.crossref_hop` is on and fires | `RetrievalPipeline._crossref_hop` |
| `document_escalation` | the escalation payload (`document_name`, `chunks_used`, `approx_tokens`, signal/score/threshold/budget) | `rag_system/agent/escalation.py` |
| `rerank_started` / `rerank_done` | `{count}` | `retrieval_pipeline.py:346-347, 394-395`, `loop.py:402-403, 418-419, 486` |
| `context_expand_started` / `context_expand_done` | `{count}` | `retrieval_pipeline.py:404-405, 438-439` |
| `prune_started` / `prune_done` | `{count}` | `retrieval_pipeline.py:453-454, 461-462` |
| `token` | `{text}` | synthesis and composition streams |
| `sub_query_token` | `{index, text, question}` | `loop.py:430` |
| `sub_query_result` | `{index, query, answer, source_documents}` | `loop.py:453-459` |
| `single_query_result` / `final_answer` | the result dict | `loop.py:397-398, 533-534` |
| `complete` | the final result dict | `api_server.py:338` |
| `error` | `{error}` | `api_server.py:344` |

## Interfaces

* Reads LanceDB tables at `storage.lancedb_uri` (`./lancedb`), table `text_pages_<index_id>` (`backend/database.py:351`) or the profile's `storage.text_table_name` (`text_pages_v4`) when a session has no linked index.
* Calls Ollama at `OLLAMA_CONFIG["host"]` — generation model for answers, utility model for routing/decomposition/verification.
* Embeddings come from `select_embedder()`: a model name containing `/` is loaded from HuggingFace in-process, anything else is treated as an Ollama tag. `_get_text_embedder()` **raises** when `embedding_model_name` is missing rather than substituting a default, because a wrong-dimensionality embedder silently returns nothing useful against an existing index (`retrieval_pipeline.py:103-116`).
* Vector search is a brute-force scan: nothing in `rag_system/` ever calls `create_index` for an ANN/IVF-PQ index. Only the full-text index is built.

## Extension points

* **New retriever** — the contract is duck-typed, not an ABC. Provide an object with `retrieve(text_query: str, table_name: str, k: int, search_type: str = "hybrid") -> List[Dict]` returning dicts with at least `chunk_id`, `text`, `score`, `document_id`, `chunk_index`, `metadata`, and return it from `RetrievalPipeline._get_dense_retriever()` (`retrieval_pipeline.py:118-133`). There is no `BaseRetriever` and no registry.
* **New reranker** — either point `reranker.model_name` / `reranker.model_type` at another `rerankers`-library model, or set `reranker.strategy` to something other than `rerankers-lib` and swap the class constructed in `_get_ai_reranker()` (`retrieval_pipeline.py:135-165`). There is no `BaseReranker`.
* **Answer prompt** — the synthesis prompt is an inline f-string at `retrieval_pipeline.py:225-250`. `_synthesize_final_answer(query, facts, *, event_callback=None)` takes no prompt-override argument; edit the literal.

## Feature surface added 2026-08-14+

* **Metadata filters / DSL** (roadmap 4.4) — a `filters` JSON object on `/chat`, `/chat/stream` and the CLI (`--filters`) is compiled by `rag_system/retrieval/filters.py::compile_filters` to a LanceDB `where` prefilter applied to both search legs (`MultiVectorRetriever.retrieve(..., where=...)`). The pipeline opens it as a thread-local `RetrievalPipeline.filter_scope`, emits `filters_applied`, and a filtered search that fails re-raises instead of returning `[]`.
* **Cross-leg dedupe** (2026-08-14) — vector/FTS candidates are de-duplicated on `(document_id, chunk_index)` in the candidate-building path (`RetrievalPipeline.retrieve_candidates`), so the same passage no longer occupies two slots.
* **FTS quote-stripping** — double quotes are stripped before the LanceDB FTS leg so a decomposer-emitted phrase cannot trip the FTS parser, and BM25 scores are read from the `_score` column (`retrievers.py`).
* **Cross-reference hop** (roadmap 4.2, off by default) — `retrieval.crossref_hop`: after reranking, chunks from documents the results cross-reference are appended (`RetrievalPipeline._crossref_hop`, index-time extraction in `rag_system/indexing/crossref.py`); hop chunks are exempt from the rerank-score seed filter by design.
* **Overview prefilter** (roadmap 4.3, off by default) — `retrieval.overview_prefilter` with `mode: "boost"` (reorder candidates toward overview-matched documents) or `"restrict"` (hide them).
* **Full-document escalation** (roadmap 4.1, off by default) — `retrieval.document_escalation`: when evidence is still weak after the retry, the top-ranked chunk's whole document is reassembled in `chunk_index` order and appended to the synthesis context, capped at `token_budget` (`rag_system/agent/escalation.py`); emits `document_escalation`.
* **Synthesis context budgeting** (2026-08-14) — rank-ordered docs are packed into an explicit token budget, `retrieval.synthesis_context_tokens` default **12000**, with sibling-span overlap suppression (`RetrievalPipeline._budget_synthesis_context`).
* **Embedding instruction override** — the `embedding_instruction` config key or the `EMBEDDING_INSTRUCTION` env var overrides the query-side instruction prefix; set it to `""` to switch the prefix off (`RetrievalPipeline._query_instruction`).

## Operational notes

* The RAG API is a single-threaded `socketserver.TCPServer` (`api_server.py:527-530`), so requests are handled one at a time. Treat it as a single-concurrent-user service.
* `RAG_AGENT` and its `RetrievalPipeline` are process-wide singletons created once at startup (`api_server.py:34-37`). Per-request overrides write into that shared config object, but the agent snapshots the config before applying them and restores it afterwards, so an override is scoped to the request that sent it.
* Changing the embedding model requires re-indexing. `VectorIndexer` raises a clear error if you try to append vectors of a different width to an existing table (`indexing/embedders.py:110-116`), and the query side will simply fail to match if the dimensions differ.

---
_Keep this document updated when stages, config keys, or the event protocol change._

---

### Documentation Triage System (`Documentation/triage_system.md`)

# 🔀 Triage / Routing System

_One deterministic gate and one LLM router, in two processes:_
* _`should_use_rag()` in `backend/server.py` (gateway, port 8000) — deterministic, no LLM call; see "Backend gate" below._
* _`Agent._triage_query_async` in `rag_system/agent/loop.py` (RAG API, port 8001) — the only LLM routing layer._

## Purpose
Decide, per query, whether to answer with:
1. **Direct LLM generation** — no retrieval, faster and cheaper; or
2. **Retrieval-Augmented Generation** — search the indexed documents first.

## Which router actually runs

| Request path | Router(s) involved |
|--------------|--------------------|
| Streaming chat (UI default): browser → `POST :8001/chat/stream` (`src/lib/api.ts:509`, toggle at `session-chat.tsx:48`, default on) | Agent router only. The backend gateway is not in this path. |
| Non-streaming chat: browser → `POST :8000/sessions/<id>/messages` → `POST :8001/chat` | Backend router first (`server.py:382`), then the agent router again inside the RAG API. |
| `POST :8000/chat` (`handle_chat`) | Neither. That endpoint always calls Ollama directly. |

On the non-streaming path the backend gate decides `use_rag` locally; when it routes to RAG it forwards the query (and `force_rag`, when set) to the RAG API, where the agent triages again and may still choose `direct_answer`. Over-sending to RAG is therefore safe.

## Agent router (`rag_system/agent/loop.py`)

Order of evaluation in `_triage_query_async` (`loop.py:175-223`):

1. **Overview routing** — `_route_via_overviews(query)` (`loop.py:602-644`). Returns `None` immediately when no overviews are loaded (`loop.py:605-607`); otherwise it builds a `DOCUMENT OVERVIEWS:` block from the first 40 loaded overviews (`loop.py:612-613`), interpolates it into the router prompt (`loop.py:615-630`) and calls the utility model with `format="json"`. Parses `{"category": ...}`, defaulting to `rag_query` on a parse failure.
2. **History short-circuit** — if the overview router returned `None` **and** the session already has chat history, the query is treated as a follow-up and routed to `rag_query` without any LLM call (`loop.py:188-193`).
3. **LLM fallback triage** — a two-way classifier (`rag_query` / `direct_answer`) on the utility model, defaulting to `rag_query` if the JSON cannot be parsed. `Agent._normalize_triage()` runs on every verdict and collapses anything that is not an explicit `direct_answer` to `rag_query`, so a small model that emits the retired `graph_query` label still lands on the RAG path.

`force_rag=true` — or a compiled metadata `filters` object on the request — skips all three: `query_type` is pinned to `rag_query` (`if force_rag or compiled_filters is not None` in `_run_async_inner`) while the `verify` / `ai_rerank` / `query_decompose` / `compose_sub_answers` / `context_expand` toggles all still apply.

Both LLM routing calls run at `temperature: 0` (deterministic routing).

There is no regex or keyword stage in the agent.

## Backend gate (`backend/server.py`)

Since the Phase-2 routing change (see `eval/decisions/phase2-gateway.md`), the gateway makes **no LLM call and reads no files** to route. `should_use_rag()` (module-level, unit-tested in `backend/test_gateway_routing.py`) evaluates in order:

1. **`force_rag`** ⇒ RAG, unconditionally (also forwarded, so agent triage is skipped too).
2. **No indexes linked** to the session ⇒ direct LLM (nothing to retrieve from).
3. **Smalltalk / assistant-meta** — a whole-message anchored allowlist (greetings, thanks, goodbyes, "who are you?"-style meta) capped at ~6 words ⇒ direct LLM.
4. **Everything else** ⇒ RAG.

The old per-message enrichment-model router (`_route_using_overviews`) and the keyword/length fallback (`_simple_pattern_routing`) were deleted — the fallback's substring matching misrouted most real document questions (`'hi'` matched *this* and *machine*). Over-sending to RAG is safe because the agent-side triage above can still answer directly; the gateway gate exists only to skip obvious non-retrieval turns at zero cost (~750 ms saved per routed message).

## Flow

```mermaid
flowchart TD
    Q["Incoming query"] --> FR{force_rag or filters?}
    FR -- yes --> RAG["Retrieval pipeline"]
    FR -- no --> OV{Overviews loaded?}
    OV -- no --> H{Chat history?}
    OV -- yes --> R1["Overview router LLM<br/>(utility model, JSON)"]
    R1 -- rag_query --> RAG
    R1 -- direct_answer --> LLM["Direct LLM answer"]
    H -- yes --> RAG
    H -- no --> R2["Fallback triage LLM<br/>(rag_query / direct_answer)"]
    R2 -- rag_query --> RAG
    R2 -- direct_answer --> LLM
```

The backend gate is not in this diagram: it is a deterministic pre-filter (force_rag → indexes → smalltalk allowlist) with no LLM call, described above.

## Overviews: where they come from

| Step | Code |
|------|------|
| Written at index time, one JSON line per document: `{"doc_id": ..., "overview": ...}` | `rag_system/indexing/overview_builder.py:33-49` |
| Default file `index_store/overviews/overviews.jsonl`; the RAG API overrides it to `index_store/overviews/<session_or_index_id>.jsonl` | `overview_builder.py:24`, `api_server.py:233-234` |
| Loaded per request by the RAG API before the agent runs | `api_server.py:362-366` → `Agent.load_overviews_for_indexes` (`loop.py:79-107`) |
| Falls back to the global `overviews.jsonl` when no per-index file exists | `loop.py:104-107` |

If no overview file exists for the session, the agent's overview router returns `None` and routing falls through to the history short-circuit or the fallback triage prompt. (The backend gate does not read overview files at all.)

## Models

Only the agent-side router costs an LLM call, on the utility model — `Agent._utility_model()` resolves `ENRICHMENT_MODEL` env var → `OLLAMA_CONFIG["enrichment_model"]` → `qwen3.5:4b`. The gateway gate is pure Python. Routing is never charged to the generation model, and a per-request `model` override does not change the routing model: the RAG API applies that override only for the duration of the request via a context manager, and the router reads `enrichment_model`, not `generation_model`.

## Configuration

| Knob | Where | Effect |
|------|-------|--------|
| `force_rag` (`forceRag`) | request body on `/chat`, `/chat/stream` (`api_server.py:191`) and on the backend's `/sessions/<id>/messages` (`server.py:381`) | Skips triage entirely and forces the RAG path. Surfaced in the UI as the "Always search documents" toggle (`session-chat.tsx:51`, default off). |

The third outcome, `graph_query`, and the `graph_strategy` config block that armed it were **removed on 2026-08-09** (roadmap item 2.5) along with the rest of the graph module. Evidence: GraphRAG loses on single-hop retrieval, its multi-hop gains are contested, and it costs 41–57× at indexing and up to ~377× in query tokens — [`research/academic-evidence-2026.md`](research/academic-evidence-2026.md) §6.

There is no global triage on/off switch and no similarity threshold. `PIPELINE_CONFIGS` has no `triage` key, and no `TRIAGE_OVERVIEW_THRESHOLD` environment variable is read anywhere.

## Failure / fallback modes

| Failure | Agent | Backend |
|---------|-------|---------|
| No overviews on disk | `_route_via_overviews` returns `None`; history short-circuit or fallback triage decides | n/a — gateway gate reads no files |
| Router LLM returns unparseable JSON / unexpected text | defaults to `rag_query` | n/a — gateway gate makes no LLM call |
| Router LLM call fails (timeout / connection / bad status) | the client catches the request error and returns `{}`, so the unparseable-JSON default applies — triage fails closed to `rag_query` | n/a |

---

_Keep this document updated whenever routing order, prompts, or fallback behaviour change._

---

### Documentation Verifier (`Documentation/verifier.md`)

# ✅ Answer Verifier

_File: `rag_system/agent/verifier.py`. Sole caller: `rag_system/agent/loop.py:560-578`._

## Objective
Assess whether an answer produced by the RAG path is **grounded** in the retrieved context snippets, and annotate the answer with the model's self-reported confidence.

Two interchangeable backends implement it. The **LLM-prompt verifier below is what
ships**; a local NLI/verifier model is opt-in via `VERIFIER_MODEL` (see
[Local verifier model](#local-verifier-model-opt-in)).

> **`[Confidence: N%]` is UX, not a measurement.** It is whatever the verifier
> emitted, rescaled to a percent. Neither backend is calibrated: an 80% does not
> mean the answer is right four times in five. Swapping the LLM prompt for an NLI
> model changes where the number comes from, not that caveat.

## Prompt
See `prompt_inventory.md` → `verifier.fact_check` (`verifier.py:25-85`). The prompt carries three few-shot examples and then a `# TASK` block into which the query, the context (clamped to the first 4000 characters at `verifier.py:76`) and the answer are injected. It is sent asynchronously with `format="json"` at `temperature: 0` — deterministic verdicts, the same pin the eval judge got.

Expected response, one line of JSON:

```jsonc
{
  "verdict": "SUPPORTED" | "NOT_SUPPORTED" | "NEEDS_CLARIFICATION",
  "is_grounded": true | false,
  "reasoning": "<short explanation>",
  "confidence_score": 0-100
}
```

It is parsed into a `VerificationResult` (`verifier.py:4-9`) with those four fields.

## Sequence

```mermaid
sequenceDiagram
    participant A as Agent._run_async
    participant V as Verifier
    participant LLM as Ollama (utility model)

    A->>A: build context_str from result["source_documents"]
    A->>V: verify_async(contextual_query, context_str, answer)
    V->>LLM: fact-check prompt (format=json)
    LLM-->>V: JSON verdict
    V-->>A: VerificationResult
    A->>A: append confidence tag to result["answer"]
```

## Call site

| Caller | Code | When it runs |
|--------|------|--------------|
| `Agent._run_async()` | `rag_system/agent/loop.py`, end of `_run_async` | After every branch (direct answer, decomposed/composed, single-query RAG), when verification is enabled **and** `result["source_documents"]` is non-empty. |

There is exactly one call site in the repository. `rag_system/pipelines/retrieval_pipeline.py` does not import or reference `Verifier`. Only the async `verify_async()` exists — the synchronous `verify()` was removed (`verifier.py:20`).

Because the check is gated on non-empty `source_documents`, the `direct_answer` route (which returns `source_documents: []`) is never verified.

## Configuration

| Knob | Where | Default | Meaning |
|------|-------|---------|---------|
| `verification.enabled` | `rag_system/main.py:80` (`default` profile) | `true` | Profile-level switch. |
| `verification.enabled` | `rag_system/main.py:109` (`fast` profile) | `false` | Verification off in the speed profile. |
| — | `loop.py:560` | `true` | Fallback used when the profile has no `verification` block. |
| `verify` | HTTP request field on `/chat` and `/chat/stream` (`api_server.py:186`) | not sent ⇒ profile value wins | Per-request override; forwarded to `Agent.run(verify=...)`. Also accepted by the backend gateway as `verify` (`backend/server.py:48`). |
| model | `loop.py`, `Agent.__init__` | utility model (`enrichment_model`, default `qwen3.5:4b`) | Which Ollama model runs the LLM-prompt verifier. Verification runs on the small model, not the answer model. |
| `verification.model` / `VERIFIER_MODEL` | pipeline config, or the env var | unset ⇒ LLM-prompt verifier | A HuggingFace model name switches the backend to a local NLI/verifier model. |
| `verification.threshold` | pipeline config | `0.5` | Score at or above which the local verifier calls an answer grounded. Ignored by the LLM-prompt backend. |
| `VERIFIER_TRUST_REMOTE_CODE` | env var | unset | Must be `1` to load a verifier that ships custom modelling code (e.g. Vectara HHEM). |

## Local verifier model (opt-in)

_Roadmap item 2.4, shipped 2026-08-09 as a **seam**: the default is unchanged._

```bash
VERIFIER_MODEL=MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli python -m rag_system.main api
```

`LocalNLIVerifier` (`rag_system/agent/verifier.py`) loads any HuggingFace
sequence-classification model **lazily on first use**, splits the answer into
sentences, scores each one against the retrieved evidence as the premise, and
takes the **minimum** — one unsupported sentence makes the answer ungrounded,
matching the binary semantics `eval/judge.py` already uses. The "supported" logit
is resolved from `id2label` (`entailment` / `consistent` / `supported` / `1`),
falling back to the last class for binary checkers.

A model that cannot be loaded **raises** with the list of names that were
checked; it does not silently fall back to the LLM prompt. A verifier that
quietly is not the verifier you configured is worse than an error.

### Availability, checked 2026-08-09

| Candidate | Verdict |
|---|---|
| **ThinknCheck** (arXiv 2604.01652, UPenn, 1B, 78.1 BAcc) | **No public weights.** The paper is real, but a HuggingFace Hub search for `thinkncheck` returns zero models and the paper links no release. Cannot be wired. |
| `ibm-granite/granite-guardian-3.3-8b` | Exists, Apache-2.0 — but 8B / ~16 GB, far over the budget this seam is for. |
| `ibm-granite/granite-guardian-hap-38m` | Exists, 38M, Apache-2.0 — but it is a **hate/abuse/profanity** RoBERTa classifier. Wrong task: it does not score answer-vs-evidence entailment. |
| `MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli` | ✅ MIT, 369 MB, no custom code. Generic NLI. |
| `lytang/MiniCheck-DeBERTa-v3-Large` | ✅ MIT, 1.74 GB, no custom code. Purpose-built grounded claim verification (the baseline ThinknCheck benchmarks against). |
| `vectara/hallucination_evaluation_model` (HHEM-2.1-open) | Apache-2.0, 438 MB, but ships custom modelling code — needs `VERIFIER_TRUST_REMOTE_CODE=1`. |

The same table is embedded in the code as `VERIFIER_AVAILABILITY_NOTES` and is
printed verbatim when a configured verifier fails to load.

The UI initialises its verify toggle to `true` (`src/components/ui/session-chat.tsx:49`), so verification is on by default for chat traffic.

## Effect on the answer

The verifier does **not** add a field to the response. It mutates the answer string (`loop.py:568-578`):

* `confidence_score > 0` → appends `" [Confidence: N%]"`.
* Additionally, when `is_grounded` is false **or** the score is below 50 → appends `" [Warning: Low confidence. Groundedness: <bool>]"`.
* `confidence_score == 0` → nothing is appended (0 is treated as a parse failure) and a warning is logged to stdout.

The API response shape is unchanged: `{"answer": ..., "source_documents": [...]}`.

## Failure modes

* Invalid JSON, a missing `response` key, or a type-mismatched verdict (a string `"85"`, `null`, the string `"false"`) → the parse/coercion in `verify_async()` fails open to `VerificationResult(False, 0)`, and because the score is 0 no tag is appended — the answer is returned unannotated. Malformed verdict JSON degrades to score 0; it cannot 500.
* HTTP-layer failures of the LLM call itself (timeout, connection error, non-200 status) are caught inside `generate_completion_async`, which returns `{}` → the same score-0 path, so the answer comes back unannotated with HTTP 200. Only **non-httpx** exceptions (e.g. `VerifierModelUnavailable` from the local-verifier seam) propagate out of `_run_async` to the API handler, which returns a 500 (or an SSE `error` event on the streaming endpoint). There is no try/except around the `verify_async` call itself.

## Cost

Verification is one extra LLM round-trip per answered query, on the utility model, with a prompt containing up to 4000 characters of context. Set `verify: false` on the request, or run the `fast` profile, to skip it.

---
_Keep updated when the schema, the gating conditions, or the answer annotations change._

---

### Documentation Research Academic Evidence 2026 (`Documentation/research/academic-evidence-2026.md`)

All four research streams are complete. Here is the deliverable.

---

# ACADEMIC STATE OF THE ART IN AGENTIC RETRIEVAL — 2025–2026

**Method / verification status.** ~200 web searches (session budget exhausted) plus ~180 direct fetches of arXiv abs/HTML pages, ACL Anthology, ICLR/OpenReview, the official BRIGHT leaderboard, the Vectara leaderboard repo, and the arXiv metadata API. Every arXiv ID below was either fetched directly or returned by arXiv's own API. Items I could not verify are flagged ⚠️ inline and listed again at the end. Citation counts come from the Semantic Scholar API (retrieved 8 Aug 2026); OpenAlex was tried first and badly undercounts preprints, so I discarded it.

---

## 1. AGENTIC / ITERATIVE RETRIEVAL TRAINING

### 1.1 The 2025 founding wave — what each actually claimed

| System | Paper | Date | Venue | Core claim |
|---|---|---|---|---|
| **DeepRetrieval** | Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, Jiawei Han. arXiv:2503.00223 | 28 Feb 2025 (v3 12 Apr) | arXiv only, cs.IR | GRPO-trained **one-shot query generation**, reward = retrieval metric, no supervised reference queries |
| **R1-Searcher** | Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Ji-Rong Wen (RUC). arXiv:2503.05592 | 7 Mar 2025 | arXiv | **Two-stage outcome-based RL**, no process rewards, no distillation cold start |
| **Search-R1** | Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, Jiawei Han (UIUC + UMass + Google Cloud AI). arXiv:2503.09516, v5 5 Aug 2025 | 12 Mar 2025 | arXiv (**1,317 citations**, 5.3k GitHub stars) | Multi-turn interleaved search+reason, **retrieved-token masking**, outcome-only reward |
| **ReSearch** | Mingyang Chen et al. arXiv:2503.19470 | 25 Mar 2025 | arXiv | End-to-end RL, search treated as part of the reasoning chain, no supervised tool-use trajectories |
| **WebDancer** | Alibaba Tongyi Lab. arXiv (WebAgent family) | May 2025 | **NeurIPS 2025** | 4-stage: data construction → trajectory sampling → SFT → RL |
| **WebSailor** | Kuan Li, Zhongwang Zhang, Huifeng Yin, … Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, Jingren Zhou (Alibaba Tongyi). arXiv:2507.02592 | 3 Jul 2025 | arXiv | High-uncertainty synthetic tasks + RFT cold start + **DUPO** RL |
| **ZeroSearch** | Hao Sun, Zile Qiao, … Fei Huang, Jingren Zhou (Alibaba). arXiv:2505.04588, v3 19 May 2026 | 7 May 2025 | arXiv | Replace the live search engine with a **simulated LLM retriever** during RL |

**Search-R1's actual numbers (verified from v5 HTML, not the abstract).** Setup: 2018 Wikipedia dump, **E5 retriever, top-3 passages**, PPO (more stable) vs GRPO (faster convergence but reward collapse). Average EM across NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, Musique, Bamboogle:

| Method | Qwen2.5-7B avg EM | Qwen2.5-3B avg EM |
|---|---|---|
| Direct | 0.181 | 0.134 |
| CoT | 0.106 | 0.015 |
| IRCoT | 0.239 | 0.181 |
| RAG | 0.304 | 0.270 |
| R1 (reason, no search) | 0.276 | 0.229 |
| **Search-R1-base** | **0.431** | **0.303** |

The widely-quoted "**41% improvement**" is over the RAG baseline. Per-dataset 7B: NQ 0.480, TriviaQA 0.638, PopQA 0.457, HotpotQA 0.433, 2Wiki 0.382, Musique **0.196**, Bamboogle 0.432.

⚠️ **Caveats that matter.** (a) The baseline is a naive top-3 single-shot RAG over a 2018 Wikipedia dump — a weak reference point by 2026 standards. (b) Musique at 0.196 shows the hard multi-hop case is barely moved. (c) The whole evaluation runs against a **fixed local E5 index**, so nothing is learned about live-web behavior. This last point is the wedge for §1.3.

**DeepRetrieval's numbers:** publication search recall **65.07%** vs 24.68% prior SOTA; trial search recall **63.18%** vs 32.11%; beats GPT-4o and Claude-3.5-Sonnet on 11 of 13 datasets with a **3B** model. ⚠️ These are literature-search domains (PubMed/ClinicalTrials.gov), not general QA; the "hacking real search engines" framing is doing a lot of work — the gains are largely query-formulation gains against a fixed API.

### 1.2 The deep-research agent line and where the open frontier sits in mid-2026

**Tongyi DeepResearch Technical Report** (56 authors, Alibaba Tongyi). arXiv:2510.24701, 28 Oct 2025, v3 18 May 2026. 30.5B total / **3.3B activated** MoE, 128K context, agentic mid-training + agentic post-training. Verified Table 1:

| Model | HLE | BrowseComp | BrowseComp-ZH | GAIA | xbench-DS | WebWalker | FRAMES |
|---|---|---|---|---|---|---|---|
| **Tongyi DeepResearch 30B-A3B** | **32.9** | 43.4 | 46.7 | **70.9** | **75.0** | **72.2** | **90.6** |
| OpenAI DeepResearch | 26.6 | **51.5** | 42.9 | 67.4 | — | — | — |
| OpenAI o3 (ReAct) | 24.9 | 49.7 | **58.1** | — | 67.0 | 71.7 | 84.0 |
| DeepSeek-V3.1 (ReAct) | 29.8 | 30.0 | 49.2 | 63.1 | 71.0 | 61.2 | 83.7 |
| Claude-4-Sonnet (ReAct) | 20.3 | 12.2 | 29.1 | 68.3 | 65.0 | 61.7 | 80.7 |
| GLM-4.5 (ReAct) | 21.2 | 26.4 | 37.5 | 66.0 | 70.0 | 65.6 | 78.9 |
| Kimi Researcher | 26.9 | — | — | — | 69.0 | — | 78.8 |

Note the split verdict: a trained 30B-A3B open model **beats OpenAI Deep Research on HLE (32.9 vs 26.6) and GAIA (70.9 vs 67.4) but loses on BrowseComp-EN (43.4 vs 51.5)**. Also note Claude-4-Sonnet's 12.2 on BrowseComp under a plain ReAct harness — prompted frontier models without a deep-research scaffold are not competitive on this task class.

**What superseded them in 2026:**

- **LiteResearcher** (Bince Qu, Wanli Li, Bo Pan, Jianyu Zhang, Zheng Liu, Pan Zhang, Wei Chen, Bo Zhang). arXiv:2604.17931, 20 Apr 2026, v5 26 Jul 2026. Builds a **"lite virtual world"** mirroring real search dynamics so RL doesn't depend on a live search API. LiteResearcher-**4B**: GAIA-Text 71.3, xbench-DS 78.0, FRAMES 83.1, WebWalker 72.7, Seal-0 41.8, HLE 22.0, BrowseComp 27.5, BrowseComp-ZH 32.5. **Parity with Claude-4.5-Sonnet on GAIA (71.3 vs 71.2) and beats Tongyi-30B on GAIA and xbench, at 4B.** Ablation (their Table 9): **SFT alone 55.58 GAIA → RL 71.3 (+15.7); xbench 64.25 → 78.0 (+13.8).** Stated limitations: 128K context exhausts on deep BrowseComp chains; gains depend on a 32M-page enriched corpus and generalization beyond it is unexplored.

- **OpenSeeker-v2** (Yuwen Du, Rui Ye, Shuo Tang, Keduan Huang, Xinyu Zhu, Yuzhu Cai, Siheng Chen). arXiv:2605.04036, 5 May 2026. **SFT only, no RL**, 10.6k trajectories: **BrowseComp 46.0, BrowseComp-ZH 58.1, HLE 34.6, xbench 78.0** at 30B/ReAct. Self-described as the first SOTA search agent at its scale from a purely academic team using only SFT.

- **WebExplorer** (Junteng Liu et al.). arXiv:2509.06501, 8 Sep 2025. **WebExplorer-8B**, 128K, up to 100 tool turns, averages **16 search turns after RL**; higher BrowseComp-en/zh than **WebSailor-72B** and best among ≤100B on WebWalkerQA and FRAMES.

**→ OPEN DEBATE #1 — is RL actually necessary?** LiteResearcher measures RL contributing **+15.7 GAIA points over its own SFT checkpoint**. OpenSeeker-v2, one month later, reaches **higher BrowseComp (46.0 vs 27.5) and HLE (34.6 vs 22.0) with pure SFT on 10.6k curated high-difficulty trajectories**. Different model scales (4B vs 30B) so it is not a clean head-to-head, but the field has no controlled experiment isolating RL from trajectory-data quality at fixed scale. This is unresolved and under-discussed.

### 1.3 The critical literature — this is the part that changes conclusions

**⭐ BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents.** Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, **Jimmy Lin** (Waterloo + CSIRO + CMU + Queensland). arXiv:2508.06600, 8 Aug 2025. **ACL 2026 Main** (aclanthology.org/2026.acl-long.1023). 155 citations.

Fixed curated corpus, human-verified supporting documents, mined hard negatives — so retriever and agent can be varied independently. Verified Table 1 accuracy:

| Agent | + BM25 | + Qwen3-Embed-8B |
|---|---|---|
| **GPT-5** | 55.90% | **70.12%** |
| o3 | 49.28% | 63.49% |
| gpt-oss-120B-high | 28.67% | 42.89% |
| Gemini 2.5 Pro | 19.04% | 28.67% |
| Claude Opus 4 | 15.54% | 36.14% |
| Claude Sonnet 4 | 14.34% | 36.75% |
| gpt-4.1 | 14.58% | 35.42% |
| **Search-R1-32B** | **3.86%** | **10.36%** |
| Qwen3-32B | 3.49% | 10.36% |

Four findings, each load-bearing:

1. **Retriever quality outweighs agent choice.** Swapping BM25 → Qwen3-Embedding-8B buys GPT-5 **+14.2 points** and roughly doubles weaker agents. Better retrievers also **reduce** search-call count.
2. **RL-trained open agents do not transfer.** Search-R1-32B scores **exactly what its untrained base Qwen3-32B scores (10.36% with the good retriever)**. The RL training bought nothing outside its training distribution. This is the single most damaging result for the Search-R1 lineage's generalization claims.
3. **The bottleneck is interleaved tool-use reasoning, not knowledge.** Authors: open models "do not substantially lag behind proprietary models in their ability to answer questions when provided with sufficient evidence." Proprietary models average **20+ search calls/query**; open models fewer than 2 despite explicit tool prompting.
4. **Reasoning-specialized retrievers underperform scaled general ones inside agentic loops.** Qwen3-Embedding-8B: **14.5% Recall@5, 20.3 nDCG@10**; ReasonIR-8B: **12.2% / 16.8**. Note the absolute ceiling — the *best* retriever gets 20.3 nDCG@10. Enormous headroom.

**⭐ Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?** Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, Gao Huang (Tsinghua). arXiv:2504.13837, 18 Apr 2025, v5 24 Nov 2025. **NeurIPS 2025 Oral**; ICML 2025 AI4MATH workshop best paper. RLVR-trained models beat base models at pass@1 but **base models overtake at large k** — the reasoning was already in the base model; RLVR narrows the sampling distribution. Six RLVR algorithms are "far from optimal in leveraging the potential of the base model." Not search-specific, but it is the theoretical frame for interpreting every "+41% over RAG" claim in this section.

**⭐ Demystifying deep search / WebDetective.** Maojia Song, Renhang Liu, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, Jingren Zhou, Dorien Herremans, Soujanya Poria. arXiv:2510.05137, 1 Oct 2025 (v3 10 Dec 2025). Two indictments of current practice: **most benchmarks leak the reasoning path in the question text**, and single-pass-rate scoring collapses distinct failure modes. Hint-free multi-hop questions in a traceable Wikipedia sandbox, **25 SOTA models**: systematic failure at knowledge utilization *despite sufficient evidence*, and near-absent appropriate refusal when evidence is missing. "Today's systems excel at executing given reasoning paths but fail when required to discover them." Their EvidenceLoop workflow (verification loops + systematic evidence tracking) is the proposed fix.

**⭐ How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1.** Yinuo Xu, Shuo Lu, Jianjie Cheng, Meng Wang, Qianlong Xie, Xingxing Wang, Ran He, Jian Liang. arXiv:2602.19526, 23 Feb 2026. Decoupled ablation of the three axes. Findings: the **"Fast Thinking" prompt template is more stable and better-performing than the Slow Thinking template used in prior work**; F1 rewards cause training collapse via answer avoidance (fixed by action-level penalties); **REINFORCE beats PPO with fewer search actions, and GRPO is the least stable**. Search-R1++ moves Qwen2.5-7B 0.403 → 0.442 and 3B 0.289 → 0.331. Read this alongside Search-R1's own PPO-over-GRPO finding — the field's optimizer consensus is unsettled.

### 1.4 The 2026 research frontier has moved to efficiency and context, not capability

Outcome-only RL **provably induces over-search**: the ratio of no-search trajectories drops toward zero while redundant-search ratio rises. Four verified responses:

- **HiPRAG** (Peilin Wu, Mian Zhang, Kun Wan, Wentian Zhao, Kaiyu He, Xinya Du, Zhiyu Chen). arXiv:2510.07794, v2 11 Apr 2026. **Accepted ICLR 2026.** Hierarchical process rewards over decomposed reasoning steps: average accuracy **65.4% (3B) / 67.2% (7B)** across 7 QA benchmarks with **over-search rate driven to 2.3%**.
- **SAAS** (Yunbo Tang, Chengyi Yang, Shiyu Liu, Zhishang Xiang, Zerui Chen, Qinggang Zhang, Jinsong Su). arXiv:2605.29796, 28 May 2026. Contrasts search-disabled vs search-enabled rollouts to model the knowledge boundary; boundary-aware trajectory penalties; stage-wise curriculum to avoid reward hacking. ⚠️ Abstract carries no numbers.
- **AutoSearch** (Jingbo Sun et al., incl. Dongbin Zhao). arXiv:2604.17337, 19 Apr 2026. Self-generated intermediate answers identify a **minimal sufficient search depth**; reward attainment, penalize over-search. ⚠️ No numbers in abstract.
- **FoldAct** (Jiaqi Shao, Yufeng Miao, Wei Zhang, Bing Luo). arXiv:2512.22733, 28 Dec 2025. Context folding for long-horizon RL: identifies gradient dilution on summary tokens, self-conditioning instability, and per-turn context recomputation. **5.19× training speedup.**
- **Erase to Improve (ERL)** (Ziliang Wang et al.). arXiv:2510.00861, v2 20 Apr 2026. Identify → erase → regenerate faulty reasoning steps. **3B: +8.48% EM / +11.56% F1; 7B: +5.38% EM / +7.22% F1** on HotpotQA/MuSiQue/2Wiki/Bamboogle.
- **Agentic-R** (Wenhan Liu, Xinyu Ma, Yutao Zhu, Yuchen Li, Daiting Shi, Dawei Yin, Zhicheng Dou). arXiv:2601.11888, 17 Jan 2026. **Bidirectional iterative co-optimization of the search agent and the retriever**, with passage utility measured by both local relevance and global answer correctness. Directionally the most interesting 2026 idea — it treats the BrowseComp-Plus finding (retriever dominates) as a training target. ⚠️ No numbers in abstract.

**Surveys.** "A Comprehensive Survey on Reinforcement Learning-based Agentic Search" — Minhua Lin, Zongyu Wu, Zhichao Xu, Hui Liu, Xianfeng Tang, Qi He, Charu Aggarwal, Hui Liu, Xiang Zhang, Suhang Wang (Penn State + Amazon + IBM). arXiv:2510.16724, 19 Oct 2025, 38pp. Also "The Landscape of Agentic Reinforcement Learning for LLMs" (arXiv:2509.02547) and "RL Foundations for Deep Research Systems: A Survey" (arXiv:2509.06733).

**Honest bottom line for §1.** RL-trained search behavior demonstrably beats prompted loops **within a fixed base model on the training distribution** (Search-R1 0.431 vs 0.304 RAG; LiteResearcher +15.7 GAIA over its own SFT). It demonstrably **does not transfer** to a new corpus/retriever (Search-R1-32B = its base model on BrowseComp-Plus). The 2026 SOTA on hard deep-research benchmarks belongs to large-scale trajectory curation (SFT or RL) plus a strong retriever — and **retriever choice moves accuracy more than agent choice does**.

---

## 2. REASONING-AWARE RETRIEVAL

### 2.1 BRIGHT and what happened to it

**BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval.** Princeton et al. arXiv:2407.12883, 16 Jul 2024. **ICLR 2025** (OpenReview ykuc5q381b). 177 citations. 1,384–1,398 real-world queries, 12 domains (economics, psychology, math, coding, robotics, StackExchange splits, theorem retrieval), corpora 7.9K–414K docs.

At release: **max nDCG@10 = 24.3**. SFR-Embedding-Mistral, then MTEB #1 at 59.0, scored **18.3**. LLM reasoning-step query augmentation helped but stayed under 30. Authors report BRIGHT is robust to data leakage — fine-tuning on the retrieval documents barely moves scores.

**The official leaderboard as of 8 Aug 2026** (brightbenchmark.github.io), short-document track, avg nDCG@10 over 12 datasets:

| Rank | System | Score | Date | Reranker |
|---|---|---|---|---|
| 1 | Mira-Reasoning-Retrieval (Forward AI Labs) | **66.9** | 22 Apr 2026 | Yes |
| 2 | INF-X-Retriever | 63.4 | 20 Dec 2025 | Yes |
| 3 | RakanEmbed4B | 52.4 | 20 Mar 2026 | Yes |
| 4 | NeMo Retriever Agentic Retrieval (NVIDIA) | 50.9 | 13 Mar 2026 | Yes |
| 5 | DIVER-v3-GroupRank | 46.8 | 13 Nov 2025 | Yes |
| 6 | BGE-Reasoner-0928 (BAAI) | 46.4 | 13 Oct 2025 | Yes |
| 7 | Lattice Hierarchical Retrieval (Google) | 42.1 | 17 Oct 2025 | Yes |
| — | BM25 + GPT-4 reasoning + reranking | 30.4 | — | Yes |
| — | **BM25 alone** | **14.5** | — | No |

Long-document track (avg Recall@1, 8 datasets, docs up to ~40k): BM25 = 11.4.

**⚠️ Four caveats that reframe the 66.9.**
1. **Every single top entry uses a reranker.** BRIGHT in 2026 measures *pipelines with heavy test-time compute*, not retrievers.
2. **The gain is mostly LLM query expansion, not retrieval.** DIVER (Duolin Sun et al., Ant Group, arXiv:2508.07995, v5 2 Apr 2026) reports **46.8 overall but 31.9 on original queries** — a 14.9-point gap attributable to iterative LLM query rewriting. BM25 alone 14.5 → BM25 + GPT-4 reasoning + rerank 30.4 makes the same point: **LLM expansion roughly doubles a 1994 lexical baseline**.
3. **Verification varies by entry.** Ranks 5–7 link arXiv papers or GitHub; ranks 1–3 link personal/company webpages. Treat the 66.9 as an unrefereed submission.
4. Compare to BrowseComp-Plus, where the best retriever inside an agentic loop reaches **20.3 nDCG@10**. Standalone BRIGHT scores and agentic-loop utility are not the same quantity.

**⭐ The reproducibility audit: Lighting the Way for BRIGHT.** Sahel Sharifymoghaddam, Yijun Ge, **Jimmy Lin**. arXiv:2509.02558, 2 Sep 2025, **v2 1 Jun 2026, SIGIR 2026 Reproducibility Track**. Findings: (a) BRIGHT's published baseline silently uses **query-side BM25 ("BM25Q")**, an undocumented detail that consistently outperforms standard BM25 on long queries — meaning much of the literature has been comparing against a mis-specified lexical baseline; (b) **BM25Q's advantage is largely BRIGHT-specific** and does not carry to five other benchmarks, while fusion with standard BM25 does; (c) an audit of the BRIGHT corpus **uncovers data-quality issues that affect evaluation**.

**Successors.**
- **BRIGHT-Pro** — "Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems." Yilun Zhao, Jinbiao Wei, Tingyu Song, Siyue Zhang, Chen Zhao, Arman Cohan. arXiv:2605.04018, 5 May 2026, **ACL 2026**. Direct critique: "benchmarks such as BRIGHT provide narrow gold sets and evaluate retrievers **in isolation**, while synthetic training corpora optimize single-passage relevance rather than **evidence portfolio construction**." Expert-annotated multi-aspect gold evidence; evaluates under **both static and agentic protocols**; introduces RTriever-Synth (aspect-decomposed, positive-conditioned hard negatives) and RTriever-4B (LoRA on Qwen3-Embedding-4B). Key claim: **agentic evaluation exposes retriever behaviors hidden by standard metrics.**
- **MM-BRIGHT** — Abdelrahman Abdallah et al. arXiv:2601.09562, 14 Jan 2026. 2,803 queries, 29 technical domains, four task types. BM25 text-only **8.5** nDCG@10; best text-only (DiVeR) **32.2**; Nomic-Vision multimodal-to-text **27.6**.

### 2.2 Reasoning-augmented retrievers and rerankers

**ReasonIR-8B** — Meta FAIR (facebookresearch/ReasonIR). arXiv:2504.20595, 29 Apr 2025. 76 citations. Synthetic pipeline generating reasoning-requiring queries plus **plausibly-related-but-unhelpful hard negatives**. **29.9 nDCG@10 on BRIGHT without reranker, 36.9 with.** Downstream: **+6.4% MMLU, +22.6% GPQA** over closed-book, beating other retrievers and search engines.

**RaDeR** — Debrup Das, Sam O'Nuallain, Razieh Rahimi. arXiv:2505.18405, 23 May 2025. Trained from **retrieval-augmented math reasoning trajectories** with self-reflective relevance evaluation; generalizes to BRIGHT and RAR-b. Two notable claims: it is the **first dense retriever to outperform BM25 when queries are chain-of-thought reasoning steps** (an admission of how bad the prior state was), and it matches/beats ReasonIR using **2.5% of ReasonIR's training data**.

**Reason-ModernColBERT** (LightOn, 149M params). **⚠️ Blog + HuggingFace model card only — no arXiv paper.** The marketing says it "outperforms all models up to 7B on BRIGHT" and beats ReasonIR-8B "by more than 2.5 nDCG on average." Its own model-card table says otherwise:

| Split group | Reason-ModernColBERT (149M) | ReasonIR-8B | BM25 |
|---|---|---|---|
| Mean StackExchange | **27.43** | 24.76 | 17.21 |
| Mean Coding | 19.79 | **22.75** | 16.15 |
| Mean Theorem | 15.38 | **24.60** | 7.17 |
| **Full BRIGHT mean** | 22.62 | **24.38** | 14.53 |

**The +2.5 claim holds only on StackExchange. On the full BRIGHT mean it loses, 22.62 vs 24.38, and loses badly on Theorem.** The "45× smaller" framing is real; the "beats it" framing is split-selective. License is cc-by-nc-4.0 (training-data restriction), so it is not commercially usable either.

**ReasonRank** — Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin, Zhicheng Dou (RUC + Baidu). arXiv:2508.07050, 9 Aug 2025, **ACL 2026 Main**. DeepSeek-R1-synthesized reasoning-intensive ranking labels; cold-start SFT then RL with a **multi-view ranking reward** for the multi-turn nature of listwise ranking. Outperforms baselines with **lower latency than pointwise rerankers**. Was #2 on BRIGHT (40.8 reranking RaDeR, Aug 2025).

**BGE-Reasoner** (BAAI + USTC, VectorSpaceLab/agentic-search). Multiple rewritten queries + ensembled reranking across model sizes. **46.4 nDCG@10 (BGE-Reasoner-0928, Oct 2025)**, held BRIGHT #1 briefly. ⚠️ GitHub/model-card, no verified paper.

### 2.3 Test-time compute for retrieval — the most interesting thread

**Rank1** — Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, Benjamin Van Durme (JHU HLTCOE). arXiv:2502.18418, 25 Feb 2025, **CoLM 2025**, 72 citations. First reranker trained to exploit test-time compute; **600,000+ R1 reasoning traces over MS MARCO** open-sourced. SOTA on reasoning and instruction-following ranking, "works remarkably well out of distribution."

**⭐ LATTICE: LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrieval.** Nilesh Gupta, Wei-Cheng Chang, Ngot Bui, Cho-Jui Hsieh, Inderjit S. Dhillon (Google). arXiv:2510.13217, 15 Oct 2025, v2 25 May 2026. **Base LATTICE with a single off-the-shelf LLM reaches 46.7 nDCG@10 on BRIGHT — matching the best fine-tuned ensemble baseline overall — and LATTICE++ (fused with cheap retrieval) reaches 49.1.** Budget behavior: "reranking offers a better tradeoff at low token budgets, but LATTICE converges to a higher asymptote after a moderate budget."

This is the strongest single result in reasoning-aware retrieval, and it is under-cited: **an untrained LLM doing hierarchical search matches purpose-trained reasoning-retrieval ensembles.** It reframes the whole area as a test-time-compute allocation problem rather than a representation-learning problem.

**Reranker-Guided Search (RGS)** — Haike Xu, Tong Chen. arXiv:2509.07163, 8 Sep 2025. Greedy search on proximity graphs to select *which* documents to send to the reranker, rather than reranking a fixed top-k. **+3.5 BRIGHT, +2.9 FollowIR, +5.1 M-BEIR**, all within a 100-document reranker budget.

**State Machine Reasoning (SMR)** — Dohyeon Lee, Yeonseok Jeong, Seung-won Hwang. arXiv:2505.23059, 29 May 2025. Discrete Refine/Rerank/Stop actions with early stopping instead of free-form CoT. On BEIR and BRIGHT: **+3.4% nDCG@10 while cutting token usage 74.4%.** Generalizes across LLMs and retrievers without task-specific tuning. The cleanest "overthinking is real in IR" result.

**Verbal-R3** — Sangkwon Park, Donghun Kang, Jisoo Mok, Sungroh Yoon (SNU). arXiv:2605.01399, 2 May 2026, **ACL 2026 Main**. A "Verbal Reranker" emitting both relevance scores and analytic narratives connecting query to context, plus **relevance-guided test-time scaling** for trajectory expansion.

**⭐ Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction.** Zhuofeng Li, Haoxiang Zhang, Cong Wei, Pan Lu, Ping Nie, Yi Lu, Yuyang Bai, Shangbin Feng, Hangxiao Zhu, Ming Zhong, Yuyu Zhang, Jianwen Xie, **Yejin Choi, James Zou, Jiawei Han, Wenhu Chen, Jimmy Lin**, Dongfu Jiang, Yu Zhang. arXiv:2605.05242, 3 May 2026. Agents search raw corpora with **grep and shell commands** — no embedding model, no vector index. Claims DCI "substantially outperforms sparse, dense, and reranking baselines" on BRIGHT and BEIR, and performs strongly on BrowseComp-Plus and multi-hop QA **without any semantic retriever**. Thesis: "retrieval quality depends not only on reasoning ability but also on the **resolution of the interface** through which models access corpora." ⚠️ **I could not extract the numeric tables** — the arXiv HTML 404s and the PDF's tables are in compressed streams. Existence, authorship, date, and qualitative claims verified; **the numbers are not**. Given the author list this deserves a follow-up read.

**Orion** — Supriti Vijay, Aman Priyanshu, Anu Vellore, Baturay Saglam, Amin Karbasi. arXiv:2511.07581, 10 Nov 2025. 350M–1.2B models doing iterative retrieval via synthetic trajectories + SFT + RL + inference-time beam search: **SciFact 77.6 (vs 72.6), BRIGHT 25.2 (vs 22.1), NFCorpus 63.2 (vs 57.8)**, beating retrievers 200–400× larger on 5 of 6 benchmarks with 3% of the training data.

**⚠️ Withdrawn paper:** "Adaptive Retrieval for Reasoning-Intensive Retrieval" (REPAIR), arXiv:2601.04618, submitted 8 Jan 2026, **withdrawn by the authors 14 Apr 2026**. It claimed +5.6pp. Do not cite it.

**→ OPEN DEBATE #2 — is reasoning-aware retrieval a training problem or a test-time-compute problem?** ReasonIR/RaDeR/BGE-Reasoner say train the retriever. LATTICE (Google) matches the best trained ensembles with an **off-the-shelf** LLM. BrowseComp-Plus says the reasoning-specialized retriever (ReasonIR-8B) **loses to a general scaled embedder (Qwen3-8B)** inside an agentic loop. SMR says a large fraction of the reasoning tokens are pure waste (−74.4% tokens for +3.4% nDCG). The training camp has never been evaluated against the test-time-compute camp under a matched compute budget.

---

## 3. RETRIEVAL ARCHITECTURE COMPONENTS

### 3.1 Late-interaction revival

**The modern models.** *GTE-ModernColBERT-v1* (LightOn, on `gte-modernbert-base`, 128-dim/token): **BEIR avg nDCG@10 = 54.67** vs answerai-colbert-small 53.79 reported / 53.35 on LightOn's rerun. LongEmbed(32k) mean 88.39. ⚠️ Vendor model-card numbers; the **+0.88 to +1.32 margin is under 1.5 points**, and the card's rerun of the competitor scores below the competitor's published figure — the classic pattern warranting skepticism.

**The best-documented efficiency table in the lineage** — *mxbai-edge-colbert-v0* (Rikiya Takehi, Benjamin Clavié, Sean Lee, Aamir Shakir; Mixedbread/Answer.AI). arXiv:2510.14880, 16 Oct 2025:

| model | params | dim | BEIR avg | LongEmbed 32k | CPU time | Mem/10k docs |
|---|---|---|---|---|---|---|
| mxbai-edge-colbert-17m | 17M | 48 | 0.490 | 0.847 | 487s | **275 MB** |
| mxbai-edge-colbert-32m | 32M | 64 | 0.521 | 0.849 | 589s | 366 MB |
| ColBERTv2 | 130M | 128 | 0.488 | 0.428 | **1540s** | **732 MB** |
| answerai-colbert-small-v1 | 33M | 96 | 0.534 | — | 621s | 549 MB |
| GTE-ModernColBERT-v1 | 130M+ | 128 | 0.547 | 0.898 | — | — |

**The efficiency lineage, verified:**
- **XTR** — Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wei Chang, Vincent Y. Zhao (Google DeepMind). arXiv:2304.01982, **NeurIPS 2023**. +2.8 BEIR nDCG@10, scoring **2–3 orders of magnitude cheaper** than ColBERT.
- **MUVERA** — Laxman Dhulipala, Majid Hadian, Rajesh Jayaram, Jason Lee, Vahab Mirrokni (Google Research). arXiv:2405.19504, 29 May 2024, **v2 8 Jun 2026 corrected the Theorem 2.1 dimension bound**. Reduces multi-vector to single-vector MIPS via asymmetric Fixed Dimensional Encodings. **2–5× fewer candidates at equal recall; 10% higher recall with 90% lower latency** vs PLAID across BEIR. Now theoretically bracketed by the same group: arXiv:2607.20393 proves near-matching lower bounds (MUVERA is near-optimal for FDE-style reductions); arXiv:2606.23475 proves multi-vector is formally more expressive than single-vector.
- **CRISP** — Veneroso, Jayaram, Rao, Hernández Ábrego, Hadian, Cer (Google Research). arXiv:2505.11471, 16 May 2025. Clustering trained *into* the model: **~3× vector reduction while beating the unpruned model**; 11× at 3.6% loss.
- **WARP** — Jan Luca Scheerer, Matei Zaharia, Christopher Potts, Gustavo Alonso, **Omar Khattab**. arXiv:2501.17788, **SIGIR 2025**. **41× latency reduction vs XTR reference; 3× over ColBERTv2/PLAID.**
- **ColBERT-serve** (arXiv:2504.14903): memory-mapped scoring, **90% RAM reduction**. **Constant-space multi-vector** — MacAvaney, Mallia, Tonellotto, **ECIR 2025** (arXiv:2504.01818): fixed vector count decoupled from doc length. ⚠️ exact deltas in PDF, unverified.
- 2026 kernels/indexes: **ColBERTSaR** (Eugene Yang, Andrew Yates, Dawn Lawrie, Mayfield, Samuel, Jha, JHU HLTCOE; arXiv:2606.05568) **50–70% smaller than a 1-bit PLAID index**; **No More K-means** (arXiv:2605.30120, **ICML 2026**) **15× faster indexing than ColBERTv2**; **TileMaxSim** (arXiv:2606.26439) 100K-candidate scoring **268ms → 1.2ms** on H100; **FLASH-MAXSIM** (IBM, arXiv:2605.29517) **9× less inference / ~100× less training memory** at ColPali scale; **FastLane** (Ramnath Kumar, Prateek Jain, Cho-Jui Hsieh, Google; arXiv:2601.06389) **up to 30× lower compute**; **LEMUR** (arXiv:2601.21853) order-of-magnitude faster MV search; **PLAID-PRF** (Xiao Wang, MacAvaney, Macdonald; arXiv:2607.18626) **+4.3% nDCG@10** from pseudo-relevance feedback over PLAID centroids.

**Honest 2026 storage/latency multiplier.** Naive ColBERTv2 at 128-dim fp16/token is **~732 MB per 10k docs** vs ~15 MB for a 768-dim single-vector index — that is the folk "50–100×". Stack the 2025–26 techniques (dim 128→48, small backbone, CRISP training-time clustering, PQ) and you land at **~3–10×**. Anyone quoting 100× in 2026 is citing ColBERTv1. Latency: PLAID's fastest measured point is **73–80.5 ms/query** on MS MARCO; WARP claims 3× on top; **~10–30 ms/query** is the current well-engineered figure.

**⭐ The strongest paper arguing late interaction is NOT worth it.** "A Reproducibility Study of PLAID" — **Sean MacAvaney & Nicola Tonellotto, SIGIR 2024 Reproducibility Track**, arXiv:2404.14989.

| PLAID setting | nprobe | t_cs | ndocs | latency | DL19 nDCG@10 |
|---|---|---|---|---|---|
| (a) | 1 | 0.50 | 256 | 80.5 ms | 0.739 |
| (b) | 2 | 0.45 | 1024 | 103.4 ms | 0.745 |
| (c) | 4 | 0.40 | 4096 | 163.9 ms | 0.745 |

**Re-ranking a BM25 candidate list with ColBERTv2 runs at as low as 9 ms/query at n=200, vs 73 ms/query for the fastest PLAID pipeline — an ~8× latency advantage — reaching RR@10 = 0.373 on MS MARCO Dev**, which the authors note beats early BERT cross-encoders. PLAID only wins at high latency where lexical recall binds. Their mechanistic finding: **most PLAID token clusters are predominantly aligned with a single token** — the centroid machinery approximates lexical matching. Their charge is methodological: prior late-interaction efficiency work **omitted the obvious baseline**. **This has not been rebutted with an updated head-to-head.**

Corroborating skepticism: *Are LLM-Based Retrievers Worth Their Cost?* (Abdallah, Holdcroft, Ali, Jatowt, **SIGIR 2026**, arXiv:2604.03676) — 14 retrievers × 12 BRIGHT tasks; large LLM bi-encoders incur substantial latency for modest gains, reasoning augmentation shows diminishing returns, and **confidence calibration is weak across all families**, so raw scores are unreliable for routing. *KaLM-Reranker-V1: **Fast but Not Late Interaction*** (arXiv:2606.22807). *MICE* (arXiv:2602.16299): **4× lower latency than standard cross-encoders while matching ColBERT-class quality**. *MINER* (arXiv:2605.06460): narrows the MV-to-dense gap to **0.2 nDCG@5**. *Your Embedding Model is SMARTer Than You Think* (arXiv:2605.24938): late interaction over **frozen hidden states** of existing single-vector models, no dedicated MV index needed.

And the proponents concede it: the **LIR @ ECIR 2026** workshop proposal (Benjamin Clavié, Xianming Li, Antoine Chaffin, **Omar Khattab**, Tom Aarsen, Manuel Faysse, Jing Li; arXiv:2511.00444) says these models pose "significant challenges of efficiency, usability, and integration" and "prohibitive storage and computational overhead," and explicitly solicits **negative or puzzling results**.

**⭐ The strongest pro-multi-vector argument is theoretical.** "On the Theoretical Limitations of Embedding-Based Retrieval" — **Orion Weller, Michael Boratko, Iftekhar Naim, Jinhyuk Lee (Google DeepMind + JHU)**, arXiv:2508.21038, v2 12 Mar 2026, **ICLR 2026**. The number of top-k document subsets returnable by *any* query is bounded by embedding dimension. LIMIT (50k docs, 1000 queries, k=2):

| model | R@2 | R@10 | R@100 |
|---|---|---|---|
| **BM25** | **97.8** | **100.0** | **100.0** |
| GTE-ModernColBERT (MV) | 23.1 | 34.6 | **54.8** |
| Promptriever Llama3 8B | 3.0 | 6.8 | 18.9 |
| GritLM 7B | 2.4 | 4.1 | 12.9 |
| Gemini Embedding | 1.6 | 3.5 | 10.0 |
| E5-Mistral 7B | 1.3 | 2.2 | 8.3 |
| Qwen3 Embedding | 0.8 | 1.8 | 4.8 |

⚠️ **LIMIT is adversarially constructed to hit the bound.** It proves an existence claim, not a claim about natural query distributions. And it is contested: **Bangachev, Bresler, Kogan, Polyanskiy (MIT)**, arXiv:2605.23556, 22 May 2026, don't dispute the bound but prove near-optimal margins are achievable at **d = O(k log(n/k))** in the sparse regime (necessary and sufficient), margin Θ(k^(-1/2)), holding to trillions of points — i.e. d≈1000 is provably near-sufficient. Separately, *Spectral Retrieval* (arXiv:2605.24764) lifts LIMIT-small R@10 from **0.33 to 0.90 without retraining**, suggesting part of the failure is a scoring-function artifact.

**→ OPEN DEBATE #3.** Weller et al. (DeepMind, ICLR'26): single-vector is dimensionally capped, MV is the escape. Bangachev et al. (MIT): the cap is far looser than worst-case. MacAvaney & Tonellotto (SIGIR'24): even granting MV's quality edge, **BM25 + ColBERT reranking dominates the Pareto frontier at deployable latencies**. No 2026 paper reconciles the three.

### 3.2 Hybrid sparse+dense fusion

**The canonical fusion-function result is still pre-2025 and still unrebutted.** "An Analysis of Fusion Functions for Hybrid Retrieval" — Sebastian Bruch, Siyu Gai, Amir Ingber, **ACM TOIS Aug 2023**, arXiv:2210.11934. RRF is **sensitive to its parameters**; convex combination is **agnostic to score-normalization choice** (min-max, z-score, any linear transform are rank-equivalent); **CC beats RRF in- and out-of-domain** and is sample-efficient. ⚠️ **Practice/theory gap:** essentially every 2025–26 applied paper uses RRF anyway, because it needs no tuning data. Nobody has re-tested.

**2025–26 evidence:**

| Paper | Date | Finding |
|---|---|---|
| From Retrieval to Generation (Abdallah, Mozafari, Piryani, Ali, Jatowt), arXiv:2502.20245 | Feb 2025 | **BEIR nDCG@10: BM25 43.42 → hybrid 52.59 (+9.17)** |
| From BM25 to Corrective RAG: Text-and-Table (Akarsu, Karaman, Mierbach), arXiv:2604.01733 | Apr 2026 | **23,088 financial QA queries** — largest here. Two-stage hybrid + neural rerank Recall@5 = **0.816**. **BM25 outperforms dense on financial documents.** Recommends hybrid RRF + cross-encoder as the minimum viable baseline |
| Dissecting Agentic RAG, arXiv:2606.21553 | Jun 2026 | **Fixed hybrid RRF beats rule-based adaptive routing (+1.8 EM, +1.9 F1)**; EM 53.2% HotpotQA. Negative result for adaptive routing |
| KohakuRAG (Yeh, Ku, Huang, Tu), arXiv:2603.07612 | Mar 2026 | ⚠️ **Contrarian: hierarchical dense alone matches hybrid; BM25 adds only +3.1pp** |
| Training-Free Lexical-Dense Fusion for Conversational Memory, arXiv:2606.04194 | Jun 2026 | Late-interaction dense + BM25: **+8.8 to +17.2 Hit@1**, no training |
| DAT: Dynamic Alpha Tuning (Hsu, Tzeng), arXiv:2503.23013 | Mar 2025 | LLM-judged per-query dense/BM25 weighting beats fixed-weight hybrid |
| Hybrid Retrieval for Hallucination Mitigation (ISTI-CNR), arXiv:2504.05324 | 2025 | On HaluBench, hybrid gives highest accuracy on fails and lowest hallucination + rejection rates |

**Where BM25 still wins outright:** LIMIT (97.8 R@2 vs 0.8 for the best dense); financial text-and-table at 23k queries; low-latency first-stage (9 ms/q + rerank beating the entire PLAID frontier below ~70 ms/q). **Where it clearly loses:** aggregate suites — HAKARI-Bench (551 tasks, 43 languages) puts BM25 at **50.24** macro nDCG@10×100 vs best sub-1B dense **64.93**; and agentic loops — BrowseComp-Plus, GPT-5 at **55.9% with BM25 vs 70.12% with Qwen3-Embedding-8B**, with the authors noting BM25's documents are "less useful in the iterative deep research process."

**→ OPEN DEBATE #4.** BM25's standing is strongly task-conditional; blanket claims fail in both directions. And **+9.17 (arXiv:2502.20245) vs +3.1pp (KohakuRAG) for the hybrid gain is a direct, unexplained, unreplicated conflict** — plausibly a corpus-structure effect where hierarchical indexing already captures what BM25 contributes.

**Reproducibility infrastructure:** Pyserini (Lin et al., SIGIR 2021) remains the reference. ⚠️ "Gosling Grows Up: Retrieval with Learned Dense and Sparse Representations Using Anserini" (Lin group, SIGIR 2025) surfaced in search only — arXiv ID unverified, not guessed. *GPUSparse* (arXiv:2606.26441): **235× speedup over Pyserini CPU at 8.8M docs** ⚠️ single-author, unreviewed.

### 3.3 Learned sparse

**SPLADE-v3** — Carlos Lassance, Hervé Déjean, Thibault Formal, Stéphane Clinchant (Naver Labs Europe). arXiv:2403.06789, 11 Mar 2024. **>40 MRR@10 on MS MARCO dev; +2% out-of-domain on BEIR** over SPLADE++. Meta-analysis over **40+ query sets**: statistically significantly better than BM25 and SPLADE++, **competitive with cross-encoder rerankers**.

**⭐ The 2026 headline: LACONIC** — Zhichao Xu, Shengyao Zhuang, Crystina Zhang, Xueguang Ma, Yijun Tian, Maitrey Mehta, **Jimmy Lin**, Vivek Srikumar (Utah + CSIRO + Waterloo). arXiv:2601.01684, 4 Jan 2026. Two-phase curriculum on Llama-3 1B/3B/8B: weakly-supervised pre-finetuning for bidirectional contextualization, then hard-negative finetuning. **8B: 60.2 nDCG on MTEB Retrieval, ranked 15th as of 1 Jan 2026, with 71% less index memory than an equivalent dense model**, running on standard CPU hardware. ⚠️ Self-reported rank; "15th" also means 14 dense models beat it.

**SPLARE** — Thibault Formal, Antoine Louis, Hervé Déjean, Stéphane Clinchant (Naver Labs). arXiv:2603.13277, **ICLR 2026**. Replaces the vocabulary projection with **sparse-autoencoder features**; SPLARE-7B posts top results on MMTEB multilingual + English retrieval.

Other verified: **CSPLADE** (Zhichao Xu, Aosong Feng, Yijun Tian, Haibo Ding, Lin Lee Cheong, Amazon; arXiv:2504.10816, **IJCNLP-AACL 2025 Main**) — 8B-scale LSR, fixes early-stage contrastive instability and unidirectional attention. **Li-LSR** (arXiv:2505.01452) — **inference-free query encoding** via table lookup, **+1–1.8 nDCG over Splade-v3-Doc**. **Sparton** (arXiv:2603.25011) — fused Triton kernel, **+33% batch size, 14% faster training**. **V-SPLADE** (Naver, arXiv:2605.30917) — inference-free multimodal LSR for production visual document search. **MILCO** (arXiv:2510.00671) — multilingual LSR via a shared English lexical space. **UEmbed** (Alibaba/Tongyi, Pengjun Xie et al., arXiv:2608.02583) — decoder-only model emitting sparse and dense simultaneously, 2B–9B. ⚠️ Skeptical note: *Understanding Wacky Weights* (Polyakov, Scells, Eickhoff, arXiv:2605.19628) finds larger vocabularies correlate with **semantically unrelated** expansion terms — SPLADE is less interpretable than assumed.

**Efficiency asymmetry worth knowing** (from HAKARI-Bench on SPLADE-v3): **document-side pruning costs only +0.01–0.04 points** at d=256→512, while **query-side reduction costs +2.5–3.6 points** at q=8→32.

**→ EMERGING SYNTHESIS.** ColBERTSaR (JHU, arXiv:2606.05568) demonstrates **ColBERT with product quantization is equivalent to learned-sparse retrieval**. Combined with MacAvaney & Tonellotto's finding that PLAID clusters align ~1:1 with tokens, there is a coherent thread arguing **late interaction and learned sparse are converging on the same mechanism** — which would dissolve Debate #3 into an implementation question. Not yet consensus.

### 3.4 Embedding model scaling and MTEB standings

⚠️ **I could not scrape the live MTEB leaderboard.** The HF Space is a client-rendered Gradio app; every fetch returned the loading shell. **I will not state a mid-2026 #1 as fact.**

**Verified anchors:** *Qwen3-Embedding-8B* — **70.58 MTEB Multilingual, No.1 as of 5 Jun 2025** (arXiv:2506.05176, Alibaba Tongyi; 0.6B/4B/8B, 119 languages). *Gemini Embedding* — **68.32 Task Mean on MTEB(Multilingual), highest at time of writing** (Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer + 43 co-authors, Google DeepMind; arXiv:2503.07891, 10 Mar 2025; size undisclosed). Both self-reported and now stale.

**Verified mid-2026 cross-vendor table** (jina-embeddings-v5-text, Jina AI, arXiv:2602.15547, 17 Feb 2026):

| model | params | MMTEB avg | MTEB-Eng | Retrieval nDCG@10 |
|---|---|---|---|---|
| Qwen3-4B (teacher) | 4B | **69.5** | **74.6** | **69.60** |
| jina-v5-text-small | 677M | 67.0 | 71.7 | 64.88 |
| jina-v5-text-nano | 239M | 65.5 | 71.0 | 63.26 |
| Qwen3-0.6B (instruct) | 596M | 64.3 | 70.5 | 64.65 |
| multilingual-e5-large-instruct | 560M | 63.2 | 65.5 | 57.12 |
| Qwen3-0.6B (generic) | 596M | 61.1 | 67.0 | — |
| embeddinggemma-300m | 308M | 61.1 | 69.7 | 62.49 |
| voyage-4-nano | 340M | 58.9 | 63.3 | 63.58 |
| jina-v3 | 572M | 58.4 | 65.7 | 55.76 |
| snowflake-arctic-embed-l-v2 | 568M | 57.0 | 63.6 | 58.36 |

⚠️ Vendor-authored; comparison set deliberately sub-1B except the teacher.

**Independent 2026 comparison: HAKARI-Bench** (Yuichi Tateno, arXiv:2606.22778, 22 Jun 2026) — 35 benchmarks / 551 tasks / 43 languages / 55 models, nano-sets validated at **Spearman 0.983 vs MTEB v2, 0.975 vs MMTEB v2, 0.973 vs full BEIR**. Top by macro nDCG@10×100: **jina-v5-text-small 64.93**, jina-v5-text-nano 63.80, **microsoft/harrier-oss-v1-0.6b 63.68**, **perplexity-ai/pplx-embed-v1-0.6b 63.64**, embeddinggemma-300m 62.58 — *BM25 baseline 50.24*. Rerankers beat all: Qwen3-Reranker-0.6B **68.03**. Quantization deltas (mean over 33 models): binary **−6.50**, int8 **−1.95**, binary+rescore **−0.93**, **int8+rescore −0.09 (effectively lossless)**. ⚠️ Single-author, ≤~1B scope, nano-sets not full benchmarks — but the only independent unified-conditions 2026 comparison found.

**Net honest answer:** among ≤1B open-weight models, **jina-embeddings-v5-text-small** sits at or near the top on two independent 2026 sources. Overall, **Qwen3-Embedding-8B / Qwen3-4B** remain the reference ceiling with **Gemini Embedding** the strongest closed model on last-verified data.

**⭐ Scaling laws.** "Scaling Laws for Embedding Dimension in Information Retrieval" — Julian Killingback, Mahta Rafiee, Madine Manas, **Hamed Zamani** (UMass CIIR). arXiv:2602.05062, 4 Feb 2026. Retrieval performance vs embedding dimension **fits a power law**, with predictive models on dimension alone and jointly with model size. **Aligned tasks: monotone improvement with diminishing returns. Misaligned tasks: unpredictable — larger embedding dimension can actively degrade results.** That second half is the non-obvious finding and directly cautions against "bigger dim is safer." Adjacent: *Retrieval Capabilities of LLMs Scale with Pretraining FLOPs* (arXiv:2508.17400); *BitNet Text Embeddings* (Zhen Li, Xin Huang, Liang Wang, Nan Yang, **Furu Wei**, MSR; arXiv:2606.25674) — extreme quantization "largely comparable" to full-precision teachers on MMTEB(eng,v2). Low end: *Bekko Embedding* (arXiv:2607.25180) — **8M active params → 56.2 MMTEB Multilingual v2; 25M → 57.5** (vs BM25's 50.24).

**⭐ The leaderboard-integrity failure, primary-source verified.** GitHub `embeddings-benchmark/mteb` **issue #3934**, "Decision: Temporary removal of the private RTEB column", **14 Jan 2026**. RTEB was built with private test sets specifically to defeat overfitting — but it was **co-developed with Voyage AI (since acquired by MongoDB), who therefore had direct access to the private evaluation data while competing on the leaderboard.** The MTEB team's words: *"the uneven playing field fundamentally undermines trust in MTEB leaderboards, which is unacceptable for a community benchmark."* No misuse alleged; the structural conflict alone forced the call. The private column was removed; it returns once the private pool is diversified with contributions from orgs that don't ship competing models.

**Peer-reviewed critiques, six independent groups reaching the same conclusion:** *On the Robustness of Multilingual Text Embedding Rankings* (Gjorgjevikj, Koroušić Seljak, Eftimov, arXiv:2605.31142) — **MTEB rankings are sensitive to dataset composition and aggregation method; conclusions lack robustness**. *MTEB-BR* (arXiv:2607.04581) — multilingual leaderboard correlates only **ρ=0.75** with Brazilian Portuguese performance. *MTEB-PT* (arXiv:2607.04071) — same for Portuguese. *LMEB* (Xinping Zhao et al., arXiv:2603.12572) — long-horizon memory results **orthogonal to MTEB; larger models don't reliably win**. *STEB* (Rivera Soto, Wegmann, Aggazzotti, arXiv:2606.31741) — semantic embeddings **consistently fail on stylistic tasks**. *PosIR* (arXiv:2601.08363, 310 datasets, 10 MMTEB SOTA models) — **position bias is pervasive and invisible to MTEB**. *CS-MTEB* (arXiv:2604.17632) — **up to 27% degradation** on code-switched queries. *SABER-Math* (arXiv:2606.29894) — "general-purpose IR benchmarks such as MTEB do not reliably predict mathematical performance." Also *HTEB* (arXiv:2605.28190) and *PTEB* (arXiv:2510.06730) proposing stochastic re-paraphrasing at eval time.

**The best-supported claim in this whole section: a high MTEB/MMTEB average does not transfer to your specific language, domain, or task.**

⚠️ **Instruction-following embeddings — partial coverage.** Search budget ran out before a dedicated pass. What is verified: instruction conditioning is worth **+3.2 MMTEB / +3.5 MTEB-Eng at identical parameter count** (Qwen3-0.6B instruct 64.3/70.5 vs generic 61.1/67.0); **Promptriever Llama3 8B was the best single-vector model on LIMIT (R@100 = 18.9 vs Qwen3's 4.8)**, i.e. instruction-following retrievers are meaningfully more robust to the dimensional bound; and **MMTEB** (Kenneth Enevoldsen + 64 co-authors, **ICLR 2025**, arXiv:2502.13595, 500+ tasks / 250+ languages) added instruction-following as a task category and found **smaller multilingual models often outperform large LLMs**.

---

## 4. CONTEXT HANDLING

### 4.1 Late chunking

**The primary paper is arXiv-only, from the vendor that sells the embeddings it was tested on.** "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models" — Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, Han Xiao (Jina AI). arXiv:2409.04701, v1 7 Sep 2024, v3 7 Jul 2025. Comments field says "11 pages, 3rd draft." **No venue.** It has an OpenReview page (74QmBTV0Zf) with no acceptance record.

Verified numbers (Table 2, nDCG@10, fixed 256-token boundaries, naive → late):

| Dataset | jina-v2-small | jina-v3 | nomic-v1 |
|---|---|---|---|
| SciFact | 64.2 → 66.1 | 71.8 → 73.2 | 70.7 → **70.6 (↓)** |
| NFCorpus | 23.5 → 30.0 | 35.6 → 36.7 | 35.3 → **35.3 (=)** |
| FiQA | 33.3 → 33.8 | 46.3 → 47.6 | 37.0 → 38.3 |
| TRECCOVID | 63.4 → 64.7 | 73.0 → 77.2 | 72.9 → 75.0 |
| **Average** | — | — | **52.2 → 54.0** |

**The headline effect is +1.5 to +1.9 nDCG@10 absolute (2.7–3.6% relative)**, and it is not uniform. The authors' own conceded failure case: late chunking **hurts on synthetic needle tasks** (Needle-8192, Passkey-8192), because contextualizing a planted needle with unrelated filler dilutes it. Their comparison against contextual retrieval (Table 4) is **a single anecdotal example with one query** — cosine 0.6343 naive → 0.8516 late → 0.8590 contextual. Not an experiment. Their argument against contextual retrieval is cost, not accuracy.

**⭐ The independent replication is mixed-to-negative.** "Reconstructing Context: Evaluating Advanced Chunking Strategies for RAG" — Carlo Merola, Jaspinder Singh. arXiv:2504.19754, **2nd Workshop on Knowledge-Enhanced IR, ECIR 2025**. NDCG@5, early → late: NFCorpus Stella-V5 0.443→0.445; jina-v3 0.374→0.380; jina-v2 0.261→0.280; **BGE-M3 0.246 → 0.070**. MSMarco, Stella-V5: **0.630 → 0.503**.

**This is the single most important refutation datapoint: late chunking is model-dependent, can catastrophically fail (BGE-M3, ~72% relative collapse), and loses badly on MSMarco with a strong non-Jina encoder.** The gains appear to hold mainly on small domain-specific corpora with Jina's own models. ⚠️ Workshop paper, small scale; the BGE-M3 collapse is plausibly a pooling-compatibility artifact. But nobody has published a rebuttal, which is itself informative. Same paper's head-to-head: contextual retrieval NDCG@5 **0.317** vs late chunking **0.309** on an NFCorpus subset — a ~2.6% relative edge at much higher cost.

**⭐ The best systematic 2026 evaluation.** "Beyond Chunk-Then-Embed: A Comprehensive Taxonomy and Evaluation of Document Chunking Strategies for IR" — Yongjie Zhou, Shuai Wang, Bevan Koopman, **Guido Zuccon** (Queensland/CSIRO). arXiv:2602.16974, 19 Feb 2026. Abstract, verbatim: *"Contextualized chunking improves in-corpus effectiveness but degrades in-document retrieval."*

- **In-corpus (BEIR, nDCG@10):** jina-v3 — **Paragraph 0.4948** > Fixed 0.4849 > Semantic 0.4726 ≈ Sentence 0.4723 > LumberChunker 0.4690 > Proposition 0.3888. **Structure-based beats LLM-guided by 5–27%.**
- **In-document (GutenQA, 3k QA over 100 books, DCG@10):** LumberChunker wins decisively — jina-v3 0.5640 vs Paragraph 0.4574.
- **Effect of late chunking:** in-corpus, Proposition **+22.87% to +26.94%**; LumberChunker +2.42% to +4.80%; structure-based **0% to +7.20%**. In-document, Paragraph **−10.76% to −62.47%**; Sentence −5.39% to −62.57%.
- Throughput: LumberChunker **1.11 docs/s vs paragraph-based 1,854 docs/s — ~1,600× slower.**

Late chunking's benefit is largest exactly where naive chunks are most context-starved, near-zero where chunks are already coherent, and **actively harmful for needle-style in-document retrieval** — independently corroborating the authors' own caveat.

**The chunking-strategy benchmark literature.** *Is Semantic Chunking Worth the Computational Cost?* (Renyi Qu, Ruixuan Tu, Forrest Sheng Bao; arXiv:2410.13070, **Findings of NAACL 2025**) — cost not justified; fixed ~200-word chunks match or beat it. *Chunking Methods on RAG: Effectiveness vs Computational Cost* (Wrocław UST, arXiv:2606.00881) — fixed-size and recursive-semantic most stable; **LumberChunker highest answer quality but completed on only ~30% of datasets** (timeouts); runtimes <1s vs **8.37 h**; DenseX 15+ h; only 5 of 8 methods completed everywhere. *A Systematic Investigation of Document Chunking Strategies* (Shaukat, Adnan, Kuhn, U. Canberra; arXiv:2603.06976) — largest sweep: 36 approaches × 6 domains × 5 embedders = 1,080 configs; **Paragraph Group Chunking best (mean nDCG@5 ≈ 0.459, P@1 24%), naive fixed-*character* worst (nDCG@5 < 0.244, P@1 ≈ 2–3%)**. *Evaluating Chunking Strategies for RAG on Academic Texts* (arXiv:2607.01852) — cluster-based semantic chunking **did not outperform** fixed-size or recursive; also warns **RAGAs faithfulness "shows limited reliability in this setup."**

**→ OPEN DEBATE #5, and it dissolves on inspection.** Four papers say simple wins; Shaukat et al. reports a ~2× nDCG spread implying chunking is a vital lever. The reconciliation: **Shaukat's worst baseline is fixed-*character* splitting (which shreds words), while the "simple wins" camp baselines against fixed-token or recursive splitting.** Consensus as of mid-2026: **paragraph- or sentence-respecting structural chunking is a strong, near-optimal, essentially free baseline; embedding-similarity semantic chunking and LLM-guided chunking do not reliably beat it in-corpus at 100–1,600× the cost.**

**Descendants.** *Context is Gold to find the Gold Passage* (Conti, Faysse, Viaud, Bosselut, Hudelot, Colombo; arXiv:2505.24782) — introduces **ConTEB** benchmark and **InSeNT** in-sequence-negative contrastive post-training combined with late-chunking pooling; the most credible academic endorsement of the pooling operator, **but it requires training — zero-shot late chunking is the weaker claim.** Also *ColChunk* visual late chunking (arXiv:2604.10167), *Graph-Aware Late Chunking for Biomedical* (arXiv:2603.22633), *pplx-embed* (Perplexity, arXiv:2602.11151 — uses late chunking in production; ⚠️ Bo Wang is a Jina co-author, not independent).

### 4.2 Contextual retrieval

**The source is a blog post. There is no paper.** Anthropic engineering blog, 19 Sep 2024. Verified claims: top-20 retrieval **failure rate 5.7% → 3.7% (−35%)** with contextual embeddings; **→ 2.9% (−49%)** adding contextual BM25; **→ 1.9% (−67%)** adding reranking. Cost **$1.02 per million document tokens**. Anthropic's own caveat: performance varies by embedding/source combination, run your own evals.

**No arXiv version, no peer review, no dataset release, and no independent reproduction of 35/49/67 exists as of August 2026.** Anyone quoting "67%" is quoting a vendor blog with an unreleased internal eval.

What academic work reports: the only head-to-head (Merola & Singh, ECIR 2025 workshop) puts contextual retrieval **~2.6% relative ahead of late chunking on an NFCorpus subset**, at substantially higher cost. Zhou et al. (2026) find **structure-based segmentation beats LLM-guided methods by 5–27% nDCG@10 in-corpus** at 1,600× the throughput; contextualization helps most for propositions (+23–27%) and barely at all for paragraphs (0 to +7.2%). ⚠️ A single-author, non-peer-reviewed Spanish-language preprint (*More Context Is Not Better: The Vector Dilution Paradox*, arXiv:2601.08851) reports an inverted-U: **moderate injection +18% recall; past a Contextualization Injection Ratio > 0.4, precision drops 22%** on targeted queries. Treat the numbers as unverified, but the shape is consistent with Zhou et al.

Cost reality the blog omits: contextual retrieval **doubles index-build storage and forces full re-contextualization whenever chunk boundaries change**.

**→ OPEN DEBATE #6.** Position A (Anthropic, unreplicated): 49–67% failure reduction, cheap. Position B (Zhou et al. 2026): LLM-side chunk processing loses to paragraph splitting in-corpus by 5–27%. Position C (Merola & Singh): best of the advanced methods, by ~1–3% on a small subset. Position D (unreviewed): over-injection costs 22% precision. **Practitioner verdict: directionally sound, most valuable for short/proposition-sized chunks in anaphora-heavy corpora (filings, codebases). The 35/49/67 figures should be cited as vendor-internal, not established.**

### 4.3 Context compression

**The lineage, all verified:**

| Method | Authors / lab | arXiv | Venue | Headline |
|---|---|---|---|---|
| LLMLingua | Jiang, Wu, Lin, Yang, Qiu (MSR) | 2310.05736 | **EMNLP 2023** | up to 20× compression, little loss |
| LongLLMLingua | Jiang, Wu, Luo, Li, Lin, Yang, Qiu (MSR) | 2310.06839 | **ACL 2024** | NaturalQuestions **+21.4% with ~4× fewer tokens**; **94.0% cost reduction** on LooGLE; **1.4–2.6× latency** speedup |
| LLMLingua-2 | Pan, Wu, Jiang, Xia, Luo, Zhang, Lin, Rühle, Yang, C.-Y. Lin, Zhao, Qiu, Zhang (MSR + Tsinghua) | 2403.12968 | **Findings ACL 2024** | **3–6× faster than prior compressors; 1.6–2.9× end-to-end latency reduction** at 2–5× |
| RECOMP | Fangyuan Xu, Weijia Shi, Eunsol Choi | 2310.04408 | ICLR 2024 | compression to **6%** with minimal loss; can emit empty string |
| xRAG | Cheng, Wang, Zhang, Ge, Chen, Wei, Zhang, Zhao (MSRA + PKU) | 2405.13792 | **NeurIPS 2024** | context → **one token**; **>10% avg improvement** on six tasks; **3.53× FLOPs reduction** |
| CompAct | Yoon, Lee, Hwang, Jeong, Kang | 2407.09014 | **EMNLP 2024** | **47× compression**, strongest on multi-hop |
| PISCO | Naver Labs Europe | 2501.16075 | **Findings ACL 2025** | **16× compression, 0–3% loss**; +8% over prior compressors; 48 h on one A100 |

**⭐ Provence is the compression result that survives scrutiny.** Nadezhda Chirkova, Thibault Formal, Vassilina Nikoulina, Stéphane Clinchant (NAVER LABS Europe). arXiv:2501.16214, 27 Jan 2025, **ICLR 2025** (confirmed: poster 29557, OpenReview TDy5Ih78b4). Sentence-level pruning as binary sequence labeling on a DeBERTa cross-encoder, **fused with the reranker** so pruning is free in a pipeline that already reranks. One hyperparameter (threshold ∈ {0.1, 0.5}) that transfers across domains.

LLM-Eval scores: **NQ 72.4 at 62.2% compression vs 71.8 full context** (pruning *improves* — denoising); **HotpotQA 56.7 @ 66.4% vs 57.0 full** (−0.3); **PopQA 59.3 @ 68.6% vs 57.8 full** (+1.5). Baselines on NQ: LLMLingua-2 **59.5** @74%; LongLLMLingua **61.3** @69%; RECOMP-extractive **70.6** @44%; DSLR **71.7** @45%. **Cross-domain across 7 datasets — NQ, HotpotQA, TyDi QA, PopQA, BioASQ (biomedical), SyllabusQA (education), RGB (news) — "negligible to no drop."** Overhead essentially zero when unified with the reranker; generation speedups 1.2–1.4× at batch 1, 1.9–2.0× at batch 256.

⚠️ Caveat: 50–80% compression is far less aggressive than CompAct's 47× or xRAG's one token. The OOD robustness comes partly from not compressing very hard.

**⭐ The systematic evaluation bug in the entire compression literature.** "Fixed RAG Compression Collapses Measured Reader Scaling" — Sugam Panthi, Rabab Abdelfattah. arXiv:2606.21807, 20 Jun 2026. A *fixed* compressor helps weak readers (removes noise they can't filter) and hurts strong readers (removes detail they could have used). Across **20 readers × 10 domain-method settings, compression gains decreased with reader baseline in 9 of 10 settings (p < 0.05)**. Generic summarization **flipped 31% of pairwise model rankings on LongMemEval-S**. A fixed HotpotQA compressor **obscured 80% of the Qwen-7B → GPT-4.1-mini improvement**. Pattern holds across compressor types and an external audit of **nine published papers**. Released `ragscale` (177k row-level transitions).

**Every "X× compression with negligible loss" number above is reader-dependent, and the strong-reader case is systematically under-reported.**

Also: *No Mean Feat: Simple, Strong Baselines for Context Compression* — Yair Feldman, **Yoav Artzi** (Cornell Tech). arXiv:2510.20797, rev 10 May 2026. Introduces **BenchPress** and shows **mean pooling and a bidirectional compression-token variant strongly outperform the widely-used causal compression-token approach** — the design underlying much of the gist-token line — across scales, datasets, and ratios. *RAISE* (arXiv:2605.30029): 13 RAG algorithms × 7 datasets, "optimization performance is highly task-dependent" with **poor cross-dataset generalization**. *Control Under Compression* (arXiv:2608.01056): reliability "diverges sharply" in the 50–35% retained-context band and compressor rankings are **not universal**.

**2026 successors:** *CORE-RAG* (Cui, Weng, Tang, Liu, Li, He, Chen, Zhang, He, Ma; arXiv:2508.19282, v4 28 May 2026, **ICML 2026**) — performance-driven compression, **at a 3% compression ratio, +3.3 EM over feeding full documents**. *ARC-Encoder* (Kyutai, arXiv:2510.20535) — 4–8× compression adapting to multiple decoders. *Sentinel* (arXiv:2505.23277) — a **0.5B proxy achieves 5× compression competitive with 7B-scale methods**. Plus ECoRAG, ACC-RAG, AttnComp, EXIT, BRIEF (+3.0 EM / +4.16 F1 on HotpotQA), and *A Unified Model and Document Representation for On-Device RAG* (Killingback, Meshi, Li, **Zamani**, Karimzadehgan; arXiv:2604.14403 — matches traditional RAG with **1/10 the context**).

**→ OPEN DEBATE #7.** The field publishes 16–47× compression with 0–3% loss and even "compression improves accuracy." Panthi & Abdelfattah show this is largely an artifact of weak readers. Feldman & Artzi show the dominant architectural choice is beaten by mean pooling. **Reading: extractive, query-conditioned, moderate compression (Provence-style, 50–70%) is genuinely robust; soft/gist/extreme compression (xRAG one-token, CompAct 47×) has not been shown to survive either a strong reader or a domain shift.**

### 4.4 Long context vs RAG — what the evidence actually says

**The 2024 axis of debate.** *Retrieval Augmented Generation or Long-Context LLMs?* — Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky (Google DeepMind + Michigan). arXiv:2407.16833, **EMNLP 2024 industry track**. When sufficiently resourced, **LC consistently beats RAG on average** — but **LC and RAG predictions are identical for >60% of queries**, and SELF-ROUTE achieves LC-comparable quality at **65% cost reduction (Gemini-1.5-Pro), 39% (GPT-4o)**.

*In Defense of RAG in the Era of Long-Context LLMs* — Tan Yu, Anbang Xu, Rama Akkiraju (NVIDIA). arXiv:2409.01666. **OP-RAG** keeps retrieved chunks in original document order rather than relevance order; reports an inverted-U in chunk count. ∞Bench EN.QA F1: Llama3.1-70B full context **34.26 @117K tokens**; GPT-4o 32.36 @117K; Gemini-1.5-Pro 43.08 @196K; SELF-ROUTE GPT-4o 34.95 @85K. **OP-RAG on Llama3.1-70B: 44.43 @16K, 45.45 @24K, 47.25 @48K.** EN.MC accuracy: full-context Llama3.1-70B 71.62 @117K vs **OP-RAG 88.65 @24K**. So **47.25 F1 with 48K tokens vs 34.26 F1 with 117K**. ⚠️ Single benchmark (book-length novel QA — the format most favorable to retrieval), single model family, arXiv-only.

**Databricks.** *Long Context RAG Performance of LLMs* — Quinn Leng, Jacob Portes, Sam Havens, **Matei Zaharia**, Michael Carbin (Databricks Mosaic). arXiv:2411.03538, **NeurIPS 2024 Workshop on Adaptive Foundation Models** (workshop, not main track). 20 models, 2K→128K, text-embedding-3-large, 512-token chunks, FAISS. Accuracy 64k → 125k: **holds up** — o1-preview 0.831→0.763, GPT-4o 0.769→0.767, Claude 3.5 Sonnet 0.741→0.706; **degrades** — Llama 3.1 405B 0.587→0.426, GPT-4 Turbo 0.623→0.560. The valuable part is that **failure modes are qualitatively distinct**: Claude 3 Sonnet refused on copyright grounds increasingly with length; Gemini 1.5 Pro tripped safety filters; DBRX summarized instead of answering above 16k; Mixtral emitted repeated nonsense; Llama 3.1 405B gave consistent wrong answers. ⚠️ Late-2024 model generation; no equally systematic 2026 redo exists.

**Effective context length ≪ advertised.**
- **RULER** — Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang, Ginsburg (NVIDIA). arXiv:2404.06654, **COLM 2024**. All models claim ≥32K; **only half maintain satisfactory performance at 32K**, despite near-perfect vanilla NIAH.
- **⭐ NoLiMa** — Modarressi, Deilamsalehy, Dernoncourt, Bui, Rossi, Yoon, Schütze (LMU + Adobe Research). arXiv:2502.05167, **ICML 2025**. Needles share **no lexical overlap** with the question — only associative links. **13 models all claiming ≥128K: at 32K tokens, 11 of 13 fall below 50% of their short-context baseline. GPT-4o: 99.3% → 69.7%.** Reasoning-enhanced models and CoT don't rescue it. **The cleanest demonstration that NIAH scores are an artifact of literal matching.**
- **HELMET** — Yen, Gao, Hou, Ding, Fleischer, Izsak, Wasserblat, Chen (Princeton + Intel Labs). arXiv:2410.02694, **ICLR 2025**. Seven application-centric categories to 128k. **"Synthetic tasks like NIAH do not reliably predict downstream performance"**; categories show low correlation with each other; the open-vs-closed gap **widens with length**.
- **LongBench v2** — arXiv:2412.15204, **ACL 2025**. 503 MCQs, 8k–2M words. **Human experts under a 15-min limit: 53.7%. Best direct-answering model: 50.1%. o1-preview with extended reasoning: 57.7%.**
- **⭐ Context Length Alone Hurts LLM Performance Despite Perfect Retrieval** — Du, Tian, Ronanki, Rongali, Bodapati, Galstyan, Wells, Schwartz, Huerta, Peng. arXiv:2510.05381, **Findings of EMNLP 2025**. **Even with perfect retrieval, accuracy degrades 13.9%–85% as input length grows — and the degradation persists when irrelevant content is replaced with whitespace or masked entirely.** Length itself, independent of distraction and retrieval quality, is a failure axis. Mitigation: prompt the model to recite retrieved evidence before answering (+up to 4% for GPT-4o on RULER).
- ⚠️ **"Context Rot"** (Kelly Hong, Anton Troynikov, Jeff Huber, Chroma, Jul 2025) is a **blog/tech report, not peer-reviewed**. Cite Du et al. (Findings EMNLP 2025) instead for the same claim.

**"Lost in the middle" has been substantially revised.** *Positional Biases Shift as Inputs Approach Context Window Limits* — Veseli, Chibane, Toneva, Koller (Saarland/MPI-SWS). arXiv:2508.07479, **COLM 2025**. **The U-curve is strongest only when the input occupies up to ~50% of the context window.** Beyond that, primacy weakens, recency holds, and it becomes a **distance-based bias**. Measuring in *relative* rather than absolute length is the methodological point most prior work got wrong. Also: *Lost in the Middle: An Emergent Property from Information Retrieval Demands* (arXiv:2510.10276); *On the Emergence of Position Bias in Transformers* (arXiv:2502.01951); and arXiv:2511.05850 reporting **Gemini 2.5 Flash shows no lost-in-the-middle effect for simple factoid QA**. Counterweight: *Stable-RAG* (arXiv:2601.02993) shows retrieval-**permutation**-induced hallucinations remain measurable.

**⭐ The most replicated practical finding of 2025–2026 is the dullest: preserve document order.** *Stronger Baselines for RAG with Long-Context LMs* — Alex Laitenberger, **Christopher D. Manning**, Nelson F. Liu (Stanford). arXiv:2506.03989, **EMNLP 2025**. **DOS RAG** ("Document's Original Structure") = retrieve-then-read preserving original passage order. It **matches or outperforms ReadAgent and RAPTOR** across long-context QA benchmarks and systematically varied token budgets; recommended as the mandatory baseline for future RAG papers. Same insight as OP-RAG, independently arrived at by a different group, and peer-reviewed.

**The cost evidence is unambiguous.** *The Token Tax of Epistemic Accuracy* — Hamilton, Singh, Wise, Yousif, Carvalho, Shan, Mayyas, Cavuoto, Megahed. arXiv:2606.20898, 18 Jun 2026. 972 answers, expert-validated manufacturing-safety benchmark: **long-context 73.1% correctness vs 65.4% for semantic RAG — at 26× the per-query token cost.** ⚠️ Small LMs, narrow domain; the gap would likely shrink with frontier models and a better-engineered RAG arm. Also: clinical EHR reasoning (arXiv:2508.14817) — **RAG with <8K tokens matches long-context**; on-device unified compression matches traditional RAG at **1/10 the context** (arXiv:2604.14403).

**And a strong result for minimal RAG:** *Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks* — Lyu, Duan, Shao, Koh, Min (UW/Berkeley/AI2). arXiv:2507.01297. A *minimal* RAG pipeline over CompactDS gives **+10% MMLU, +33% MMLU Pro, +14% GPQA, +19% MATH** across 8B–70B, matching or beating Google Search and agentic RAG.

**→ OPEN DEBATE #8 — LC vs RAG.** Google DeepMind (EMNLP 2024) and Li et al. (arXiv:2501.01880) say LC wins on average. NVIDIA/OP-RAG and Lyu et al. say a properly ordered RAG beats LC at a fraction of the tokens. LaRA (arXiv:2502.09977, 2,326 test cases) says the question is ill-posed — it depends on parameter size, long-text ability, context length, task type, and chunk characteristics. Hamilton et al. quantify it as +7.7 points for 26× cost. **The reconciliation the 2025–26 evidence supports: most published "LC beats RAG" results used a weak RAG arm (relevance-ordered, badly chunked, small top-k). Fix ordering (DOS/OP-RAG) and recall and RAG closes most of the gap at 3–25× lower cost. Meanwhile LC has an independent problem — Du et al. show accuracy falls 13.9–85% with length even under perfect retrieval with distractors literally blanked out — so "just use the long window" is not stable as context grows.** Note also Li et al.'s under-quoted finding that **summarization-based retrieval performs comparably to LC while chunk-based retrieval lags** — the RAG side of most comparisons is handicapped by chunking, not by retrieval.

---

## 5. VERIFICATION, GROUNDING, ATTRIBUTION

### 5.1 Claim-level attribution

**The benchmarks.** *ALCE* (Tianyu Gao, Howard Yen, Jiatong Yu, Danqi Chen; **EMNLP 2023**, arXiv:2305.14627) — citation recall + precision via a TRUE-style NLI model; **~50% of the best models' generations on ELI5 are not fully supported by their own cited passages**. *AttrScore* (Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, Huan Sun; **Findings EMNLP 2023**, arXiv:2305.06311) — defines the attributable/extrapolatory/contradictory taxonomy. ⚠️ **Per-model F1 numbers not extractable from the abstract or Anthology page — do not quote a specific AttrScore F1.** *HAGRID* (Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, **Jimmy Lin**; arXiv:2307.16883). *AttributionBench* (arXiv:2402.15089, Findings ACL 2024, OSU NLP) — ⚠️ from a search summary: **even a fine-tuned GPT-3.5 reaches only ~80% macro-F1** on binary supportedness. *LFRQA / RAG-QA Arena* (arXiv:2407.13998, EMNLP 2024, AWS AI Labs) — 26K queries, 7 domains; ⚠️ frequently misdescribed as an attribution benchmark, it is an answer-quality arena.

**⭐ The single most important 2026 result: attribution evaluators do NOT transfer.** "Do LLM Attribution Metrics Transfer? Auditing RAG Evaluation Across Datasets and Constructs" — Tianyu Ding, Aditya Nannapaneni, Juan Pablo De la Cruz Weinstein. arXiv:2606.23915, 22 Jun 2026. 8 scorers × 3 constructs; 1,610 AttributionBench + 2,150 HAGRID examples.

- **No scorer stayed inside the 95% CI across all datasets within any construct.**
- **Metric rankings invert across datasets: Kendall tau = −0.64, p = 0.031.** An actual reversal, not noise.
- **One NLI scorer: AUROC 0.90 on short-claim AttributedQA → 0.53 (chance) on long-form LFQA.** The standard NLI citation checker used by ALCE-style pipelines is **at chance on long-form answers**.
- **BERTScore was the best scorer on LFQA at 0.91 AUROC** — the "dumb" metric beat entailment models exactly where entailment should shine.
- Selecting a scorer by average cross-dataset performance: **mean regret 0.172 AUROC** under leave-one-dataset-out.
- LLM judges avoid total collapses but cost **~100×** and are non-deterministic.

⚠️ Single preprint, not peer-reviewed; 8 scorers is decent but not exhaustive. Still the strongest available evidence that **any single reported attribution-evaluator score is a property of the dataset, not the evaluator.**

**⭐ Citation quality in deployed deep-research agents.** "Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents" — Hailey Onweller, Elias Lumer, Austin Huber, Pia Ramchandani, Vamse Kumar Subbiah, Corey Feld. arXiv:2605.06635, 7 May 2026.
- Frontier models: **>94% link accessibility, >80% topical relevance, but only 39–77% of citations are factually accurate against the source content.**
- **Fewer than half of open-source models can produce a cited report at all** one-shot.
- **Fact-check accuracy drops ~42% on average across two frontier models as tool calls scale from 2 to 150.** Longer research trajectories, monotonically worse grounding.

⚠️ Preprint; the fact-check judge is itself an LLM (so §5.1's transfer problem applies to this paper's own instrument); "two frontier models" is thin. But *links resolve, topics match, claims don't* is the most actionable attribution finding of 2026 — and the tool-call scaling result is directly relevant to agentic retrieval design.

**Does citation generation degrade answer quality? Yes, and granularity is the knob.** "Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation" — Hexuan Wang, Jingyu Zhang, **Benjamin Van Durme, Daniel Khashabi** (JHU). arXiv:2604.01432, Apr 2026. **Forcing sentence-level citations degrades attribution quality by 16–276% relative to the optimal granularity**, across model sizes; quality **peaks at intermediate (paragraph-level) granularity**; **the penalty for fine-grained constraints grows with model scale**. At the right granularity you get both attribution and correctness. Proposed mechanism: attention dilution during synthesis plus atomic units fracturing the semantic context needed to synthesize.

### 5.2 Hallucination detection — the numbers

**Benchmarks.** *RAGTruth* (Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, Tong Zhang; **ACL 2024**, arXiv:2401.00396) — **~18,000 naturally generated RAG responses with word-level manual hallucination annotations** from 6 LLMs across QA / data-to-text / summarization. Still the de facto standard; ⚠️ generators are 2023-vintage, a real staleness concern for 2026. *HaluBench* (via Lynx) — 15,000 samples across HaluEval, DROP, CovidQA, PubMedQA, FinanceBench, RAGTruth; human validation agreement **0.90–0.96**; ⚠️ constructed partly by **semantic perturbation**, a distribution unlike naturally-occurring hallucination. *FaithBench* (**NAACL 2025 short**, Vectara-affiliated) — deliberately built from summaries **where SOTA detectors including GPT-4o-as-judge disagree**; **most SOTA detectors score near 50% (chance)**; ⚠️ adversarial by construction, so ~50% is partly definitional, and the team also ships a competing detector.

**⭐ TRIVIA+ / the 2026 benchmark-hygiene paper.** "Rethinking Evaluation for LLM Hallucination Detection: A Desiderata, A New RAG-based Benchmark, New Insights" — Wenbo Chen, Veena Padmanabhan, Tootiya Giyahchi, Elaine Wong, Leman Akoglu. arXiv:2605.11330, 11 May 2026. RAG-based with **"the longest context in the literature"** plus **four synthetic label-noise sets**. Three findings: current detectors have ample room on RAG tasks; **plain LLM-as-a-Judge is competitive with specialized detectors**; **label noise degrades detection and shifts detector rankings unpredictably**. ⚠️ **I could not extract the AUROC table** (PDF compressed streams) — qualitative findings only.

**Detector numbers I can stand behind.** *LettuceDetect* — Ádám Kovács, Gábor Recski (KR Labs / TU Wien). arXiv:2502.17125, 24 Feb 2025. ModernBERT-based, 8k context, token-level classification. RAGTruth example-level F1:

| System | F1 |
|---|---|
| GPT-4 (prompt-based) | 63.4% |
| Luna (Galileo encoder) | 65.4% |
| **LettuceDetect-large** | **79.22%** |
| Fine-tuned Llama-2-13B | 78.7% |
| **Fine-tuned Llama-3-8B (prior SOTA)** | **83.9%** |

**LettuceDetect is NOT SOTA on RAGTruth — fine-tuned Llama-3-8B at 83.9% is.** LettuceDetect is the best *cost-adjusted* detector: ~30× smaller, 30–60 examples/sec on one GPU, +14.8% relative over Luna. The paper itself says "competitive with." The widely-repeated "LettuceDetect is SOTA" framing is wrong.

*Lynx* — Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannappan, **Douwe Kiela**, Rebecca Qian (Patronus AI). arXiv:2407.08488. HaluBench accuracy:

| Model | Overall | HaluEval | DROP | CovidQA | PubMedQA | FinanceBench | RAGTruth |
|---|---|---|---|---|---|---|---|
| **Lynx 70B** | **87.4%** | 88.4 | 86.4 | 97.5 | 90.4 | 81.4 | 80.2 |
| GPT-4o | 86.5% | 87.9 | 84.3 | 95.0 | 82.1 | **85.3** | **84.3** |
| Lynx 8B | 82.9% | 85.7 | 77.8 | 96.3 | 85.2 | 72.5 | 80.0 |
| Claude-3-Sonnet | 78.8% | 84.5 | 84.3 | 95.0 | 82.9 | 69.7 | 79.1 |
| RAGAS Faithfulness | 66.9% | — | — | — | — | — | — |

⚠️ **The marketing omits that the 87.4 vs 86.5 win is 0.9 points, and GPT-4o beats Lynx-70B on FinanceBench (85.3 vs 81.4) and RAGTruth (84.3 vs 80.2).** Lynx's margin comes almost entirely from PubMedQA and CovidQA — a benchmark-composition win. Vendor-authored, vendor-constructed benchmark, perturbation-synthesized hallucinations. "Lynx 2.0" is blog-announced with no paper found.

**Vectara HHEM leaderboard — vendor-run, flagged.** Fetched github.com/vectara/hallucination-leaderboard. **Last updated 11 May 2026; current scoring model HHEM-2.3** (not 2.1). Methodology: **>7,700 source articles, 50 to 24,000 words**, across news/tech/science/medicine/legal/sports/business/education; instruction "Summarize using only the information in the given passage. Do not infer"; **temperature 0**; reports hallucination rate, factual consistency, and **answer rate**. Top as of 11 May 2026: antgroup/finix_s1_32b **1.8%** (99.5% answer rate); openai/gpt-5.4-nano **3.1%** (100%); google/gemini-2.5-flash-lite **3.3%**; microsoft/Phi-4 **3.7%** but only **80.7% answer rate**; meta-llama/Llama-3.3-70B **4.1%**.

⚠️ Caveats that matter: it is **graded by the vendor's own product**, so it is simultaneously a benchmark and a product demo; it measures **summarization faithfulness only**; the **answer-rate column is the tell** (3.7% at 80.7% answer rate is not comparable to 3.1% at 100%); a 32B model from a payments company topping a board of frontier models is the pattern you'd expect from optimization-toward-the-metric. Vectara's own repo text: determining hallucinations is "impossible without a reference source," and the problem "is far from solved." ⚠️ Figures circulating in SEO summaries (GPT-5 Pro ~1.0%, Claude Opus 4.7 ~1.2%) **do not appear on the leaderboard I fetched — do not use them.**

**Semantic entropy and its 2026 pressure.** *SelfCheckGPT* (Manakul, Liusie, Gales; EMNLP 2023) is now used almost exclusively as a baseline; the standard criticism is that surface-token divergence conflates paraphrase with contradiction. *Semantic entropy* (Farquhar, Kossen, Kuhn, Gal, **Nature 2024**) clusters samples by bidirectional entailment into meaning classes. *Semantic Entropy Probes* (Kossen et al., arXiv:2406.15927) approximate it in a **single forward pass**.

2026 pressure (⚠️ all from an arXiv listing fetch, IDs as returned, abstracts not individually opened): arXiv:2607.16868 — **+7.1% AUROC over semantic entropy** via logical graphs, arguing entailment clustering is the wrong equivalence relation; arXiv:2606.10198 — **+5–20 points** via geometry; arXiv:2605.04295 — **0.88 vs 0.65 AUROC** on TriviaQA with adaptive conformal SE; **arXiv:2605.05166 "The First Token Knows"** — **first-token confidence matches or modestly exceeds semantic self-consistency with zero sampling overhead** (the sharpest efficiency critique); arXiv:2607.07670 — 5-sample SE reaches only **0.71–0.83 AUROC at 5× inference cost**, losing to activation probes; arXiv:2603.22812 — **~50% fewer samples** via variance-based early termination; arXiv:2606.24115 — clustering underperforms plain token statistics in a VLM medical domain.

**Honest synthesis:** nobody has published a clean refutation, and the *idea* (uncertainty over meanings, not tokens) is not seriously contested. What is contested is (a) whether the cost is justified vs single-pass probes, (b) whether bidirectional-entailment clustering is the right equivalence relation, (c) whether it transfers across domains. Also: **it is gray-box** — it needs token probabilities, unusable behind most commercial APIs.

**Probing / internal states.** *LLMs Know More Than They Show* — Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, **Yonatan Belinkov**. **ICLR 2025**, arXiv:2410.02707. Four findings, and the third is the one people skip: (1) internal representations encode more truthfulness signal than thought; (2) the signal is **concentrated in specific tokens**; (3) **error detectors fail to generalize across datasets — truthfulness encoding is not universal but multifaceted**; (4) models can internally encode the right answer and still emit the wrong one. Counter-current: arXiv:2510.09033 argues the probe signal is largely **recall/familiarity, not truthfulness**. ⚠️ Active dispute; both cannot be right in their strong forms.

*ReDeEP* (Zhongxiang Sun, Xiaoxue Zang, Kai Zheng, Yang Song, Jun Xu, Xiao Zhang, Weijie Yu, Han Li; arXiv:2410.11414) — mechanistic claim that RAG hallucination occurs when **Knowledge FFNs over-weight parametric knowledge in the residual stream while Copying Heads fail to integrate retrieved context**. ⚠️ **The abstract carries no numbers**; a third-party 2026 paper (arXiv:2605.07209) reports **ReDeEP token-level AUC 0.73 on RAGTruth**, beating it by 7.4–10.3 points — treat as unverified secondary reporting.

**Which methods generalize? None of them, cleanly.** Three independent results converge: attribution scorers invert rankings across datasets (τ = −0.64) and one drops to chance; internal-state error detectors fail to generalize (Orgad et al., ICLR 2025); detector rankings shift unpredictably under label noise and plain LLM-as-a-Judge is competitive (TRIVIA+). **The practical implication is uniform: validate a detector on your own target distribution. Picking by published averages carries ~0.17 AUROC mean regret. And the honest baselines to beat are LLM-as-a-Judge and BERTScore, not SelfCheckGPT.**

### 5.3 Do verification passes measurably help?

**The negative camp.** *Large Language Models Cannot Self-Correct Reasoning Yet* — Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, **Denny Zhou** (Google DeepMind/Research). **ICLR 2024**, arXiv:2310.01798. **"LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction."** The methodological contribution is showing prior reported gains came from **oracle labels leaking in** (the loop was told when to stop) or from baselines not given equal compute.

*On the Self-Verification Limitations of LLMs on Reasoning and Planning Tasks* — Kaya Stechly, Karthik Valmeekam, **Subbarao Kambhampati**. **ICLR 2025**. ⚠️ Search returned arXiv:2402.08115; I could not verify that ID. Finding: LLM self-critique does not deliver iterative improvement; sound *external* verifiers do.

*SELF-[IN]CORRECT* (arXiv:2404.04298) gives the mechanism: models are no better at **discriminating** among their own candidates than at generating a good one. If generation and discrimination are equally strong, a self-critique loop has no information advantage.

**⭐ *Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback*** — Dongwei Jiang, Alvin Zhang, Andrew Wang, Nicholas Andrews, **Daniel Khashabi**. arXiv:2506.11930, v2 21 Sep 2025. Uncomfortable for the "just add external feedback" fix: **even under near-perfect, complete external feedback, models plateau below the achievable ceiling** across math, knowledge, scientific, and multi-domain reasoning including Claude 3.7 with extended thinking. Progressive temperature sampling and explicit rejection of prior wrong answers **still fail**. **Model confidence, measured via semantic entropy, predicts feedback resistance** — high-confidence wrong answers are the ones that won't budge. ⚠️ No numeric ceilings in the abstract.

*Decomposing LLM Self-Correction: The Accuracy-Correction Paradox and Error Depth Hypothesis* (arXiv:2601.00828, Jan 2026) — **weaker models show 1.6× higher intrinsic correction rates than stronger models**. Strong models' remaining errors are *deep* errors self-correction structurally cannot reach, so raw "correction rate" is inversely correlated with base capability and is an actively misleading metric.

**The positive camp.** *CorrectBench* (Guiyao Tie et al., arXiv:2510.16062) — self-correction does improve accuracy, especially on complex reasoning. But three quiet negatives inside: **reasoning models (DeepSeek-R1) show limited additional gain at high time cost** (RL-trained reasoning already internalizes the loop); combining strategies helps at an efficiency cost; and **plain CoT is competitive on accuracy *and* efficiency.**

**⭐ The strongest RAG-specific positive, and it credits the verifier not the reflection.** *Self-Correcting RAG (NLI-guided MCTS)* — Shijia Xu, Zhou Wu, Xiaolong Jia, Yu Wang, Kai Liu, April Xiaowen Dong. arXiv:2604.10734, 12 Apr 2026:

| Metric | Standard RAG | Self-Correcting RAG |
|---|---|---|
| Attribution Precision | 0.52 | **0.85** |
| Contradiction Rate | 0.15 | **0.04** |
| Supportability | 0.65 | **0.88** |
| EM (avg, 6 datasets) | 25.8% | **37.1%** |
| F1 (avg, 6 datasets) | 36.1% | **45.8%** |

**The ablation is load-bearing: MMKP context selection alone gives EM 34.5% but attribution precision only 0.58. NLI-guided MCTS alone gives attribution precision 0.82 at EM 31.2%.** Better context alone barely moves faithfulness; **the verifier does essentially all the faithfulness work.** Datasets: NQ, PopQA, MuSiQue, 2Wiki, HotpotQA (1,000 queries each) + MultiHop-RAG (2,556). ⚠️ Preprint; "Attribution Precision" is itself an automatic scorer, which §5.1 says may not transfer.

*CRAG* (Yan et al., **ICLR 2024**, arXiv:2401.15884) — lightweight T5 retrieval evaluator triggering correct/ambiguous/incorrect actions with web-search fallback. **2026 reproduction** (arXiv:2603.16169) finds two things: (a) the original relied on the Google Search API and closed weights; swapping in Wikipedia API + Phi-3-mini **reproduces comparable performance**; (b) **SHAP analysis shows CRAG's T5 evaluator primarily keys on named-entity alignment, not semantic similarity** — the "retrieval quality evaluator" is substantially a lexical entity-overlap detector. A meaningful deflation even though the numbers replicate. *Self-RAG* (Asai et al., ICLR 2024) trains the critique in via reflection tokens; ⚠️ **I found no rigorous 2025–26 independent reproduction.**

**→ THE HONEST NET VERDICT.** The apparent conflict dissolves along one axis:
1. **Intrinsic self-correction — no external signal, no oracle stopping — does not reliably help and sometimes hurts.** Huang et al. (ICLR 2024) canonical; Stechly/Kambhampati extends to planning; SELF-[IN]CORRECT gives the mechanism; the 2026 error-depth paper shows the apparent gains shrink as base models strengthen. **Nothing in 2025–26 overturns this.**
2. **Verification against an external, sound signal does help substantially in RAG.** The NLI-MCTS ablation is the cleanest demonstration: verifier alone moves attribution precision 0.52 → 0.82; better retrieval alone moves it 0.52 → 0.58. **The gain is in the verifier, not in the reflection.**
3. **But external feedback is not a solved fix.** Feedback Friction: models resist even near-perfect feedback, and semantic-entropy confidence predicts which errors are unfixable.
4. **If you are running a modern reasoning model, a bolted-on critique pass is likely a latency tax** (CorrectBench).
5. **Verification-loop gains are hard to measure honestly**, because the faithfulness metrics used to score them are the same ones §5.1 showed don't transfer.

**Practical distillation: add a verification pass only when it consumes something the generator did not already see** — retrieved evidence checked by an independent NLI model, a search result, an executor, a sound checker. A pass that re-reads the model's own output and asks "are you sure?" is a latency tax.

### 5.4 Groundedness benchmarks in 2026

**FACTS Benchmark Suite (Google DeepMind + Kaggle), released 9 Dec 2025.** Supersedes the standalone FACTS Grounding leaderboard by absorbing it. Four pillars: Parametric (2,104), Search (1,884, standardized web-search tool, often multi-hop), Multimodal (1,522), Grounding v2. **3,513 examples publicly released** (⚠️ the four listed sizes exceed this — presumably the public split only; I could not resolve the inconsistency). **Kaggle owns the private held-out sets, runs the evaluations, and hosts the leaderboard** — the right structural answer to contamination and the main reason to cite FACTS in 2026.

**Results: 15 leading models tested. Gemini 3 Pro leads at 68.8% FACTS Score. Every model — Claude, GPT, Llama included — scores below 70%.**

⚠️ **Google builds the benchmark and Google's model tops it.** Kaggle's custody of the private sets mitigates contamination but not design bias. The earlier FACTS Grounding used an **ensemble of three frontier judges** (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet) to dilute single-judge bias; the blog does not state whether the Suite keeps that ensemble. ⚠️ Older grounding-only figures circulating (Gemini 2.0 Flash Exp 83.6%, Claude 3.5 Sonnet 79.4%, GPT-4o 78.8%) come from a vendor blog, not DeepMind, and I could not confirm them against Kaggle (page rendered empty). Note the ~80% grounding-only and ~69% composite are **different scales measuring different things** — not a trend.

⚠️ 2026 entrants surfaced but not fetched: **LayerRAG-Bench** (arXiv:2607.27353) — argues groundedness-only evaluation produces **false positives** when answers look grounded but fail at the evidence, tool-contract, authorization, or session-state layer (the right critique for agentic RAG); **From Binary Groundedness to Support Relations** (Sarkar, Poelitz, Kewenig, **Microsoft Research**, arXiv:2604.08082) — argues the binary supported/unsupported framing is wrong.

**Where frontier models actually sit, in one paragraph.** On the most rigorous contamination-controlled composite (FACTS Suite, private held-out sets): **no model exceeds 70%; the leader is 68.8%.** On grounding-in-provided-documents, the best models sit around 80–84% (older leaderboard). On summarization faithfulness with an explicit "do not infer" instruction at temp 0, the best hallucinate on **~2–4%** of summaries (vendor-scored). On RAG QA faithfulness, error rates are several times higher. **On citation-level factual support in deep research agents, only 39–77% of citations actually support their claim, degrading ~42% as tool calls scale from 2 to 150.** The spread between those numbers *is* the story: the more constrained the grounding task, the better models look, and **the number that matters most for production agentic RAG is the worst one**.

---

## 6. GRAPHRAG — DID THE EVIDENCE HOLD UP?

### 6.1 The origin paper never got a venue

"From Local to Global: A Graph RAG Approach to Query-Focused Summarization" — Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, Jonathan Larson (**Microsoft Research**). arXiv:2404.16130, v1 24 Apr 2024, v2 19 Feb 2025. **1,875 citations.**

Two corpora — podcast transcripts (~1M tokens, 8,564 entities, 20,691 edges) and news (~1.7M tokens, 15,754 entities, 19,520 edges). **72–83% comprehensiveness win rates and 62–82% diversity win rates vs naive vector RAG**, judged pairwise by an LLM.

⚠️ **I could not verify any peer-reviewed venue.** No journal-ref or venue comment on arXiv through v2 (Feb 2025). As best I can establish, **the single most influential GraphRAG paper remained an arXiv preprint / MSR tech report and was never published at ACL/EMNLP/NeurIPS/ICLR.** The field built on an unrefereed baseline. And the numbers are **LLM-as-judge pairwise preferences on comprehensiveness and diversity** — not accuracy, not F1, not human evaluation — over **LLM-generated evaluation questions**, with no cost figures.

### 6.2 The successor line

| System | Paper | Venue | Headline |
|---|---|---|---|
| **RAPTOR** | Sarthi, Abdullah, Tuli, Khanna, Goldie, **Manning** (Stanford), arXiv:2401.18059 | **ICLR 2024** | **+20% on QuALITY** with GPT-4; +2% over DPR, +5.1% over BM25 |
| **LightRAG** | Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, Chao Huang (HKU), arXiv:2410.05779 | **EMNLP 2025** | Dual-level graph retrieval; lower cost + faster incremental update than GraphRAG |
| **HippoRAG 2** | Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, Yu Su (Ohio State), arXiv:2502.14802 | **ICML 2025** | **+7% associative memory** over SOTA embeddings. **The abstract itself concedes prior graph methods' "performance on more basic factual memory tasks drops considerably below standard RAG"** |
| **GraphReader** | arXiv:2406.14550 | **Findings EMNLP 2024** | 4k-context GraphReader beats GPT-4-128k on LV-Eval across 16k–256k |
| **Think-on-Graph 2.0** | arXiv:2407.10805 | **ICLR 2025** | +5.51% over CoK on HotpotQA |
| **KAG** | Ant Group / OpenSPG, arXiv:2409.13731 | **ACM Web Conf 2025** | Schema-constrained KG to fix OpenIE noise in HippoRAG/GraphRAG |
| **PathRAG** | arXiv:2502.14902 | ⚠️ venue unverified | Flow-based path pruning. Key framing: **the problem with GraphRAG is redundancy, not insufficiency** |
| **Youtu-GraphRAG** | Junnan Dong, Siyu An, … Xiao Huang, Yunsheng Wu, Di Yin, Xing Sun (Tencent Youtu + Monash + HK PolyU), arXiv:2508.19855 | ⚠️ GitHub says ICLR 2026; arXiv comments do not | **+16.62% accuracy, −90.71% token cost** in graph construction |
| **KET-RAG** | arXiv:2502.09304 | **KDD 2025** | **20% indexing cost reduction** |
| **LazyGraphRAG** | Microsoft Research, 25 Nov 2024 | ⚠️ **blog, not a paper** | Indexing cost **identical to vector RAG, 0.1% of full GraphRAG**; comparable global-query quality at **>700× lower query cost** |

That last row is the tell. **You don't build LazyGraphRAG unless the original was unaffordable.**

### 6.3 The skeptical line — this is the substantive part

**⭐ RAG vs. GraphRAG: A Systematic Evaluation and Key Insights** — Haoyu Han, Li Ma, Yu Wang, Harry Shomer, Yongjia Lei, Zhisheng Qi, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, **Charu C. Aggarwal**, Jiliang Tang (**Michigan State + Meta + IBM** — *not* NVIDIA; I found no NVIDIA study of this name). arXiv:2502.11371, v1 17 Feb 2025, **v3 4 Mar 2026.**

- **Natural Questions (single-hop):** RAG **64.78 F1** > Community-GraphRAG(Local) 63.01 > HippoRAG2 61.03. **Plain RAG wins.**
- **HotpotQA:** HippoRAG2 63.01 > Community-GraphRAG(Local) 61.66 > RAG 60.04.
- **MultiHop-RAG (accuracy):** HippoRAG2 **70.27** > Community-GraphRAG(Local) 69.01 > RAG 67.02. **The multi-hop win is ~+3 points, not a category change.**
- **Complementarity: 13.6% of MultiHop-RAG queries are GraphRAG-only wins, 11.6% are RAG-only wins** — nearly symmetric.
- **Summarization (SQuALITY, ROUGE-2 F1): RAG 10.08, Community-GraphRAG(Local) 10.10, Community-GraphRAG(Global) 6.99.** The global/community mode — Microsoft GraphRAG's entire selling point — **loses badly on a real summarization metric.**
- **Cost (MultiHop-RAG):** construction time RAG **135s** vs KG-GraphRAG **7,702s** vs Community-GraphRAG **5,560s** — a **41–57× indexing penalty**. Retrieval latency 1,724s / 14,434s / 1,249s. Storage 127 / 117 / 165 MB.
- Hybrid integration: **+6.4%** on MultiHop-RAG with Llama-3.1-70B.
- **Methodological warning: "position bias is clearly present in LLM-as-a-Judge evaluations"** — directly undercutting Edge et al.'s win-rate methodology.

**⭐ When to use Graphs in RAG / GraphRAG-Bench** — Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, Jinsong Su. arXiv:2506.05690, v1 6 Jun 2025, **v3 22 Feb 2026**. ⚠️ GitHub labels it ICLR 2026; unverified from arXiv. The abstract opens with the negative framing: *"recent studies report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks."*

Across 7 frameworks (MS-GraphRAG, HippoRAG, HippoRAG2, LightRAG, Fast-GraphRAG, RAPTOR, Lazy-GraphRAG):
- **Level 1 fact retrieval, Novel:** Basic RAG w/ rerank **60.92%** vs best GraphRAG (HippoRAG2) 60.14% — *graph loses.*
- **Level 2 complex reasoning, Novel:** Basic RAG 42.93% → HippoRAG2 **53.38% (+24.4% relative)** — the clearest genuine graph win.
- **Level 3 contextual summarization, Novel:** Basic RAG 51.30% → MS-GraphRAG **64.40%**.
- **Evidence recall inverts:** on fact retrieval, Basic RAG recall **83.21%** vs HippoRAG2 70.29%. Graph retrieval is *worse at finding the right evidence* for simple questions.
- **The cost table is the headline — average prompt tokens per query:**

| Method | Novel | Medical |
|---|---|---|
| Vanilla RAG | **879** | **954** |
| HippoRAG2 | 1,008 | 1,020 |
| RAPTOR | 3,441 | 3,510 |
| HippoRAG | 7,208 | 7,342 |
| LightRAG | 100,832 | 100,310 |
| **MS-GraphRAG (global)** | **331,375** | 332,881 |

**Microsoft GraphRAG global search burns ~377× the prompt tokens of vanilla RAG per query** — inference cost, on top of indexing cost. Conclusion: graph "introduces redundant information, which in turn degrades context relevance."

Also: **GraphRAG-Bench dataset paper** (Yilin Xiao, Junnan Dong, Chuang Zhou, Su Dong, Qian-wen Zhang, Di Yin, Xing Sun, Xiao Huang; arXiv:2506.02404) — college-level questions across 16 disciplines / 20 textbooks, 9 methods, scoring reasoning coherence not just final answers.

**⭐ Do We Still Need GraphRAG? Benchmarking RAG and GraphRAG for Agentic Search Systems** — Dongzhe Fan, Zheyi Xue, Siyuan Liu, Qiaoyu Tan. arXiv:2604.09666, **1 Apr 2026.** The 2026 reframing: does agentic multi-round retrieval make explicit graphs redundant? Standardized LLM backbone, retrieval budget, inference protocol, full test sets.
- **Single-shot:** general QA — graph gives only **+0.47 avg**. Multi-hop — graph gives **+27.23 avg** (Contain-EM).
- **With training-free agentic search:** dense RAG + GraphSearch **narrows the multi-hop gap by 32.3%**.
- **With GRPO agentic search:** dense RAG becomes best on Natural Questions; graph backends still lead multi-hop.
- Verdict: agentic search "substantially improves dense RAG and narrows the gap... Nevertheless, GraphRAG remains advantageous for complex multi-hop reasoning... **when its offline cost is amortized.**"

**⭐ The harshest result: BM25 Wins at Scale: A Scaling Study of RAG Paradigms** — Pengyu Wang, Benfeng Xu, Shaohan Wang, Mingxuan Du, Xin Zeng, Huarui Wu, Lei Zhang, Licheng Zhang. arXiv:2607.26497, **29 Jul 2026** (v3 31 Jul). 28 strictly nested corpus tiers spanning ~450× expansion, fixed questions and reference docs.
- A file-system agent leads at small scale but **costs 39× more query tokens** at the largest tier.
- **BM25 overtakes it around 10M corpus tokens and leads at every larger shared tier, with a margin approaching 20 points at full scale.**
- **Graph-based RAG "encounters construction walls before deployment scale, and its scalable variants remain below BM25 at shared tiers."** At real corpus sizes GraphRAG cannot even be built, and the cheap variants lose to lexical search from 1994.
- Conclusion: "lexical retrieval is the strongest scalable default, while agentic reasoning works best **after** ranked discovery rather than in place of it."

⚠️ I searched for a paper titled "GraphRAG under fire" and **found no such academic paper.** Not asserting it exists.

### 6.4 Honest 2026 verdict

**The evidence did not stall — it substantially deflated and re-scoped.**

1. **Single-hop / factual lookup: graph loses.** 64.78 vs 63.01 F1 (NQ); 60.92% vs 60.14% (GraphRAG-Bench). Consistent across independent groups. The HippoRAG 2 authors concede it in their own abstract.
2. **Multi-hop: graph genuinely wins, but the magnitude is wildly contested — +3 points (controlled same-backbone, MultiHop-RAG) to +24% relative (GraphRAG-Bench) to +27 EM (RAGSearch single-shot).** The spread *is* the finding: it depends almost entirely on whether the vector baseline is well-tuned and whether the questions are natively bridge-entity questions.
3. **Global/sensemaking summarization: the original claim is the least replicated.** Edge et al. reported 72–83% comprehensiveness win rates on LLM-judge preference; the controlled ROUGE-2 replication gives Community-GraphRAG Global **6.99 vs RAG 10.08** — global search *loses*. GraphRAG-Bench does find graph wins on Level-3 summarization. Verdict: **graph helps on breadth/coverage summarization, but the specific "global search over community summaries" mechanism is not well supported, and the original evidence rested on an LLM-judge protocol later shown to have position bias.**
4. **Cost is the decisive variable, not accuracy.** Indexing 41–57× (135s → 5,560–7,702s); inference **331k prompt tokens/query vs 879** (~377×). Microsoft's own LazyGraphRAG (0.1% indexing, >700× lower query cost), Youtu's −90.71% construction tokens, and KET-RAG's −20% are the same admission from three different groups.
5. **The 2026 threat isn't better vector RAG — it's agentic search and BM25.** Agentic retrieval closes 32.3% of the multi-hop gap with no graph; at deployment scale, graph can't be constructed at all.
6. **The rule the literature converges on:** build a graph only when (a) queries are genuinely multi-hop over bridge entities, (b) the corpus is small/static enough to amortize indexing, and (c) you have already lost to a hybrid BM25+dense+reranker baseline. The 13.6%/11.6% complementarity says the right answer is **hybrid routing**, not graph-everything.

---

## 7. MULTI-TURN / MEMORY-AUGMENTED RETRIEVAL AGENTS

### 7.1 The load-bearing context paper

**LLMs Get Lost In Multi-Turn Conversation** — Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer Neville (**Microsoft Research + Salesforce Research**). arXiv:2505.06120, 9 May 2025. **378 citations.**

- **Average −39% performance across six generation tasks**, single-turn → multi-turn, for *every* top open- and closed-weight LLM tested. **200,000+ simulated conversations.**
- **The decomposition is the important part: the drop is a minor loss in aptitude and a large increase in unreliability.** Models aren't dumber multi-turn — they're erratic.
- Mechanism: LLMs "make assumptions in early turns and prematurely attempt to generate final solutions, on which they overly rely."

**Why this matters for memory systems:** multi-turn failure is substantially a *generation-side* pathology, not purely a retrieval/recall pathology. A memory layer that only fixes recall addresses part of the problem. Papers claiming large multi-turn wins from memory alone should be read against this.

### 7.2 Benchmarks

**LongMemEval** — Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu (UCLA / Tencent AI Lab / UCSD). arXiv:2410.10813, **ICLR 2025** (confirmed in arXiv comments + proceedings). 500 curated questions, freely scalable histories; five abilities including **knowledge updates and abstention**. **30% accuracy drop** for commercial assistants and long-context LLMs on sustained interaction.

**LongMemEval-V2** — Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang. arXiv:2605.12493, **12 May 2026**. 451 questions, up to **500 trajectories / 115M tokens**. **AgentRunbook-C 72.5%; AgentRunbook-R (RAG-based memory) 48.5%; off-the-shelf coding agent baseline 69.3%.** Note: **the RAG-memory approach loses to no memory system at all by 21 points.**

**LoCoMo** — Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang. **ACL 2024**, arXiv:2402.17753. ~600 turns / 16K tokens avg over up to 32 sessions. **This is the benchmark the entire industry memory-scoring war is fought on, and it is small and partly defective — see §7.4.**

**MemoryAgentBench** — Yuanzhe Hu, Yu Wang, Julian McAuley (UCSD). arXiv:2507.05257, latest 28 Jun 2026. Four competencies: accurate retrieval, test-time learning, long-range understanding, **conflict resolution / selective forgetting**. "Current methods fall short of mastering all four." Memory agents on GPT-4o reach only **~60% on single-hop conflict resolution.**

**MTRAG** — Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, Marina Danilevsky (**IBM Research**). arXiv:2501.03468. ⚠️ IBM's publications page lists it as **TACL**; arXiv v1 has no venue comment. **110 fully human-generated conversations, avg 7.7 turns, 4 domains, 842 tasks.** SOTA RAG systems struggle specifically on **later turns, unanswerable questions, and non-standalone questions**. Now the basis of **SemEval-2026 Task 8 (MTRAGEval)**.

**HELMET** (**ICLR 2025**) covers multi-turn categories; key relevant finding: **synthetic tasks like NIAH do not reliably predict downstream performance**, and category scores correlate poorly with each other.

**⭐ Memora: From Recall to Forgetting** — Md Nayem Uddin, Kumar Shubham, Eduardo Blanco, Chitta Baral, Gengyu Wang. arXiv:2604.20006, **21 Apr 2026**. Weeks-to-months conversations; introduces **FAMA (Forgetting-Aware Memory Accuracy)**, penalizing reliance on obsolete or invalidated memory. **Evaluating 4 LLMs and 6 memory agents: "frequent reuse of invalid memories and failures to reconcile evolving memories. Memory agents offer marginal improvements."** The strongest 2026 negative result in this space.

Also *PerLTQA* (arXiv:2402.16288, 8,593 questions / 30 characters, semantic + episodic memory).

### 7.3 Systems

| System | Paper | Venue | Numbers |
|---|---|---|---|
| **MemGPT / Letta** | Packer et al., arXiv:2310.08560 | preprint | OS-style paged memory; established DMR as its eval |
| **Zep / Graphiti** | Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, Daniel Chalef. arXiv:2501.13956, 20 Jan 2025 | preprint, no venue comment | **DMR 94.8% vs MemGPT 93.4%**; LongMemEval **up to +18.5%**, **−90% latency** |
| **Mem0** | Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, Deshraj Yadav. arXiv:2504.19413, 28 Apr 2025 | ⚠️ arXiv shows no venue; secondary sources say ECAI 2025 (unverified) | **+26% relative LLM-as-Judge over OpenAI memory**; graph variant +~2%; **−91% p95 latency**, **>90% token cost saving** vs full-context |
| **MemoRAG** | arXiv:2409.05591 | — | Light long-range model builds global memory + draft clues; expensive model answers |
| **Cognitive Workspace** | arXiv:2508.13171 | preprint | **58.6% avg memory reuse vs 0% for traditional RAG**; 17–18% net efficiency gain despite 3.3× operations; p<0.001, Cohen's d>23. ⚠️ **Effect sizes that large are a red flag for a self-defined metric, and "0% for RAG" is true by construction** |
| **Memory-R1** | arXiv:2508.19828 | preprint | RL (PPO/GRPO) over ADD/UPDATE/DELETE/NOOP memory ops. On LLaMA-3.1-8B: **+48% F1, +69% BLEU-1, +37% LLM-as-Judge**, trained on only **152 QA pairs**. Beats Mem0, LangMem, A-MEM |

### 7.4 ⚠️ CONTESTED — the Mem0 vs Zep benchmark dispute

A real, documented, unresolved public dispute. **Provenance warning: the underlying claims are in arXiv papers; the rebuttals live in blog posts and a GitHub issue, not in peer review.**

1. **Zep (arXiv:2501.13956, Jan 2025)** claims DMR 94.8% vs MemGPT 93.4% and LongMemEval +18.5% / −90% latency; separately publicized **~84% on LoCoMo**.
2. **Mem0 (arXiv:2504.19413, Apr 2025)** benchmarks 10 approaches on LoCoMo: **Mem0 at 67.13% LLM-as-Judge, +26% relative over OpenAI memory, and Zep at 65.99%.**
3. **Mem0 → Zep, GitHub issue getzep/zep-papers#5, filed 8 May 2025 by Deshraj Yadav (Mem0 CTO):** alleges Zep's 84% is invalid because Zep **included questions from LoCoMo's excluded adversarial 5th category in the numerator while excluding them from the denominator**; also alleges Zep modified the system prompt with timestamp-favoring instructions not given to baselines, changed the retrieval template vs its own prior DMR work, and reported a **single run** rather than Mem0's 10-run-with-variance standard. **Mem0's re-run of Zep: 58.44% ± 0.20 — a 25.56-point reduction.** ⚠️ As of the fetch, the issue was open with no Zep response in-thread.
4. **Zep → Mem0, blog "Lies, Damn Lies, and Statistics":** alleges Mem0 misconfigured Zep three ways — assigned the **user role to both participants** in a graph designed for single user-assistant interaction; passed **timestamps appended to message text instead of Zep's dedicated `created_at` field**, breaking temporal reasoning; ran **searches sequentially rather than in parallel**, inflating latency. Zep's corrected numbers: **75.14% ± 0.17** (vs Mem0's reported 65.99%), p95 search latency **0.632s** vs Mem0's reported 0.778s.
5. **Zep also attacks the benchmark:** LoCoMo conversations are only **16k–26k tokens**, contain **no knowledge-update tests**, and have data-quality defects — missing ground truths, multimodal errors, incorrect speaker attribution, ambiguous questions.

**Net state of the Zep-on-LoCoMo number: 84% (Zep original) → 65.99% (Mem0's measurement) → 75.14% (Zep's correction) → 58.44% (Mem0's re-run of Zep's own pipeline). A ~26-point spread on the same system and same benchmark, with no neutral adjudication.**

**Honest reading: both parties are vendors self-reporting on a 2024 academic benchmark that both agree is partly broken. No corrected figure has been independently replicated in a refereed venue. Treat all LoCoMo numbers from memory vendors as unreliable.** The academic side has moved on — MemoryAgentBench, LongMemEval-V2, and Memora were all built partly because LoCoMo is inadequate.

**⭐ Independent counter-evidence.** "Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents" — Natchanon Pollertlam, Witchayut Kornsuwannawit. arXiv:2603.04814, **5 Mar 2026**. Mem0-based fact memory vs long-context GPT-5-mini on LongMemEval, LoCoMo, PersonaMemv2.
- **Long-context GPT-5-mini achieves *higher* factual recall on both LongMemEval and LoCoMo.** Memory is only competitive on PersonaMemv2.
- The real finding is the cost model: long-context cost grows per-turn even under prompt caching; memory read cost is roughly fixed after a one-time write. **At 100k context, memory becomes cheaper after ~10 turns**, break-even falling as context grows.
- **Interpretation: as of 2026, memory systems are an economic argument, not an accuracy argument** — directly contradicting Mem0's paper framing.

⚠️ **Two false attributions circulating in blog aggregators, corrected:** (a) "Letta scored 49.0% on LongMemEval in independent evaluation (arXiv 2603.04814)" — **that paper does not evaluate Letta or Zep at all**, only Mem0 vs long-context GPT-5-mini. Do not use it. (b) "Mem0's new algorithm hits 93.4% on LongMemEval vs Zep 63.8%" traces to **Mem0's own 2026 marketing blog**, not a paper.

### 7.5 Query rewriting vs end-to-end retrieval in multi-turn

The field splits into **conversational query rewriting (CQR)** — produce a standalone de-contextualized query — and **conversational dense retrieval (CDR)** — encode the whole session end-to-end.

**TREC iKAT** 2024 and 2025. iKAT 2025 has a **SIGIR 2026 resource paper** (ACM DOI 10.1145/3805712.3808591) and a NIST track overview; the 2025 edition moved to a **live API where systems must rewrite, retrieve, and ground in real time per turn**. ⚠️ **Methodological caveat from the track itself: manual runs use the human rewrite as input, which advantages them and leaves assessment less complete for automatic reformulations — so CQR-vs-ceiling comparisons are biased.** Participant papers: RALI@TREC iKAT 2024 (arXiv:2412.07998), CFDA & CLIP @ TREC iKAT 2025 (arXiv:2509.15588), Adaptive Personalized Conversational IR (arXiv:2508.08634).

**⭐ The clearest 2026 evidence comes from SemEval-2026 Task 8 (MTRAGEval), built on IBM's MTRAG.**
- **Sifei @ SemEval-2026 Task 8** (arXiv:2606.28352): training-free hybrid dense+sparse retrieval with **controlled query rewriting** + cross-encoder reranking → **0.5453 nDCG@5, 3rd of 38 teams**, vs strongest baseline 0.4795.
- **The most useful negative finding: controlled conversational rewriting combined with last-turn concatenation gives consistent gains across domains, while retrieval-oriented rewrites — keyword lists, hypothetical-document expansion — consistently HURT**, by distorting intent and over-amplifying rare terms.
- Others: uva-irlab-conv (arXiv:2606.11945 — learned sparse + listwise reranking, five complementary LLM reformulations fused via variance-aware nested RRF); H-RAG (arXiv:2605.00631); AILS-NTUA (arXiv:2603.10524). Earlier: arXiv:2406.18960.

**Verdict: as of 2026, CQR has not been displaced by end-to-end conversational dense retrieval — every competitive MTRAGEval system rewrites. But the *type* of rewrite matters and the naive "make it retrieval-friendly" instinct is empirically wrong: decontextualize and preserve intent, keep the raw last turn, don't keyword-ify.**

### 7.6 Honest 2026 verdict

1. **The problem is real and large:** −39% single→multi-turn (200k+ sims), −30% on LongMemEval, MTRAG failure concentrated on later/non-standalone/unanswerable turns.
2. **The cause is not purely retrieval.** Laban et al.'s decomposition — unreliability, not aptitude — means a memory layer can only fix part of it.
3. **Vendor memory-system accuracy claims are not trustworthy right now.** A 26-point swing on one system on one benchmark with no neutral referee, and both sides agree the benchmark is inadequate.
4. **Independent 2026 work is unkind to memory systems.** Memora: 6 agents, "marginal improvements," frequent reuse of invalidated memory. LongMemEval-V2: RAG-based memory 48.5% vs a plain coding agent at 69.3%. arXiv:2603.04814: long-context GPT-5-mini beats Mem0 on factual recall on both LongMemEval and LoCoMo.
5. **The surviving case for memory is economic, not qualitative:** fixed per-turn read cost vs context cost growing even under caching; break-even ~10 turns at 100k context.
6. **The unsolved competency is not recall — it's update and forgetting.** ~60% on single-hop conflict resolution with GPT-4o; Memora's whole FAMA metric exists because systems keep citing superseded facts; LongMemEval flagged knowledge-updates and abstention in Oct 2024 and they remain the failure modes in 2026.
7. **Most promising direction with real numbers: learned memory management.** Memory-R1 (+48% F1 / +69% BLEU-1 / +37% judge on LLaMA-3.1-8B with 152 training pairs) suggests the ADD/UPDATE/DELETE policy is learnable and is where the headroom is — consistent with (6).

---

## 8. CROSS-CUTTING OBSERVATIONS AND THE FULL LIST OF OPEN DEBATES

### 8.1 The recurring arc

Every one of the seven areas shows the same shape. **A 2024 preprint from a named lab makes a large claim on an LLM-as-judge or vendor-internal evaluation** (Edge et al.'s 72–83% GraphRAG win rates; Anthropic's 67% failure reduction; Jina's late chunking; Zep/Mem0's LoCoMo scores; Search-R1's +41%). **A successor wave builds on it.** Then **2025–2026 controlled, same-backbone, full-test-set replications with standardized budgets shrink the effect to a few points, relocate it to a narrow query class, and reveal the cost was the real story.**

**In every area, the strongest 2026 papers are benchmark, reproducibility, and cost papers — not method papers.** BrowseComp-Plus (ACL 2026), Lighting the Way for BRIGHT (SIGIR 2026 Repro), the PLAID reproduction (SIGIR 2024 Repro), Fixed RAG Compression Collapses Measured Reader Scaling, Do LLM Attribution Metrics Transfer, RAG vs GraphRAG, BM25 Wins at Scale, Memora, and the MTEB RTEB governance decision. If you read only ten things from this report, read those.

### 8.2 A second recurring pattern: the "obvious baseline was omitted"

This charge appears independently in at least six places and is the most reliable predictor of a deflated result:
- PLAID papers omitted BM25 + ColBERT reranking (MacAvaney & Tonellotto).
- BRIGHT baselines silently used an undocumented BM25 variant (Sharifymoghaddam, Ge, Lin).
- Long-context-beats-RAG papers used relevance-ordered, badly-chunked RAG arms (DOS RAG / OP-RAG).
- Self-correction papers leaked oracle stopping signals and under-resourced their baselines (Huang et al., ICLR 2024).
- Compression papers evaluated on weak readers (Panthi & Abdelfattah).
- Chunking papers baselined against fixed-*character* splitting rather than sentence-aware splitting.

### 8.3 The complete open-debate list

1. **Is RL necessary for agentic search, or is it trajectory-data quality?** LiteResearcher measures +15.7 GAIA from RL over its own SFT; OpenSeeker-v2 beats it on BrowseComp/HLE with **pure SFT** on 10.6k curated trajectories. No controlled experiment at matched scale exists.
2. **Do RL-trained search agents generalize?** BrowseComp-Plus: Search-R1-32B scores **exactly its untrained base model's score** on an out-of-distribution corpus. Unrebutted.
3. **Reasoning-aware retrieval: training problem or test-time-compute problem?** LATTICE (Google) matches the best fine-tuned ensembles with an off-the-shelf LLM. Never evaluated against the training camp at matched compute.
4. **Is the single-vector dimensional bound practically binding?** Weller et al. (DeepMind, ICLR'26) yes; Bangachev et al. (MIT) the bound is far looser; Spectral Retrieval lifts LIMIT-small R@10 0.33→0.90 without retraining.
5. **Does late interaction earn its cost?** SIGIR'24 reproduction says BM25+ColBERT rerank at 9 ms/q beats the fastest PLAID at 73 ms/q. Not directly rebutted with an updated head-to-head.
6. **Are late interaction and learned sparse the same thing?** ColBERTSaR proves quantized ColBERT ≡ learned sparse; PLAID clusters align 1:1 with tokens. Emerging synthesis, not consensus.
7. **RRF vs convex combination.** Bruch et al. (TOIS 2023) showed CC wins; every 2025–26 applied paper uses RRF anyway; nobody has re-tested.
8. **Does BM25 still beat dense?** Yes on LIMIT and financial tables and at low latency; no by ~15 points on aggregate suites and by ~14 points in agentic loops. Task-conditional.
9. **How much does hybrid add?** +9.17 nDCG (arXiv:2502.20245) vs +3.1pp (KohakuRAG). Unexplained, unreplicated.
10. **How much does chunking strategy matter?** Resolves once you notice the disagreeing papers use different worst-case baselines (fixed-character vs fixed-token).
11. **Does contextual retrieval work as advertised?** No independent reproduction of 35/49/67 exists.
12. **Are compression numbers real?** Reader-strength confound flips 31% of model rankings and hides 80% of a reader upgrade.
13. **LC vs RAG.** Reconciles as "most LC-wins papers used a weak RAG arm," but LC has its own length-alone degradation problem.
14. **Is "lost in the middle" still true?** Largely trained away for simple retrieval in frontier models; order-sensitivity for composition over many chunks persists.
15. **Does self-verification help?** Resolved along the intrinsic/external axis, but two live sub-disputes: whether external feedback is sufficient (Feedback Friction says no), and whether reported self-correction gains survive controlling for base capability (error-depth says no).
16. **Do internal states encode truthfulness or just recall?** Orgad et al. (ICLR 2025) vs arXiv:2510.09033. Both cannot be right in their strong forms.
17. **Is semantic entropy worth its cost?** No refutation, but three 2026 papers claim single-pass alternatives match or beat it.
18. **Are specialized hallucination detectors better than LLM-as-a-Judge?** TRIVIA+ says LLM-as-Judge is competitive. Every vendor detector paper says otherwise. Note who benefits from each answer.
19. **Do finer citations help or hurt?** Sentence-level requirements degrade attribution 16–276%, and the penalty grows with model scale — contradicting prevailing benchmark design.
20. **Did GraphRAG pay off?** Multi-hop win contested between +3 points and +27 EM; global-summarization claim not replicated; cost is 41–57× indexing and ~377× inference; and at scale it cannot be built.
21. **Do memory systems beat long context?** Independent 2026 work says long context wins on recall; memory wins only on cost past ~10 turns.

---

## 9. WHAT I COULD NOT VERIFY — DO NOT CITE THESE AS FACTS

**Numbers/claims:**
- Current (Aug 2026) MTEB/MMTEB #1. The leaderboard is a client-rendered Gradio Space; every fetch returned the loading shell.
- Numeric tables in arXiv:2605.05242 (grep-based Direct Corpus Interaction vs BM25/dense/rerankers). Existence, authorship (incl. Yejin Choi, Jiawei Han, Jimmy Lin), date, and qualitative claims verified; **the numbers are not.** Worth a follow-up given the author list.
- AttrScore per-model F1 and its human-agreement figure.
- ReDeEP's own reported AUROC (abstract omits it; 0.73 is third-party).
- TRIVIA+ detector AUROC table (PDF tables unreadable).
- Exact figures inside arXiv:2504.01818, arXiv:2604.03676, arXiv:2403.06789.
- 2026 numbers for NV-Embed, Stella, BGE-M3 successors, Voyage flagship tiers.
- Current Kaggle FACTS Grounding standings (page rendered empty).
- HHEM figures for GPT-5 Pro / Claude Opus 4.7 that appear in SEO summaries but **not** on the actual Vectara leaderboard.
- Lynx 2.0 numbers (blog-only). Galileo Hallucination Index methodology.
- "BEIR is no longer zero-shot because researchers train on it" and "MTEB has 400+ models with marginal differences" — search-snippet only.
- SAAS, AutoSearch, and Agentic-R report no numbers in their abstracts.

**Venues:**
- **A peer-reviewed venue for Edge et al. (GraphRAG, arXiv:2404.16130) — likely never formally published.**
- ICLR 2026 for GraphRAG-Bench (2506.05690) and Youtu-GraphRAG (2508.19855) — asserted in GitHub titles only.
- ECAI 2025 for Mem0 (2504.19413) — secondary sources only.
- TACL for MTRAG — IBM's page says so; arXiv has no comment.
- PathRAG (2502.14902).
- arXiv ID for Stechly/Valmeekam/Kambhampati ICLR 2025 (search said 2402.08115; unverified).
- "Gosling Grows Up" (SIGIR 2025) — arXiv ID not verified and deliberately not guessed.

**Corrections to things circulating as fact:**
- There is **no NVIDIA "RAG vs GraphRAG" study**; the systematic evaluation is arXiv:2502.11371 from **Michigan State + Meta + IBM**.
- There is **no academic paper titled "GraphRAG under fire."**
- **"Letta 49.0% on LongMemEval per arXiv:2603.04814" is false** — that paper evaluates Mem0 vs long-context only.
- **arXiv:2601.04618 (REPAIR) was withdrawn by its authors on 14 Apr 2026.** Do not cite its +5.6pp.
- **Reason-ModernColBERT has no paper**, and its "beats ReasonIR-8B by 2.5 nDCG" claim holds only on the StackExchange splits; it loses on the full BRIGHT mean.
- **LettuceDetect is not SOTA on RAGTruth**; fine-tuned Llama-3-8B at 83.9% is.

**Citation counts** (Semantic Scholar API, 8 Aug 2026; the API was rate-limited for most of this session and OpenAlex badly undercounts preprints, so this is a partial set): GraphRAG 1,875 · Search-R1 1,317 · LLMs Get Lost 378 · BRIGHT 177 (ICLR) · BrowseComp-Plus 155 · ReasonIR 76 · Rank1 72. GitHub traction as a secondary proxy: Alibaba-NLP/DeepResearch 19.8k stars · Search-R1 5.3k · FlashRAG 3.5k · PyLate 877 · BrowseComp-Plus 327 · ReasonIR 230.

---

--- METRICS ---
- Files Extracted: 23
- Estimated Token Budget: ~134564 tokens
- Canonical Reference: https://codewiki.google/github.com/PromtEngineer/localGPT