## File: README.md

---

```bash
pip install pixelrag
```

The two core operations — **render** a page to screenshots, **search** a visual index:

```bash
# Render any page or document to screenshot tiles
pixelshot https://en.wikipedia.org/wiki/Python --output ./tiles

# Search a hosted index of 8.28M Wikipedia pages — no setup, runs against the live API
curl -X POST https://api.pixelrag.ai/search \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
```

> **Live, hosted endpoint** — [`https://api.pixelrag.ai`](https://api.pixelrag.ai/status) serves a
> pre-built index of **8.28M Wikipedia pages**. No setup, no API key. It even takes an image as the query
> ([visual search](https://pixelrag.ai/docs#search)) — see the **[API reference →](https://pixelrag.ai/docs)**.

Or try it in the browser at **[pixelrag.ai](https://pixelrag.ai)**, or run the demo notebook in
Colab [](https://colab.research.google.com/github/StarTrail-org/PixelRAG/blob/main/demos/quickstart.ipynb) — it
renders a page and searches the hosted index, with the images inline.

## What it is

PixelRAG renders documents — web pages, PDFs, images — as screenshots and retrieves over the
images directly. Visual structure that HTML parsing throws away — tables, charts, layout,
infographics — stays intact, so the reader model can actually answer questions about it.
Wikipedia's 8.28M articles ship as a pre-built index; the pipeline itself is general-purpose.

## Give Claude eyes

The renderer also ships as a Claude Code plugin — the **pixelbrowse** skill. Instead of fetching
raw HTML, Claude screenshots a page with `pixelshot` and _reads the image_, so it sees
charts, diagrams, tables, and layout the way a person does.

Install it — no clone needed. Install the `pixelshot` CLI so it's on your `PATH`
(use `uv tool` or `pipx` to keep it isolated yet always available to Claude — a
plain `pip install` into a project venv may leave `pixelshot` off `PATH`):

```bash
uv tool install pixelrag                            # pixelshot on PATH (or: pipx install pixelrag)
claude plugin marketplace add StarTrail-org/PixelRAG
claude plugin install pixelbrowse@pixelrag-plugins
```

Then just ask Claude to look at a page:

```bash
claude -p "screenshot https://news.ycombinator.com and summarize the top stories"
claude -p "screenshot https://arxiv.org/abs/2404.12387 and explain the key findings"
```

Or use the slash command in an interactive session: `/screenshot https://example.com`.
No MCP server, no backend: the skill just calls `pixelshot` (Playwright/CDP) on your machine.

## How it works



Text-based RAG parses the page to text chunks and **loses the table** — the reader can't find the
answer. PixelRAG renders the page to **screenshot tiles**, retrieves the right tile, and the reader
reads the number straight off the image.

Two pieces make this work: (1) rendering documents to images instead of parsing them to text, and
(2) a `Qwen3-VL-Embedding` model, LoRA-fine-tuned on screenshot data, that embeds page images into
a space where visual content is retrievable.

## Pipelines

Capture is the standalone `pixelshot` command; the rest of the pipeline runs through the
`pixelrag` umbrella — `pixelrag <stage>`. Install only the stages you need:

| Command                                    | What it does                                                    | Install                         |
| ------------------------------------------ | --------------------------------------------------------------- | ------------------------------- |
| `pixelshot`                                | Document → image tiles (Playwright CDP, PDF)                    | `pip install pixelrag`          |
| `pixelrag chunk` · `embed` · `build-index` | Tiles → vectors → FAISS index                                   | `pip install 'pixelrag[embed]'` |
| `pixelrag index`                           | Orchestrates the full pipeline: source → ingest → embed → index | `pip install 'pixelrag[index]'` |
| `pixelrag serve`                           | FAISS search API (FastAPI, CPU or GPU)                          | `pip install 'pixelrag[serve]'` |

```
render ←── index ──→ embed       serve (independent)       train → serve (HTTP)
```

**`train` is a separate uv project** with its own pinned env (`torch==2.9.1+cu129`,
`transformers==4.57.1`, cuDNN 9.20) — install it from inside `train/`, not from the root.

### Search a pre-built index

```bash
pip install 'pixelrag[serve]'

# Download a pre-built index from Hugging Face. The dataset repo holds four FAISS indexes
# (base/LoRA Wikipedia pixel, Wikipedia text, news pixel); grab just the base one (~217G) here.
huggingface-cli download StarTrail-org/pixelrag-faiss-indexes \
  --repo-type dataset --include "search_index_normed_v2/*" --local-dir ./index

# Serve, then query
pixelrag serve --index-dir ./index/search_index_normed_v2 --port 30001

curl -X POST http://localhost:30001/search \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
```

### Build an index from your own documents

Works on **Linux (CUDA)** and **macOS (Apple Silicon / MPS)** — `device: auto` picks the best backend.

```bash
pip install 'pixelrag[index]'

# Create pixelrag.yaml
cat > pixelrag.yaml << 'EOF'
source:
  type: local
  path: ./my_docs

embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: auto          # cuda on Linux, mps on macOS, cpu as fallback

output: ./my_index
EOF

# Build, then serve
pixelrag index build
pixelrag serve --index-dir ./my_index --port 30001
```


**Try it: index a PDF and search it locally**

No GPU required — runs on macOS (Apple Silicon) or any machine with Python 3.10+.

```bash
pip install 'pixelrag[index]'

# 1. Grab a sample PDF (or use your own)
curl -L -o paper.pdf https://raw.githubusercontent.com/StarTrail-org/PixelRAG/main/assets/pixelrag-paper.pdf

# 2. Create config (device: auto picks MPS on Mac, CUDA on Linux)
cat > pixelrag.yaml << 'EOF'
source:
  type: local
  path: ./paper.pdf

embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: auto

output: ./paper_index
EOF

# 3. Build the index (~3 min on Apple M-series, ~1 min on GPU)
pixelrag index build

# 4. Serve it
pixelrag serve --index-dir ./paper_index --port 30001

# 5. Search — should return page 2 (the overview diagram)
curl -X POST http://localhost:30001/search \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"text": "Overview of PixelRAG and the diagram"}], "n_docs": 1}'
```



### Render a page programmatically

```python
from pixelrag_render import render_url

# render a single page to tiles — e.g. for an agent to read
tiles = render_url("https://en.wikipedia.org/wiki/Python", "./tiles")
```

The same rendering is available as a CLI — `pixelshot` ships with `pip install pixelrag`:

```bash
# Web page → tiles (headless Chromium via CDP)
pixelshot https://en.wikipedia.org/wiki/Python -o ./tiles

# PDF → tiles (requires poppler; install the pdf extra: pip install 'pixelrag[pdf]')
curl -sL -o paper.pdf https://arxiv.org/pdf/2503.09516
pixelshot paper.pdf -o ./tiles --dpi 200

# URLs and local files can be mixed freely
pixelshot https://github.com/StarTrail-org/PixelRAG paper.pdf -o ./tiles
```

> **Chrome on Windows/macOS** — the bundled turbo `headless_shell` auto-installs on
> **linux-x64** only. Elsewhere, `pixelshot` uses your system Chrome/Chromium (or
> Playwright's Chromium), auto-detected from the standard install locations. Point it at
> a specific binary with `CHROME_PATH=/path/to/chrome` if it isn't found automatically.
> Each render runs in an isolated, throwaway Chrome profile, so it works even while you
> have Chrome open.

### Embed tools (standalone)

Each stage runs independently, without the orchestrator:

```bash
pip install 'pixelrag[embed]'

pixelrag chunk --tiles-dir ./tiles
pixelrag embed --shard-dir ./tiles --output-dir ./embeddings --gpu-ids 0,1
pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index
```

### Qdrant backend

[Qdrant](https://qdrant.tech) is an open-source vector search engine for high-performance and massive scale. [FAISS](https://ai.meta.com/tools/faiss/) remains the default for local indexes. Use Qdrant for configurable quantization, disk-backed vectors, payload filtering, and one collection shared by multiple PixelRAG servers.

Quantization compresses vectors to reduce memory use and speed up search, with a recall tradeoff that depends on the method and settings.

To configure quantization, pass any Qdrant `quantization_config` object in a JSON file.
See Qdrant's [quantization guide](https://qdrant.tech/documentation/manage-data/quantization/#setting-up-quantization-in-qdrant) for supported methods and parameters.

```json
{
  "scalar": {
    "type": "int8",
    "quantile": 0.99,
    "always_ram": true
  }
}
```

```bash
pip install 'pixelrag[serve,qdrant]'   # or 'pixelrag[index,qdrant]'

# Build against a Qdrant server.
# Start one locally with: docker run -p 6333:6333 qdrant/qdrant
pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index \
    --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag \
    --qdrant-quantization-config ./quantization.json

# Add documents to an existing collection.
pixelrag build-index --embeddings-dir ./more --output-dir ./index \
    --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag --append

# Replace an existing collection and its configuration.
pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index \
    --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag --recreate

# Serve the collection. PixelRAG reads the backend from summary.json.
pixelrag serve --index-dir ./index --qdrant-url http://localhost:6333 \
    --qdrant-client-config ./qdrant-client.json --port 30001
```

Configure the orchestrator in `pixelrag.yaml`:

```yaml
index:
  backend: qdrant
  qdrant_url: http://localhost:6333
  collection: pixelrag
  client_config: ./qdrant-client.json
  quantization_config: ./quantization.json
  # Set append: true or recreate: true when the collection already exists.
```

### Training

Fine-tuning lives in `train/` — a **separate uv project** (`wiki-screenshot-training`) with its own
pinned env. It LoRA-fine-tunes `Qwen/Qwen3-VL-Embedding-2B` for webpage retrieval; run it from
inside `train/` (`cd train && uv sync`). See [`train/README.md`](train/README.md) for the full recipe.

You don't need to retrain to use the model — the trained adapters are published at
[`Chrisyichuan/wiki-screenshot-embedding-lora`](https://huggingface.co/Chrisyichuan/wiki-screenshot-embedding-lora/tree/main/lora_vit/ckpt200).

We also release the full training set
([`Chrisyichuan/screenshot-training-natural-filtered-v2`](https://huggingface.co/datasets/Chrisyichuan/screenshot-training-natural-filtered-v2)),
so you can adapt other backbones yourself — a larger Qwen, or any other embedding model.
The data curation pipeline (LLM-augmented query generation, filtering, hard-negative mining)
is documented in [`train/docs/synthetic_data_pipeline.md`](train/docs/synthetic_data_pipeline.md).

## Citation

If you find PixelRAG useful, please cite our paper:

```bibtex
@misc{wang2026pixelragwebscreenshotsbeat,
      title={PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation}, 
      author={Yichuan Wang and Zhifei Li and Zirui Wang and Paul Teiletche and Lesheng Jin and Matei Zaharia and Joseph E. Gonzalez and Sewon Min},
      year={2026},
      eprint={2606.28344},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2606.28344}, 
}
```

## Acknowledgments

Thanks to [Rulin Shao](https://rulinshao.github.io/) for support.

Thanks also to [Claude Code](https://github.com/anthropics/claude-code) and
[OpenAI Codex](https://github.com/openai/codex) for supporting open-source contributors with credits and plans,
which we earned by working on [LEANN](https://github.com/StarTrail-org/LEANN).

This work is done by the [Berkeley Sky Computing Lab](https://sky.cs.berkeley.edu/),
[BAIR](https://bair.berkeley.edu/), and the [Berkeley NLP Group](https://nlp.cs.berkeley.edu/).

## License

Apache-2.0

---

## File: chromium/README.md

# Chromium Screenshot Patches

Custom Chromium patches for high-throughput headless screenshot capture.
Adds three CDP parameters to `Page.captureScreenshot` and a helper method.

## Patches

**`screenshot-patches.diff`** — 222 lines, 5 files:

| Feature | Description |
|---------|-------------|
| `rawFilePath` | Write raw BGRA pixels to a file path (async ThreadPool). Bypasses PNG/JPEG encoding. |
| `directClip` | Capture clip region directly via `CopyFromSurface(src_rect)` without modifying viewport/emulation state. |
| `skipRedraw` | Lightweight `ForceRedrawWithCallback` → `CopyFromSurface`. Skips the full `GetSnapshotFromBrowser` presentation feedback wait. |

## Usage (CDP)

```javascript
// rawFilePath — write raw BGRA to /dev/shm (28MB for 875×8192)
await cdp("Page.captureScreenshot", {
    rawFilePath: "/dev/shm/tile.raw",
    fromSurface: true,
    optimizeForSpeed: true,
    clip: { x: 0, y: 0, width: 875, height: 8192, scale: 1 }
});

// directClip — capture region without emulation change
await cdp("Page.captureScreenshot", {
    directClip: true,
    clip: { x: 0, y: 1024, width: 875, height: 1024, scale: 1 }
});

// skipRedraw — ForceRedrawWithCallback then CopyFromSurface
await cdp("Page.captureScreenshot", {
    skipRedraw: true,
    rawFilePath: "/dev/shm/tile.raw",
    clip: { x: 0, y: 0, width: 875, height: 8192, scale: 1 }
});
```

## Raw file format

12-byte header + pixel data:
```
offset 0:  uint32 width
offset 4:  uint32 height
offset 8:  uint32 rowBytes (= width × 4)
offset 12: BGRA pixel data (rowBytes × height bytes)
```

Read in Python:
```python
import struct
from PIL import Image

data = open("tile.raw", "rb").read()
w, h, rb = struct.unpack_from(" out/Release/args.gn << 'EOF'
is_debug = false
is_official_build = true
is_component_build = false
symbol_level = 0
blink_symbol_level = 0
chrome_pgo_phase = 0
EOF

gn gen out/Release

# 4. Build
autoninja -C out/Release chrome
# ~20 min on 224 cores, ~2 hours on 16 cores
```

Build output: `out/Release/chrome` (~476MB)

## Compatibility

- Chromium 150.x (May 2026). May apply cleanly to nearby versions.
- `is_official_build=true` required for competitive performance (10x vs debug).
- All patches are in `content/browser/devtools/protocol/` (CDP layer) and
  `content/browser/renderer_host/` (widget host). No rendering engine changes.

---

## File: demos/e2e/README.md

# End-to-End Demo: Wikipedia → Search

Builds a visual search index from Wikipedia articles and queries it.

## Quick Start

```bash
cd pixelrag
uv run python demos/e2e/run.py
```

This will:
1. Start kiwix-serve with Simple English Wikipedia
2. Capture 100 article screenshots (~20s)
3. Chunk tiles into 1024px strips (~1s)
4. Embed chunks with Qwen3-VL on CPU (~5 min for 100 articles)
5. Build a FAISS index
6. Start a search API and run sample queries

## Prerequisites

- Simple English Wikipedia ZIM at `~/pixelrag-data/zim/wikipedia_en_simple.zim`
  (download: `curl -L https://download.kiwix.org/zim/wikipedia/wikipedia_en_simple_all_nopic_2026-05.zim -o ~/pixelrag-data/zim/wikipedia_en_simple.zim`)
- kiwix-serve binary at `.local/bin/kiwix-serve`

## Configuration

Edit `pixelrag.yaml` in this directory to change:
- Number of articles (`limit`)
- Embedding device (`cpu` or `cuda`)
- Output location

---

## File: demos/render/README.md

# Ingest Demo: Heterogeneous Documents

Captures screenshots from a mix of URLs, PDFs, and HTML files in one call.

## Run

```bash
cd pixelrag
uv run python demos/render/run.py
```

## What it does

1. Fetches 3 Wikipedia articles (URLs)
2. Renders 2 local HTML files (created on the fly)
3. Produces tiled JPEG screenshots for all 5
4. Shows a summary of tiles, sizes, and timing

---

## File: deploy/README.md

# Deploy

How PixelRAG runs in production. (Host-specific details — the actual machine, secrets, live
state — are intentionally **not** in this public repo; they live in an out-of-repo file the
root `CLAUDE.md` imports on the deploy host.)

## Topology
- **Frontend** (`web/`) — deployed on Vercel from `main`, automatically.
- **Search API** (`serve/`) — `pixelrag serve` behind nginx, run **blue-green** (two slots).
- **Chat agent** (`web/agent-server.mjs`) — a Node service that calls the search API.

## CD — self-hosted runner (egress-only)
Continuous deployment uses a **self-hosted GitHub Actions runner** on the deploy box. The runner
dials *out* to GitHub (no inbound access, no SSH keys or hostnames in the repo/secrets), so the
machine stays private and works behind a firewall.

`.github/workflows/deploy.yml` triggers **only** on push to `main` (post-merge = trusted code) and
manual dispatch — never `pull_request` — so fork PRs can't run on the runner. All CI stays on
GitHub-hosted runners.

On each deploy, `deploy/deploy.sh` fast-forwards the checkout and restarts **only what changed**:
- `uv.lock` → `uv sync`
- `web/agent-server.mjs` → restart the agent (cheap)
- `serve/**` → flagged only (a search-index reload is expensive — use blue-green instead)

It refuses to run unless the checkout is on a clean `main` (the deploy box doubles as a dev
machine, so it must never clobber in-progress work).

## Blue-green search API
Two slots — **blue** and **green** — each an independent `pixelrag serve` on its own port, fronted
by an nginx `upstream` (`pixelrag-api-upstream.conf` → `/etc/nginx/conf.d/`). To roll out a new
index or model with zero downtime:

1. Bring up the idle slot with the new config (`pixelrag-api-green.service` is the green slot).
2. `deploy/api-switch.sh ` — health- and smoke-checks the target, flips the nginx upstream
   with a graceful reload (no dropped connections), and repoints + restarts the agent.
3. **Rollback** = `deploy/api-switch.sh <other-port>`.

This is preferred over restarting a slot in place, which reloads the (large) FAISS index and would
mean minutes of downtime.

## Files
- `deploy.sh` — CD restart logic (invoked by the Deploy workflow)
- `api-switch.sh` — blue-green cutover + rollback
- `pixelrag-api.service`, `pixelrag-api-green.service` — the two search-API slots
- `pixelrag-agent.service` — the chat agent
- `pixelrag-api-upstream.conf` — nginx upstream the switch script rewrites