PixelRAG Documentation

File: README.md

bash

pip install pixelrag

The two core operations — render a page to screenshots, search a visual index:

bash

# Render any page or document to screenshot tiles
pixelshot https://en.wikipedia.org/wiki/Python --output ./tiles

# Search a hosted index of 8.28M Wikipedia pages — no setup, runs against the live API
curl -X POST https://api.pixelrag.ai/search \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'

Live, hosted endpoint — https://api.pixelrag.ai serves a
pre-built index of 8.28M Wikipedia pages. No setup, no API key. It even takes an image as the query
(visual search) — see the API reference →.

Or try it in the browser at pixelrag.ai, or run the demo notebook in
Colab — it
renders a page and searches the hosted index, with the images inline.

What it is

PixelRAG renders documents — web pages, PDFs, images — as screenshots and retrieves over the
images directly. Visual structure that HTML parsing throws away — tables, charts, layout,
infographics — stays intact, so the reader model can actually answer questions about it.
Wikipedia's 8.28M articles ship as a pre-built index; the pipeline itself is general-purpose.

Give Claude eyes

The renderer also ships as a Claude Code plugin — the pixelbrowse skill. Instead of fetching
raw HTML, Claude screenshots a page with pixelshot and reads the image, so it sees
charts, diagrams, tables, and layout the way a person does.

Install it — no clone needed. Install the pixelshot CLI so it's on your PATH
(use uv tool or pipx to keep it isolated yet always available to Claude — a
plain pip install into a project venv may leave pixelshot off PATH):

bash

uv tool install pixelrag                            # pixelshot on PATH (or: pipx install pixelrag)
claude plugin marketplace add StarTrail-org/PixelRAG
claude plugin install pixelbrowse@pixelrag-plugins

Then just ask Claude to look at a page:

bash

claude -p "screenshot https://news.ycombinator.com and summarize the top stories"
claude -p "screenshot https://arxiv.org/abs/2404.12387 and explain the key findings"

Or use the slash command in an interactive session: /screenshot https://example.com.
No MCP server, no backend: the skill just calls pixelshot (Playwright/CDP) on your machine.

How it works

Text-based RAG parses the page to text chunks and loses the table — the reader can't find the
answer. PixelRAG renders the page to screenshot tiles, retrieves the right tile, and the reader
reads the number straight off the image.

Two pieces make this work: (1) rendering documents to images instead of parsing them to text, and
(2) a Qwen3-VL-Embedding model, LoRA-fine-tuned on screenshot data, that embeds page images into
a space where visual content is retrievable.

Pipelines

Capture is the standalone pixelshot command; the rest of the pipeline runs through the
pixelrag umbrella — pixelrag <stage>. Install only the stages you need:

Command	What it does	Install
`pixelshot`	Document → image tiles (Playwright CDP, PDF)	`pip install pixelrag`
`pixelrag chunk` · `embed` · `build-index`	Tiles → vectors → FAISS index	`pip install 'pixelrag[embed]'`
`pixelrag index`	Orchestrates the full pipeline: source → ingest → embed → index	`pip install 'pixelrag[index]'`
`pixelrag serve`	FAISS search API (FastAPI, CPU or GPU)	`pip install 'pixelrag[serve]'`

text

render ←── index ──→ embed       serve (independent)       train → serve (HTTP)

train is a separate uv project with its own pinned env (torch==2.9.1+cu129,
transformers==4.57.1, cuDNN 9.20) — install it from inside train/, not from the root.

Search a pre-built index

bash

pip install 'pixelrag[serve]'

# Download a pre-built index from Hugging Face. The dataset repo holds four FAISS indexes
# (base/LoRA Wikipedia pixel, Wikipedia text, news pixel); grab just the base one (~217G) here.
huggingface-cli download StarTrail-org/pixelrag-faiss-indexes \
  --repo-type dataset --include "search_index_normed_v2/*" --local-dir ./index

# Serve, then query
pixelrag serve --index-dir ./index/search_index_normed_v2 --port 30001

curl -X POST http://localhost:30001/search \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'

Build an index from your own documents

Works on Linux (CUDA) and macOS (Apple Silicon / MPS) — device: auto picks the best backend.

bash

pip install 'pixelrag[index]'

# Create pixelrag.yaml
cat > pixelrag.yaml << 'EOF'
source:
  type: local
  path: ./my_docs

embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: auto          # cuda on Linux, mps on macOS, cpu as fallback

output: ./my_index
EOF

# Build, then serve
pixelrag index build
pixelrag serve --index-dir ./my_index --port 30001

Try it: index a PDF and search it locally

No GPU required — runs on macOS (Apple Silicon) or any machine with Python 3.10+.

bash

pip install 'pixelrag[index]'

# 1. Grab a sample PDF (or use your own)
curl -L -o paper.pdf https://raw.githubusercontent.com/StarTrail-org/PixelRAG/main/assets/pixelrag-paper.pdf

# 2. Create config (device: auto picks MPS on Mac, CUDA on Linux)
cat > pixelrag.yaml << 'EOF'
source:
  type: local
  path: ./paper.pdf

embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: auto

output: ./paper_index
EOF

# 3. Build the index (~3 min on Apple M-series, ~1 min on GPU)
pixelrag index build

# 4. Serve it
pixelrag serve --index-dir ./paper_index --port 30001

# 5. Search — should return page 2 (the overview diagram)
curl -X POST http://localhost:30001/search \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"text": "Overview of PixelRAG and the diagram"}], "n_docs": 1}'

Render a page programmatically

python

from pixelrag_render import render_url

# render a single page to tiles — e.g. for an agent to read
tiles = render_url("https://en.wikipedia.org/wiki/Python", "./tiles")

The same rendering is available as a CLI — pixelshot ships with pip install pixelrag:

bash

# Web page → tiles (headless Chromium via CDP)
pixelshot https://en.wikipedia.org/wiki/Python -o ./tiles

# PDF → tiles (requires poppler; install the pdf extra: pip install 'pixelrag[pdf]')
curl -sL -o paper.pdf https://arxiv.org/pdf/2503.09516
pixelshot paper.pdf -o ./tiles --dpi 200

# URLs and local files can be mixed freely
pixelshot https://github.com/StarTrail-org/PixelRAG paper.pdf -o ./tiles

Chrome on Windows/macOS — the bundled turbo headless_shell auto-installs on
linux-x64 only. Elsewhere, pixelshot uses your system Chrome/Chromium (or
Playwright's Chromium), auto-detected from the standard install locations. Point it at
a specific binary with CHROME_PATH=/path/to/chrome if it isn't found automatically.
Each render runs in an isolated, throwaway Chrome profile, so it works even while you
have Chrome open.

Embed tools (standalone)

Each stage runs independently, without the orchestrator:

bash

pip install 'pixelrag[embed]'

pixelrag chunk --tiles-dir ./tiles
pixelrag embed --shard-dir ./tiles --output-dir ./embeddings --gpu-ids 0,1
pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index

Qdrant backend

Qdrant is an open-source vector search engine for high-performance and massive scale. FAISS remains the default for local indexes. Use Qdrant for configurable quantization, disk-backed vectors, payload filtering, and one collection shared by multiple PixelRAG servers.

Quantization compresses vectors to reduce memory use and speed up search, with a recall tradeoff that depends on the method and settings.

To configure quantization, pass any Qdrant quantization_config object in a JSON file.
See Qdrant's quantization guide for supported methods and parameters.

json

{
  "scalar": {
    "type": "int8",
    "quantile": 0.99,
    "always_ram": true
  }
}

bash

pip install 'pixelrag[serve,qdrant]'   # or 'pixelrag[index,qdrant]'

# Build against a Qdrant server.
# Start one locally with: docker run -p 6333:6333 qdrant/qdrant
pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index \
    --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag \
    --qdrant-quantization-config ./quantization.json

# Add documents to an existing collection.
pixelrag build-index --embeddings-dir ./more --output-dir ./index \
    --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag --append

# Replace an existing collection and its configuration.
pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index \
    --backend qdrant --qdrant-url http://localhost:6333 --collection pixelrag --recreate

# Serve the collection. PixelRAG reads the backend from summary.json.
pixelrag serve --index-dir ./index --qdrant-url http://localhost:6333 \
    --qdrant-client-config ./qdrant-client.json --port 30001

Configure the orchestrator in pixelrag.yaml:

yaml

index:
  backend: qdrant
  qdrant_url: http://localhost:6333
  collection: pixelrag
  client_config: ./qdrant-client.json
  quantization_config: ./quantization.json
  # Set append: true or recreate: true when the collection already exists.

Training

Fine-tuning lives in train/ — a separate uv project (wiki-screenshot-training) with its own
pinned env. It LoRA-fine-tunes Qwen/Qwen3-VL-Embedding-2B for webpage retrieval; run it from
inside train/ (cd train && uv sync). See train/README.md for the full recipe.

You don't need to retrain to use the model — the trained adapters are published at
Chrisyichuan/wiki-screenshot-embedding-lora.

We also release the full training set
(Chrisyichuan/screenshot-training-natural-filtered-v2),
so you can adapt other backbones yourself — a larger Qwen, or any other embedding model.
The data curation pipeline (LLM-augmented query generation, filtering, hard-negative mining)
is documented in train/docs/synthetic_data_pipeline.md.

Citation

If you find PixelRAG useful, please cite our paper:

bibtex

@misc{wang2026pixelragwebscreenshotsbeat,
      title={PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation}, 
      author={Yichuan Wang and Zhifei Li and Zirui Wang and Paul Teiletche and Lesheng Jin and Matei Zaharia and Joseph E. Gonzalez and Sewon Min},
      year={2026},
      eprint={2606.28344},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2606.28344}, 
}

Acknowledgments

Thanks to Rulin Shao for support.

Thanks also to Claude Code and
OpenAI Codex for supporting open-source contributors with credits and plans,
which we earned by working on LEANN.

This work is done by the Berkeley Sky Computing Lab,
BAIR, and the Berkeley NLP Group.

License

Apache-2.0

File: chromium/README.md

Chromium Screenshot Patches

Custom Chromium patches for high-throughput headless screenshot capture.
Adds three CDP parameters to Page.captureScreenshot and a helper method.

Patches

screenshot-patches.diff — 222 lines, 5 files:

Feature	Description
`rawFilePath`	Write raw BGRA pixels to a file path (async ThreadPool). Bypasses PNG/JPEG encoding.
`directClip`	Capture clip region directly via `CopyFromSurface(src_rect)` without modifying viewport/emulation state.
`skipRedraw`	Lightweight `ForceRedrawWithCallback` → `CopyFromSurface`. Skips the full `GetSnapshotFromBrowser` presentation feedback wait.

Usage (CDP)

javascript

// rawFilePath — write raw BGRA to /dev/shm (28MB for 875×8192)
await cdp("Page.captureScreenshot", {
    rawFilePath: "/dev/shm/tile.raw",
    fromSurface: true,
    optimizeForSpeed: true,
    clip: { x: 0, y: 0, width: 875, height: 8192, scale: 1 }
});

// directClip — capture region without emulation change
await cdp("Page.captureScreenshot", {
    directClip: true,
    clip: { x: 0, y: 1024, width: 875, height: 1024, scale: 1 }
});

// skipRedraw — ForceRedrawWithCallback then CopyFromSurface
await cdp("Page.captureScreenshot", {
    skipRedraw: true,
    rawFilePath: "/dev/shm/tile.raw",
    clip: { x: 0, y: 0, width: 875, height: 8192, scale: 1 }
});

Raw file format

12-byte header + pixel data:

text

offset 0:  uint32 width
offset 4:  uint32 height
offset 8:  uint32 rowBytes (= width × 4)
offset 12: BGRA pixel data (rowBytes × height bytes)

Read in Python:

python

import struct
from PIL import Image

data = open("tile.raw", "rb").read()
w, h, rb = struct.unpack_from(" out/Release/args.gn << 'EOF'
is_debug = false
is_official_build = true
is_component_build = false
symbol_level = 0
blink_symbol_level = 0
chrome_pgo_phase = 0
EOF

gn gen out/Release

# 4. Build
autoninja -C out/Release chrome
# ~20 min on 224 cores, ~2 hours on 16 cores

Build output: out/Release/chrome (~476MB)

Compatibility

Chromium 150.x (May 2026). May apply cleanly to nearby versions.
is_official_build=true required for competitive performance (10x vs debug).
All patches are in content/browser/devtools/protocol/ (CDP layer) and
content/browser/renderer_host/ (widget host). No rendering engine changes.

File: demos/e2e/README.md

End-to-End Demo: Wikipedia → Search

Builds a visual search index from Wikipedia articles and queries it.

Quick Start

bash

cd pixelrag
uv run python demos/e2e/run.py

This will:

Start kiwix-serve with Simple English Wikipedia
Capture 100 article screenshots (~20s)
Chunk tiles into 1024px strips (~1s)
Embed chunks with Qwen3-VL on CPU (~5 min for 100 articles)
Build a FAISS index
Start a search API and run sample queries

Prerequisites

Simple English Wikipedia ZIM at ~/pixelrag-data/zim/wikipedia_en_simple.zim
(download: curl -L https://download.kiwix.org/zim/wikipedia/wikipedia_en_simple_all_nopic_2026-05.zim -o ~/pixelrag-data/zim/wikipedia_en_simple.zim)
kiwix-serve binary at .local/bin/kiwix-serve

Configuration

Edit pixelrag.yaml in this directory to change:

Number of articles (limit)
Embedding device (cpu or cuda)
Output location

File: demos/render/README.md

Ingest Demo: Heterogeneous Documents

Captures screenshots from a mix of URLs, PDFs, and HTML files in one call.

Run

bash

cd pixelrag
uv run python demos/render/run.py

What it does

Fetches 3 Wikipedia articles (URLs)
Renders 2 local HTML files (created on the fly)
Produces tiled JPEG screenshots for all 5
Shows a summary of tiles, sizes, and timing

File: deploy/README.md

Deploy

How PixelRAG runs in production. (Host-specific details — the actual machine, secrets, live
state — are intentionally not in this public repo; they live in an out-of-repo file the
root CLAUDE.md imports on the deploy host.)

Topology

Frontend (web/) — deployed on Vercel from main, automatically.
Search API (serve/) — pixelrag serve behind nginx, run blue-green (two slots).
Chat agent (web/agent-server.mjs) — a Node service that calls the search API.

CD — self-hosted runner (egress-only)

Continuous deployment uses a self-hosted GitHub Actions runner on the deploy box. The runner
dials out to GitHub (no inbound access, no SSH keys or hostnames in the repo/secrets), so the
machine stays private and works behind a firewall.

.github/workflows/deploy.yml triggers only on push to main (post-merge = trusted code) and
manual dispatch — never pull_request — so fork PRs can't run on the runner. All CI stays on
GitHub-hosted runners.

On each deploy, deploy/deploy.sh fast-forwards the checkout and restarts only what changed:

uv.lock → uv sync
web/agent-server.mjs → restart the agent (cheap)
serve/** → flagged only (a search-index reload is expensive — use blue-green instead)

It refuses to run unless the checkout is on a clean main (the deploy box doubles as a dev
machine, so it must never clobber in-progress work).

Blue-green search API

Two slots — blue and green — each an independent pixelrag serve on its own port, fronted
by an nginx upstream (pixelrag-api-upstream.conf → /etc/nginx/conf.d/). To roll out a new
index or model with zero downtime:

Bring up the idle slot with the new config (pixelrag-api-green.service is the green slot).
deploy/api-switch.sh — health- and smoke-checks the target, flips the nginx upstream
with a graceful reload (no dropped connections), and repoints + restarts the agent.
Rollback = deploy/api-switch.sh <other-port>.

This is preferred over restarting a slot in place, which reloads the (large) FAISS index and would
mean minutes of downtime.

Files

deploy.sh — CD restart logic (invoked by the Deploy workflow)
api-switch.sh — blue-green cutover + rollback
pixelrag-api.service, pixelrag-api-green.service — the two search-API slots
pixelrag-agent.service — the chat agent
pixelrag-api-upstream.conf — nginx upstream the switch script rewrites

File: README.md Copy Chapter

What it is Copy Chapter

Give Claude eyes Copy Chapter

How it works Copy Chapter

Pipelines Copy Chapter

Search a pre-built index Copy Section

Build an index from your own documents Copy Section

Render a page programmatically Copy Section

Embed tools (standalone) Copy Section

Qdrant backend Copy Section

Training Copy Section

Citation Copy Chapter

Acknowledgments Copy Chapter

License Copy Chapter

File: chromium/README.md Copy Chapter

Chromium Screenshot Patches

Patches Copy Chapter

Usage (CDP) Copy Chapter

Raw file format Copy Chapter

Compatibility Copy Chapter

File: demos/e2e/README.md Copy Chapter

End-to-End Demo: Wikipedia → Search

Quick Start Copy Chapter

Prerequisites Copy Chapter

Configuration Copy Chapter

File: demos/render/README.md Copy Chapter

Ingest Demo: Heterogeneous Documents

Run Copy Chapter

What it does Copy Chapter

File: deploy/README.md Copy Chapter

Deploy

Topology Copy Chapter

CD — self-hosted runner (egress-only) Copy Chapter

Blue-green search API Copy Chapter

Files Copy Chapter

Explore Related Technologies & Topics

File: README.md

What it is

Give Claude eyes

How it works

Pipelines

Search a pre-built index

Build an index from your own documents

Render a page programmatically

Embed tools (standalone)

Qdrant backend

Training

Citation

Acknowledgments

License

File: chromium/README.md

Patches

Usage (CDP)

Raw file format

Compatibility

File: demos/e2e/README.md

Quick Start

Prerequisites

Configuration

File: demos/render/README.md

Run

What it does

File: deploy/README.md

Topology

CD — self-hosted runner (egress-only)

Blue-green search API

Files