## File: README.md [](https://www.surfsense.com/) # SurfSense: The Open-Source NotebookLM Alternative for Open Web Research SurfSense is the **open-source NotebookLM alternative for AI agents**, an open web research platform with live data connectors. Your agents research the live web with structured data from **Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, Indeed, and any page on the open web**, through one **REST API** or **MCP server**. Scheduled and event-triggered agents turn what they find into briefs and alerts, and a built-in knowledge base keeps every finding searchable with citations. > [!NOTE] > **📢 A note for our NotebookLM-alternative users** > > For the past couple of months we built SurfSense as the best general research agent for your own knowledge, and that chapter earned us a community we are genuinely proud of. Agentic tools like Claude, OpenCode, Hermes, and OpenClaw have now proven that agents are the future, and reasoning over a static index is becoming something every capable agent does out of the box. What agents still lack is **live data from the places where answers actually live, and the workflows around it**. That is where we are pointing all of our energy: giving agents the primitives to research the open web. > > **Nothing you rely on is going away.** Your knowledge base, chat with citations, reports, podcasts, presentations, automations, and collaborative chats all keep working, and self-hosting stays free and open source. Read the full announcement on [our changelog](https://www.surfsense.com/changelog). ## Table of contents - [Why agents need SurfSense](#why-agents-need-surfsense) - [Live data connectors](#live-data-connectors) - [Quick start](#quick-start) - [Everything else in the box](#everything-else-in-the-box) - [How SurfSense compares](#how-surfsense-compares) - [Roadmap](#roadmap) - [Contribute](#contribute) ## Why agents need SurfSense Ask any capable agent "what is Reddit saying about this product since launch?" or "what do the reviews of these ten places actually complain about?" and it has nowhere trustworthy to look. Official platform APIs are rate-limited, priced for enterprises, or missing entirely; scraping plumbing is brittle; and driving a browser with an LLM burns minutes and tokens per page. SurfSense gives agents the primitives instead: - **One typed surface for wherever the data lives.** Every connector is a REST endpoint returning structured JSON — posts, comments, transcripts, reviews, SERPs, pages. No rate-limit roulette, no HTML parsing, no browser loop. - **An MCP server** that exposes every connector as a native tool (`surfsense_reddit_scrape`, `surfsense_google_search`, and more) to Claude, Cursor, or any agent framework. - **An agent harness**, not just raw data: retries, structured output, and credit metering are built in, so agents go from a question to a cited brief without you building the plumbing. - **Open source and self-hostable**, so your research stays on your own infrastructure. ## Live data connectors | Connector | What your agents get | Learn more | |---|---|---| | **Reddit** | Posts, comments, and subreddit streams without the official API's rate limits | [Reddit Scraper API](https://www.surfsense.com/reddit) | | **YouTube** | Videos, transcripts, and comment threads at scale | [YouTube Scraper API](https://www.surfsense.com/youtube) | | **Instagram** | Public profiles, posts, and reels without the Graph API | [Instagram Scraper API](https://www.surfsense.com/instagram) | | **TikTok** | Videos, comments, hashtags, and profiles without Research API approval | [TikTok Scraper API](https://www.surfsense.com/tiktok) | | **Google Maps** | Places, ratings, and reviews for local business research | [Google Maps Scraper API](https://www.surfsense.com/google-maps) | | **Google Search** | Live SERPs for search research and monitoring | [Google Search API](https://www.surfsense.com/google-search) | | **Indeed** | Public job postings with salaries and full descriptions, by search or company | [Indeed Scraper API](https://www.surfsense.com/indeed) | | **Amazon** | Public product data: prices, ratings, offers, sellers, and best-seller ranks | [Amazon Product API](https://www.surfsense.com/amazon) | | **Walmart** | Public product data plus the full review history: prices, ratings, sellers, and variants | [Walmart Product & Review API](https://www.surfsense.com/walmart) | | **Web Crawl** | Any page on the open web as clean, structured content | [Web Crawling API](https://www.surfsense.com/web-crawl) | | **External MCP Connectors** | Bring any MCP server to your agents, with one-click OAuth for Notion, Slack, Jira, and more | [External MCP Connectors](https://www.surfsense.com/external-mcp-connectors) | The connector catalog is growing beyond social platforms and search; every new source lands as a typed endpoint on the same API and MCP server. Billing is pay as you go: connectors bill per item actually returned, crawls per page successfully fetched, and failed calls are never billed. Self-hosted installs run with billing off. See [pricing](https://www.surfsense.com/pricing). ## Quick start ### Call a connector from code Every connector is a REST endpoint you can call from any language with your SurfSense API key: ```bash curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \ -H "Authorization: Bearer $SURFSENSE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "search_queries": ["your brand"], "community": "SaaS", "sort": "top", "time_filter": "week" }' ``` Each [connector page](https://www.surfsense.com/connectors) has copy-paste examples in Python, JavaScript, Go, PHP, Ruby, Java, and C#. ### Give the tools to your agents over MCP Add the SurfSense MCP server to Claude, Cursor, or your own agent framework: ```json { "mcpServers": { "surfsense": { "url": "https://mcp.surfsense.com/mcp", "headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" } } } } ``` Your agent can now call every connector as a native tool. See the [SurfSense MCP server](https://www.surfsense.com/mcp-server) page for the full tool list, or run the server locally from [`surfsense_mcp`](./surfsense_mcp). ### Use the cloud Go to [surfsense.com](https://www.surfsense.com), log in, and ask the agent for live web data in plain English. New accounts start with $5 of free credit and no subscription. ### Self-host for free Run the entire platform, connectors, agents, automations, and the MCP server on your own infrastructure. Self-hosted installs ship with billing off, so scraping, crawling, and agent runs are limited only by your hardware and the model keys you bring. **Prerequisites:** [Docker Desktop](https://www.docker.com/products/docker-desktop/) must be installed and running. For Linux/macOS: ```bash curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash ``` For Windows: ```bash irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex ``` The install script sets up [Watchtower](https://github.com/nicholas-fedor/watchtower) automatically for daily auto-updates. To skip it, add the `--no-watchtower` flag. For Docker Compose, manual installation, and other deployment options, see the [docs](https://www.surfsense.com/docs/). ## Everything else in the box The research workspace that made SurfSense the leading open-source NotebookLM alternative is still here, and everything your agents gather lands in it. **Knowledge base** - Upload PDFs, Office docs, images, and audio, or sync **Google Drive, OneDrive, and Dropbox**. 50+ file formats supported. - Hybrid semantic and full-text search with cited, Perplexity-style answers. - AI file sorting auto-organizes documents by source, date, and topic. **Deliverable studio** - Downloadable AI artifacts for reports, resumes, documents, spreadsheets, and presentations. - Two-host AI podcasts from any document or folder in under 20 seconds. - Editable slide decks, narrated video overviews, and AI image generation. **Automations** - Run full agent turns on a schedule or in response to events, described in plain English, with results written back to Notion, Slack, Linear, and Jira. **Team collaboration** - Real-time collaborative AI chats with comments and mentions. - RBAC with Owner, Admin, Editor, and Viewer roles. **Desktop app** Native AI assistance in every application on your computer. Download from the [latest release](https://github.com/MODSetter/SurfSense/releases/latest). - **General Assist**: launch SurfSense from any app with a global shortcut. - **Quick Assist**: select text anywhere, then ask AI to explain, rewrite, or act on it. - **Screenshot Assist**: capture any region of your screen and ask AI about it. - **Watch Local Folder**: auto-sync a local folder to your knowledge base. Point it at your Obsidian vault to keep your notes searchable. **No vendor lock-in** - 100+ LLMs via the OpenAI spec and LiteLLM, including GPT-5.5, Claude Sonnet 5, and Gemini 3.1 Pro. - 6,000+ embedding models and all major rerankers. - Full local and private LLM support (vLLM, Ollama), so your data stays yours. ## Video Agent Sample https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a ## Podcast Agent Sample https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7 ## How to collaborate in real time (Beta) 1. Go to the Manage Members page and create an invite. 2. A teammate joins and that workspace becomes shared. 3. Make a chat shared and work in it together in real time, with comments to tag teammates. ## How SurfSense compares SurfSense is the only open-source product that combines a NotebookLM-style research workspace for people with live-data primitives for agents. Here is how that stacks up against each class of tool. **vs browser agents (Browserbase, Browser Use).** Browser agents drive a real browser with an LLM in the loop — the right tool when a task needs clicking, logging in, or filling forms. But most research is read-only retrieval, and for retrieval the LLM-in-a-browser loop costs you minutes and thousands of tokens per page. A SurfSense connector call is one HTTP request: seconds, deterministic, and zero tokens spent deciding where to click. **vs scraping APIs (Firecrawl).** Scraping APIs are great at turning a generic page into markdown, but a markdown blob still leaves your agent parsing structure out of prose, and they degrade on bot-protected platforms like Reddit, TikTok, and Instagram. SurfSense connectors return platform-native structured items — posts, comments, transcripts, reviews — and bill only for items actually returned; failed calls are never billed. **vs search APIs (Exa, Tavily, Parallel).** Search APIs answer from a web index, which is the right tool for "find me pages about X." They cannot pull a Reddit thread's comments, TikTok reactions, YouTube transcripts, or Google Maps reviews — the places where the answer often actually lives. **vs scraper marketplaces (Apify).** Marketplaces give you thousands of community actors, each with its own schema, quality, and pricing. SurfSense is one typed API and one MCP server with an agent harness and a research workspace behind it, and it is open source. ### SurfSense vs Google NotebookLM Still comparing us as a NotebookLM alternative? Here is the honest breakdown. | Feature | Google NotebookLM | SurfSense | |---------|-------------------|-----------| | **Live web data for agents** | No | Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, Indeed, and web crawl connectors via REST API and MCP | | **MCP server** | No | Every connector exposed as a native agent tool, plus bring-your-own MCP servers with one-click OAuth apps | | **Sources per Notebook** | 50 (Free) to 600 (Ultra, $249.99/mo) | Unlimited | | **Number of Notebooks** | 100 (Free) to 500 (paid tiers) | Unlimited | | **Source Size Limit** | 500,000 words / 200MB per source | No limit | | **Pricing** | Free tier; Pro $19.99/mo, Ultra $249.99/mo | Free and open source to self-host; cloud is pay as you go with $5 free credit | | **LLM Support** | Google Gemini only | 100+ LLMs via OpenAI spec & LiteLLM | | **Embedding Models** | Google only | 6,000+ embedding models, all major rerankers | | **Local / Private LLMs** | Not available | Full support (vLLM, Ollama), your data stays yours | | **Self Hostable** | No | Yes, Docker one-liner or full Docker Compose | | **Open Source** | No | Yes | | **Knowledge Base Sources** | Google Drive, YouTube, websites | File uploads, Google Drive, OneDrive, Dropbox, local folder sync, and crawled pages | | **File Format Support** | PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, images, web URLs, YouTube | 50+ formats: documents, images, videos via LlamaCloud, Unstructured, or Docling (local) | | **Search** | Semantic search | Hybrid semantic + full-text with hierarchical indices & reciprocal rank fusion | | **Cited Answers** | Yes | Yes, Perplexity-style cited responses | | **Agentic Architecture** | No | Yes, powered by [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) with planning, subagents, and file system access | | **AI Automations & Agents** | No | Scheduled workflows, event triggers, and chat-built no-code automations with write-back to Notion, Slack, Linear & Jira | | **Real-Time Multiplayer** | Shared notebooks with Viewer/Editor roles (no real-time chat) | RBAC with Owner / Admin / Editor / Viewer roles, real-time chat & comment threads | | **Video Generation** | Cinematic Video Overviews via Veo 3 (Ultra only) | Available (NotebookLM is better here, actively improving) | | **Presentation Generation** | Better looking slides but not editable | Editable, slide-based presentations | | **Podcast Generation** | Audio Overviews with customizable hosts and languages | Available with multiple TTS providers (NotebookLM is better here, actively improving) | | **Desktop App** | No | Native app with General Assist, Quick Assist, Screenshot Assist, and local folder sync | ## Feature requests and future **SurfSense is actively being developed.** While it's not yet production-ready, you can help us speed up the process. Join the [SurfSense Discord](https://discord.gg/ejRNvftDp9) and help shape the future of SurfSense! ## Roadmap Stay up to date with our development progress and upcoming features. Check out our public roadmap and contribute your ideas or feedback: **Roadmap Discussion:** [SurfSense 2026 Roadmap](https://github.com/MODSetter/SurfSense/discussions/565) **Kanban Board:** [SurfSense Project Board](https://github.com/users/MODSetter/projects/3) ## Contribute All contributions welcome, from stars and bug reports to backend improvements. See [CONTRIBUTING.md](CONTRIBUTING.md) to get started. Thanks to all our Surfers: ## Star History --- --- --- --- --- ## File: surfsense_browser_extension/README.md # SurfSense Cross Browser Extension Use this guide to build for your browser https://docs.plasmo.com/framework/workflows/build This is a [Plasmo extension](https://docs.plasmo.com/) project bootstrapped with [`plasmo init`](https://www.npmjs.com/package/plasmo). ## Getting Started First, run the development server: ```bash pnpm dev # or npm run dev ``` Open your browser and load the appropriate development build. For example, if you are developing for the chrome browser, using manifest v3, use: `build/chrome-mv3-dev`. You can start editing the popup by modifying `popup.tsx`. It should auto-update as you make changes. To add an options page, simply add a `options.tsx` file to the root of the project, with a react component default exported. Likewise to add a content page, add a `content.ts` file to the root of the project, importing some module and do some logic, then reload the extension on your browser. For further guidance, [visit our Documentation](https://docs.plasmo.com/) ## Making production build Run the following: ```bash pnpm build # or npm run build ``` This should create a production bundle for your extension, ready to be zipped and published to the stores. ## Submit to the webstores The easiest way to deploy your Plasmo extension is to use the built-in [bpp](https://bpp.browser.market) GitHub action. Prior to using this action however, make sure to build your extension and upload the first version to the store to establish the basic credentials. Then, simply follow [this setup instruction](https://docs.plasmo.com/framework/workflows/submit) and you should be on your way for automated submission! --- ## File: surfsense_web/content/docs/index.mdx --- title: Documentation icon: BookOpen --- import { Card, Cards } from 'fumadocs-ui/components/card'; import { Download, Container, Wrench, Cable, BookOpen, FlaskConical, Heart, MessageCircle, Cpu } from 'lucide-react'; Welcome to **SurfSense's Documentation!** Here, you'll find everything you need to get the most out of SurfSense. ## Getting Started The fastest way to run SurfSense is the [Docker installation](/docs/docker-installation), one command and you're up. If you want full control over each component (or want to contribute), use the [manual installation](/docs/manual-installation). } title="Docker Installation" description="One command, all dependencies pre-configured (recommended)" href="/docs/docker-installation" /> } title="Manual Installation" description="Set up SurfSense from source, component by component" href="/docs/manual-installation" /> } title="Connectors" description="Built-in scraper APIs plus Notion, Slack, Google, Jira, and more" href="/docs/connectors" /> } title="Local Models" description="Connect local model servers like Ollama and LM Studio" href="/docs/local-models" /> } title="Messaging Channels" description="Chat with SurfSense from Telegram, WhatsApp, Slack, and Discord" href="/docs/messaging-channels" /> } title="Testing" description="Running and writing tests for SurfSense" href="/docs/testing" /> } title="Code of Conduct" description="Community guidelines and expectations" href="/docs/code-of-conduct" /> --- ## File: surfsense_web/content/docs/installation.mdx --- title: Installation icon: Download --- import { Card, Cards } from 'fumadocs-ui/components/card'; import { Container, Wrench } from 'lucide-react'; SurfSense runs as a Python backend + Next.js frontend. You can either spin it up with Docker or set everything up yourself. **Not sure which to pick?** Docker gets you up and running fastest. Go manual if you want to build SurfSense with us or need full control over the stack. } title="Docker Installation" description="All dependencies pre-configured, up and running in minutes" href="/docs/docker-installation" /> } title="Manual Installation" description="Set up each component yourself for full control" href="/docs/manual-installation" /> --- ## File: surfsense_web/content/docs/manual-installation.mdx --- title: Manual Installation icon: Wrench --- This guide sets up SurfSense without Docker (except for zero-cache, which is simplest to run as a container). Choose this path if you want to contribute to SurfSense or need full control over each component. If you just want to run SurfSense, the [Docker installation](/docs/docker-installation) is much faster. ## What You'll Need - **Python 3.12+** — backend runtime - **Node.js 20+** and **pnpm** — frontend runtime - **PostgreSQL 14+** with the **pgvector** extension — database - **Redis** — message broker for background tasks - **Docker** — to run zero-cache (the real-time sync server) - **Git** — to clone the repository - **uv** — Python package manager ([install instructions](https://docs.astral.sh/uv/getting-started/installation/)) Clone the repository first: ```bash git clone https://github.com/MODSetter/SurfSense.git cd SurfSense ``` ## Decisions to Make Up Front **Authentication.** SurfSense supports local email/password auth (the default, no setup needed) or Google OAuth login. For Google login you need an OAuth client from the [Google Cloud Console](https://console.cloud.google.com/apis/credentials) — the [Google connectors guide](/docs/connectors/external/google) walks through creating one. **Document parsing (ETL).** SurfSense converts uploaded files with one of three services: - **Docling** (default) — runs locally, no API key, privacy-friendly. Supports PDF, Office docs, images, HTML, CSV. - **Unstructured** — needs an API key from [Unstructured Platform](https://platform.unstructured.io/). Supports 34+ formats. - **LlamaCloud** — needs an API key from [LlamaCloud](https://cloud.llamaindex.ai/). Supports 50+ formats. You only need one. Docling is the easiest way to start. ## Backend Setup ### 1. Configure the Environment Copy the example environment file: **Linux/macOS:** ```bash cd surfsense_backend cp .env.example .env ``` **Windows (PowerShell):** ```powershell cd surfsense_backend Copy-Item -Path .env.example -Destination .env ``` `.env.example` is the source of truth for configuration — every variable is documented inline with comments and sensible defaults. At minimum, set your PostgreSQL connection string and a JWT secret key (generate one with `openssl rand -base64 32`). Everything else — auth type, ETL service, embeddings, TTS/STT, connector credentials — is optional and explained in the file itself. For separate embedding servers, set `EMBEDDING_BASE_URL` with Chonkie/LiteLLM embedding models; `OLLAMA_EMBEDDING_BASE_URL` is also supported as an Ollama-specific fallback. ### 2. Install Dependencies ```bash # From surfsense_backend/ uv sync ``` ### 3. Configure PostgreSQL for Zero Sync SurfSense uses [Rocicorp Zero](https://zero.rocicorp.dev/) for real-time updates (notifications, document status, chat comments, indexing progress). Zero replicates data from PostgreSQL via **logical replication**, which requires a one-time PostgreSQL configuration change. Edit your `postgresql.conf` (typical locations: `/etc/postgresql//main/postgresql.conf` on Linux, `/usr/local/var/postgres/postgresql.conf` on macOS via Homebrew, `C:\Program Files\PostgreSQL\\data\postgresql.conf` on Windows) and set: ```ini wal_level = logical max_replication_slots = 10 max_wal_senders = 10 ``` Then restart PostgreSQL: **Linux:** ```bash sudo systemctl restart postgresql ``` **macOS (Homebrew):** ```bash brew services restart postgresql ``` **Windows (PowerShell, replace `17` with your major version):** ```powershell Restart-Service postgresql-x64-17 ``` Verify the change: ```bash psql -U postgres -d surfsense -c "SHOW wal_level;" # Should return: logical ``` **Managed databases (RDS, Supabase, Cloud SQL, etc.):** enable logical replication via your provider's parameter group (e.g. `rds.logical_replication=1` on RDS) and grant your database user the `REPLICATION` privilege: ```sql ALTER USER surfsense WITH REPLICATION; GRANT CREATE ON DATABASE surfsense TO surfsense; ``` ### 4. Run Database Migrations This creates the schema **and** the `zero_publication` that zero-cache needs to start. Skipping this step will cause zero-cache to crash-loop with `Unknown or invalid publications. Specified: [zero_publication]`. ```bash # From surfsense_backend/ uv run alembic upgrade head ``` Verify the publication was created: ```bash psql -U postgres -d surfsense -c "SELECT pubname FROM pg_publication;" # Should include: zero_publication ``` ### 5. Start Redis **Linux:** ```bash sudo systemctl start redis # or run directly redis-server ``` **macOS (Homebrew):** ```bash brew services start redis ``` **Windows — run Redis in Docker (easiest):** ```powershell docker run -d --name redis -p 6379:6379 redis:latest ``` Verify it's running: ```bash redis-cli ping # Should return: PONG ``` ### 6. Start the Celery Worker In a new terminal, start the worker that handles background tasks (document indexing, connector syncs): **Linux/macOS:** ```bash cd surfsense_backend DEFAULT_Q="${CELERY_TASK_DEFAULT_QUEUE:-surfsense}" uv run celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues="${DEFAULT_Q},${DEFAULT_Q}.connectors,${DEFAULT_Q}.gateway" ``` **Windows (PowerShell):** ```powershell cd surfsense_backend uv run celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues="surfsense,surfsense.connectors,surfsense.gateway" ``` Optionally, run [Flower](https://flower.readthedocs.io/) in another terminal to monitor tasks at [http://localhost:5555](http://localhost:5555): ```bash uv run celery -A celery_worker.celery_app flower --port=5555 ``` ### 7. Start Celery Beat (Scheduler) In another terminal, start the scheduler that triggers periodic tasks (like scheduled connector syncs). Without it, scheduled tasks won't run. ```bash cd surfsense_backend uv run celery -A celery_worker.celery_app beat --loglevel=info ``` ### 8. Run the Backend ```bash # From surfsense_backend/ uv run main.py # Or with hot reloading for development uv run main.py --reload ``` You should see the server running on `http://localhost:8000`. ## Zero-Cache Setup **zero-cache** is the Rocicorp Zero server that sits between PostgreSQL and the browser. It streams real-time updates to all connected clients via WebSocket. Without it, the UI sits on stale data. For an overview of how Zero works, see the [Real-Time Sync with Zero](/docs/how-to/zero-sync) guide. ### 1. Run Zero-Cache via Docker **Linux/macOS:** ```bash docker run -d --name surfsense-zero-cache \ -p 4848:4848 \ --add-host=host.docker.internal:host-gateway \ -e ZERO_UPSTREAM_DB="postgresql://postgres:postgres@host.docker.internal:5432/surfsense?sslmode=disable" \ -e ZERO_CVR_DB="postgresql://postgres:postgres@host.docker.internal:5432/surfsense?sslmode=disable" \ -e ZERO_CHANGE_DB="postgresql://postgres:postgres@host.docker.internal:5432/surfsense?sslmode=disable" \ -e ZERO_REPLICA_FILE=/data/zero.db \ -e ZERO_ADMIN_PASSWORD=surfsense-zero-admin \ -e ZERO_APP_PUBLICATIONS=zero_publication \ -e ZERO_NUM_SYNC_WORKERS=4 \ -e ZERO_UPSTREAM_MAX_CONNS=20 \ -e ZERO_CVR_MAX_CONNS=30 \ -e ZERO_QUERY_URL="http://host.docker.internal:3000/api/zero/query" \ -e ZERO_MUTATE_URL="http://host.docker.internal:3000/api/zero/mutate" \ -e ZERO_QUERY_FORWARD_COOKIES=true \ -v surfsense-zero-cache:/data \ rocicorp/zero:1.6.0 ``` **Windows (PowerShell):** ```powershell docker run -d --name surfsense-zero-cache ` -p 4848:4848 ` --add-host=host.docker.internal:host-gateway ` -e ZERO_UPSTREAM_DB="postgresql://postgres:postgres@host.docker.internal:5432/surfsense?sslmode=disable" ` -e ZERO_CVR_DB="postgresql://postgres:postgres@host.docker.internal:5432/surfsense?sslmode=disable" ` -e ZERO_CHANGE_DB="postgresql://postgres:postgres@host.docker.internal:5432/surfsense?sslmode=disable" ` -e ZERO_REPLICA_FILE=/data/zero.db ` -e ZERO_ADMIN_PASSWORD=surfsense-zero-admin ` -e ZERO_APP_PUBLICATIONS=zero_publication ` -e ZERO_NUM_SYNC_WORKERS=4 ` -e ZERO_UPSTREAM_MAX_CONNS=20 ` -e ZERO_CVR_MAX_CONNS=30 ` -e ZERO_QUERY_URL="http://host.docker.internal:3000/api/zero/query" ` -e ZERO_MUTATE_URL="http://host.docker.internal:3000/api/zero/mutate" ` -e ZERO_QUERY_FORWARD_COOKIES=true ` -v surfsense-zero-cache:/data ` rocicorp/zero:1.6.0 ``` **Adjustments for your setup:** - Replace `postgres:postgres` in the connection URLs with your actual database user and password. - On Linux without Docker Desktop, `host.docker.internal` may not resolve. Either keep the `--add-host=host.docker.internal:host-gateway` flag (Docker 20.10+) or replace `host.docker.internal` with your host's IP / `--network=host` + `localhost`. - For production / custom domains, set `ZERO_QUERY_URL` and `ZERO_MUTATE_URL` to your public frontend URL (e.g. `https://app.yourdomain.com/api/zero/query`). ### 2. Verify Zero-Cache ```bash curl http://localhost:4848/keepalive # Should return HTTP 200 # Tail logs to confirm initial replication completed without errors docker logs -f surfsense-zero-cache ``` ### Alternative: Let Docker Manage All Dependencies If you'd rather have Docker manage Postgres, Redis, and zero-cache together (while still running the backend and frontend natively), the repository ships a deps-only compose file. **Run alembic migrations on the host first** so `zero_publication` exists before zero-cache starts: ```bash cd surfsense_backend uv run alembic upgrade head cd ../docker docker compose -f docker-compose.deps-only.yml up -d ``` The deps-only stack exposes zero-cache on port `4848`. Point the frontend at it by setting the zero-cache URL in `surfsense_web/.env` (see the comments in `surfsense_web/.env.example`). ## Frontend Setup ### 1. Configure the Environment **Linux/macOS:** ```bash cd surfsense_web cp .env.example .env ``` **Windows (PowerShell):** ```powershell cd surfsense_web Copy-Item -Path .env.example -Destination .env ``` As with the backend, `.env.example` documents every option inline. Make sure the auth type and ETL service match what you configured for the backend, and that the backend URL points at `http://localhost:8000`. ### 2. Install Dependencies and Run ```bash # Install pnpm if you don't have it npm install pnpm pnpm install pnpm run dev ``` The frontend should now be running at [http://localhost:3000](http://localhost:3000). ## Browser Extension (Optional) The SurfSense browser extension saves any webpage — including those behind authentication — straight into your knowledge base. ```bash cd surfsense_browser_extension cp .env.example .env # set the backend URL, documented inline pnpm install pnpm build # Chrome (default) pnpm build --target=firefox # or Firefox pnpm build --target=edge # or Edge ``` Load the built extension in your browser's developer mode and configure it with your SurfSense API key. See the [Plasmo build docs](https://docs.plasmo.com/framework/workflows/build#with-a-specific-target) for details. ## Verify Your Installation 1. Open [http://localhost:3000](http://localhost:3000) and sign in. 2. Create a workspace and upload a document. 3. Watch the upload status update live without refreshing — this confirms zero-cache is wired up correctly. 4. Chat with your uploaded content. ## Troubleshooting - **Database connection issues**: Verify PostgreSQL is running and pgvector is installed. - **Redis connection issues**: `redis-cli ping` should return `PONG`. Check the Redis URL in your backend `.env`. - **Celery worker issues**: Make sure the worker is running in a separate terminal and check its logs. - **File upload failures**: Validate your ETL service API key, or use Docling which needs none. - **Real-time updates not working / stale UI**: Verify zero-cache is running (`curl http://localhost:4848/keepalive` returns 200). Open browser DevTools → Console and look for WebSocket errors. Confirm the zero-cache URL in `surfsense_web/.env` matches the running zero-cache address. - **Zero-cache stuck on `Unknown or invalid publications. Specified: [zero_publication]`**: You skipped `uv run alembic upgrade head`. Run it from `surfsense_backend/`, then `docker restart surfsense-zero-cache`. - **Zero-cache crashes with `_zero.tableMetadata` errors**: A previous run left a half-built SQLite replica behind. Start fresh: `docker rm -f surfsense-zero-cache && docker volume rm surfsense-zero-cache`, then re-run the command from [Zero-Cache Setup](#zero-cache-setup). - **`wal_level` is not `logical`**: zero-cache requires logical replication. Set it in `postgresql.conf`, restart PostgreSQL, and verify with `SHOW wal_level;`. - **Backend `/ready` returns 503**: The readiness probe verifies `zero_publication` exists. Run `uv run alembic upgrade head` to create it. ## Next Steps - Set up [connectors](/docs/connectors) to bring in your tools and services. - Connect [local models](/docs/local-models) like Ollama or LM Studio. - For production, put a reverse proxy in front, add SSL, and set up database backups.