SurfSense Documentation

Open-source NotebookLM alternative. Research the open web with live data(Reddit, YT, IG, TikTok, Indeed, Google Search, Maps etc) through one platform, API...

15,651 stars Python Python 1 file · ~8,248 tokens #agent#agents#ai#fastapi#langchain#langgraph#nextjs#notebooklm
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy SurfSense on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name surfsense -p 5432:5432 --restart always pgvector/pgvector:pg17
Configured Services & Ports
⚙️ db
image: pgvector/pgvector:pg17
🔌 Ports: ${POSTGRES_PORT:-5432}:5432
⚙️ pgadmin
image: dpage/pgadmin4
🔌 Ports: ${PGADMIN_PORT:-5050}:80
⚙️ redis
image: redis:8-alpine
🔌 Ports: ${REDIS_PORT:-6379}:6379
⚙️ searxng
image: searxng/searxng:2026.3.13-3c1f68c59
🔌 Ports: ${SEARXNG_PORT:-8888}:8080
⚙️ zero-cache
image: rocicorp/zero:1.6.0
🔌 Ports: ${ZERO_CACHE_PORT:-4848}:4848
⚙️ opensandbox-server
image: opensandbox/server:v0.2.2
🔌 Ports: ${OPENSANDBOX_PORT:-8080}:8080
⚙️ azurite
image: mcr.microsoft.com/azure-storage/azurite:3.33.0
🔌 Ports: ${AZURITE_BLOB_PORT:-10000}:10000
docker-compose.yml
yaml (docker-compose.yml) 
# =============================================================================
# SurfSense — Dependencies only (no backend / frontend / Celery images)
# =============================================================================
# Postgres, Redis, pgAdmin, Zero — run API + Next + Celery on the host.
# Celery is not Dockerized here: use `uv run` from surfsense_backend/ (no extra
# backend image build just for workers).
#
# From repo root (SurfSense/):
#   docker compose -f docker/docker-compose.deps-only.yml up -d
#
# Compose variable substitution uses `docker/.env` (copy from .env.example).
# Bind mounts use ./postgresql.conf in this directory.
#
# Local Celery (from surfsense_backend/, after Redis is up):
#   uv run celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues=surfsense,surfsense.connectors
#   uv run celery -A celery_worker.celery_app beat --loglevel=info
#
# Host setup:
#   - Backend .env: DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5432/surfsense
#   - Backend .env: CELERY_BROKER_URL / REDIS_APP_URL → redis://localhost:6379/0
#   - Web .env: NEXT_PUBLIC_ZERO_CACHE_URL=http://localhost:${ZERO_CACHE_PORT:-4848}
#
# IMPORTANT — schema migrations:
#   This compose file does NOT build the backend image and therefore cannot
#   run a `migrations` service. You MUST run alembic on the host before
#   bringing zero-cache up, or zero-cache will crash-loop with
#   `Unknown or invalid publications. Specified: [zero_publication]`.
#
#   First-time / after-pull:
#     cd surfsense_backend && uv run alembic upgrade head
#
#   The other compose files (docker-compose.yml, docker-compose.dev.yml)
#   handle this automatically via a dedicated `migrations` service.
# =============================================================================

name: surfsense-deps

services:
  db:
    image: pgvector/pgvector:pg17
    ports:
      - "${POSTGRES_PORT:-5432}:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./postgresql.conf:/etc/postgresql/postgresql.conf:ro
    environment:
      - POSTGRES_USER=${DB_USER:-postgres}
      - POSTGRES_PASSWORD=${DB_PASSWORD:-postgres}
      - POSTGRES_DB=${DB_NAME:-surfsense}
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-postgres} -d ${DB_NAME:-surfsense}"]
      interval: 10s
      timeout: 5s
      retries: 5

  pgadmin:
    image: dpage/pgadmin4
    ports:
      - "${PGADMIN_PORT:-5050}:80"
    environment:
      - PGADMIN_DEFAULT_EMAIL=${PGADMIN_DEFAULT_EMAIL:[email protected]}
      - PGADMIN_DEFAULT_PASSWORD=${PGADMIN_DEFAULT_PASSWORD:-surfsense}
    volumes:
      - pgadmin_data:/var/lib/pgadmin
    depends_on:
      - db

  redis:
    image: redis:8-alpine
    ports:
      - "${REDIS_PORT:-6379}:6379"
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  # Google Search scraper fallback. The host-run backend reaches it with
  # SEARXNG_URL=http://localhost:8888.
  searxng:
    image: searxng/searxng:2026.3.13-3c1f68c59
    ports:
      - "${SEARXNG_PORT:-8888}:8080"
    volumes:
      - ./searxng:/etc/searxng
    environment:
      - SEARXNG_SECRET=${SEARXNG_SECRET:-surfsense-searxng-secret}
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/healthz"]
      interval: 10s
      timeout: 5s
      retries: 5

  # NOTE: zero-cache requires the `zero_publication` Postgres publication to
  # exist before it starts. In this deps-only stack there is no backend
  # container to run migrations, so you must run `uv run alembic upgrade head`
  # from `surfsense_backend/` on the host BEFORE `docker compose up -d`.
  zero-cache:
    image: rocicorp/zero:1.6.0
    ports:
      - "${ZERO_CACHE_PORT:-4848}:4848"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      db:
        condition: service_healthy
    environment:
      - ZERO_UPSTREAM_DB=postgresql://${DB_USER:-postgres}:${DB_PASSWORD:-postgres}@db:5432/${DB_NAME:-surfsense}?sslmode=${DB_SSLMODE:-disable}
      - ZERO_CVR_DB=postgresql://${DB_USER:-postgres}:${DB_PASSWORD:-postgres}@db:5432/${DB_NAME:-surfsense}?sslmode=${DB_SSLMODE:-disable}
      - ZERO_CHANGE_DB=postgresql://${DB_USER:-postgres}:${DB_PASSWORD:-postgres}@db:5432/${DB_NAME:-surfsense}?sslmode=${DB_SSLMODE:-disable}
      - ZERO_REPLICA_FILE=/data/zero.db
      - ZERO_ADMIN_PASSWORD=${ZERO_ADMIN_PASSWORD:-surfsense-zero-admin}
      - ZERO_APP_PUBLICATIONS=${ZERO_APP_PUBLICATIONS:-zero_publication}
      - ZERO_AUTO_RESET=${ZERO_AUTO_RESET:-true}
      - ZERO_NUM_SYNC_WORKERS=${ZERO_NUM_SYNC_WORKERS:-4}
      - ZERO_UPSTREAM_MAX_CONNS=${ZERO_UPSTREAM_MAX_CONNS:-20}
      - ZERO_CVR_MAX_CONNS=${ZERO_CVR_MAX_CONNS:-30}
      - ZERO_QUERY_URL=${ZERO_QUERY_URL:-http://host.docker.internal:3000/api/zero/query}
      - ZERO_MUTATE_URL=${ZERO_MUTATE_URL:-http://host.docker.internal:3000/api/zero/mutate}
      - ZERO_QUERY_FORWARD_COOKIES=${ZERO_QUERY_FORWARD_COOKIES:-true}
      - ZERO_QUERY_API_KEY=${ZERO_QUERY_API_KEY:-}
      - ZERO_AUTH_REVALIDATE_INTERVAL_SECONDS=${ZERO_AUTH_REVALIDATE_INTERVAL_SECONDS:-60}
      - ZERO_AUTH_RETRANSFORM_INTERVAL_SECONDS=${ZERO_AUTH_RETRANSFORM_INTERVAL_SECONDS:-60}
    volumes:
      - zero_cache_data:/data
    restart: unless-stopped
    stop_grace_period: 300s
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4848/keepalive"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 600s

  # Sandbox control plane for code execution / artifact generation. It spawns
  # sandbox containers on the host daemon as siblings, so it needs the socket.
  # The host-run backend reaches it with OPENSANDBOX_DOMAIN=localhost:8080.
  opensandbox-server:
    image: opensandbox/server:v0.2.2
    ports:
      - "${OPENSANDBOX_PORT:-8080}:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./opensandbox/sandbox.toml:/etc/opensandbox/config.toml:ro
      - opensandbox_data:/data
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      - OPENSANDBOX_SERVER_API_KEY=${OPENSANDBOX_API_KEY:-surfsense-dev-sandbox}
    mem_limit: ${OPENSANDBOX_SERVER_MEMORY_LIMIT:-512m}
    restart: unless-stopped
    # The runtime image ships neither curl nor wget; python is on PATH.
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=3).status == 200 else 1)"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 15s

  # OPTIONAL — Azurite emulates Azure Blob Storage for testing the Azure
  # original-file backend. The default filesystem backend needs none of this.
  # To exercise it, set in surfsense_backend/.env:
  #   FILE_STORAGE_BACKEND=azure
  #   AZURE_STORAGE_CONTAINER=surfsense-documents
  #   AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:${AZURITE_BLOB_PORT:-10000}/devstoreaccount1;
  # The backend creates blobs on upload; create the container once first
  # (Azure CLI / Storage Explorer), then upload a document.
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite:3.33.0
    command: azurite-blob --blobHost 0.0.0.0 --blobPort 10000
    ports:
      - "${AZURITE_BLOB_PORT:-10000}:10000"
    volumes:
      - azurite_data:/data
    restart: unless-stopped

volumes:
  postgres_data:
    name: surfsense-deps-postgres
  pgadmin_data:
    name: surfsense-deps-pgadmin
  redis_data:
    name: surfsense-deps-redis
  zero_cache_data:
    name: surfsense-deps-zero-cache
  azurite_data:
    name: surfsense-deps-azurite
  opensandbox_data:
    name: surfsense-deps-opensandbox

File: README.md

SurfSense: The Open-Source NotebookLM Alternative for Open Web Research

SurfSense is the open-source NotebookLM alternative for AI agents, an open web research platform with live data connectors. Your agents research the live web with structured data from Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, Indeed, and any page on the open web, through one REST API or MCP server. Scheduled and event-triggered agents turn what they find into briefs and alerts, and a built-in knowledge base keeps every finding searchable with citations.

[!NOTE]
📢 A note for our NotebookLM-alternative users

For the past couple of months we built SurfSense as the best general research agent for your own knowledge, and that chapter earned us a community we are genuinely proud of. Agentic tools like Claude, OpenCode, Hermes, and OpenClaw have now proven that agents are the future, and reasoning over a static index is becoming something every capable agent does out of the box. What agents still lack is live data from the places where answers actually live, and the workflows around it. That is where we are pointing all of our energy: giving agents the primitives to research the open web.

Nothing you rely on is going away. Your knowledge base, chat with citations, reports, podcasts, presentations, automations, and collaborative chats all keep working, and self-hosting stays free and open source. Read the full announcement on our changelog.

Table of contents

Why agents need SurfSense

Ask any capable agent "what is Reddit saying about this product since launch?" or "what do the reviews of these ten places actually complain about?" and it has nowhere trustworthy to look. Official platform APIs are rate-limited, priced for enterprises, or missing entirely; scraping plumbing is brittle; and driving a browser with an LLM burns minutes and tokens per page. SurfSense gives agents the primitives instead:

  • One typed surface for wherever the data lives. Every connector is a REST endpoint returning structured JSON — posts, comments, transcripts, reviews, SERPs, pages. No rate-limit roulette, no HTML parsing, no browser loop.
  • An MCP server that exposes every connector as a native tool (surfsense_reddit_scrape, surfsense_google_search, and more) to Claude, Cursor, or any agent framework.
  • An agent harness, not just raw data: retries, structured output, and credit metering are built in, so agents go from a question to a cited brief without you building the plumbing.
  • Open source and self-hostable, so your research stays on your own infrastructure.

Live data connectors

Connector What your agents get Learn more
Reddit Posts, comments, and subreddit streams without the official API's rate limits Reddit Scraper API
YouTube Videos, transcripts, and comment threads at scale YouTube Scraper API
Instagram Public profiles, posts, and reels without the Graph API Instagram Scraper API
TikTok Videos, comments, hashtags, and profiles without Research API approval TikTok Scraper API
Google Maps Places, ratings, and reviews for local business research Google Maps Scraper API
Google Search Live SERPs for search research and monitoring Google Search API
Indeed Public job postings with salaries and full descriptions, by search or company Indeed Scraper API
Amazon Public product data: prices, ratings, offers, sellers, and best-seller ranks Amazon Product API
Walmart Public product data plus the full review history: prices, ratings, sellers, and variants Walmart Product & Review API
Web Crawl Any page on the open web as clean, structured content Web Crawling API
External MCP Connectors Bring any MCP server to your agents, with one-click OAuth for Notion, Slack, Jira, and more External MCP Connectors

The connector catalog is growing beyond social platforms and search; every new source lands as a typed endpoint on the same API and MCP server.

Billing is pay as you go: connectors bill per item actually returned, crawls per page successfully fetched, and failed calls are never billed. Self-hosted installs run with billing off. See pricing.

Quick start

Call a connector from code

Every connector is a REST endpoint you can call from any language with your SurfSense API key:

bash
curl -X POST "$SURFSENSE_API_URL/workspaces/$WORKSPACE_ID/scrapers/reddit/scrape" \
  -H "Authorization: Bearer $SURFSENSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "search_queries": ["your brand"],
    "community": "SaaS",
    "sort": "top",
    "time_filter": "week"
  }'

Each connector page has copy-paste examples in Python, JavaScript, Go, PHP, Ruby, Java, and C#.

Give the tools to your agents over MCP

Add the SurfSense MCP server to Claude, Cursor, or your own agent framework:

json
{
  "mcpServers": {
    "surfsense": {
      "url": "https://mcp.surfsense.com/mcp",
      "headers": { "Authorization": "Bearer ${SURFSENSE_API_KEY}" }
    }
  }
}

Your agent can now call every connector as a native tool. See the SurfSense MCP server page for the full tool list, or run the server locally from surfsense_mcp.

Use the cloud

Go to surfsense.com, log in, and ask the agent for live web data in plain English. New accounts start with $5 of free credit and no subscription.

Self-host for free

Run the entire platform, connectors, agents, automations, and the MCP server on your own infrastructure. Self-hosted installs ship with billing off, so scraping, crawling, and agent runs are limited only by your hardware and the model keys you bring.

Prerequisites: Docker Desktop must be installed and running.

For Linux/macOS:

bash
curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash

For Windows:

bash
irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex

The install script sets up Watchtower automatically for daily auto-updates. To skip it, add the --no-watchtower flag. For Docker Compose, manual installation, and other deployment options, see the docs.

Everything else in the box

The research workspace that made SurfSense the leading open-source NotebookLM alternative is still here, and everything your agents gather lands in it.

Knowledge base

  • Upload PDFs, Office docs, images, and audio, or sync Google Drive, OneDrive, and Dropbox. 50+ file formats supported.
  • Hybrid semantic and full-text search with cited, Perplexity-style answers.
  • AI file sorting auto-organizes documents by source, date, and topic.

Deliverable studio

  • Downloadable AI artifacts for reports, resumes, documents, spreadsheets, and presentations.
  • Two-host AI podcasts from any document or folder in under 20 seconds.
  • Editable slide decks, narrated video overviews, and AI image generation.

Automations

  • Run full agent turns on a schedule or in response to events, described in plain English, with results written back to Notion, Slack, Linear, and Jira.

Team collaboration

  • Real-time collaborative AI chats with comments and mentions.
  • RBAC with Owner, Admin, Editor, and Viewer roles.

Desktop app

Native AI assistance in every application on your computer. Download from the latest release.

  • General Assist: launch SurfSense from any app with a global shortcut.
  • Quick Assist: select text anywhere, then ask AI to explain, rewrite, or act on it.
  • Screenshot Assist: capture any region of your screen and ask AI about it.
  • Watch Local Folder: auto-sync a local folder to your knowledge base. Point it at your Obsidian vault to keep your notes searchable.

No vendor lock-in

  • 100+ LLMs via the OpenAI spec and LiteLLM, including GPT-5.5, Claude Sonnet 5, and Gemini 3.1 Pro.
  • 6,000+ embedding models and all major rerankers.
  • Full local and private LLM support (vLLM, Ollama), so your data stays yours.

Video Agent Sample

https://github.com/user-attachments/assets/012a7ffa-6f76-4f06-9dda-7632b470057a

Podcast Agent Sample

https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7

How to collaborate in real time (Beta)

  1. Go to the Manage Members page and create an invite.

  2. A teammate joins and that workspace becomes shared.

  3. Make a chat shared and work in it together in real time, with comments to tag teammates.

How SurfSense compares

SurfSense is the only open-source product that combines a NotebookLM-style research workspace for people with live-data primitives for agents. Here is how that stacks up against each class of tool.

vs browser agents (Browserbase, Browser Use). Browser agents drive a real browser with an LLM in the loop — the right tool when a task needs clicking, logging in, or filling forms. But most research is read-only retrieval, and for retrieval the LLM-in-a-browser loop costs you minutes and thousands of tokens per page. A SurfSense connector call is one HTTP request: seconds, deterministic, and zero tokens spent deciding where to click.

vs scraping APIs (Firecrawl). Scraping APIs are great at turning a generic page into markdown, but a markdown blob still leaves your agent parsing structure out of prose, and they degrade on bot-protected platforms like Reddit, TikTok, and Instagram. SurfSense connectors return platform-native structured items — posts, comments, transcripts, reviews — and bill only for items actually returned; failed calls are never billed.

vs search APIs (Exa, Tavily, Parallel). Search APIs answer from a web index, which is the right tool for "find me pages about X." They cannot pull a Reddit thread's comments, TikTok reactions, YouTube transcripts, or Google Maps reviews — the places where the answer often actually lives.

vs scraper marketplaces (Apify). Marketplaces give you thousands of community actors, each with its own schema, quality, and pricing. SurfSense is one typed API and one MCP server with an agent harness and a research workspace behind it, and it is open source.

SurfSense vs Google NotebookLM

Still comparing us as a NotebookLM alternative? Here is the honest breakdown.

Feature Google NotebookLM SurfSense
Live web data for agents No Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, Indeed, and web crawl connectors via REST API and MCP
MCP server No Every connector exposed as a native agent tool, plus bring-your-own MCP servers with one-click OAuth apps
Sources per Notebook 50 (Free) to 600 (Ultra, $249.99/mo) Unlimited
Number of Notebooks 100 (Free) to 500 (paid tiers) Unlimited
Source Size Limit 500,000 words / 200MB per source No limit
Pricing Free tier; Pro $19.99/mo, Ultra $249.99/mo Free and open source to self-host; cloud is pay as you go with $5 free credit
LLM Support Google Gemini only 100+ LLMs via OpenAI spec & LiteLLM
Embedding Models Google only 6,000+ embedding models, all major rerankers
Local / Private LLMs Not available Full support (vLLM, Ollama), your data stays yours
Self Hostable No Yes, Docker one-liner or full Docker Compose
Open Source No Yes
Knowledge Base Sources Google Drive, YouTube, websites File uploads, Google Drive, OneDrive, Dropbox, local folder sync, and crawled pages
File Format Support PDFs, Docs, Slides, Sheets, CSV, Word, EPUB, images, web URLs, YouTube 50+ formats: documents, images, videos via LlamaCloud, Unstructured, or Docling (local)
Search Semantic search Hybrid semantic + full-text with hierarchical indices & reciprocal rank fusion
Cited Answers Yes Yes, Perplexity-style cited responses
Agentic Architecture No Yes, powered by LangChain Deep Agents with planning, subagents, and file system access
AI Automations & Agents No Scheduled workflows, event triggers, and chat-built no-code automations with write-back to Notion, Slack, Linear & Jira
Real-Time Multiplayer Shared notebooks with Viewer/Editor roles (no real-time chat) RBAC with Owner / Admin / Editor / Viewer roles, real-time chat & comment threads
Video Generation Cinematic Video Overviews via Veo 3 (Ultra only) Available (NotebookLM is better here, actively improving)
Presentation Generation Better looking slides but not editable Editable, slide-based presentations
Podcast Generation Audio Overviews with customizable hosts and languages Available with multiple TTS providers (NotebookLM is better here, actively improving)
Desktop App No Native app with General Assist, Quick Assist, Screenshot Assist, and local folder sync

Feature requests and future

SurfSense is actively being developed. While it's not yet production-ready, you can help us speed up the process.

Join the SurfSense Discord and help shape the future of SurfSense!

Roadmap

Stay up to date with our development progress and upcoming features. Check out our public roadmap and contribute your ideas or feedback:

Roadmap Discussion: SurfSense 2026 Roadmap

Kanban Board: SurfSense Project Board

Contribute

All contributions welcome, from stars and bug reports to backend improvements. See CONTRIBUTING.md to get started.

Thanks to all our Surfers:

Star History

File: surfsense_browser_extension/README.md

SurfSense Cross Browser Extension

Use this guide to build for your browser https://docs.plasmo.com/framework/workflows/build

This is a Plasmo extension project bootstrapped with plasmo init.

Getting Started

First, run the development server:

bash
pnpm dev
# or
npm run dev

Open your browser and load the appropriate development build. For example, if you are developing for the chrome browser, using manifest v3, use: build/chrome-mv3-dev.

You can start editing the popup by modifying popup.tsx. It should auto-update as you make changes. To add an options page, simply add a options.tsx file to the root of the project, with a react component default exported. Likewise to add a content page, add a content.ts file to the root of the project, importing some module and do some logic, then reload the extension on your browser.

For further guidance, visit our Documentation

Making production build

Run the following:

bash
pnpm build
# or
npm run build

This should create a production bundle for your extension, ready to be zipped and published to the stores.

Submit to the webstores

The easiest way to deploy your Plasmo extension is to use the built-in bpp GitHub action. Prior to using this action however, make sure to build your extension and upload the first version to the store to establish the basic credentials. Then, simply follow this setup instruction and you should be on your way for automated submission!

File: surfsense_web/content/docs/index.mdx

title: Documentation icon: BookOpen

import { Card, Cards } from 'fumadocs-ui/components/card';
import { Download, Container, Wrench, Cable, BookOpen, FlaskConical, Heart, MessageCircle, Cpu } from 'lucide-react';

Welcome to SurfSense's Documentation! Here, you'll find everything you need to get the most out of SurfSense.

Getting Started

The fastest way to run SurfSense is the Docker installation, one command and you're up. If you want full control over each component (or want to contribute), use the manual installation.

} title="Docker Installation" description="One command, all dependencies pre-configured (recommended)" href="/docs/docker-installation" /> } title="Manual Installation" description="Set up SurfSense from source, component by component" href="/docs/manual-installation" /> } title="Connectors" description="Built-in scraper APIs plus Notion, Slack, Google, Jira, and more" href="/docs/connectors" /> } title="Local Models" description="Connect local model servers like Ollama and LM Studio" href="/docs/local-models" /> } title="Messaging Channels" description="Chat with SurfSense from Telegram, WhatsApp, Slack, and Discord" href="/docs/messaging-channels" /> } title="Testing" description="Running and writing tests for SurfSense" href="/docs/testing" /> } title="Code of Conduct" description="Community guidelines and expectations" href="/docs/code-of-conduct" />

File: surfsense_web/content/docs/installation.mdx

title: Installation icon: Download

import { Card, Cards } from 'fumadocs-ui/components/card';
import { Container, Wrench } from 'lucide-react';

SurfSense runs as a Python backend + Next.js frontend. You can either spin it up with Docker or set everything up yourself.

Not sure which to pick? Docker gets you up and running fastest. Go manual if you want to build SurfSense with us or need full control over the stack.

} title="Docker Installation" description="All dependencies pre-configured, up and running in minutes" href="/docs/docker-installation" /> } title="Manual Installation" description="Set up each component yourself for full control" href="/docs/manual-installation" />

File: surfsense_web/content/docs/manual-installation.mdx

title: Manual Installation icon: Wrench

This guide sets up SurfSense without Docker (except for zero-cache, which is simplest to run as a container). Choose this path if you want to contribute to SurfSense or need full control over each component. If you just want to run SurfSense, the Docker installation is much faster.

What You'll Need

  • Python 3.12+ — backend runtime
  • Node.js 20+ and pnpm — frontend runtime
  • PostgreSQL 14+ with the pgvector extension — database
  • Redis — message broker for background tasks
  • Docker — to run zero-cache (the real-time sync server)
  • Git — to clone the repository
  • uv — Python package manager (install instructions)

Clone the repository first:

bash
git clone https://github.com/MODSetter/SurfSense.git
cd SurfSense

Decisions to Make Up Front

Authentication. SurfSense supports local email/password auth (the default, no setup needed) or Google OAuth login. For Google login you need an OAuth client from the Google Cloud Console — the Google connectors guide walks through creating one.

Document parsing (ETL). SurfSense converts uploaded files with one of three services:

  • Docling (default) — runs locally, no API key, privacy-friendly. Supports PDF, Office docs, images, HTML, CSV.
  • Unstructured — needs an API key from Unstructured Platform. Supports 34+ formats.
  • LlamaCloud — needs an API key from LlamaCloud. Supports 50+ formats.

You only need one. Docling is the easiest way to start.

Backend Setup

1. Configure the Environment

Copy the example environment file:

Linux/macOS:

bash
cd surfsense_backend
cp .env.example .env

Windows (PowerShell):

powershell
cd surfsense_backend
Copy-Item -Path .env.example -Destination .env

.env.example is the source of truth for configuration — every variable is documented inline with comments and sensible defaults. At minimum, set your PostgreSQL connection string and a JWT secret key (generate one with openssl rand -base64 32). Everything else — auth type, ETL service, embeddings, TTS/STT, connector credentials — is optional and explained in the file itself.

For separate embedding servers, set EMBEDDING_BASE_URL with Chonkie/LiteLLM embedding models; OLLAMA_EMBEDDING_BASE_URL is also supported as an Ollama-specific fallback.

2. Install Dependencies

bash
# From surfsense_backend/
uv sync

3. Configure PostgreSQL for Zero Sync

SurfSense uses Rocicorp Zero for real-time updates (notifications, document status, chat comments, indexing progress). Zero replicates data from PostgreSQL via logical replication, which requires a one-time PostgreSQL configuration change.

Edit your postgresql.conf (typical locations: /etc/postgresql/<version>/main/postgresql.conf on Linux, /usr/local/var/postgres/postgresql.conf on macOS via Homebrew, C:\Program Files\PostgreSQL\<version>\data\postgresql.conf on Windows) and set:

ini
wal_level = logical
max_replication_slots = 10
max_wal_senders = 10

Then restart PostgreSQL:

Linux:

bash
sudo systemctl restart postgresql

macOS (Homebrew):

bash
brew services restart postgresql

Windows (PowerShell, replace 17 with your major version):

powershell
Restart-Service postgresql-x64-17

Verify the change:

bash
psql -U postgres -d surfsense -c "SHOW wal_level;"
# Should return: logical

Managed databases (RDS, Supabase, Cloud SQL, etc.): enable logical replication via your provider's parameter group (e.g. rds.logical_replication=1 on RDS) and grant your database user the REPLICATION privilege:

sql
ALTER USER surfsense WITH REPLICATION;
GRANT CREATE ON DATABASE surfsense TO surfsense;

4. Run Database Migrations

This creates the schema and the zero_publication that zero-cache needs to start. Skipping this step will cause zero-cache to crash-loop with Unknown or invalid publications. Specified: [zero_publication].

bash
# From surfsense_backend/
uv run alembic upgrade head

Verify the publication was created:

bash
psql -U postgres -d surfsense -c "SELECT pubname FROM pg_publication;"
# Should include: zero_publication

5. Start Redis

Linux:

bash
sudo systemctl start redis
# or run directly
redis-server

macOS (Homebrew):

bash
brew services start redis

Windows — run Redis in Docker (easiest):

powershell
docker run -d --name redis -p 6379:6379 redis:latest

Verify it's running:

bash
redis-cli ping
# Should return: PONG

6. Start the Celery Worker

In a new terminal, start the worker that handles background tasks (document indexing, connector syncs):

Linux/macOS:

bash
cd surfsense_backend
DEFAULT_Q="${CELERY_TASK_DEFAULT_QUEUE:-surfsense}"
uv run celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues="${DEFAULT_Q},${DEFAULT_Q}.connectors,${DEFAULT_Q}.gateway"

Windows (PowerShell):

powershell
cd surfsense_backend
uv run celery -A celery_worker.celery_app worker --loglevel=info --concurrency=1 --pool=solo --queues="surfsense,surfsense.connectors,surfsense.gateway"

Optionally, run Flower in another terminal to monitor tasks at http://localhost:5555:

bash
uv run celery -A celery_worker.celery_app flower --port=5555

7. Start Celery Beat (Scheduler)

In another terminal, start the scheduler that triggers periodic tasks (like scheduled connector syncs). Without it, scheduled tasks won't run.

bash
cd surfsense_backend
uv run celery -A celery_worker.celery_app beat --loglevel=info

8. Run the Backend

bash
# From surfsense_backend/
uv run main.py

# Or with hot reloading for development
uv run main.py --reload

You should see the server running on http://localhost:8000.

Zero-Cache Setup

zero-cache is the Rocicorp Zero server that sits between PostgreSQL and the browser. It streams real-time updates to all connected clients via WebSocket. Without it, the UI sits on stale data. For an overview of how Zero works, see the Real-Time Sync with Zero guide.

1. Run Zero-Cache via Docker

Linux/macOS:

bash
docker run -d --name surfsense-zero-cache \
  -p 4848:4848 \
  --add-host=host.docker.internal:host-gateway \
  -e ZERO_UPSTREAM_DB="postgresql://postgres:[email protected]:5432/surfsense?sslmode=disable" \
  -e ZERO_CVR_DB="postgresql://postgres:[email protected]:5432/surfsense?sslmode=disable" \
  -e ZERO_CHANGE_DB="postgresql://postgres:[email protected]:5432/surfsense?sslmode=disable" \
  -e ZERO_REPLICA_FILE=/data/zero.db \
  -e ZERO_ADMIN_PASSWORD=surfsense-zero-admin \
  -e ZERO_APP_PUBLICATIONS=zero_publication \
  -e ZERO_NUM_SYNC_WORKERS=4 \
  -e ZERO_UPSTREAM_MAX_CONNS=20 \
  -e ZERO_CVR_MAX_CONNS=30 \
  -e ZERO_QUERY_URL="http://host.docker.internal:3000/api/zero/query" \
  -e ZERO_MUTATE_URL="http://host.docker.internal:3000/api/zero/mutate" \
  -e ZERO_QUERY_FORWARD_COOKIES=true \
  -v surfsense-zero-cache:/data \
  rocicorp/zero:1.6.0

Windows (PowerShell):

powershell
docker run -d --name surfsense-zero-cache `
  -p 4848:4848 `
  --add-host=host.docker.internal:host-gateway `
  -e ZERO_UPSTREAM_DB="postgresql://postgres:[email protected]:5432/surfsense?sslmode=disable" `
  -e ZERO_CVR_DB="postgresql://postgres:[email protected]:5432/surfsense?sslmode=disable" `
  -e ZERO_CHANGE_DB="postgresql://postgres:[email protected]:5432/surfsense?sslmode=disable" `
  -e ZERO_REPLICA_FILE=/data/zero.db `
  -e ZERO_ADMIN_PASSWORD=surfsense-zero-admin `
  -e ZERO_APP_PUBLICATIONS=zero_publication `
  -e ZERO_NUM_SYNC_WORKERS=4 `
  -e ZERO_UPSTREAM_MAX_CONNS=20 `
  -e ZERO_CVR_MAX_CONNS=30 `
  -e ZERO_QUERY_URL="http://host.docker.internal:3000/api/zero/query" `
  -e ZERO_MUTATE_URL="http://host.docker.internal:3000/api/zero/mutate" `
  -e ZERO_QUERY_FORWARD_COOKIES=true `
  -v surfsense-zero-cache:/data `
  rocicorp/zero:1.6.0

Adjustments for your setup:

  • Replace postgres:postgres in the connection URLs with your actual database user and password.
  • On Linux without Docker Desktop, host.docker.internal may not resolve. Either keep the --add-host=host.docker.internal:host-gateway flag (Docker 20.10+) or replace host.docker.internal with your host's IP / --network=host + localhost.
  • For production / custom domains, set ZERO_QUERY_URL and ZERO_MUTATE_URL to your public frontend URL (e.g. https://app.yourdomain.com/api/zero/query).

2. Verify Zero-Cache

bash
curl http://localhost:4848/keepalive
# Should return HTTP 200

# Tail logs to confirm initial replication completed without errors
docker logs -f surfsense-zero-cache

Alternative: Let Docker Manage All Dependencies

If you'd rather have Docker manage Postgres, Redis, and zero-cache together (while still running the backend and frontend natively), the repository ships a deps-only compose file. Run alembic migrations on the host first so zero_publication exists before zero-cache starts:

bash
cd surfsense_backend
uv run alembic upgrade head
cd ../docker
docker compose -f docker-compose.deps-only.yml up -d

The deps-only stack exposes zero-cache on port 4848. Point the frontend at it by setting the zero-cache URL in surfsense_web/.env (see the comments in surfsense_web/.env.example).

Frontend Setup

1. Configure the Environment

Linux/macOS:

bash
cd surfsense_web
cp .env.example .env

Windows (PowerShell):

powershell
cd surfsense_web
Copy-Item -Path .env.example -Destination .env

As with the backend, .env.example documents every option inline. Make sure the auth type and ETL service match what you configured for the backend, and that the backend URL points at http://localhost:8000.

2. Install Dependencies and Run

bash
# Install pnpm if you don't have it
npm install pnpm

pnpm install
pnpm run dev

The frontend should now be running at http://localhost:3000.

Browser Extension (Optional)

The SurfSense browser extension saves any webpage — including those behind authentication — straight into your knowledge base.

bash
cd surfsense_browser_extension
cp .env.example .env   # set the backend URL, documented inline

pnpm install
pnpm build                     # Chrome (default)
pnpm build --target=firefox    # or Firefox
pnpm build --target=edge       # or Edge

Load the built extension in your browser's developer mode and configure it with your SurfSense API key. See the Plasmo build docs for details.

Verify Your Installation

  1. Open http://localhost:3000 and sign in.
  2. Create a workspace and upload a document.
  3. Watch the upload status update live without refreshing — this confirms zero-cache is wired up correctly.
  4. Chat with your uploaded content.

Troubleshooting

  • Database connection issues: Verify PostgreSQL is running and pgvector is installed.
  • Redis connection issues: redis-cli ping should return PONG. Check the Redis URL in your backend .env.
  • Celery worker issues: Make sure the worker is running in a separate terminal and check its logs.
  • File upload failures: Validate your ETL service API key, or use Docling which needs none.
  • Real-time updates not working / stale UI: Verify zero-cache is running (curl http://localhost:4848/keepalive returns 200). Open browser DevTools → Console and look for WebSocket errors. Confirm the zero-cache URL in surfsense_web/.env matches the running zero-cache address.
  • Zero-cache stuck on Unknown or invalid publications. Specified: [zero_publication]: You skipped uv run alembic upgrade head. Run it from surfsense_backend/, then docker restart surfsense-zero-cache.
  • Zero-cache crashes with _zero.tableMetadata errors: A previous run left a half-built SQLite replica behind. Start fresh: docker rm -f surfsense-zero-cache && docker volume rm surfsense-zero-cache, then re-run the command from Zero-Cache Setup.
  • wal_level is not logical: zero-cache requires logical replication. Set it in postgresql.conf, restart PostgreSQL, and verify with SHOW wal_level;.
  • Backend /ready returns 503: The readiness probe verifies zero_publication exists. Run uv run alembic upgrade head to create it.

Next Steps

  • Set up connectors to bring in your tools and services.
  • Connect local models like Ollama or LM Studio.
  • For production, put a reverse proxy in front, add SSL, and set up database backups.

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#agent Browse agent catalog → #agents Browse agents catalog → #ai Browse ai catalog → #fastapi Browse fastapi catalog → #langchain Browse langchain catalog → #langgraph Browse langgraph catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →