Repository: ArchiveBox/ArchiveBox

Stars: 27248

CLAUDE.md

Claude Code Development Guide for ArchiveBox

Quick Start

bash

Set up dev environment (always use uv, never pip directly)

uv sync --dev --all-extras
Run tests as non-root user (required - ArchiveBox always refuses to run as root)

sudo -u testuser bash -c 'source .venv/bin/activate && python -m pytest archivebox/tests/ -v'

Development Environment Setup

Prerequisites

- Python 3.11+ (3.13 recommended)
- uv package manager
- A non-root user for running tests (e.g., testuser)

Install Dependencies

bash

uv sync --dev --all-extras  # Always use uv, never pip directly

Activate Virtual Environment

bash

source .venv/bin/activate

Common Gotchas

#### File Permissions
New files created by root need permissions fixed for testuser:

bash

chmod 644 archivebox/tests/test_*.py

#### DATA_DIR Environment Variable
ArchiveBox commands must run inside a data directory. Tests use temp directories - the run_archivebox() helper sets DATA_DIR automatically.

Code Style Guidelines

Naming Conventions for Grep-ability

Use consistent naming for everything to enable easy grep-ability and logical grouping:

Principle: Fewest unique names. If you must create a new unique name, make it grep and group well.

Examples:

python

Filesystem migration methods - all start with fs_

def fs_migration_needed() -> bool: ...
def fs_migrate() -> None: ...
def _fs_migrate_from_0_7_0_to_0_8_0() -> None: ...
def _fs_migrate_from_0_8_0_to_0_9_0() -> None: ...
def _fs_next_version(current: str) -> str: ...
Logging methods - ALL must start with log_ or _log

def log_migration_start(snapshot_id: str) -> None: ...
def _log_error(message: str) -> None: ...
def log_validation_result(ok: bool, msg: str) -> None: ...

Rules:
- Group related functions with common prefixes
- Use _ prefix for internal/private helpers within the same family
- ALL logging-related methods MUST start with log_ or _log
- Search for all migration functions: grep -r "def.fs_.(" archivebox/
- Search for all logging: grep -r "def.log_.(" archivebox/

Minimize Unique Names and Data Structures

Do not invent new data structures, variable names, or keys if possible. Try to use existing field names and data structures exactly to keep the total unique data structures and names in the codebase to an absolute minimum.

Example - GOOD:

python

Binary has overrides field

binary = Binary(overrides={'TIMEOUT': '60s'})
Binary reuses the same field name and structure

class Binary(models.Model):
    overrides = models.JSONField(default=dict)  # Same name, same structure

Example - BAD:

python

Don't invent new names like custom_bin_cmds, binary_overrides, etc.

class Binary(models.Model):
    custom_bin_cmds = models.JSONField(default=dict)  # ❌ New unique name

Principle: If you're storing the same conceptual data (e.g., overrides), use the same field name across all models and keep the internal structure identical. This makes the codebase predictable and reduces cognitive load.

Testing

CRITICAL: Never Run as Root

ArchiveBox has a root check that prevents running as root user. All ArchiveBox commands (including tests) must run as non-root user inside a data directory:

bash

Run all migration tests

sudo -u testuser bash -c 'source /path/to/.venv/bin/activate && python -m pytest archivebox/tests/test_migrations_*.py -v'
Run specific test file

sudo -u testuser bash -c 'source .venv/bin/activate && python -m pytest archivebox/tests/test_migrations_08_to_09.py -v'
Run single test

sudo -u testuser bash -c 'source .venv/bin/activate && python -m pytest archivebox/tests/test_migrations_fresh.py::TestFreshInstall::test_init_creates_database -xvs'

Test File Structure

text

archivebox/tests/
├── test_migrations_helpers.py    # Schemas, seeding functions, verification helpers
├── test_migrations_fresh.py      # Fresh install tests
├── test_migrations_04_to_09.py   # 0.4.x → 0.9.x migration tests
├── test_migrations_07_to_09.py   # 0.7.x → 0.9.x migration tests
└── test_migrations_08_to_09.py   # 0.8.x → 0.9.x migration tests

Test Writing Standards

#### NO MOCKS - Real Tests Only
Tests must exercise real code paths:
- Create real SQLite databases with version-specific schemas
- Seed with realistic test data
- Run actual python -m archivebox commands via subprocess
- Query SQLite directly to verify results

If something is hard to test: Modify the implementation to make it easier to test, or fix the underlying issue. Never mock, skip, simulate, or exit early from a test because you can't get something working inside the test.

#### NO SKIPS
Never use @skip, skipTest, or pytest.mark.skip. Every test must run. If a test is difficult, fix the code or test environment - don't disable the test.

#### Strict Assertions
- init command must return exit code 0 (not [0, 1])
- Verify ALL data is preserved, not just "at least one"
- Use exact counts (==) not loose bounds (>=)

Example Test Pattern

python

def test_migration_preserves_snapshots(self):
    """Migration should preserve all snapshots."""
    result = run_archivebox(self.work_dir, ['init'], timeout=45)
    self.assertEqual(result.returncode, 0, f"Init failed: {result.stderr}")    ok, msg = verify_snapshot_count(self.db_path, expected_count)
    self.assertTrue(ok, msg)

Testing Gotchas

#### Extractors Disabled for Speed
Tests disable all extractors via environment variables for faster execution:

python

env['SAVE_TITLE'] = 'False'
env['SAVE_FAVICON'] = 'False'
... etc

#### Timeout Settings
Use appropriate timeouts for migration tests (45s for init, 60s default).

Plugin Testing & Code Coverage

Target: 80-90% coverage for critical plugins (screenshot, chrome, singlefile, dom)

bash

Run plugin tests with coverage (both Python + JavaScript)

bash bin/test_plugins.sh screenshot
View coverage reports

bash bin/test_plugins.sh --coverage-report
Or individual reports:

coverage report --show-missing --include='archivebox/plugins/' --omit='/tests/*'

#### Plugin Test Structure

Tests are completely isolated from ArchiveBox - they replicate production directory structure in temp dirs:

python

Correct production paths:

Crawl:    DATA_DIR/users/{username}/crawls/YYYYMMDD/example.com/{crawl-id}/{plugin}/

Snapshot: DATA_DIR/users/{username}/snapshots/YYYYMMDD/example.com/{snapshot-uuid}/{plugin}/
with tempfile.TemporaryDirectory() as tmpdir:
    data_dir = Path(tmpdir)
    # Crawl-level plugin (e.g., chrome launcher)
    crawl_dir = data_dir / 'users' / 'testuser' / 'crawls' / '20240101' / 'example.com' / 'crawl-123'
    chrome_dir = crawl_dir / 'chrome'
    chrome_dir.mkdir(parents=True)
    # Snapshot-level plugin (e.g., screenshot)
    snapshot_dir = data_dir / 'users' / 'testuser' / 'snapshots' / '20240101' / 'example.com' / 'snap-456'
    screenshot_dir = snapshot_dir / 'screenshot'
    screenshot_dir.mkdir(parents=True)    # Run hook in its output directory
    result = subprocess.run(
        ['node', str(SCREENSHOT_HOOK), '--url=https://example.com'],
        cwd=str(screenshot_dir),
        env={get_test_env(), 'EXTRA_CONTEXT': '{"snapshot_id":"snap-456"}'},
        capture_output=True,
        timeout=120
    )

#### Coverage Improvement Loop

To improve from ~20% to 80%+:

1. Run tests: bash bin/test_plugins.sh screenshot → Shows: 19.1% (13/68 ranges)
2. Identify gaps: Check hook file for untested paths (session connection vs fallback, config branches, error cases)
3. Add tests: Test both execution paths (connect to session + launch own browser), skip conditions, error cases, config variations
4. Verify: Re-run tests → Should show: 85%+ (58+/68 ranges)

Critical: JavaScript hooks have TWO paths that both must be tested (connect to session ~50% + launch browser ~30% + shared ~20%). Testing only one path = max 50% coverage possible!

Database Migrations

Generate and Apply Migrations

bash

Generate migrations (run from archivebox subdirectory)

cd archivebox
./manage.py makemigrations
Apply migrations to test database

cd data/
archivebox init

Schema Versions

- 0.4.x: First Django version. Tags as comma-separated string, no ArchiveResult model
- 0.7.x: Tag model with M2M, ArchiveResult model, AutoField PKs
- 0.8.x: Crawl/Seed models, UUID PKs, status fields, depth/retry_at
- 0.9.x: Seed model removed, seed_id FK removed from Crawl

Testing a Migration Path

1. Create SQLite DB with source version schema (from test_migrations_helpers.py)
2. Seed with realistic test data using seed_0_X_data()
3. Run archivebox init to trigger migrations
4. Verify data preservation with verify_* functions
5. Test CLI commands work post-migration (status, list, add, etc.)

Squashed Migrations

When testing 0.8.x (dev branch), you must record ALL replaced migrations:

python

The squashed migration replaces these - all must be recorded

('core', '0023_alter_archiveresult_options_archiveresult_abid_and_more'),
('core', '0024_auto_20240513_1143'),
... all 52 migrations from 0023-0074 ...

('core', '0023_new_schema'),  # Also record the squashed migration itself

Migration Strategy

- Squashed migrations for clean installs
- Individual migrations recorded for upgrades from dev branch
- replaces attribute in squashed migrations lists what they replace

Migration Gotchas

#### Circular FK References in Schemas
SQLite handles circular references with IF NOT EXISTS. Order matters less than in other DBs.

Plugin System Architecture

Plugin Dependency Rules

Like other plugins, chrome plugins ARE NOT ALLOWED TO DEPEND ON ARCHIVEBOX OR DJANGO.
However, they are allowed to depend on two shared files ONLY:
- archivebox/plugins/chrome/chrome_utils.js ← source of truth API for all basic chrome ops
- archivebox/plugins/chrome/tests/chrome_test_utils.py ← use for your tests, do not implement launching/killing/pid files/cdp/etc. in python, just extend this file as needed.

Chrome-Dependent Plugins

Many plugins depend on Chrome/Chromium via CDP (Chrome DevTools Protocol). When checking for script name references or debugging Chrome-related issues, check these plugins:

Main puppeteer-based chrome installer + launcher plugin:
- chrome - Core Chrome integration (CDP, launch, navigation)

Metadata extraction using chrome/chrome_utils.js / CDP:
- dns - DNS resolution info
- ssl - SSL certificate info
- headers - HTTP response headers
- redirects - Capture redirect chains
- staticfile - Direct file downloads (e.g. if the url itself is a .png, .exe, .zip, etc.)
- responses - Capture network responses
- consolelog - Capture console.log output
- title - Extract page title
- accessibility - Extract accessibility tree
- seo - Extract SEO metadata

Extensions installed using chrome/chrome_utils.js / controlled using CDP:
- ublock - uBlock Origin ad blocking
- istilldontcareaboutcookies - Cookie banner dismissal
- twocaptcha - 2captcha CAPTCHA solver integration

Page-alteration plugins to prepare the content for archiving:
- modalcloser - Modal dialog dismissal
- infiniscroll - Infinite scroll handler

Main Extractor Outputs:
- dom - DOM snapshot extraction
- pdf - Generate PDF snapshots
- screenshot - Generate screenshots
- singlefile - SingleFile archival, can be single-file-cli that launches chrome, or singlefile extension running inside chrome

Crawl URL parsers (post-process dom.html, singlefile.html, staticfile, responses, headers, etc. for URLs to re-emit as new queued Snapshots during recursive crawling):
- parse_dom_outlinks - Extract outlinks from DOM (special, uses CDP to directly query browser)
- parse_html_urls - Parse URLs from HTML (doesn't use chrome directly, just reads dom.html)
- parse_jsonl_urls - Parse URLs from JSONL (doesn't use chrome directly, just reads dom.html)
- parse_netscape_urls - Parse Netscape bookmark format (doesn't use chrome directly, just reads dom.html)

Finding Chrome-Dependent Plugins

bash

Find all files containing "chrom" (case-insensitive)

grep -ri "chrom" archivebox/plugins//on_. --include=".*" 2>/dev/null | cut -d: -f1 | sort -u
Or get just the plugin names

grep -ri "chrom" archivebox/plugins//on_. --include=".*" 2>/dev/null | cut -d/ -f3 | sort -u

Note: This list may not be complete. Always run the grep command above when checking for Chrome-related script references or debugging Chrome integration issues.

Architecture Notes

Crawl Model (0.9.x)

- Crawl groups multiple Snapshots from a single add command
- Each add creates one Crawl with one or more Snapshots
- Seed model was removed - crawls now store URLs directly

Code Coverage

Overview

Coverage tracking is enabled for passive collection across all contexts:
- Unit tests (pytest)
- Integration tests
- Dev server (manual testing)
- CLI usage

Coverage data accumulates in .coverage file and can be viewed/analyzed to find dead code.

Install Coverage Tools

bash

uv sync --dev  # Installs pytest-cov and coverage

Running with Coverage

#### Unit Tests

bash

Run tests with coverage

pytest --cov=archivebox --cov-report=term archivebox/tests/
Or run specific test file

pytest --cov=archivebox --cov-report=term archivebox/tests/test_migrations_08_to_09.py

#### Dev Server with Coverage

bash

Start dev server with coverage tracking

coverage run --parallel-mode -m archivebox server
Or CLI commands

coverage run --parallel-mode -m archivebox init
coverage run --parallel-mode -m archivebox add https://example.com

#### Manual Testing (Always-On)
To enable coverage during ALL Python executions (passive tracking):

bash

Option 1: Use coverage run wrapper

coverage run --parallel-mode -m archivebox [command]
Option 2: Set environment variable (tracks everything)

export COVERAGE_PROCESS_START=pyproject.toml
Now all Python processes will track coverage

archivebox server
archivebox add https://example.com

Viewing Coverage

#### Text Report (Quick View)

bash

Combine all parallel coverage data

coverage combine
View summary

coverage report
View detailed report with missing lines

coverage report --show-missing
View specific file

coverage report --include="archivebox/core/models.py" --show-missing

#### JSON Report (LLM-Friendly)

bash

Generate JSON report

coverage json
View the JSON

cat coverage.json | jq '.files | keys'  # List all files
Find files with low coverage

cat coverage.json | jq -r '.files | to_entries[] | select(.value.summary.percent_covered < 50) | "\(.key): \(.value.summary.percent_covered)%"'
Find completely uncovered files (dead code candidates)

cat coverage.json | jq -r '.files | to_entries[] | select(.value.summary.percent_covered == 0) | .key'
Get missing lines for a specific file

cat coverage.json | jq '.files["archivebox/core/models.py"].missing_lines'

#### HTML Report (Visual)

bash

Generate interactive HTML report

coverage html
Open in browser

open htmlcov/index.html

Isolated Runs

To measure coverage for specific scenarios:

bash

1. Reset coverage data

coverage erase
2. Run your isolated test/scenario

pytest --cov=archivebox archivebox/tests/test_migrations_fresh.py
OR

coverage run --parallel-mode -m archivebox add https://example.com
3. View results

coverage combine
coverage report --show-missing
4. Optionally export for analysis

coverage json

Finding Dead Code

bash

1. Run comprehensive tests + manual testing to build coverage

pytest --cov=archivebox archivebox/tests/
coverage run --parallel-mode -m archivebox server  # Use the app manually
coverage combine
2. Find files with 0% coverage (strong dead code candidates)

coverage json
cat coverage.json | jq -r '.files | to_entries[] | select(.value.summary.percent_covered == 0) | .key'
3. Find files with <10% coverage (likely dead code)

cat coverage.json | jq -r '.files | to_entries[] | select(.value.summary.percent_covered < 10) | "\(.key): \(.value.summary.percent_covered)%"' | sort -t: -k2 -n
4. Generate detailed report for analysis

coverage report --show-missing > coverage_report.txt

Tips

- Parallel mode (--parallel-mode): Allows multiple processes to track coverage simultaneously without conflicts
- Combine: Always run coverage combine before viewing reports to merge parallel data
- Reset: Use coverage erase to start fresh for isolated measurements
- Branch coverage: Enabled by default - tracks if both branches of if/else are executed
- Exclude patterns: Config in pyproject.toml excludes tests, migrations, type stubs

Debugging Tips

Check Migration State

bash

sqlite3 /path/to/index.sqlite3 "SELECT app, name FROM django_migrations WHERE app='core' ORDER BY id;"

Check Table Schema

bash

sqlite3 /path/to/index.sqlite3 "PRAGMA table_info(core_snapshot);"

Verbose Test Output

bash

sudo -u testuser bash -c 'source .venv/bin/activate && python -m pytest archivebox/tests/test_migrations_08_to_09.py -xvs 2>&1 | head -200'

Kill Zombie Chrome Processes

bash

./bin/kill_chrome.sh

README.md

<div align="center" style="text-align: center; width: 100%">
<img src="https://archivebox.io/icon.png" height="90px"/>
<h1>ArchiveBox Open-source self-hosted web archiving.</h1>

▶️ <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart">Quickstart</a> | <a href="https://demo.archivebox.io">Demo</a> | <a href="https://github.com/ArchiveBox/ArchiveBox">GitHub</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Documentation</a> | <a href="#background--motivation">Info & Motivation</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community">Community</a>

</div>
<hr/>

ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.

We aim to make your data immediately useful, and kept in formats that other programs can read directly. As output, we save standard HTML, PNG, PDF, TXT, JSON, WARC, SQLite, all guaranteed to be readable for decades to come. ArchiveBox also has a CLI, REST API, and webhooks so you can set up integrations with other services.

Without active preservation effort, everything on the internet eventually disappears or degrades.

ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...

➡️ Get ArchiveBox with pip install archivebox on Linux/macOS, or via Docker ⭐️ on any OS.

Once installed, you can interact with it through the: Browser Extension, CLI, self-hosted web interface, Python API, or filesystem.

<hr/>

📥 You can feed ArchiveBox URLs one at a time, or schedule regular imports from your bookmarks or history, social media feeds or RSS, link-saving services like Pocket/Pinboard, our Browser Extension, and more.
See <a href="#input-formats">Input Formats</a> for a full list of supported input formats...

It saves snapshots of the URLs you feed it in several redundant formats.
It also detects any content featured inside pages & extracts it out into a folder:
- 🌐 HTML/Any websites ➡️ original HTML+CSS+JS, singlefile HTML, screenshot PNG, PDF, WARC, title, article text, favicon, headers, ...
- 🎥 Social Media/News ➡️ post content TXT, comments, title, author, images, ...
- 🎬 YouTube/SoundCloud/etc. ➡️ MP3/MP4s, subtitles, metadata, thumbnail, ...
- 💾 Github/Gitlab/etc. links ➡️ clone of GIT source code, README, images, ...
- ✨ and more, see Output Formats below...

You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the pip-installed CLI, Python API, and SQLite3 APIs.
All the ways of using it are equivalent, and provide matching features like adding tags, scheduling regular crawls, viewing logs, and more...

<hr/>

🛠️ ArchiveBox uses standard tools like Chrome, wget, & yt-dlp, and stores data in ordinary files & folders.
(no complex proprietary formats, all data is readable without needing to run ArchiveBox)

The goal is to sleep soundly knowing the part of the internet you care about will be automatically preserved in durable, easily accessible formats for decades after it goes down.

<hr/>

📦  Install ArchiveBox using your preferred method: docker / pip / apt / etc. (see full Quickstart below).

<details>
  <summary>Expand for quick copy-pastable install commands...   ⤵️</summary>
 
<pre lang="bash"><code style="white-space: pre-line"># Option A: Get ArchiveBox with Docker Compose (recommended):
mkdir -p ~/archivebox/data && cd ~/archivebox
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml # edit options in this file as-needed
docker compose run archivebox init --install

docker compose run archivebox add 'https://example.com'

docker compose run archivebox help

docker compose up

Option B: Or use it as a plain Docker container:

mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -it -v $PWD:/data archivebox/archivebox init --install

docker run -it -v $PWD:/data archivebox/archivebox add 'https://example.com'

docker run -it -v $PWD:/data archivebox/archivebox help

docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox

Option C: Or install it with your preferred pkg manager (see Quickstart below for apt, brew, and more)

pip install archivebox
mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init --install

archivebox add 'https://example.com'

archivebox help

archivebox server 0.0.0.0:8000

Option D: Or use the optional auto setup script to install it

curl -fsSL 'https://get.archivebox.io' | bash
</code></pre>
 
Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️ 
Set <code>LISTEN_HOST</code> to change the base domain; <code>web.</code> and <code>admin.</code> subdomains are used automatically.
</details>

<div align="center" style="text-align: center">
 
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic">   <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/b2765a33-0d1e-4019-a1db-920c7e00e20e" height="75px" alt="logo" align="top"/>   <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic">
 
<a href="https://demo.archivebox.io">Demo</a> | <a href="#screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a>
 
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
 
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/8d67382c-e0ce-4286-89f7-7915f09b930c" width="22%" alt="cli init screenshot" align="top">
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/dad2bc51-e7e5-484e-bb26-f956ed692d16" width="22%" alt="cli init screenshot" align="top">
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e8e0b6f8-8fdf-4b7f-8124-c10d8699bdb2" width="22%" alt="server snapshot admin screenshot" align="top">
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ace0954a-ddac-4520-9d18-1c77b1ec50b2" width="28.6%" alt="server snapshot details page screenshot" align="top"/>
 
</div>

Key Features

- Free & open source, own your own data & maintain your privacy by self-hosting
- Powerful CLI with modular dependencies and support for Google Drive/NFS/SMB/S3/B2/etc.
- Comprehensive documentation, active development, and rich community
- Extracts a wide variety of content out-of-the-box: media (yt-dlp), articles (readability), code (git), etc.
- Supports scheduled/realtime importing from many types of sources
- Uses standard, durable, long-term formats like HTML, JSON, PDF, PNG, MP4, TXT, and WARC
- Powerful CLI, self-hosted web UI, Python API (BETA), REST API (ALPHA), or desktop app
- Saves all pages to archive.org as well by default for redundancy (can be disabled for local-only mode)
- Advanced users: support for archiving content requiring login/paywall/cookies (see wiki security caveats!)
- Planned: support for running JS during archiving to adblock, autoscroll, modal-hide, thread-expand

🤝 Professional Integration

ArchiveBox is free for everyone to self-host, but we also provide support, security review, and custom integrations to help NGOs, governments, and other organizations run ArchiveBox professionally:

- Journalists:
crawling during research, preserving cited pages, fact-checking & review
- Lawyers:
collecting & preserving evidence, detecting changes, tagging & review
- Researchers:
analyzing social media trends, getting LLM training data, crawling pipelines
- Individuals:
saving bookmarks, preserving portfolio content, legacy / memoirs archival
- Governments:
snapshotting public service sites, recordkeeping compliance

Contact us if your org wants help using ArchiveBox professionally.

We offer: setup & support, CAPTCHA/ratelimit unblocking, SSO, audit logging/chain-of-custody, and more

ArchiveBox is a 🏛️ 501(c)(3) nonprofit FSP and all our work supports open-source development.

Quickstart

🖥  Supported OSs: Linux/BSD, macOS, Windows (Docker)   👾  CPUs: amd64 (x86_64), arm64, arm7 (raspi>=3)

#### ✳️  Easy Setup

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code> (macOS/Linux/Windows)   👈  recommended   (click to expand)</summary>
 
👍 Docker Compose is recommended for the easiest install/update UX + best security + all <a href="#dependencies">extras</a> out-of-the-box.
 
<ol>
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
<li>Download the <a href="https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/docker-compose.yml" download><code>docker-compose.yml</code></a> file into a new empty directory (can be anywhere).
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox

Read and edit docker-compose.yml options as-needed after downloading

curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
</code></pre></li>
<li>Run the initial setup to create an admin user (or set ADMIN_USER/PASS in docker-compose.yml)
<pre lang="bash"><code style="white-space: pre-line">docker compose run archivebox init --install
</code></pre></li>
<li>Next steps: Start the server then login to the Web UI <a href="http://127.0.0.1:8000">http://127.0.0.1:8000</a> ⇢ Admin.
<pre lang="bash"><code style="white-space: pre-line">docker compose up

completely optional, CLI can always be used without running a server

docker compose run [-T] archivebox [subcommand] [--help]

docker compose run archivebox add 'https://example.com'
docker compose run archivebox help
</code></pre>
For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. ➡️
</li>
</ol>

See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#sql-shell-usage">filesystem/SQL/Python</a> to manage your archive.
 
</details>

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker run</code> (macOS/Linux/Windows)</summary>
 
<ol>
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
<li>Create a new empty directory and initialize your collection (can be anywhere).
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -v $PWD:/data -it archivebox/archivebox init --install
</code></pre>
</li>
<li>Optional: Start the server then login to the Web UI <a href="http://127.0.0.1:8000">http://127.0.0.1:8000</a> ⇢ Admin.
<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox

completely optional, CLI can always be used without running a server

docker run -v $PWD:/data -it [subcommand] [--help]

docker run -v $PWD:/data -it archivebox/archivebox help
</code></pre>
For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. ➡️
</li>
</ol>

See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
 
</details>

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" height="28px" align="top"/> <code>bash</code> auto-setup script (macOS/Linux)</summary>
 
<ol>
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (optional, highly recommended but not required).</li>
<li>Run the automatic setup script.
<pre lang="bash"><code style="white-space: pre-line">curl -fsSL 'https://get.archivebox.io' | bash</code></pre>
For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-b-automatic-setup-script">Install: Bare Metal</a> in the Wiki. ➡️
</li>
</ol>

See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive. 
See <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/setup.sh"><code>setup.sh</code></a> for the source code of the auto-install script. 
See <a href="https://docs.sweeting.me/s/against-curl-sh">"Against curl | sh as an install method"</a> blog post for my thoughts on the shortcomings of this install method.
 
</details>

#### 🛠  Package Manager Setup

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="Pip" height="28px" align="top"/> <code>pip</code> (macOS/Linux/BSD)</summary>
 
<ol>

<li>Install <a href="https://realpython.com/installing-python/">Python >= v3.13</a> and <a href="https://nodejs.org/en/download/package-manager/">Node >= v22</a> on your system (if not already installed).</li>
<li>Install the ArchiveBox package using <code>pip3</code> (or <a href="https://docs.astral.sh/uv/guides/tools/#running-tools"><code>uvx</code></a>).
<pre lang="bash"><code style="white-space: pre-line">pip3 install --upgrade archivebox
archivebox version

install any missing extras shown using apt/brew/pkg/etc. see Wiki for instructions

[email protected] node curl wget git ripgrep ...

</code></pre>
See the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install">Install: Bare Metal</a> Wiki for full install instructions for each OS...
</li>
<li>Create a new empty directory and initialize your collection (can be anywhere).
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data # for example
archivebox init --install # instantialize a new collection

(--setup auto-installs and link JS dependencies: singlefile, readability, mercury, etc.)

</code></pre>
</li>
<li>Optional: Start the server then login to the Web UI <a href="http://127.0.0.1:8000">http://127.0.0.1:8000</a> ⇢ Admin.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000

completely optional, CLI can always be used without running a server

archivebox [subcommand] [--help]

archivebox help
</code></pre>
</li>
</ol>

See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive. 
 
See the <a href="https://github.com/ArchiveBox/pip-archivebox"><code>pip-archivebox</code></a> repo for more details about this distribution.
 
</details>

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117448075-49597580-af0c-11eb-91ba-f34fff10096b.png" alt="aptitude" height="28px" align="top"/> <code>apt</code> (Ubuntu/Debian/etc.)</summary>
 
<ol>
<li>Download and install the <code>.deb</code> package from the <a href="https://github.com/ArchiveBox/ArchiveBox/releases">latest release</a>.
<pre lang="bash"><code style="white-space: pre-line"># download the .deb for your architecture (amd64 or arm64)
ARCH="$(dpkg --print-architecture)"
VERSION="$(curl -fsSL https://api.github.com/repos/ArchiveBox/ArchiveBox/releases/latest | python3 -c "import sys,json; print(json.load(sys.stdin)['tag_name'].lstrip('v'))")"
curl -fsSL "https://github.com/ArchiveBox/ArchiveBox/releases/latest/download/archivebox_${VERSION}_${ARCH}.deb" -o /tmp/archivebox.deb
sudo apt install /tmp/archivebox.deb
archivebox version # make sure all dependencies are installed
</code></pre>
</li>
<li>Create a new empty directory and initialize your collection (can be anywhere).
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init --install
</code></pre>
 
</li>
<li>Optional: Start the server then login to the Web UI <a href="http://127.0.0.1:8000">http://127.0.0.1:8000</a> ⇢ Admin.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000

completely optional, CLI can always be used without running a server

archivebox [subcommand] [--help]

archivebox help
</code></pre>
</li>
</ol>
See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive. 
See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for more details about this distribution.
 
</details>

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117447803-f2ec3700-af0b-11eb-87d3-671d114f011d.png" alt="homebrew" height="28px" align="top"/> <code>brew</code> (macOS only)</summary>
 
<ol>
<li>Install <a href="https://brew.sh/#install">Homebrew</a> on your system (if not already installed).</li>
<li>Install the ArchiveBox package using <code>brew</code>.
<pre lang="bash"><code style="white-space: pre-line">brew tap archivebox/archivebox
brew install archivebox
archivebox version # make sure all dependencies are installed
</code></pre>
See the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-c-bare-metal-setup">Install: Bare Metal</a> Wiki for more granular instructions for macOS... ➡️
</li>
<li>Create a new empty directory and initialize your collection (can be anywhere).
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init --install
</code></pre>
</li>
<li>Optional: Start the server then login to the Web UI <a href="http://127.0.0.1:8000">http://127.0.0.1:8000</a> ⇢ Admin.
<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000

completely optional, CLI can always be used without running a server

archivebox [subcommand] [--help]

archivebox help
</code></pre> 
</li>
</ol>

See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive. 
See the <a href="https://github.com/ArchiveBox/homebrew-archivebox"><code>homebrew-archivebox</code></a> repo for more details about this distribution.
 
</details>

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/118077361-f0616580-b381-11eb-973c-ee894a3349fb.png" alt="Arch" height="28px" align="top"/> <code>pacman</code> / <img src="https://user-images.githubusercontent.com/511499/118077946-29e6a080-b383-11eb-94f0-d4871da08c3f.png" alt="FreeBSD" height="28px" align="top"/> <code>pkg</code> / <img src="https://user-images.githubusercontent.com/511499/118077861-002d7980-b383-11eb-86a7-5936fad9190f.png" alt="Nix" height="28px" align="top"/> <code>nix</code> (Arch/FreeBSD/NixOS/more)</summary>

Warning: These are contributed by external volunteers and may lag behind the official pip channel.

<ul>
<li>Arch: <a href="https://aur.archlinux.org/packages/archivebox/"><code>yay -S archivebox</code></a> (contributed by <a href="https://github.com/imlonghao"><code>@imlonghao</code></a>, maintained by <a href="https://github.com/jasongodev"><code>@jasongodev</code></a>)</li>
<li>FreeBSD: <a href="https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup"><code>curl -fsSL 'https://get.archivebox.io' | bash</code></a> (uses <code>pkg</code> + <code>pip3</code> under-the-hood)</li>
<li>Nix: <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/misc/archivebox/default.nix"><code>nix-env --install archivebox</code></a> (contributed by <a href="https://github.com/siraben"><code>@siraben</code></a>)</li>
<li>Guix: <a href="https://packages.guix.gnu.org/packages/archivebox/"><code>guix install archivebox</code></a> (contributed by <a href="https://github.com/rakino"><code>@rakino</code></a>)</li>
<li>More: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/new">contribute another distribution...!</a></li>
</ul>
See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
 
</details>

#### 🎗  Other Options

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker</code> + <img src="https://user-images.githubusercontent.com/511499/117447263-4316c980-af0b-11eb-928d-eaf1292ac646.png" alt="Electron" height="28px" align="top"/> <code>electron</code> Desktop App (macOS/Linux/Windows)</summary>
 
<ol>
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
<li>Download a binary release for your OS or build the native app from source 
<ul>
<li>macOS: <a href="https://github.com/ArchiveBox/ArchiveBox/releases/download/v0.6.2/Electron-ArchiveBox-macOS-x64-0.6.2.app.zip" download><code>ArchiveBox.app.zip</code></a></li>
<li>Linux: <code>ArchiveBox.deb</code> (alpha: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
<li>Windows: <code>ArchiveBox.exe</code> (beta: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
</ul>
</li>
</ol>
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/575ef92f-bb3e-4a7c-a4ba-986c1fd76ecf" width="320px">
 
✨ Alpha (contributors wanted!): for more info, see the: <a href="https://github.com/ArchiveBox/electron-archivebox">Electron ArchiveBox</a> repo.
 
</details>

<details>
<summary><img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0c46e949-00fe-49c8-a613-ee14501c014c" alt="Self-hosting Platforms" height="28px" align="top"/> TrueNAS / UNRAID / YunoHost / Cloudron / etc. (self-hosting solutions)</summary>

Warning: These are contributed by external volunteers and may lag behind the official pip channel.

<ul>
<li><s>TrueNAS: <a href="https://truecharts.org/charts/stable/archivebox/">Official ArchiveBox TrueChart</a> / <a href="https://dev.to/finloop/setting-up-archivebox-on-truenas-scale-1788">Custom App Guide</a></s> (<a href="https://truecharts.org/news/scale-deprecation/">TrueCharts is discontinued</a>, wait for <a href="https://forums.truenas.com/t/the-future-of-electric-eel-and-apps/5409/">Electric Eel</a>)</li>
<li><a href="https://unraid.net/community/apps?q=archivebox#r">UnRaid</a></li>
<li><a href="https://community-scripts.github.io/ProxmoxVE/scripts?id=archivebox">Proxmox</a></li>
<li><a href="https://github.com/YunoHost-Apps/archivebox_ynh">Yunohost</a></li>
<li><a href="https://www.cloudron.io/store/io.archivebox.cloudronapp.html">Cloudron</a></li>
<li><a href="https://docs.saltbox.dev/sandbox/apps/archivebox/">Saltbox</a></li>
<li><a href="https://portainer-templates.as93.net/archivebox">Portainer</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/pull/922/files#diff-00f0606e18b2618c3cc1667ca7c2b703b537af690ca71eba1330633587dcb1ee">AppImage</a></li>
<li><a href="https://runtipi.io/docs/apps-available#:~:text=for%20AI%20Chats.-,ArchiveBox,Open%20source%20self%2Dhosted%20web%20archiving.,-Atuin%20Server">Runtipi</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/issues/986">Umbrel</a> (need contributors...)</li>

<li>More: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/new">contribute another distribution...!</a></li>
</ul>
See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
 
</details>

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117448723-1663b180-af0d-11eb-837f-d43959227810.png" alt="paid" height="27px" align="top"/> Paid hosting solutions (cloud VPS)</summary>
 
<ul>
<li><a href="https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102">
<img src="https://img.shields.io/badge/Custom_Development-ArchiveBox.io-%231a1a1a.svg?style=flat" height="22px"/>
</a> (<a href="https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102">get hosting, support, and feature customization directly from us</a>)</li>
<li><a href="https://monadical.com">
<img src="https://img.shields.io/badge/General_Dev_Consulting-Monadical.com-%231a1a1a.svg?style=flat" height="22px"/>
</a> (<a href="https://monadical.com/contact-us.html">generalist consultancy that has ArchiveBox experience</a>)</li>
 
Other providers of paid ArchiveBox hosting (not officially endorsed): 
 
<li><a href="https://elest.io/open-source/archivebox"><img src="https://img.shields.io/badge/Managed_Hosting-Elest.io-%23193f7e.svg?style=flat" height="22px"/></a></li>
<li><a href="https://www.stellarhosted.com/archivebox/"><img src="https://img.shields.io/badge/Semi_Managed_Hosting-StellarHosted.com-%23193f7e.svg?style=flat" height="22px"/></a> (USD $29-250/mo, <a href="https://www.stellarhosted.com/archivebox/#pricing">pricing</a>)</li>
<li><a href="https://www.pikapods.com/pods?run=archivebox"><img src="https://img.shields.io/badge/Semi_Managed_Hosting-PikaPods.com-%2343a047.svg?style=flat" height="22px"/></a> (from USD $2.6/mo)</li>
<li><a href="https://m.do.co/c/cbc4c0c17840">
<img src="https://img.shields.io/badge/Unmanaged_VPS-DigitalOcean.com-%232f7cf7.svg?style=flat" height="22px"/>
</a> (USD $5-50+/mo, <a href="https://m.do.co/c/cbc4c0c17840">🎗  referral link</a>, <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-compose-on-ubuntu-20-04">instructions</a>)</li>
<li><a href="https://www.vultr.com/?ref=7130289">
<img src="https://img.shields.io/badge/Unmanaged_VPS-Vultr.com-%232337a8.svg?style=flat" height="22px"/>
</a> (USD $2.5-50+/mo, <a href="https://www.vultr.com/?ref=7130289">🎗  referral link</a>, <a href="https://www.vultr.com/docs/install-docker-compose-on-ubuntu-20-04">instructions</a>)</li>
<li><a href="https://fly.io/">
<img src="https://img.shields.io/badge/Unmanaged_App-Fly.io-%239a2de6.svg?style=flat" height="22px"/>
</a> (USD $10-50+/mo, <a href="https://fly.io/docs/hands-on/start/">instructions</a>)</li>
<li><a href="https://railway.app/template/2Vvhmy">
<img src="https://img.shields.io/badge/Unmanaged_App-Railway-%23A11BE6.svg?style=flat" height="22px"/>
</a> (USD $0-5+/mo)</li>
<li><a href="https://aws.amazon.com/marketplace/pp/Linnovate-Open-Source-Innovation-Support-For-Archi/B08RVW6MJ2"><img src="https://img.shields.io/badge/Unmanaged_VPS-AWS-%23ee8135.svg?style=flat" height="22px"/></a> (USD $60-200+/mo)</li>
<li><a href="https://azuremarketplace.microsoft.com/en-us/marketplace/apps/meanio.archivebox?ocid=gtmrewards_whatsnewblog_archivebox_vol118"><img src="https://img.shields.io/badge/Unmanaged_VPS-Azure-%237cb300.svg?style=flat" height="22px"/></a> (USD $60-200+/mo)</li>
 
Referral links marked 🎗 provide $5-10 of free credit for new users and help pay for our <a href="https://demo.archivebox.io">demo server</a> hosting costs.
</ul>

For more discussion on managed and paid hosting options see here: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/531">Issue #531</a>.

</details>

#### ➡️  Next Steps

- Import URLs from some of the supported Input Formats or view the supported Output Formats...
- (Optional) Create a persona and import browser cookies to archive logged-in sites: archivebox persona create --import=chrome personal
- Tweak your UI or archiving behavior Configuration, read about some of the Caveats, or Troubleshoot
- Read about the Dependencies used for archiving, the Upgrading Process, or the Archive Layout on disk...
- Or check out our full Documentation or Community Wiki...

Usage

#### ⚡️  <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage">CLI Usage</a>

ArchiveBox commands can be run in a terminal directly on your host, or via Docker/Docker Compose.
(depending on how you chose to install it above)

bash

mkdir -p ~/archivebox/data   # create a new data dir anywhere
cd ~/archivebox/data         # IMPORTANT: cd into the directory
archivebox [subcommand] [--help]

archivebox version
archivebox help
equivalent: docker compose run archivebox [subcommand] [--help]

docker compose run archivebox help
equivalent: docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]

docker run -it -v $PWD:/data archivebox/archivebox help
optional: import your browser cookies into a persona for logged-in archiving

archivebox persona create --import=chrome personal
supported: chrome/chromium/brave/edge (Chromium-based only)

use --profile to target a specific profile (e.g. Default, Profile 1)

re-running import merges/dedupes cookies.txt (by domain/path/name) but replaces chrome_user_data

#### ArchiveBox Subcommands

- archivebox help/version to see the list of available subcommands / currently installed version info
- archivebox setup/init/config/status/shell/manage to administer your collection
- archivebox add/schedule to pull in fresh URLs from bookmarks/history/RSS/etc.
- archivebox list/update/remove to manage existing Snapshots in your collection

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" height="22px" align="top"/> CLI Usage Examples: non-Docker</summary>
 
<pre lang="bash"><code style="white-space: pre-line">

make sure you have pip-installed ArchiveBox and it's available in your $PATH first

archivebox [subcommand] [--help]

archivebox init --install # safe to run init multiple times (also how you update versions)
archivebox version # get archivebox version info + check dependencies
archivebox help # get list of archivebox subcommands that can be run
archivebox add --depth=1 'https://news.ycombinator.com'
</code></pre>
For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage">Usage: CLI Usage</a> wiki. ➡️
</details>

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="22px" align="top"/> CLI Usage Examples: Docker Compose</summary>
 
<pre lang="bash"><code style="white-space: pre-line">

make sure you have `docker-compose.yml` from the Quickstart instructions first

docker compose run archivebox [subcommand] [--help]

docker compose run archivebox init --install
docker compose run archivebox version
docker compose run archivebox help
docker compose run archivebox add --depth=1 'https://news.ycombinator.com'

to start webserver: docker compose up

</code></pre>
For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage">Usage: Docker Compose CLI</a> wiki. ➡️
</details>

<details>
<summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="22px" align="top"/> CLI Usage Examples: Docker</summary>
 
<pre lang="bash"><code style="white-space: pre-line">

make sure you create and cd into in a new empty directory first

docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]

docker run -v $PWD:/data -it archivebox/archivebox init --install
docker run -v $PWD:/data -it archivebox/archivebox version
docker run -v $PWD:/data -it archivebox/archivebox help
docker run -v $PWD:/data -it archivebox/archivebox add --depth=1 'https://news.ycombinator.com'

to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox

</code></pre>
For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1">Usage: Docker CLI</a> wiki. ➡️
</details>

<details>
<summary>🗄  SQL/Python/Filesystem Usage</summary>
<pre lang="bash"><code style="white-space: pre-line">
archivebox shell # explore the Python library API in a REPL
sqlite3 ./index.sqlite3 # run SQL queries directly on your index
ls ./archive/*/index.html # or inspect snapshot data directly on the filesystem
</code></pre>
For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage">Python Shell</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#sql-shell-usage">SQL API</a>, and <a href="https://github.com/ArchiveBox/ArchiveBox#archive-layout">Disk Layout</a> wikis. ➡️
</details>

<details>
<summary>🖥  Web UI & API Usage</summary>
<pre lang="bash"><code style="white-space: pre-line">

Start the server on bare metal (pip/apt/brew/etc):

archivebox manage createsuperuser # create a new admin user via CLI
archivebox server 0.0.0.0:8000 # start the server

Or with Docker Compose:

nano docker-compose.yml # setup initial ADMIN_USERNAME & ADMIN_PASSWORD
docker compose up # start the server

Or with a Docker container:

docker run -v $PWD:/data -it archivebox/archivebox archivebox manage createsuperuser
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox
</code></pre>

Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️ 
Set <code>LISTEN_HOST</code> to change the base domain; <code>web.</code> and <code>admin.</code> subdomains are used automatically.
 
For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage">Usage: Web UI</a> wiki. ➡️
 
Optional: Change permissions to allow non-logged-in users

<pre lang="bash"><code style="white-space: pre-line">
archivebox config --set PUBLIC_ADD_VIEW=True # allow guests to submit URLs
archivebox config --set PUBLIC_SNAPSHOTS=True # allow guests to see snapshot content
archivebox config --set PUBLIC_INDEX=True # allow guests to see list of all snapshots

or

docker compose run archivebox config --set ...

restart the server to apply any config changes

</code></pre>
</details>

TIP

Whether in Docker or not, ArchiveBox commands work the same way, and can be used to access the same data on-disk.

For example, you could run the Web UI in Docker Compose, and run one-off commands with pip-installed ArchiveBox.

<details>
<summary>Expand to show comparison...</summary>

<pre lang="bash"><code style="white-space: pre-line">
archivebox add --depth=1 'https://example.com' # add a URL with pip-installed archivebox on the host
docker compose run archivebox add --depth=1 'https://example.com' # or w/ Docker Compose
docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://example.com' # or w/ Docker, all equivalent
</code></pre>

For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker">Docker</a> wiki. ➡️

</details>

<div align="center" style="text-align: center">
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
 
<a href="https://demo.archivebox.io">DEMO: <code>https://demo.archivebox.io</code></a> 
<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration">Configuration</a> | <a href="#Caveats">Caveats</a>
 
</div>

---

Overview

Input Formats: How to pass URLs into ArchiveBox for saving

- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a>
Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox browsers.

- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files
ArchiveBox supports injesting URLs in any text-based format.

- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/32b494e6-4de1-4984-8d88-dc02f18e5c34" height="22px"/> From manually exported browser history or browser bookmarks (in Netscape format)
Instructions: <a href="https://support.google.com/chrome/answer/96816?hl=en">Chrome</a>, <a href="https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer">Firefox</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/assets/511499/24ad068e-0fa6-41f4-a7ff-4c26fc91f71a">Safari</a>, <a href="https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows">IE</a>, <a href="https://help.opera.com/en/latest/features/#bookmarks:~:text=Click%20the%20import/-,export%20button,-on%20the%20bottom">Opera</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a>

- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/4f7bd318-265c-4235-ad25-38be89946b12" height="22px"/> From URLs visited through a MITM Proxy with archivebox-proxy
Provides realtime archiving of all traffic from any device going through the proxy.

- <img src="https://getpocket.com/favicon.ico" height="22px"/> From bookmarking services or social media (e.g. Twitter bookmarks, Reddit saved posts, etc.)
Instructions: <a href="https://getpocket.com/export">Pocket</a>, <a href="https://pinboard.in/export/">Pinboard</a>, <a href="https://www.instapaper.com/user">Instapaper</a>, <a href="https://shaarli.readthedocs.io/en/master/Usage/#importexport">Shaarli</a>, <a href="https://www.groovypost.com/howto/howto/export-delicious-bookmarks-xml/">Delicious</a>, <a href="https://github.com/csu/export-saved-reddit">Reddit Saved</a>, <a href="https://doc.wallabag.org/en/user/import/wallabagv2.html">Wallabag</a>, <a href="http://help.unmark.it/import-export">Unmark.it</a>, <a href="https://www.addictivetips.com/web/onetab-save-close-all-chrome-tabs-to-restore-export-or-import/">OneTab</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/issues/648">Firefox Sync</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a>

bash

archivebox add --help

archivebox add 'https://example.com/some/page'
archivebox add --parser=generic_rss < ~/Downloads/some_feed.xml
archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12'
echo 'http://example.com' | archivebox add
echo 'any text with <a href="https://example.com">urls</a> in it' | archivebox add
if using Docker, add -i when piping stdin:

echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox add

if using Docker Compose, add -T when piping stdin / stdout:

echo 'https://example.com' | docker compose run -T archivebox add

See the Usage: CLI page for documentation and examples.

It also includes a built-in scheduled import feature with archivebox schedule, handled by the same orchestrator that powers archivebox server, so you can pull in URLs from RSS feeds and websites regularly without a separate cron container.

Output Formats: What ArchiveBox saves for each URL

For each web page added, ArchiveBox creates a Snapshot folder and preserves its content as ordinary files inside the folder (e.g. HTML, PDF, PNG, JSON, etc.).

It uses all available methods out-of-the-box, but you can disable extractors and fine-tune the configuration as-needed.

<details>
<summary>Expand to see the full list of ways it saves each page...</summary>

<code>data/archive/{Snapshot.id}/</code> 
<ul>
<li>Index: <code>index.html</code> & <code>index.json</code> HTML and JSON index files containing metadata and details</li>
<li>Title, Favicon, Headers Response headers, site favicon, and parsed site title</li>
<li>SingleFile: <code>singlefile.html</code> HTML snapshot rendered with headless Chrome using SingleFile</li>
<li>Wget Clone: <code>example.com/page-name.html</code> wget clone of the site with <code>warc/TIMESTAMP.gz</code></li>
<li>Chrome Headless <ul>
<li>PDF: <code>output.pdf</code> Printed PDF of site using headless chrome</li>
<li>Screenshot: <code>screenshot.png</code> 1440x900 screenshot of site using headless chrome</li>
<li>DOM Dump: <code>output.html</code> DOM Dump of the HTML after rendering using headless chrome</li>
</ul></li>
<li>Article Text: <code>article.html/json</code> Article text extraction using Readability & Mercury</li>
<li>Archive.org Permalink: <code>archive.org.txt</code> A link to the saved site on archive.org</li>
<li>Audio & Video: <code>media/</code> all audio/video files + playlists, including subtitles & metadata w/ <code>yt-dlp</code></li>
<li>Source Code: <code>git/</code> clone of any repository found on GitHub, Bitbucket, or GitLab links</li>
<li>More coming soon! See the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap">Roadmap</a>...</li>
</ul>
</details>

Configuration

ArchiveBox can be configured via environment variables, by using the archivebox config CLI, or by editing ./ArchiveBox.conf.
 
<details>
<summary>Expand to see examples...</summary>
<pre lang="bash"><code style="white-space: pre-line">archivebox config # view the entire config
archivebox config --get CHROME_BINARY # view a specific value
 
archivebox config --set CHROME_BINARY=chromium # persist a config using CLI

OR

echo CHROME_BINARY=chromium >> ArchiveBox.conf # persist a config using file

OR

env CHROME_BINARY=chromium archivebox ... # run with a one-off config
</code></pre>
These methods also work the same way when run inside Docker, see the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration">Docker Configuration</a> wiki page for details.
</details>

The configuration is documented here: Configuration Wiki, and loaded from: archivebox/config/.

<a name="most-common-options-to-tweak"></a>
<details>
<summary>Expand to see the most common options to tweak...</summary>
<pre lang="bash"><code style="white-space: pre-line">

e.g. archivebox config --set TIMEOUT=120

or docker compose run archivebox config --set TIMEOUT=120

TIMEOUT=240 # default: 60 add more seconds on slower networks
CHECK_SSL_VALIDITY=False # default: True False = allow saving URLs w/ bad SSL
 
PUBLIC_INDEX=True # default: True whether anon users can view index
PUBLIC_SNAPSHOTS=True # default: True whether anon users can view pages
PUBLIC_ADD_VIEW=False # default: False whether anon users can add new URLs
 
USER_AGENT="Mozilla/5.0 ..." # change this to get around bot blocking
</code></pre>
</details>

Dependencies

To achieve high-fidelity archives in as many situations as possible, ArchiveBox depends on a variety of 3rd-party libraries and tools that specialize in extracting different types of content.

Under-the-hood, ArchiveBox uses Django to power its Web UI, Django Ninja for the REST API, and SQlite + the filesystem to provide fast & durable metadata storage w/ deterministic upgrades.

ArchiveBox bundles industry-standard tools like Google Chrome, wget, yt-dlp, readability, etc. internally, and its operation can be tuned, secured, and extended as-needed for many different applications.

<details>
<summary>Expand to learn more about ArchiveBox's internals & dependencies...</summary>

<blockquote>
TIP: For better security while running ArchiveBox, and to avoid polluting your host system with a bunch of sub-dependencies that you need to keep up-to-date,it is strongly recommended to use the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker">⭐️ official Docker image</a> which provides everything in an easy container with simple one-liner upgrades.
</blockquote>

<ul>
<li>Language: Python <code>>=3.13</code></li>
<li>Backend: <a href="https://www.djangoproject.com/">Django</a> + <a href="https://django-ninja.dev/">Django-Ninja</a> for REST API</li>
<li>Frontend: <a href="https://docs.djangoproject.com/en/6.0/ref/contrib/admin/">Django Admin</a> + Vanilla HTML, CSS, JS</li>
<li>Web Server: <a href="https://www.djangoproject.com/">Django</a> + <a href="https://github.com/django/daphne/"><code>daphne</code></a> (ASGI)</li>
<li>Database: <a href="https://docs.djangoproject.com/en/6.0/ref/databases/#sqlite-notes">Django ORM</a> saving to <a href="https://www.sqlite.org/mostdeployed.html">SQLite3</a> <code>./data/index.sqlite3</code></li>
<li>Job Queue: Custom orchestrator using <code>supervisord</code> for worker management</li>
<li>Build/test/lint: <a href="https://github.com/astral-sh/uv"><code>uv</code></a> / <code>pyright</code>+<code>ty</code>+<code>pytest</code> / <code>ruff</code></li>
<li>Subdependencies: <a href="https://github.com/ArchiveBox/abx-pkg"><code>abx-pkg</code></a> installs apt/brew/pip/npm pkgs at runtime (e.g. <code>yt-dlp</code>, <code>singlefile</code>, <code>readability</code>, <code>git</code>)</li>
</ul>

These optional subdependencies used for archiving sites include:

<ul>
<li><code>chromium</code> / <code>chrome</code> (for screenshots, PDF, DOM HTML, and headless JS scripts)</li>
<li><code>node</code> & <code>npm</code> (for readability, mercury, and singlefile)</li>
<li><code>wget</code> (for plain HTML, static files, and WARC saving)</li>
<li><code>curl</code> (for fetching headers, favicon, and posting to Archive.org)</li>
<li><code>yt-dlp</code> or <code>youtube-dl</code> (for audio, video, and subtitles)</li>
<li><code>git</code> (for cloning git repos)</li>
<li><code>singlefile</code> (for saving into a self-contained html file)</li>
<li><code>postlight/parser</code> (for discussion threads, forums, and articles)</li>
<li><code>readability</code> (for articles and long text content)</li>
<li>and more as we grow...</li>
</ul>

You don't need to install every dependency to use ArchiveBox. ArchiveBox will automatically disable extractors that rely on dependencies that aren't installed, based on what is configured and available in your <code>$PATH</code>.

If not using Docker, make sure to keep the dependencies up-to-date yourself and check that ArchiveBox isn't reporting any incompatibility with the versions you install.

<pre lang="bash"><code style="white-space: pre-line">#install python3 and archivebox with your system package manager

apt/brew/pip/etc install ... (see Quickstart instructions above)

which -a archivebox # see where you have installed archivebox
archivebox install # auto install all the extractors and extras
archivebox --version # see info and check validity of installed dependencies
</code></pre>

Installing directly on Windows without Docker or WSL/WSL2/Cygwin is not officially supported (I cannot respond to Windows support tickets), but some advanced users have reported getting it working.

<h4>Learn More</h4>
<ul>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies">Wiki: Install (Dependencies)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install">Wiki: Chromium Install</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives">Wiki: Upgrading or Merging Archives</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#installing">Wiki: Troubleshooting (Installing)</a></li>
</ul>

</details>

Archive Layout

All of ArchiveBox's state (SQLite DB, content, config, logs, etc.) is stored in a single folder per collection.

<details>
<summary>Expand to learn more about the layout of Archivebox's data on-disk...</summary>

Data folders can be created anywhere (~/archivebox/data or $PWD/data as seen in our examples), and you can create as many data folders as you want to hold different collections.
All <code>archivebox</code> CLI commands are designed to be run from inside an ArchiveBox data folder, starting with <code>archivebox init</code> to initialize a new collection inside an empty directory.

<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data # just an example, can be anywhere
archivebox init</code></pre>

The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard <code>index.sqlite3</code> database in the root of the data folder (it can also be <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html">exported as static JSON/HTML</a>), and the archive snapshots are organized by date-added timestamp in the <code>data/archive/</code> subfolder.

<pre lang="bash"><code style="white-space: pre-line">data/
index.sqlite3
ArchiveBox.conf
archive/
...
1617687755/
index.html
index.json
screenshot.png
media/some_video.mp4
warc/1617687755.warc.gz
git/somerepo.git
...
</code></pre>

Each snapshot subfolder <code>data/archive/TIMESTAMP/</code> includes a static <code>index.json</code> and <code>index.html</code> describing its contents, and the snapshot extractor outputs are plain files within the folder.

<h4>Learn More</h4>
<ul>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage">Wiki: Setting Up Storage (SMB, NFS, S3, B2, Google Drive, etc.)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Disk-Layout">Wiki: Usage (Disk Layout)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#large-archives">Wiki: Usage (Large Archives)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#output-folder">Wiki: Security Overview (Output Folder)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives">Wiki: Upgrading or Merging Archives</a></li>
</ul>

</details>

Static Archive Exporting

You can export your index as static HTML using archivebox list (so you can view it without an ArchiveBox server).

<details>
<summary>Expand to learn how to export your ArchiveBox collection...</summary>

<blockquote>
NOTE: These exports are not paginated, exporting many URLs or the entire archive at once may be slow. Use the filtering CLI flags on the <code>archivebox list</code> command to export specific Snapshots or ranges.
</blockquote>

<pre lang="bash"><code style="white-space: pre-line"># archivebox list --help
archivebox list --html --with-headers > index.html # export to static html table
archivebox list --json --with-headers > index.json # export to json blob
archivebox list --csv=timestamp,url,title > index.csv # export to csv spreadsheet

(if using Docker Compose, add the -T flag when piping)

docker compose run -T archivebox list --html 'https://example.com' > index.json

</code></pre>

The paths in the static exports are relative, make sure to keep them next to your ./archive folder when backing them up or viewing them.

<h4>Learn More</h4>

<ul>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html">Wiki: Publishing Your Archive (Exporting as Static HTML)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing">Wiki: Security Overview (Publishing)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#public_index--public_snapshots--public_add_view">Wiki: Configuration (<code>PUBLIC_INDEX</code>, <code>PUBLIC_SNAPSHOTS</code>, <code>PUBLIC_ADD_VIEW</code>)</a></li>
</ul>

</details>

Caveats

Archiving Private Content

If you're importing pages with private content or URLs containing secret tokens you don't want public (e.g Google Docs, paywalled content, unlisted videos, etc.), you may want to disable some of the extractor methods to avoid leaking that content to 3rd party APIs or the public.

<details>
<summary>Expand to learn about privacy, permissions, and user accounts...</summary>

<pre lang="bash"><code style="white-space: pre-line"># don't save private content to ArchiveBox, e.g.:
archivebox add 'https://docs.google.com/document/d/12345somePrivateDocument'
archivebox add 'https://vimeo.com/somePrivateVideo'

restrict the main index, Snapshot content, and Add Page to authenticated users as-needed:

archivebox config --set PUBLIC_INDEX=False
archivebox config --set PUBLIC_SNAPSHOTS=False
archivebox config --set PUBLIC_ADD_VIEW=False
archivebox manage createsuperuser
</code></pre>

<blockquote>
CAUTION: Assume anyone viewing your archives will be able to see any cookies, session tokens, or private URLs passed to ArchiveBox during archiving.
Make sure to secure your ArchiveBox data and don't share snapshots with others without stripping out sensitive headers and content first.
</blockquote>

<h4>Learn More</h4>

<ul>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">Wiki: Chromium Install (Setting Up a User Profile)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir">Wiki: Configuration (<code>CHROME_USER_DATA_DIR</code>)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#cookies_file">Wiki: Configuration (<code>COOKIES_FILE</code>)</a></li>
</ul>

</details>

Security Risks of Viewing Archived JS

Be aware that malicious archived JS can access the contents of other pages in your archive when viewed. Because the Web UI serves all viewed snapshots from a single domain, they share a request context and typical CSRF/CORS/XSS/CSP protections do not work to prevent cross-site request attacks. See the Security Overview page and Issue #239 for more details.

<details>
<summary>Expand to see risks and mitigations...</summary>

<pre lang="bash"><code style="white-space: pre-line"># visiting an archived page with malicious JS:
https://127.0.0.1:8000/archive/1602401954/example.com/index.html

example.com/index.js can now make a request to read everything from:

https://127.0.0.1:8000/index.html
https://127.0.0.1:8000/archive/*

then example.com/index.js can send it off to some evil server

</code></pre>

<blockquote>
NOTE: Only the <code>wget</code> & <code>dom</code> extractor methods execute archived JS when viewing snapshots, all other archive methods produce static output that does not execute JS on viewing. 
If you are worried about these issues ^ you should disable these extractors using: <code>archivebox config --set SAVE_WGET=False SAVE_DOM=False</code>.
</blockquote>

<h4>Learn More</h4>
<ul>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/issues/239">ArchiveBox Github Issue: #239</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/security/advisories/GHSA-cr45-98w9-gwqx">Security Advisory: <code>CVE-2023-45815</code></a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing">Wiki: Security Overview (Publishing)</a></li>
</ul>

</details>

Working Around Sites that Block Archiving

For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) actively block archiving or bots in general. There are a number of approaches to work around this, and we also provide <a href="https://docs.monadical.com/s/archivebox-consulting-services">consulting services</a> to help here.

<details>
<summary>Click to learn how to set up user agents, cookies, and site logins...</summary>

<ul>
<li>Set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent"><code>CHROME_USER_AGENT</code>, <code>WGET_USER_AGENT</code>, <code>CURL_USER_AGENT</code></a> to impersonate a real browser (by default, ArchiveBox reveals that it's a bot when using the default user agent settings)</li>
<li>Set up a logged-in browser session for archiving using <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile"><code>CHROME_USER_DATA_DIR</code> & <code>COOKIES_FILE</code></a></li>
<li>Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g. 
<code>reddit.com/some/url</code> -> <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
</ul>

In the future we plan on adding support for running JS scripts during archiving to block ads, cookie popups, modals, and fix other issues. Follow here for progress: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/51">Issue #51</a>.

</details>

Saving Multiple Snapshots of a Single URL

ArchiveBox appends a hash with the current date https://example.com#2020-10-24 to differentiate when a single URL is archived multiple times.

<details>
<summary>Click to learn how the <code>Re-Snapshot</code> feature works...</summary>

Because ArchiveBox uniquely identifies snapshots by URL, it must use a workaround to take multiple snapshots of the same URL (otherwise they would show up as a single Snapshot entry). It makes the URLs of repeated snapshots unique by adding a hash with the archive date at the end:

<pre lang="bash"><code style="white-space: pre-line">archivebox add 'https://example.com#2020-10-24'
...
archivebox add 'https://example.com#2020-10-25'
</code></pre>

The <img src="https://user-images.githubusercontent.com/511499/115942091-73c02300-a476-11eb-958e-5c1fc04da488.png" alt="Re-Snapshot Button" height="24px"/> button in the Admin UI is a shortcut for this hash-date multi-snapshotting workaround.

Improved support for saving multiple snapshots of a single URL without this hash-date workaround will be <a href="https://github.com/ArchiveBox/ArchiveBox/issues/179">added eventually</a> (along with the ability to view diffs of the changes between runs).

<h4>Learn More</h4>

<ul>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/issues/179">ArchiveBox Issues: #179</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#explanation-of-buttons-in-the-web-ui---admin-snapshots-list">Wiki: Usage (Explanation of Web UI Buttons)</a></li>
</ul>

</details>

Storage Requirements

Because ArchiveBox is designed to ingest a large volume of URLs with multiple copies of each URL stored by different 3rd-party tools, it can be quite disk-space intensive. There are also some special requirements when using filesystems like NFS/SMB/FUSE.

<details>
<summary>Click to learn more about ArchiveBox's filesystem and hosting requirements...</summary>

<ul>
<li>ArchiveBox can use anywhere from ~1gb per 1000 Snapshots, to ~50gb per 1000 Snapshots, mostly dependent on whether you're saving video/audio using <code>YTDLP_ENABLED=True</code> and whether you lower <code>YTDLP_MAX_SIZE=750m</code>.</li>
<li>Disk usage can be reduced by using a compressed/<a href="https://www.ixsystems.com/blog/ixsystems-and-klara-systems-celebrate-valentines-day-with-a-heartfelt-donation-of-fast-dedupe-to-openzfs-and-truenas/">deduplicated</a> filesystem like <a href="https://www.reddit.com/r/zfs/comments/t9cexx/a_simple_real_world_zfs_compression_speed_an/">ZFS</a>/BTRFS, or by turning off extractors methods you don't need. You can also deduplicate content with a tool like <a href="https://github.com/adrianlopezroche/fdupes"><code>fdupes</code></a> or <a href="https://github.com/pauldreik/rdfind"><code>rdfind</code></a>.
</li>
<li>Don't store large collections on older filesystems like EXT3/FAT as they may not be able to handle more than 50k directory entries in the <code>data/archive/</code> folder.
</li>
<li>Try to keep the <code>data/index.sqlite3</code> file on local drive (not a network mount) or SSD for maximum performance, however the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, you may need to set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid"><code>PUID</code> & <code>PGID</code></a> and <a href="https://github.com/ArchiveBox/ArchiveBox/issues/1304">disable <code>root_squash</code></a> on your fileshare server.
</li>
</ul>

<h4>Learn More</h4>

<ul>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Disk-Layout">Wiki: Usage (Disk Layout)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#output-folder">Wiki: Security Overview (Output-Folder)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#large-archives">Wiki: Usage (Large Archives)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid">Wiki: Configuration (<code>PUID</code> & <code>GUID</code>)</a></li>
<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root">Wiki: Security Overview (Do Not Run as Root)</a></li>
</ul>

</details>

---

Screenshots

Background & Motivation

ArchiveBox aims to enable more of the internet to be saved from deterioration by empowering people to self-host their own archives. The intent is for all the web content you care about to be viewable with common software in 50 - 100 years without needing to run ArchiveBox or other specialized software to replay it.

<details>
<summary>Click to read more about why archiving is important and how to do it ethically...</summary>

Vast treasure troves of knowledge are lost every day on the internet to link rot. As a society, we have an imperative to preserve some important parts of that treasure, just like we preserve our books, paintings, and music in physical libraries long after the originals go out of print or fade into obscurity.

Whether it's to resist censorship by saving news articles before they get taken down or edited, or just to save a collection of early 2010's flash games you loved to play, having the tools to archive internet content enables to you save the stuff you care most about before it disappears.

<div align="center" style="text-align: center">
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/71e36bc5-1c94-44e2-92b6-405fa898c734" width="40%"/> 
Image from <a href="https://perma.cc/">Perma.cc</a>... 
</div>

The balance between the permanence and ephemeral nature of content on the internet is part of what makes it beautiful. I don't think everything should be preserved in an automated fashion--making all content permanent and never removable, but I do think people should be able to decide for themselves and effectively archive specific content that they care about, just like libraries do. Without the work of archivists saving physical books, manuscrips, and paintings we wouldn't have any knowledge of our ancestors' history. I believe archiving the web is just as important to provide the same benefit to future generations.

ArchiveBox's stance is that duplication of other people's content is only ethical if it:

- A. doesn't deprive the original creators of revenue and
- B. is responsibly curated by an individual/institution.

In the U.S., <a href="https://guides.library.oregonstate.edu/copyright/libraries">libraries, researchers, and archivists</a> are allowed to duplicate copyrighted materials under <a href="https://libguides.ala.org/copyright/fairuse">"fair use"</a> for <a href="https://guides.cuny.edu/cunyfairuse/librarians#:~:text=One%20of%20these%20specified%20conditions,may%20be%20liable%20for%20copyright">private study, scholarship, or research</a>. Archive.org's non-profit preservation work is <a href="https://blog.archive.org/2024/03/01/fair-use-in-action-at-the-internet-archive/">covered under fair use</a> in the US, and they properly handle <a href="https://cardozoaelj.com/2015/03/20/use-of-copyright-law-to-take-down-revenge-porn/">unethical content</a>/<a href="https://help.archive.org/help/rights/">DMCA</a>/<a href="https://gdpr.eu/right-to-be-forgotten/#:~:text=An%20individual%20has%20the%20right,that%20individual%20withdraws%20their%20consent.">GDPR</a> removal requests to maintain good standing in the eyes of the law.

As long as you A. don't try to profit off pirating copyrighted content and B. have processes in place to respond to removal requests, many countries allow you to use software like ArchiveBox to ethically and responsibly archive any web content you can view. That being said, ArchiveBox is not liable for how you choose to operate the software. You must research your own local laws and regulations, and get proper legal council if you plan to host a public instance (start by putting your DMCA/GDPR contact info in <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#footer_info"><code>FOOTER_INFO</code></a> and changing your instance's branding using <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#custom_templates_dir"><code>CUSTOM_TEMPLATES_DIR</code></a>).

</details>

Comparison to Other Projects

Check out our community wiki for a list of alternative web archiving tools and orgs.

ArchiveBox gained momentum in the internet archiving industry because it uniquely combines 3 things:

- it's distributed: users own their data instead of entrusting it to one big central provider
- it's future-proof: saving in multiple formats and extracting out raw TXT, PNG, PDF, MP4, etc. files
- it's extensible: with powerful APIs, flexible storage, and a big community adding new extractors regularly

<details>
<summary>Expand for a more direct comparison to Archive.org and specific open-source alternatives...</summary>

ArchiveBox tries to be a robust, set-and-forget archiving solution suitable for archiving RSS feeds, bookmarks, or your entire browsing history (beware, it may be too big to store), including private/authenticated content that you wouldn't otherwise share with a centralized service like Archive.org.

<h3>Comparison With Centralized Public Archives</h3>

Not all content is suitable to be archived on a centralized, publicly accessible platform. Archive.org doesn't offer the ability to save things behind login walls for good reason, as the content may not have been intended for a public audience. ArchiveBox exists to fill that gap by letting everyone save what they have access to on an individual basis, and to encourage decentralized archiving that's less succeptible to censorship or natural disasters.

By having users store their content locally or within their organizations, we can also save much larger portions of the internet than a centralized service has the disk capacity to handle. The eventual goal is to work towards federated archiving where users can share portions of their collections with each other, and with central archives on a case-by-case basis.

<h3>Comparison With Other Self-Hosted Archiving Options</h3>

ArchiveBox differentiates itself from similar self-hosted projects by providing both a comprehensive CLI interface for managing your archive, a Web UI that can be used either independently or together with the CLI, and a simple on-disk data format that can be used without either.

If you want better fidelity for very complex interactive pages with heavy JS/streams/API requests, check out ArchiveWeb.page and ReplayWeb.page.

If you want more bookmark categorization and note-taking features, check out Memex, Hoarder, LinkWarden, Archivy, or LinkAce.

If you need more advanced recursive spider/crawling ability beyond --depth=1, check out Browsertrix, Photon, or Scrapy and pipe the outputted URLs into ArchiveBox.

For more alternatives, see our list here...

ArchiveBox is neither the highest fidelity nor the simplest tool available for self-hosted archiving, rather it's a jack-of-all-trades that tries to do most things well by default. We encourage you to try these other tools made by our friends if ArchiveBox isn't suited to your needs.

</details>

Internet Archiving Ecosystem

<details>
<summary>Our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community">Community Wiki</a> strives to be a comprehensive index of the web archiving industry...</summary>

- Community Wiki
- Web Archiving Software
_List of ArchiveBox alternatives and open source projects in the internet archiving space._
- Awesome-Web-Archiving Lists
_Community-maintained indexes of archiving tools and institutions like iipc/awesome-web-archiving._
- Reading List
_Articles, posts, and blogs relevant to ArchiveBox and web archiving in general._
- Communities
_A collection of the most active internet archiving communities and initiatives._
- Check out the ArchiveBox Roadmap and Changelog
- Learn why archiving the internet is important by reading the "On the Importance of Web Archiving" blog post.
- Reach out to me for questions and comments via @ArchiveBoxApp or @theSquashSH on Twitter

</details>

Need help building a custom archiving solution?

✨ Hire the team that built Archivebox to solve archiving for your org. (@ArchiveBoxApp)

Documentation

We use the ArchiveBox GitHub Wiki for documentation.

There is also a mirror available on <a href="https://archivebox.readthedocs.io/en/latest/">Read the Docs</a> (though it's sometimes outdated).

✏️ You can submit docs changes & suggestions in our dedicated repo ArchiveBox/docs.

Getting Started

- Quickstart
- Install
- Docker
- Usage
- Configuration
- Supported Sources
- Supported Outputs
- Scheduled Archiving

Advanced

- Security Overview
- Cookies & Sessions Setup (archiving sites that require logins)
- Setting up the Search Backends (choosing ripgrep, Sonic, or FTS5)
- Setting up Local/Remote Storages (S3/B2/Google Drive/SMB/NFS/etc.)
- Setting up Authentication & Permissions (SSO/LDAP/OAuth/API Keys/etc.)
- Publishing Your Archive (sharing your archive server with others)
- Chromium Install Options (installing and configuring ArchiveBox's Chrome)
- Upgrading or Merging Archives
- Troubleshooting

Developers

- Developer Documentation
- Python API
- REST API (alpha)

More Info

- Bug Tracker
- Roadmap
- Changelog
- Donations
- Background & Motivation
- Web Archiving Community

---

ArchiveBox Development

All contributions to ArchiveBox are welcomed! Check our issues and Roadmap for things to work on, and please open an issue to discuss your proposed implementation before working on things! Otherwise we may have to close your PR if it doesn't align with our roadmap.

For low hanging fruit / easy first tickets, see: <a href="https://github.com/ArchiveBox/ArchiveBox/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22help+wanted%22">ArchiveBox/Issues #good first ticket #help wanted</a>.

Python API Documentation: https://docs.archivebox.io/en/dev/archivebox.html#module-archivebox.main

Internal Architecture Diagrams: https://github.com/ArchiveBox/ArchiveBox/wiki/ArchiveBox-Architecture-Diagrams

Setup the dev environment

<details><summary>Click to expand...</summary>

#### 1. Setup the monorepo

First make sure you have uv installed: https://docs.astral.sh/uv/getting-started/installation/

bash

git clone https://github.com/ArchiveBox/monorepo
cd monorepo
./bin/setup.sh
activate the monorepo venv, then cd into archivebox

source .venv/bin/activate
cd archivebox

Repos included in monorepo setup:

- https://github.com/ArchiveBox/abxbus
- https://github.com/ArchiveBox/abx-pkg
- https://github.com/ArchiveBox/abx-plugins
- https://github.com/ArchiveBox/abx-dl
- https://github.com/ArchiveBox/ArchiveBox

#### 2. Option A: Install the Python, JS, and system dependencies directly on your machine

bash

Install ArchiveBox runtime dependencies

mkdir -p data && cd data
archivebox init
archivebox install         # detect and install all extractor dependencies
Run the development server w/ autoreloading (but no bg workers)

archivebox server --debug --reload 0.0.0.0:8000
Run the production server (with bg workers but no autoreloading)

archivebox server 0.0.0.0:8000

#### 2. Option B: Build the docker container and use that for development instead

bash

Optional: develop via docker by mounting the code dir into the container

if you edit e.g. ./archivebox/core/models.py on the docker host, runserver

inside the container will reload and pick up your changes

./bin/build_docker.sh dev
docker run -it -v $PWD/data:/data archivebox/archivebox:dev init --install
Run the development server w/ autoreloading (but no bg workers)

docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --debug --reload 0.0.0.0:8000
Run the production server (with bg workers but no autoreloading)

docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server
(remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)

When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.

</details>

Common development tasks

See the ./bin/ folder and read the source of the bash scripts within.
You can also run all these in Docker. For more examples see the GitHub Actions CI/CD tests that are run: .github/workflows/*.yaml.

#### Run in DEBUG mode

<details><summary>Click to expand...</summary>

bash

set up persistent DEBUG=True for all runs

archivebox config --set DEBUG=True
OR you can run a dev server with DEBUG=True in a few ways:

archivebox server --debug --reload 0.0.0.0:8000
or

archivebox server --debug 0.0.0.0:8000
or

env DEBUG=True daphne -b 0.0.0.0 -p 8000 archivebox.core.asgi:application

https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running

</details>

#### Install and run a specific GitHub branch

<details><summary>Click to expand...</summary>

##### Use a Pre-Built Image

If you're looking for the latest dev Docker image, it's often available pre-built on Docker Hub, simply pull and use archivebox/archivebox:dev.

bash

docker pull archivebox/archivebox:dev
docker run archivebox/archivebox:dev version
verify the BUILD_TIME and COMMIT_HASH in the output are recent

##### Build Branch from Source

You can also build and run any branch yourself from source, for example to build & use dev locally:

bash

docker-compose.yml:

services:
    archivebox:
        image: archivebox/archivebox:dev
        build: 'https://github.com/ArchiveBox/ArchiveBox.git#dev'
        ...
or with plain Docker:

docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
docker run -it -v $PWD:/data archivebox:dev init
or with pip:

pip install 'git+https://github.com/pirate/ArchiveBox@dev'
npm install 'git+https://github.com/ArchiveBox/ArchiveBox.git#dev'
archivebox install

</details>

#### Run the linters / tests

<details><summary>Click to expand...</summary>

bash

./bin/lint.sh
./bin/test.sh

(uses ruff, pyright, ty, and pytest -s)

</details>

#### Make DB migrations, enter Django shell, other dev helper commands

<details><summary>Click to expand...</summary>

bash

generate the database migrations after changes to models.py

cd archivebox/
./manage.py makemigrations
enter a python shell or a SQL shell

cd path/to/test/data/
archivebox shell
archivebox manage dbshell
generate a graph of the ORM models

brew install graphviz
pip install pydot graphviz
archivebox manage graph_models -a -o orm.png
open orm.png
list all models with field db info and methods

archivebox manage list_model_info --all --signature --db-type --field-class
print all django settings

archivebox manage print_settings
archivebox manage print_settings --format=yaml    # pip install pyyaml
autogenerate an admin.py from given app models

archivebox manage admin_generator core > core/admin.py
dump db data to a script that re-populates it

archivebox manage dumpscript core > scripts/testdata.py
archivebox manage reset core
archivebox manage runscript testdata
resetdb and clear all data!

archivebox manage reset_db
use django-tui to interactively explore commands

uv pip install django-tui
ensure django-tui is in INSTALLED_APPS: core/settings.py

archivebox manage tui

- https://django-extensions.readthedocs.io/en/latest/command_extensions.html
- https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running
- https://github.com/anze3db/django-tui (explore manage.py commands as TUI)
- https://github.com/bloomberg/memray (advanced python profiler)
- https://github.com/laixintao/flameshow (display flamegraphs in terminal)
- https://github.com/taliraj/django-migrations-tui (explore migrations as TUI)

</details>

#### Contributing a new extractor

Extractors are maintained in a separate repo here: https://github.com/ArchiveBox/abx-plugins (included in monorepo setup).

Copy a similar plugin as a template to modify, then open a new PR to add it in that repo.

#### Build the docs, pip package, and docker image

<details><summary>Click to expand...</summary>

(Normally CI takes care of this, but these scripts can be run to do it manually)

bash

./bin/build.sh
or individually:

./bin/build_docs.sh
./bin/build_pip.sh
./bin/build_docker.sh

</details>

#### Roll a release

<details><summary>Click to expand...</summary>

(Normally CI takes care of this, but these scripts can be run to do it manually)

bash

./bin/release.sh
or individually:

./bin/release_docs.sh
./bin/release_pip.sh
./bin/release_docker.sh

</details>

---

Repository: ArchiveBox/ArchiveBox

Stars: 27248

CLAUDE.md Copy Chapter

Claude Code Development Guide for ArchiveBox

Quick Start Copy Chapter

Set up dev environment (always use uv, never pip directly)

Run tests as non-root user (required - ArchiveBox always refuses to run as root)

Development Environment Setup Copy Chapter

Prerequisites Copy Section

Install Dependencies Copy Section

Activate Virtual Environment Copy Section

Common Gotchas Copy Section

Code Style Guidelines Copy Chapter

Naming Conventions for Grep-ability Copy Section

Filesystem migration methods - all start with fs_

Logging methods - ALL must start with log_ or _log

Minimize Unique Names and Data Structures Copy Section

Binary has overrides field

Binary reuses the same field name and structure

Don't invent new names like custom_bin_cmds, binary_overrides, etc.

Testing Copy Chapter

CRITICAL: Never Run as Root Copy Section

Run all migration tests

Run specific test file

Run single test

Test File Structure Copy Section

Test Writing Standards Copy Section

Example Test Pattern Copy Section

Testing Gotchas Copy Section

... etc

Plugin Testing & Code Coverage Copy Section

Run plugin tests with coverage (both Python + JavaScript)

View coverage reports

Or individual reports:

Correct production paths:

Crawl: DATA_DIR/users/{username}/crawls/YYYYMMDD/example.com/{crawl-id}/{plugin}/

Snapshot: DATA_DIR/users/{username}/snapshots/YYYYMMDD/example.com/{snapshot-uuid}/{plugin}/

Database Migrations Copy Chapter

Generate and Apply Migrations Copy Section

Generate migrations (run from archivebox subdirectory)

Apply migrations to test database

Schema Versions Copy Section

Testing a Migration Path Copy Section

Squashed Migrations Copy Section

The squashed migration replaces these - all must be recorded

... all 52 migrations from 0023-0074 ...

Migration Strategy Copy Section

Migration Gotchas Copy Section

Plugin System Architecture Copy Chapter

Plugin Dependency Rules Copy Section

Chrome-Dependent Plugins Copy Section

Finding Chrome-Dependent Plugins Copy Section

Find all files containing "chrom" (case-insensitive)

Or get just the plugin names

Architecture Notes Copy Chapter

Crawl Model (0.9.x) Copy Section

Code Coverage Copy Chapter

Overview Copy Section

Install Coverage Tools Copy Section

Running with Coverage Copy Section

Run tests with coverage

Or run specific test file

Start dev server with coverage tracking

Or CLI commands

Option 1: Use coverage run wrapper

Option 2: Set environment variable (tracks everything)

Now all Python processes will track coverage

Viewing Coverage Copy Section

Combine all parallel coverage data

View summary

View detailed report with missing lines

View specific file

Generate JSON report

View the JSON

Find files with low coverage

Find completely uncovered files (dead code candidates)

Get missing lines for a specific file

Generate interactive HTML report

Open in browser

Isolated Runs Copy Section

CLAUDE.md

Quick Start

Development Environment Setup

Prerequisites

Install Dependencies

Activate Virtual Environment

Common Gotchas

Code Style Guidelines

Naming Conventions for Grep-ability

Minimize Unique Names and Data Structures

Testing

CRITICAL: Never Run as Root

Test File Structure

Test Writing Standards

Example Test Pattern

Testing Gotchas

Plugin Testing & Code Coverage

Database Migrations

Generate and Apply Migrations

Schema Versions

Testing a Migration Path

Squashed Migrations

Migration Strategy

Migration Gotchas

Plugin System Architecture

Plugin Dependency Rules

Chrome-Dependent Plugins

Finding Chrome-Dependent Plugins

Architecture Notes

Crawl Model (0.9.x)

Code Coverage

Overview

Install Coverage Tools

Running with Coverage

Viewing Coverage

Isolated Runs

Finding Dead Code

Tips

Debugging Tips

Check Migration State

Check Table Schema

Verbose Test Output

Kill Zombie Chrome Processes

README.md

Key Features

🤝 Professional Integration

Usage

make sure you have `docker-compose.yml` from the Quickstart instructions first

Input Formats: How to pass URLs into ArchiveBox for saving