# Technical Documentation: benbjohnson/litestream

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/benbjohnson/litestream)

### Overview (`README.md`)

Litestream



[](https://hub.docker.com/r/litestream/litestream/)
==========

Litestream is a standalone disaster recovery tool for SQLite. It runs as a
background process and safely replicates changes incrementally to another file
or S3. Litestream only communicates with SQLite through the SQLite API so it
will not corrupt your database.

If you need support or have ideas for improving Litestream, please visit
[GitHub Issues](https://github.com/benbjohnson/litestream/issues).
Please visit the [Litestream web site](https://litestream.io) for installation
instructions and documentation.

If you find this project interesting, please consider starring the project on
GitHub.

Source database changes
-----------------------

When Litestream begins replicating a database, it creates an internal
`_litestream_lock` table in the source SQLite database. This is expected and
safe to leave in place. Litestream uses the table to acquire SQLite's write
lock while coordinating synchronization around WAL checkpoints. The writes
occur in transactions that are rolled back, so Litestream does not leave rows
in the table.

The table is part of the source database, not a separate table in the replica
destination. Because the source schema is backed up, restored databases also
contain it. Its creation changes the source database schema and can change its
page count or bytes, so a database under Litestream should not be expected to
remain byte-identical to its pre-Litestream state.

Do not drop `_litestream_lock` while Litestream is running. Synchronization
around a checkpoint can fail until Litestream reinitializes the database.
Restarting Litestream recreates the missing table automatically.

Contributing
------------

We welcome bug reports, fixes, and patches! Please see our [Contributing Guide](CONTRIBUTING.md) for details on how to contribute.

Security
--------

Please do not open a public issue for security vulnerabilities. Report them
privately through GitHub's [private vulnerability reporting](https://github.com/benbjohnson/litestream/security/advisories/new),
which keeps the report visible only to you and the maintainers until a fix is
released. See our [Security Policy](SECURITY.md) for what to include and what to
expect.

Acknowledgements
----------------

I want to give special thanks to individuals who invest much of their time and
energy into the project to help make it better:

- Thanks to [Cory LaNou](https://twitter.com/corylanou) for giving early feedback and testing when Litestream was still pre-release.
- Thanks to [Michael Lynch](https://github.com/mtlynch) for digging into issues and contributing to the documentation.
- Thanks to [Kurt Mackey](https://twitter.com/mrkurt) for feedback and testing.
- Thanks to [Sam Weston](https://twitter.com/cablespaghetti) for figuring out how to run Litestream on Kubernetes and writing up the docs for it.
- Thanks to [Rafael](https://github.com/netstx) & [Jungle Boogie](https://github.com/jungle-boogie) for helping to get OpenBSD release builds working.
- Thanks to [Simon Gottschlag](https://github.com/simongottschlag), [Marin](https://github.com/supermarin),[Victor Björklund](https://github.com/victorbjorklund), [Jonathan Beri](https://twitter.com/beriberikix) [Yuri](https://github.com/yurivish), [Nathan Probst](https://github.com/nprbst), [Yann Coleu](https://github.com/yanc0), and [Nicholas Grilly](https://twitter.com/ngrilly) for frequent feedback, testing, & support.

Huge thanks to fly.io for their support and for contributing credits for testing and development!

---

### Overview (`grafana/README.md`)

# Litestream Grafana Dashboard

This directory contains a Grafana dashboard for monitoring Litestream metrics.

## Prerequisites

1. Litestream configured with metrics endpoint enabled in `litestream.yml`:

   ```yaml
   addr: ":9090"
   ```

2. Prometheus configured to scrape Litestream metrics:

   ```yaml
   scrape_configs:
     - job_name: 'litestream'
       static_configs:
         - targets: ['localhost:9090']
   ```

3. Grafana with Prometheus data source configured

## Installation

1. Open Grafana and navigate to **Dashboards** → **Import**
2. Upload the `litestream-dashboard.json` file or paste its contents
3. Select your Prometheus data source
4. Click **Import**

## Metrics Included

The dashboard monitors the following key metrics:

- **Database & WAL Size**: Current size of the database and Write-Ahead Log
- **Total WAL Bytes Written**: Cumulative bytes written to shadow WAL
- **Sync Operations**: Rate of sync operations and any sync errors
- **Sync Duration**: Time spent syncing shadow WAL
- **Checkpoint Operations**: Rate of checkpoint operations by mode
- **Checkpoint Errors**: Any checkpoint errors that occur
- **Transaction ID**: Current transaction ID for each database
- **Replica Operations**: Operations performed by replica type (GET/PUT)
- **Replica Throughput**: Bytes transferred by replica operations

## Configuration

The dashboard uses template variables:

- `datasource`: Select your Prometheus data source
- `job`: Select the Prometheus job name (defaults to "litestream")

## Support

For issues or improvements to this dashboard, please open an issue at:
<https://github.com/benbjohnson/litestream/issues>

---

### Overview (`scripts/README.md`)

# Utility Scripts

Utility scripts for Litestream testing and distribution.

## Overview

This directory contains utility scripts for post-test analysis and packaging. All long-running soak tests have been migrated to Go integration tests in `tests/integration/`.

> **Note:** For all soak tests (2-8 hours), see the Go-based test suite in [tests/integration/](../tests/integration/README.md). The bash soak tests have been migrated to Go for better maintainability and cross-platform support

## Prerequisites

```bash
go build -o bin/litestream ./cmd/litestream
go build -o bin/litestream-test ./cmd/litestream-test
```

## Available Scripts

### analyze-test-results.sh

Post-test analysis tool for examining overnight test results.

```bash
./scripts/analyze-test-results.sh /tmp/litestream-overnight-<timestamp>
```

**Analyzes:**
1. Test duration and timeline
2. Compaction statistics
3. Checkpoint frequency
4. Error analysis and categorization
5. Performance metrics
6. Database growth patterns
7. Replica file statistics
8. Final validation results

**Output:**
- Analysis report written to `<test-dir>/analysis-report.txt`
- Console summary with key findings
- Error categorization and counts
- Performance statistics

**What it Reports:**
- Total test duration
- Number of compactions by interval
- Checkpoint count and frequency
- Error types and severity
- Database size growth
- WAL file patterns
- Replica size and file counts
- Success/failure summary

**Use Cases:**
- Post-overnight-test analysis
- Comparing test runs
- Identifying performance trends
- Debugging test failures
- Documenting test results

### setup-homebrew-tap.sh

Homebrew tap setup script for packaging and distribution.

```bash
./scripts/setup-homebrew-tap.sh
```

**Purpose:** Automates Homebrew tap setup for Litestream distribution. Not a test script per se, but part of the release process.

## Usage

### Analyzing Test Results

```bash
ls /tmp/litestream-overnight-* -dt | head -1

./scripts/analyze-test-results.sh $(ls /tmp/litestream-overnight-* -dt | head -1)
```

## Test Duration Guide

| Duration | Use Case | Test Type | Expected Results |
|----------|----------|-----------|------------------|
| 5 minutes | CI/CD smoke test | Go integration tests | Basic functionality |
| 30 minutes | Short integration | Go integration tests | Pattern detection |
| 2-8 hours | Soak testing | Go soak tests (local only) | Full validation |

> **Note:** All soak tests are now Go-based in `tests/integration/`. See [tests/integration/README.md](../tests/integration/README.md) for details on running comprehensive, MinIO, and overnight S3 soak tests.

## Monitoring and Debugging

### Real-time Monitoring

All tests create timestamped directories in `/tmp/`:

```bash
LATEST=$(ls /tmp/litestream-* -dt | head -1)
tail -f $LATEST/logs/monitor.log
tail -f $LATEST/logs/litestream.log
```

### Key Metrics to Watch

**Database Growth:**
- Should grow steadily
- WAL file size should cycle (grow, checkpoint, reset)

**Replica Statistics:**
- Snapshot count should increase over time
- LTX file count should grow then stabilize (compaction)
- Replica size should be similar to database size

**Operations:**
- Compactions should occur at scheduled intervals
- Checkpoints should happen regularly
- Sync operations should complete successfully

**Errors:**
- Should be minimal or zero
- Transient errors OK if recovered
- Persistent errors indicate issues

### Common Issues

#### Test Fails to Start

Check binaries:
```bash
ls -la bin/litestream bin/litestream-test
```

Rebuild if needed:
```bash
go build -o bin/litestream ./cmd/litestream
go build -o bin/litestream-test ./cmd/litestream-test
```

#### S3 Test Fails

Verify credentials:
```bash
aws s3 ls s3://$S3_BUCKET/
```

Check environment variables:
```bash
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
echo $S3_BUCKET
```

#### High Error Counts

Check log for error details:
```bash
grep -i error /tmp/litestream-*/logs/litestream.log | head -20
```

#### Validation Fails

Compare databases manually:
```bash
sqlite3 /tmp/litestream-*/test.db "SELECT COUNT(*) FROM test_data"
sqlite3 /tmp/litestream-*/restored.db "SELECT COUNT(*) FROM test_data"
```

### Stopping Tests Early

Go tests can be interrupted with Ctrl+C. They will cleanup gracefully via defer statements.

## Test Artifacts

All tests create timestamped directories with comprehensive artifacts:

```
/tmp/litestream-overnight-<timestamp>/
├── logs/
│   ├── litestream.log      # Litestream replication log
│   ├── load.log            # Load generator log
│   ├── monitor.log         # Real-time monitoring log
│   ├── populate.log        # Initial population log
│   └── validate.log        # Final validation log
├── test.db                 # Source database
├── test.db-wal             # Write-ahead log
├── test.db-shm             # Shared memory file
├── replica/                # Replica directory (file tests)
│   └── ltx/               # LTX files
└── restored.db            # Restored database for validation
```

## Integration with Go Tests

These utility scripts complement the Go integration test suite:

**Test Locations:**
- `tests/integration/` → All integration and soak tests (Go-based)
- `cmd/litestream-test/scripts/` → Scenario and debugging tests (bash, being phased out)
- `scripts/` → Utilities only (this directory)

**Testing Workflow:**
1. Run quick integration tests during development
2. Run full integration test suite before major changes
3. Run soak tests (2-8h) locally before releases: `TestComprehensiveSoak`, `TestMinIOSoak`, `TestOvernightS3Soak`
4. Analyze results with `analyze-test-results.sh`

## Related Documentation

- [Go Integration Tests](../tests/integration/README.md) - Complete Go-based test suite including soak tests
- [litestream-test CLI Tool](../cmd/litestream-test/README.md) - Testing harness documentation
- [Scenario Test Scripts](../cmd/litestream-test/scripts/README.md) - Focused test scenarios
- [S3 Retention Testing](../cmd/litestream-test/S3-RETENTION-TESTING.md) - S3-specific testing

---

### Overview (`_examples/library/README.md`)

# Litestream Library Usage Examples

These examples demonstrate how to use Litestream as a Go library instead of as a
standalone CLI tool.

## API Stability Warning

The Litestream library API is not considered stable and may change between
versions. The CLI interface is more stable for production use. Use the library
API at your own risk, and pin to specific versions.

**Note (POSIX platforms):** All POSIX platforms (Linux, macOS, BSD, etc.) use
per-process locks for SQLite, not per-handle locks. If you open the same
database with two different SQLite driver implementations in the same process
and close one of them, you can hit locking issues. You **must** use
`modernc.org/sqlite` for your app since Litestream uses it internally.

## Important Constraints

When using Litestream as a library, be aware of these critical requirements:

1. **Required Driver**: You must use `modernc.org/sqlite`. Litestream uses this
   driver internally, and mixing drivers causes lock conflicts on POSIX systems.

2. **Lifecycle Management**: You cannot call `litestream.DB.Close()` or
   `Replica.Stop(true)` while your application still has open database
   connections. Either close all your app's database connections first, or only
   close Litestream when your process is shutting down.

3. **PRAGMA Configuration**: Use DSN parameters (e.g.,
   `?_pragma=busy_timeout(5000)`) instead of `PRAGMA` statements via
   `ExecContext`. An `sql.DB` is a connection pool, and `ExecContext` only
   applies the PRAGMA to one random connection from the pool.

## Examples

### Basic (File Backend)

The simplest example using local filesystem replication.

```bash
cd basic
go run main.go
```

This creates:

- `myapp.db` - The SQLite database
- `replica/` - Directory containing replicated LTX files

### S3 Backend

A more complete example showing the restore-on-startup pattern with S3.

```bash
cd s3

# Set required environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export LITESTREAM_BUCKET="your-bucket-name"
export LITESTREAM_PATH="databases/myapp"  # optional, defaults to "litestream"
export AWS_REGION="us-east-1"             # optional, defaults to "us-east-1"

go run main.go
```

This example:

1. Checks if the local database exists
2. If not, attempts to restore from S3
3. Starts background replication to S3
4. Inserts sample data every 2 seconds
5. Gracefully shuts down on Ctrl+C

## Core API Pattern

```go
import (
    "context"
    "database/sql"
    "github.com/benbjohnson/litestream"
    "github.com/benbjohnson/litestream/file"  // or s3, gs, abs, etc.
    _ "modernc.org/sqlite"
)

// 1. Create database wrapper
db := litestream.NewDB("/path/to/db.sqlite")

// 2. Create replica client
client := file.NewReplicaClient("/path/to/replica")
// OR from URL:
// client, _ := litestream.NewReplicaClientFromURL("s3://bucket/path")

// 3. Attach replica to database
replica := litestream.NewReplicaWithClient(db, client)
db.Replica = replica
client.Replica = replica // file backend only; preserves ownership/permissions

// 4. Create compaction levels (L0 required, plus at least one more)
levels := litestream.CompactionLevels{
    {Level: 0},
    {Level: 1, Interval: 10 * time.Second},
}

// 5. Create Store to manage DB and background compaction
store := litestream.NewStore([]*litestream.DB{db}, levels)

// 6. Open Store (opens all DBs, starts background monitors)
if err := store.Open(ctx); err != nil { ... }
defer store.Close(context.Background())

// 7. Open your app's SQLite connection for normal database operations
// Use DSN params for PRAGMAs to ensure they apply to all connections in the pool
dsn := fmt.Sprintf("file:%s?_pragma=busy_timeout(5000)&_pragma=journal_mode(wal)", "/path/to/db.sqlite")
sqlDB, err := sql.Open("sqlite", dsn)
if err != nil { ... }
```

## Restore Pattern

```go
// Create replica without database for restore
replica := litestream.NewReplicaWithClient(nil, client)

opt := litestream.NewRestoreOptions()
opt.OutputPath = "/path/to/restored.db"
// Optional: point-in-time restore
// opt.Timestamp = time.Now().Add(-1 * time.Hour)

if err := replica.Restore(ctx, opt); err != nil {
    if errors.Is(err, litestream.ErrTxNotAvailable) || errors.Is(err, litestream.ErrNoSnapshots) {
        // No backup available, create fresh database
    }
    return err
}
```

## Supported Backends

- `file` - Local filesystem
- `s3` - AWS S3 and S3-compatible storage
- `gs` - Google Cloud Storage
- `abs` - Azure Blob Storage
- `oss` - Alibaba Cloud OSS
- `sftp` - SFTP servers
- `nats` - NATS JetStream
- `webdav` - WebDAV servers

## Key Configuration Options

### Store Settings

```go
store.SnapshotInterval  = 24 * time.Hour  // How often to create snapshots
store.SnapshotRetention = 24 * time.Hour  // How long to keep snapshots
store.L0Retention       = 5 * time.Minute // How long to keep L0 files after compaction
```

### DB Settings

```go
db.MonitorInterval    = 1 * time.Second   // How often to check for changes
db.CheckpointInterval = 1 * time.Minute   // Time-based checkpoint interval
db.MinCheckpointPageN = 1000              // Page threshold for checkpoint
db.BusyTimeout        = 1 * time.Second   // SQLite busy timeout
```

### Replica Settings

```go
replica.SyncInterval   = 1 * time.Second  // Time between syncs
replica.MonitorEnabled = true             // Auto-sync in background
```

## Resources

- [Litestream Documentation](https://litestream.io)
- [GitHub Repository](https://github.com/benbjohnson/litestream)

---

### Overview (`cmd/litestream-test/README.md`)

# litestream-test

A CLI testing harness for Litestream that provides tools for database population, load generation, and replication validation.

## Overview

`litestream-test` is a purpose-built tool for testing Litestream's replication functionality across various scenarios. It provides commands for:

- Quickly populating databases to specific sizes
- Generating continuous load with configurable patterns
- Shrinking databases to test compaction scenarios
- Validating replication integrity after restore

## Installation

```bash
go build -o bin/litestream-test ./cmd/litestream-test
```

## Commands

### populate

Quickly populate a database to a target size with structured test data.

```bash
litestream-test populate -db <path> [options]
```

**Options:**
- `-db` - Database path (required)
- `-target-size` - Target database size (default: "100MB", examples: "1GB", "500MB", "50MB")
- `-row-size` - Average row size in bytes (default: 1024)
- `-batch-size` - Rows per transaction (default: 1000)
- `-table-count` - Number of tables to create (default: 1)
- `-index-ratio` - Percentage of columns to index, 0.0-1.0 (default: 0.2)
- `-page-size` - SQLite page size in bytes (default: 4096)

**Examples:**
```bash
litestream-test populate -db /tmp/test.db -target-size 1GB
litestream-test populate -db /tmp/test.db -target-size 50MB -batch-size 10000
litestream-test populate -db /tmp/test.db -target-size 1.5GB -page-size 4096
```

**Use Cases:**
- Creating test databases of specific sizes
- Testing SQLite lock page boundary (1GB with 4KB pages)
- Generating initial data before replication tests

### load

Generate continuous write and read load on a database with configurable patterns.

```bash
litestream-test load -db <path> [options]
```

**Options:**
- `-db` - Database path (required, must exist)
- `-write-rate` - Writes per second (default: 100)
- `-duration` - How long to run (default: 1m, examples: "30s", "5m", "2h", "8h")
- `-pattern` - Write pattern (default: "constant")
  - `constant` - Steady write rate
  - `burst` - Periodic bursts of activity
  - `random` - Random write intervals
  - `wave` - Sinusoidal pattern simulating varying load
- `-payload-size` - Size of each write in bytes (default: 1024)
- `-read-ratio` - Read/write ratio, 0.0-1.0 (default: 0.2)
- `-workers` - Number of concurrent workers (default: 1)

**Examples:**
```bash
litestream-test load -db /tmp/test.db -write-rate 50 -duration 5m
litestream-test load -db /tmp/test.db -write-rate 100 -duration 2h -pattern wave
litestream-test load -db /tmp/test.db -write-rate 200 -duration 8h -workers 4 -pattern burst
```

**Use Cases:**
- Stress testing replication under sustained load
- Testing checkpoint behavior with various patterns
- Simulating production workloads for overnight tests
- Testing concurrent operations with multiple workers

### shrink

Shrink a database by deleting data, useful for testing compaction scenarios.

```bash
litestream-test shrink -db <path> [options]
```

**Use Cases:**
- Testing database shrinkage and compaction
- Simulating data deletion scenarios
- Testing retention cleanup behavior

### validate

Validate that a replica can be restored and matches the source database.

```bash
litestream-test validate [options]
```

**Options:**
- `-source-db` - Original database path (required)
- `-replica-url` - Replica URL to validate (e.g., "file:///path", "s3://bucket/path")
- `-restored-db` - Path for restored database (default: source-db + ".restored")
- `-check-type` - Type of validation (default: "quick")
  - `quick` - Fast row count comparison
  - `integrity` - SQLite PRAGMA integrity_check
  - `checksum` - Full database checksum comparison
  - `full` - All validation types
- `-ltx-continuity` - Check LTX file continuity (default: false)
- `-config` - Litestream config file path (alternative to replica-url)

**Examples:**
```bash
litestream-test validate -source-db /tmp/test.db -replica-url file:///tmp/replica
litestream-test validate -source-db /tmp/test.db -replica-url s3://bucket/path -check-type full
litestream-test validate -source-db /tmp/test.db -config /tmp/litestream.yml -ltx-continuity
```

**Use Cases:**
- Verifying replication integrity after tests
- Testing restore functionality
- Validating data consistency across replicas
- Checking LTX file continuity

### version

Show version information.

```bash
litestream-test version
```

## Usage Patterns

### Basic Test Workflow

```bash
litestream-test populate -db /tmp/test.db -target-size 100MB

litestream replicate /tmp/test.db file:///tmp/replica &
LITESTREAM_PID=$!

litestream-test load -db /tmp/test.db -duration 5m -write-rate 50

kill $LITESTREAM_PID
wait

litestream-test validate -source-db /tmp/test.db -replica-url file:///tmp/replica
```

### Overnight Test Pattern

```bash
litestream-test populate -db /tmp/test.db -target-size 100MB

litestream replicate -config litestream.yml &

litestream-test load -db /tmp/test.db \
  -duration 8h \
  -write-rate 100 \
  -pattern wave \
  -workers 4
```

### Large Database with Lock Page Testing

```bash
litestream-test populate -db /tmp/test.db \
  -target-size 1.5GB \
  -page-size 4096 \
  -batch-size 10000

litestream replicate /tmp/test.db s3://bucket/path &

litestream-test validate -source-db /tmp/test.db \
  -replica-url s3://bucket/path \
  -check-type full
```

## Integration with Test Scripts

The `litestream-test` tool is used by all scripts in the `scripts/` directory. These scripts orchestrate full test scenarios:

- `scripts/*.sh` - Use litestream-test for database operations
- `scripts/verify-test-setup.sh` - Checks that litestream-test is built
- See `scripts/README.md` for detailed test scenario documentation

## Related Documentation

- [Test Scripts Documentation](./scripts/README.md) - Comprehensive test scenarios
- [S3 Retention Testing](./S3-RETENTION-TESTING.md) - S3-specific test documentation
- [Top-level Integration Scripts](../../scripts/README.md) - Long-running test documentation

## Development

### Building

```bash
go build -o bin/litestream-test ./cmd/litestream-test
```

### Testing

```bash
go test ./cmd/litestream-test/...
```

### Adding New Commands

1. Create a new file in `cmd/litestream-test/` (e.g., `mycommand.go`)
2. Implement the command struct and Run method
3. Add command to switch statement in `main.go`
4. Update this README with command documentation

---

### Overview (`packages/python/README.md`)

# litestream-vfs

Litestream VFS extension for SQLite — distributed as a Python wheel.

This package bundles the [Litestream](https://litestream.io) VFS shared library
so you can load it directly into a Python `sqlite3` connection.

## Installation

```bash
pip install litestream-vfs
```

## Usage

```python
import sqlite3
import litestream_vfs

conn = sqlite3.connect(":memory:")
litestream_vfs.load(conn)
```

To get the path to the shared library (for use with other SQLite bindings):

```python
path = litestream_vfs.loadable_path()
```

## Platform Support

| Platform | Architecture |
|----------|-------------|
| Linux | x86_64, aarch64 |
| macOS | x86_64, arm64 |

## License

Apache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).

---

### Overview (`packages/ruby/README.md`)

# litestream-vfs

Litestream VFS extension for SQLite — distributed as a Ruby gem.

This gem bundles the [Litestream](https://litestream.io) VFS shared library
so you can load it directly into a SQLite connection.

## Installation

```bash
gem install litestream-vfs
```

Or add to your Gemfile:

```ruby
gem "litestream-vfs"
```

## Usage

```ruby
require "litestream_vfs"

db = SQLite3::Database.new(":memory:")
LitestreamVfs.load(db)
```

To get the path to the shared library:

```ruby
path = LitestreamVfs.loadable_path
```

## Platform Support

| Platform | Architecture |
|----------|-------------|
| Linux | x86_64, aarch64 |
| macOS | x86_64, arm64 |

## License

Apache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).

---

### Overview (`tests/cpu-usage/README.md`)

# CPU Usage Testing

This directory contains test scripts and configurations for measuring Litestream's idle CPU usage, particularly for validating the fixes in issue #992.

## Files

- `test-cpu-usage.sh` - Automated CPU monitoring script
- `litestream-test-polling.yml` - Config for testing with S3 replication

## Prerequisites

1. Build Litestream binary:
   ```bash
   cd ../..
   go build -o bin/litestream ./cmd/litestream
   ```

2. Set up AWS credentials in `.envrc` at repo root:
   ```bash
   export AWS_ACCESS_KEY_ID="your-key-id"
   export AWS_SECRET_ACCESS_KEY="your-secret-key"
   export AWS_REGION="us-east-2"
   export S3_BUCKET="your-test-bucket"
   ```

3. Have `sqlite3` CLI installed

## Usage

From this directory, run:

```bash
# Test for 60 seconds
./test-cpu-usage.sh 60

# Longer test (5 minutes)
./test-cpu-usage.sh 300
```

## What It Tests

The script:
1. Creates a test SQLite database at `/tmp/test.db`
2. Starts Litestream with S3 replication
3. Monitors CPU usage every second using `ps`
4. Calculates average CPU usage
5. Verifies S3 replication is working
6. Outputs results and detailed CSV log

## Expected Results

Based on testing for PR #993:

- **With S3 transport fix:** ~0.0067% CPU (99% improvement)
- **Original (v0.5.6):** ~0.7% CPU

The S3 transport fix achieves near-zero idle CPU usage, validating the fix.

## Output

Results are printed to stdout and detailed logs are saved to:
- `/tmp/litestream-cpu-log.csv` - Per-second CPU measurements

## Notes

- Tests require real S3 credentials and will upload data to your bucket
- Test database is created at `/tmp/test.db` and cleaned up on each run
- CPU measurements are instantaneous snapshots, not averages over intervals
- Longer test durations (5-10 minutes) provide more stable averages

---

### Overview (`tests/integration/README.md`)

# Integration Tests

Go-based integration tests for Litestream. These tests replace the previous bash-based test scripts with proper Go testing infrastructure.

## Overview

This package contains comprehensive integration tests organized by test type:

- **scenario_test.go** - Core functionality scenarios (fresh start, integrity, deletion, failover)
- **concurrent_test.go** - Concurrency and stress tests (rapid checkpoints, WAL growth, concurrent ops, busy timeout)
- **quick_test.go** - Quick validation tests (30 minutes configurable)
- **overnight_test.go** - Long-running stability tests (8+ hours)
- **boundary_test.go** - Edge cases (1GB boundary, different page sizes)
- **helpers.go** - Shared test utilities and helpers
- **fixtures.go** - Test data generators and scenarios

## Prerequisites

Build the required binaries:

```bash
go build -o bin/litestream ./cmd/litestream
go build -o bin/litestream-test ./cmd/litestream-test
```

## Running Tests

### Quick Tests (Default)

Run fast integration tests suitable for CI:

```bash
go test -v -tags=integration -timeout=30m ./tests/integration/... \
  -run="TestFreshStart|TestDatabaseIntegrity|TestRapidCheckpoints"
```

### All Scenario Tests

Run all scenario tests (excluding long-running):

```bash
go test -v -tags=integration -timeout=1h ./tests/integration/...
```

### Long-Running Tests

Run overnight and boundary tests:

```bash
go test -v -tags="integration,long" -timeout=10h ./tests/integration/... \
  -run="TestOvernight|Test1GBBoundary"
```

## Soak Tests

Long-running soak tests live alongside the other integration tests and share the same helpers. They are excluded from CI by default and are intended for release validation or targeted debugging.

### Overview

| Test | Tags | Defaults | Purpose | Extra Requirements |
| --- | --- | --- | --- | --- |
| `TestComprehensiveSoak` | `integration,soak` | 2h duration, 50 MB DB, 500 writes/s | File-backed end-to-end stress | Litestream binaries in `./bin` |
| `TestMinIOSoak` | `integration,soak,docker` | 2h duration, 5 MB DB (short=2 m), 100 writes/s | S3-compatible replication via MinIO | Docker daemon, `docker` CLI |
| `TestSoakReplicateRestore` | `integration,soak,docker` | 5m duration (short=1 m), 100 writes/s, restore every 30s | Issue #1164 repro: stop→restore→integrity_check cycles against MinIO | Docker daemon, `docker` CLI |
| `TestOvernightS3Soak` | `integration,soak,aws` | 8h duration, 50 MB DB | Real S3 replication & restore | AWS credentials, `aws` CLI |

All soak tests support `go test -test.short` to scale the default duration down to roughly two minutes for smoke verification.

### Environment Variables

| Variable | Default | Description |
| --- | --- | --- |
| `SOAK_AUTO_PURGE` | `yes` for non-interactive shells; prompts otherwise | Controls whether MinIO buckets are cleared before each run. Set to `no` to retain objects between runs. |
| `SOAK_KEEP_TEMP` | unset | When set (any value), preserves the temporary directory and artifacts (database, config, logs) instead of removing them after the test completes. |
| `SOAK_DEBUG` | `0` | Streams command stdout/stderr (database population, load generation, docker helpers) directly to the console. Without this the output is captured and only shown on failure. |
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `S3_BUCKET`, `AWS_REGION` | required for `aws` tag | Provide credentials and target bucket for the overnight S3 soak. Region defaults to `us-east-1` if unset. |

### Example Commands

File-based soak (full length):

```bash
go test -v -tags="integration,soak" \
  -run=TestComprehensiveSoak -timeout=3h ./tests/integration
```

File-based soak (short mode with preserved artifacts and debug logging):

```bash
SOAK_KEEP_TEMP=1 SOAK_DEBUG=1 go test -v -tags="integration,soak" \
  -run=TestComprehensiveSoak -test.short -timeout=1h ./tests/integration
```

MinIO soak (short mode, auto-purges bucket, preserves results):

```bash
SOAK_AUTO_PURGE=yes SOAK_KEEP_TEMP=1 go test -v -tags="integration,soak,docker" \
  -run=TestMinIOSoak -test.short -timeout=20m ./tests/integration
```

Overnight S3 soak (full duration):

```bash
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export S3_BUCKET=your-bucket
export AWS_REGION=us-east-1

go test -v -tags="integration,soak,aws" \
  -run=TestOvernightS3Soak -timeout=10h ./tests/integration
```

### Tips

- Run with `-v` to view the 60-second progress updates and final status summary. Without `-v`, progress output is suppressed by Go’s test runner.
- When prompted about purging a MinIO bucket, answering “yes” clears the bucket via `minio/mc` before the run; “no” allows you to inspect lingering objects from previous executions.
- `SOAK_KEEP_TEMP=1` is especially useful when investigating failures—the helper prints the preserved path so you can inspect databases, configs, and logs.
- The monitoring infrastructure automatically prints additional status blocks when error counts change, making `SOAK_DEBUG=1` optional for most workflows.

### Specific Tests

Run individual test functions:

```bash
# Fresh start test
go test -v -tags=integration ./tests/integration/... -run=TestFreshStart

# Rapid checkpoints test
go test -v -tags=integration ./tests/integration/... -run=TestRapidCheckpoints

# 1GB boundary test
go test -v -tags=integration ./tests/integration/... -run=Test1GBBoundary
```

### Short Mode

Run abbreviated versions with `-short`:

```bash
go test -v -tags=integration -short ./tests/integration/...
```

This reduces test durations by 10x (e.g., 8 hours becomes 48 minutes).

## Test Categories

### Scenario Tests

Core functionality tests that run in seconds to minutes:

- `TestFreshStart` - Starting replication before database exists
- `TestDatabaseIntegrity` - Complex schema and data integrity
- `TestDatabaseDeletion` - Source database deletion during replication

### Concurrent Tests

Stress and concurrency tests:

- `TestRapidCheckpoints` - Rapid checkpoint operations under load
- `TestWALGrowth` - Large WAL file handling (100MB+)
- `TestConcurrentOperations` - Multiple databases replicating simultaneously
- `TestBusyTimeout` - Database busy timeout and lock handling

### Quick Tests

Configurable duration validation (default 30 minutes):

- `TestQuickValidation` - Comprehensive validation with wave pattern load

### Overnight Tests

Long-running stability tests (default 8 hours):

- `TestOvernightFile` - 8-hour file-based replication test
- `TestOvernightComprehensive` - 8-hour comprehensive test with large database

### Boundary Tests

Edge case and boundary condition tests:

- `Test1GBBoundary` - SQLite 1GB lock page boundary (page #262145 with 4KB pages)
- `TestLockPageWithDifferentPageSizes` - Lock page handling with various page sizes

## CI Integration

### Automatic (Pull Requests)

Quick tests run automatically on PRs modifying Go code:

```yaml
- Quick integration tests (TestFreshStart, TestDatabaseIntegrity, TestRapidCheckpoints)
- Timeout: 30 minutes
```

### Manual Workflows

Trigger via GitHub Actions UI:

**Quick Tests:**
```
workflow_dispatch → test_type: quick
```

**All Scenario Tests:**
```
workflow_dispatch → test_type: all
```

**Long-Running Tests:**
```
workflow_dispatch → test_type: long
```

## Test Infrastructure

### Helpers (helpers.go)

- `SetupTestDB(t, name)` - Create test database instance
- `TestDB.Create()` - Create database with WAL mode
- `TestDB.Populate(size)` - Populate to target size
- `TestDB.StartLitestream()` - Start replication
- `TestDB.StopLitestream()` - Stop replication
- `TestDB.Restore(path)` - Restore from replica
- `TestDB.Validate(path)` - Full validation (integrity, checksum, data)
- `TestDB.QuickValidate(path)` - Quick validation
- `TestDB.GenerateLoad(...)` - Generate database load
- `GetTestDuration(t, default)` - Get configurable test duration
- `RequireBinaries(t)` - Check for required binaries

### Fixtures (fixtures.go)

- `DefaultLoadConfig()` - Load generation configuration
- `DefaultPopulateConfig()` - Database population configuration
- `CreateComplexTestSchema(db)` - Multi-table schema with foreign keys
- `PopulateComplexTestData(db, ...)` - Populate complex data
- `LargeWALScenario()` - Large WAL test scenario
- `RapidCheckpointsScenario()` - Rapid checkpoint scenario

## Test Artifacts

Tests create temporary directories via `t.TempDir()`:

```
/tmp/<test-temp-dir>/
├── <name>.db          # Test database
├── <name>.db-wal      # WAL file
├── <name>.db-shm      # Shared memory
├── replica/           # Replica directory
│   └── ltx/0/        # LTX files
├── litestream.log     # Litestream output
└── *-restored.db      # Restored databases
```

Artifacts are automatically cleaned up after tests complete.

## Debugging Tests

### View Litestream Logs

```go
log, err := db.GetLitestreamLog()
fmt.Println(log)
```

### Check for Errors

```go
errors, err := db.CheckForErrors()
for _, e := range errors {
    t.Logf("Error: %s", e)
}
```

### Inspect Replica

```go
fileCount, _ := db.GetReplicaFileCount()
t.Logf("LTX files: %d", fileCount)
```

### Check Database Size

```go
size, _ := db.GetDatabaseSize()
t.Logf("DB size: %.2f MB", float64(size)/(1024*1024))
```

## Migration from Bash

This is part of an ongoing effort to migrate bash test scripts to Go integration tests. This migration improves maintainability, enables CI integration, and provides platform independence.

### Test Directory Organization

Three distinct test locations serve different purposes:

**`tests/integration/` (this directory)** - Go-based integration and soak tests:
- Quick integration tests: `scenario_test.go`, `concurrent_test.go`, `boundary_test.go`
- Soak tests (2-8 hours): `comprehensive_soak_test.go`, `minio_soak_test.go`, `overnight_s3_soak_test.go`
- All tests use proper Go testing infrastructure with build tags

**`scripts/` (top-level)** - Utility scripts only (soak tests migrated to Go):
- `analyze-test-results.sh` - Post-test analysis utility
- `setup-homebrew-tap.sh` - Packaging script (not a test)

**`cmd/litestream-test/scripts/`** - Scenario and debugging bash scripts (being phased out):
- Bug reproduction scripts for specific issues (#752, #754)
- Format & upgrade tests for version compatibility
- S3 retention tests with Python mock
- Quick validation and setup utilities

### Migration Status

**Migrated from `scripts/` (5 scripts):**
- `test-quick-validation.sh` → `quick_test.go::TestQuickValidation` (CI: ✅)
- `test-overnight.sh` → `overnight_test.go::TestOvernightFile` (CI: ❌ too long)
- `test-comprehensive.sh` → `comprehensive_soak_test.go::TestComprehensiveSoak` (CI: ❌ soak test)
- `test-minio-s3.sh` → `minio_soak_test.go::TestMinIOSoak` (CI: ❌ soak test, requires Docker)
- `test-overnight-s3.sh` → `overnight_s3_soak_test.go::TestOvernightS3Soak` (CI: ❌ soak test, 8 hours)

**Migrated from `cmd/litestream-test/scripts/` (9 scripts):**
- `test-fresh-start.sh` → `scenario_test.go::TestFreshStart`
- `test-database-integrity.sh` → `scenario_test.go::TestDatabaseIntegrity`
- `test-database-deletion.sh` → `scenario_test.go::TestDatabaseDeletion`
- `test-replica-failover.sh` → NOT MIGRATED (feature removed from Litestream)
- `test-rapid-checkpoints.sh` → `concurrent_test.go::TestRapidCheckpoints`
- `test-wal-growth.sh` → `concurrent_test.go::TestWALGrowth`
- `test-concurrent-operations.sh` → `concurrent_test.go::TestConcurrentOperations`
- `test-busy-timeout.sh` → `concurrent_test.go::TestBusyTimeout`
- `test-1gb-boundary.sh` → `boundary_test.go::Test1GBBoundary`

**Remaining Bash Scripts:**

_scripts/_ (2 scripts remaining):
- `analyze-test-results.sh` - Post-test analysis utility (may stay as bash)
- `setup-homebrew-tap.sh` - Packaging script (not a test)

_cmd/litestream-test/scripts/_ (16 scripts remaining):
- Bug reproduction scripts: `reproduce-critical-bug.sh`, `test-754-*.sh`, `test-v0.5-*.sh`
- Format & upgrade tests: `test-format-isolation.sh`, `test-upgrade-*.sh`, `test-massive-upgrade.sh`
- S3 retention tests: `test-s3-retention-*.sh` (4 scripts, use Python S3 mock)
- Utility: `verify-test-setup.sh`

### Why Some Tests Aren't in CI

Per industry best practices, CI tests should complete in < 1 hour (ideally < 10 minutes):
- ✅ **Quick tests** (< 5 min) - Run on every PR
- ❌ **Soak tests** (2-8 hours) - Run locally before releases only
- ❌ **Long-running tests** (> 30 min) - Too slow for CI feedback loop

Soak tests are migrated to Go for maintainability but run **locally only**. See "Soak Tests" section below.

## Soak Tests (Long-Running Stability Tests)

Soak tests run for 2-8 hours to validate long-term stability under sustained load. These tests are **NOT run in CI** per industry best practices (effective CI requires tests to complete in < 1 hour).

### Purpose

Soak tests validate:
- Long-term replication stability
- Memory leak detection over time
- Compaction effectiveness across multiple cycles
- Checkpoint behavior under sustained load
- Recovery from transient issues
- Storage growth patterns

### When to Run Soak Tests

- ✅ Before major releases
- ✅ After significant replication changes
- ✅ To reproduce stability issues
- ✅ For performance benchmarking
- ❌ NOT on every commit (too slow for CI)

### Running Soak Tests Locally

**File-based comprehensive test (2 hours):**
```bash
go test -v -tags="integration,soak" -timeout=3h -run=TestComprehensiveSoak ./tests/integration/
```

**MinIO S3 test (2 hours, requires Docker):**
```bash
# Ensure Docker is running
go test -v -tags="integration,soak,docker" -timeout=3h -run=TestMinIOSoak ./tests/integration/
```

**Overnight S3 test (8 hours, requires AWS):**
```bash
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export S3_BUCKET=your-test-bucket
export AWS_REGION=us-east-1

go test -v -tags="integration,soak,aws" -timeout=10h -run=TestOvernightS3Soak ./tests/integration/
```

**Run all soak tests:**
```bash
go test -v -tags="integration,soak,docker,aws" -timeout=15h ./tests/integration/
```

### Adjust Duration for Testing

Tests respect the `-test.short` flag to run abbreviated versions:

```bash
# Run comprehensive test for 30 minutes instead of 2 hours
go test -v -tags="integration,soak" -timeout=1h -run=TestComprehensiveSoak ./tests/integration/ -test.short
```

### Soak Test Build Tags

Soak tests use multiple build tags to control execution:

- `integration` - Required for all integration tests
- `soak` - Marks long-running stability tests (2-8 hours)
- `docker` - Requires Docker (MinIO test)
- `aws` - Requires AWS credentials (S3 tests)

### Monitoring Soak Tests

All soak tests log progress every 60 seconds:

```bash
# Watch test progress in real-time
go test -v -tags="integration,soak" -run=TestComprehensiveSoak ./tests/integration/ 2>&1 | tee soak-test.log
```

Metrics reported during execution:
- Database size and WAL size
- Row count
- Replica statistics (snapshots, LTX segments)
- Operation counts (checkpoints, compactions, syncs)
- Error counts
- Write rate

### Soak Test Summary

| Test | Duration | Requirements | What It Tests |
|------|----------|--------------|---------------|
| TestComprehensiveSoak | 2h | None | File-based replication with aggressive compaction |
| TestMinIOSoak | 2h | Docker | S3-compatible storage via MinIO container |
| TestSoakReplicateRestore | 5m | Docker | Issue #1164 repro: periodic stop→restore→integrity_check |
| TestOvernightS3Soak | 8h | AWS credentials | Real S3 replication, overnight stability |

## Benefits Over Bash

1. **Type Safety** - Compile-time error checking
2. **Better Debugging** - Use standard Go debugging tools
3. **Code Reuse** - Shared helpers and fixtures
4. **Parallel Execution** - Tests can run concurrently
5. **CI Integration** - Run automatically on PRs
6. **Test Coverage** - Measure code coverage
7. **Consistent Patterns** - Standard Go testing conventions
8. **Better Error Messages** - Structured, clear reporting
9. **Platform Independent** - Works on Linux, macOS, Windows
10. **IDE Integration** - Full editor support

## Contributing

When adding new integration tests:

1. Use appropriate build tags (`//go:build integration` or `//go:build integration && long`)
2. Call `RequireBinaries(t)` to check prerequisites
3. Use `SetupTestDB(t, name)` for test setup
4. Call `defer db.Cleanup()` for automatic cleanup
5. Log test progress with descriptive messages
6. Use `GetTestDuration(t, default)` for configurable durations
7. Add test to CI workflow if appropriate
8. Update this README with new test documentation

## Related Documentation

- [cmd/litestream-test README](../../cmd/litestream-test/README.md) - Testing harness CLI
- [scripts/README.md](../../scripts/README.md) - Legacy bash test scripts
- [GitHub Issue #798](https://github.com/benbjohnson/litestream/issues/798) - Migration tracking

---

### Overview (`cmd/litestream-test/scripts/README.md`)

# Litestream Test Scripts

Comprehensive test scripts for validating Litestream functionality across various scenarios. These scripts use the `litestream-test` CLI tool to orchestrate complex testing scenarios.

## Prerequisites

```bash
go build -o bin/litestream ./cmd/litestream
go build -o bin/litestream-test ./cmd/litestream-test

./cmd/litestream-test/scripts/verify-test-setup.sh
```

## Quick Reference

> **Note:** Some tests have been migrated to Go integration tests in `tests/integration/`. See [tests/integration/README.md](../../tests/integration/README.md) for the Go-based test suite.

| Script | Purpose | Duration | Status |
|--------|---------|----------|--------|
| verify-test-setup.sh | Environment validation | ~5s | ✅ Stable |
| reproduce-critical-bug.sh | Checkpoint during downtime bug | ~2min | 🐛 Reproduces #752 |
| test-754-s3-scenarios.sh | Issue #754 S3 vs file replication | ~10min | 🐛 Tests #754 |
| test-754-restore-focus.sh | Issue #754 restore focus | ~5min | 🐛 Tests #754 |
| test-simple-754-reproduction.sh | Minimal #754 reproduction | ~3min | 🐛 Tests #754 |
| test-v0.5-flag-reproduction.sh | ltx v0.5.0 flag issue | ~5min | 🐛 Tests #754 |
| test-v0.5-restart-scenarios.sh | v0.5 restart scenarios | ~5min | 🐛 Tests #754 |
| test-format-isolation.sh | Format version isolation | ~3min | ✅ Stable |
| test-quick-format-check.sh | Quick format validation | ~30s | ✅ Stable |
| test-upgrade-v0.3-to-v0.5.sh | v0.3 to v0.5 upgrade | ~10min | ✅ Stable |
| test-upgrade-large-db.sh | Large database upgrade | ~15min | ✅ Stable |
| test-massive-upgrade.sh | Massive database upgrade | ~20min | ✅ Stable |
| test-s3-retention-cleanup.sh | Basic S3 retention | ~8min | ✅ Stable |
| test-s3-retention-small-db.sh | S3 retention 50MB | ~8min | ✅ Stable |
| test-s3-retention-large-db.sh | S3 retention 1.5GB | ~20min | ✅ Stable |
| test-s3-retention-comprehensive.sh | Full S3 retention suite | ~30min | ✅ Stable |
| test-s3-access-point.sh | S3 Access Point ARN support | ~2min | ✅ Stable |

## Test Categories

### Setup & Validation

#### verify-test-setup.sh
Verifies that the test environment is properly configured with required binaries and dependencies.

```bash
./cmd/litestream-test/scripts/verify-test-setup.sh
```

**Checks:**
- Litestream binary exists
- litestream-test binary exists
- SQLite3 available
- Python dependencies for S3 mock

### Bug Reproduction Scripts

#### reproduce-critical-bug.sh
Reproduces checkpoint during downtime bug that causes restore failures.

```bash
./cmd/litestream-test/scripts/reproduce-critical-bug.sh
```

**Reproduces:** Issue #752

**Scenario:**
1. Litestream is killed (simulating crash)
2. Writes continue and a checkpoint occurs
3. Litestream is restarted
4. Restore fails with "nonsequential page numbers" error

**Expected:** Database should restore successfully
**Actual:** Restore fails, causing data loss

#### test-754-s3-scenarios.sh
Tests Issue #754 flag compatibility with S3 replication versus file replication.

```bash
./cmd/litestream-test/scripts/test-754-s3-scenarios.sh
```

**Tests:**
- S3 replica behavior with ltx v0.5.0
- File replica behavior comparison
- LTX file cleanup and retention
- Flag compatibility issues

#### test-754-restore-focus.sh
Focused testing of Issue #754 restore failures.

```bash
./cmd/litestream-test/scripts/test-754-restore-focus.sh
```

**Tests:**
- Restore failures with pre-existing databases
- Flag mismatch detection
- Recovery scenarios

#### test-simple-754-reproduction.sh
Minimal reproduction case for Issue #754.

```bash
./cmd/litestream-test/scripts/test-simple-754-reproduction.sh
```

**Reproduces:** Issue #754 with minimal steps for debugging

#### test-v0.5-flag-reproduction.sh
Reproduces ltx v0.5.0 flag compatibility issue.

```bash
./cmd/litestream-test/scripts/test-v0.5-flag-reproduction.sh
```

**Tests:**
- Pre-existing database behavior with ltx v0.5.0
- Flag mismatch scenarios
- Upgrade path issues

#### test-v0.5-restart-scenarios.sh
Tests various restart scenarios with ltx v0.5.0.

```bash
./cmd/litestream-test/scripts/test-v0.5-restart-scenarios.sh
```

**Tests:**
- Clean restart
- Restart after checkpoint
- Restart with pending data
- Flag persistence across restarts

### Format & Upgrade Tests

#### test-format-isolation.sh
Tests isolation between different LTX format versions.

```bash
./cmd/litestream-test/scripts/test-format-isolation.sh
```

**Tests:**
- Multiple format versions coexisting
- Format detection and handling
- Migration between formats
- Backward compatibility

#### test-quick-format-check.sh
Quick validation of LTX format handling.

```bash
./cmd/litestream-test/scripts/test-quick-format-check.sh
```

**Duration:** ~30 seconds

**Tests:**
- Format version detection
- Basic format integrity
- Quick validation workflow

#### test-upgrade-v0.3-to-v0.5.sh
Tests upgrade path from ltx v0.3 to v0.5.

```bash
./cmd/litestream-test/scripts/test-upgrade-v0.3-to-v0.5.sh
```

**Tests:**
- Migration from v0.3 to v0.5
- Data preservation during upgrade
- Flag handling in upgrade process
- Backward compatibility verification

#### test-upgrade-large-db.sh
Tests upgrade process with large databases (1GB+).

```bash
./cmd/litestream-test/scripts/test-upgrade-large-db.sh
```

**Tests:**
- Large database upgrade performance
- Data integrity during upgrade
- Lock page handling in upgrades
- Resource usage during migration

#### test-massive-upgrade.sh
Tests upgrade with very large databases and long-running scenarios.

```bash
./cmd/litestream-test/scripts/test-massive-upgrade.sh
```

**Tests:**
- Multi-GB database upgrades
- Extended migration scenarios
- Performance under scale
- Memory and disk usage

### S3 & Retention Tests

For detailed S3 retention testing documentation, see [S3-RETENTION-TESTING.md](../S3-RETENTION-TESTING.md).

#### test-s3-access-point.sh
Tests S3 Access Point ARN support (Issue #923). Verifies that Access Point ARNs work automatically without manual endpoint configuration.

```bash
export LITESTREAM_S3_ACCESS_POINT_ARN='arn:aws:s3:us-east-2:123456789012:accesspoint/my-access-point'
./cmd/litestream-test/scripts/test-s3-access-point.sh
```

**Tests:**
- Replication to S3 Access Point using ARN
- Automatic endpoint resolution (UseARNRegion)
- Restore from Access Point ARN
- Data integrity verification

**Environment Variables:**
- `LITESTREAM_S3_ACCESS_POINT_ARN` - Full ARN of the S3 Access Point (required)
- `LITESTREAM_S3_REGION` - AWS region (optional, extracted from ARN)
- `LITESTREAM_S3_PREFIX` - Path prefix in bucket (optional)
- AWS credentials via standard methods (env vars, credentials file, IAM role)

#### test-s3-retention-cleanup.sh
Basic S3 LTX retention cleanup testing.

```bash
./cmd/litestream-test/scripts/test-s3-retention-cleanup.sh
```

**Tests:**
- Basic retention cleanup behavior
- Old LTX file removal
- Retention period enforcement

#### test-s3-retention-small-db.sh
S3 retention testing with 50MB database.

```bash
./cmd/litestream-test/scripts/test-s3-retention-small-db.sh
```

**Configuration:**
- Database size: 50MB
- Retention period: 2 minutes
- Duration: ~8 minutes

**Tests:**
- Small database retention cleanup
- Quick retention cycles
- S3 mock integration

#### test-s3-retention-large-db.sh
S3 retention testing with 1.5GB database crossing lock page boundary.

```bash
./cmd/litestream-test/scripts/test-s3-retention-large-db.sh
```

**Configuration:**
- Database size: 1.5GB
- Page size: 4KB (lock page at #262145)
- Retention period: 3 minutes
- Duration: ~20 minutes

**Tests:**
- Large database retention cleanup
- Lock page boundary handling
- Extended monitoring
- Scale behavior

#### test-s3-retention-comprehensive.sh
Master script running both small and large database retention tests with analysis.

```bash
./cmd/litestream-test/scripts/test-s3-retention-comprehensive.sh

./cmd/litestream-test/scripts/test-s3-retention-comprehensive.sh --small-only
./cmd/litestream-test/scripts/test-s3-retention-comprehensive.sh --large-only
./cmd/litestream-test/scripts/test-s3-retention-comprehensive.sh --no-cleanup
```

**Duration:** ~30 minutes for full suite

**Features:**
- Runs both small and large DB tests
- Comparative analysis
- Detailed reports
- Configurable execution

## Usage Patterns

### Running Individual Tests

```bash
./cmd/litestream-test/scripts/test-fresh-start.sh
```

### Verify Environment First

```bash
./cmd/litestream-test/scripts/verify-test-setup.sh
./cmd/litestream-test/scripts/test-rapid-checkpoints.sh
```

### Running Multiple Tests

```bash
for script in test-fresh-start.sh test-rapid-checkpoints.sh test-database-integrity.sh; do
    echo "Running $script..."
    ./cmd/litestream-test/scripts/$script
    echo ""
done
```

### S3 Testing with Local Mock

The S3 tests automatically use the Python S3 mock server (`./etc/s3_mock.py`) for isolated testing:

```bash
./cmd/litestream-test/scripts/test-s3-retention-small-db.sh
```

### Debugging Failed Tests

Most tests create logs in `/tmp/`:

```bash
tail -f /tmp/fresh-test.log

tail -f /tmp/checkpoint-cycle.log

grep -i error /tmp/*.log
```

## Test Artifacts

Tests typically create artifacts in `/tmp/`:

- **Databases:** `/tmp/*-test.db`
- **Replicas:** `/tmp/*-replica/`
- **Logs:** `/tmp/*-test.log`
- **Configs:** `/tmp/*.yml`
- **Restored DBs:** `/tmp/*-restored.db`

## Test Results & Analysis

Historical test results and analysis are stored in `.local/test-results/` (git-ignored):

- `final-test-summary.md` - Comprehensive test findings
- `validation-results-after-ltx-v0.5.0.md` - ltx v0.5.0 impact analysis
- `comprehensive-test-findings.md` - Initial test results
- `critical-bug-analysis.md` - Detailed bug analysis

## Key Findings Summary

### Performance ✅
- Successfully handles 400+ writes/second
- Manages 100MB+ WAL files
- Multiple concurrent databases replicate cleanly

### Fresh Databases ✅
- Work perfectly with ltx v0.5.0
- Clean replication and restore
- No flag issues

### Pre-existing Databases ❌
- Broken due to ltx flag compatibility (#754)
- Restore failures
- Upgrade path issues

### Checkpoint During Downtime ❌
- Worse with ltx v0.5.0 (#752)
- Causes restore failures
- Data loss risk

### S3 Retention ✅
- LTX cleanup works correctly
- Handles lock page boundary
- Scale testing successful

## Related Issues

- [#752](https://github.com/benbjohnson/litestream/issues/752) - Checkpoint during downtime bug
- [#753](https://github.com/benbjohnson/litestream/issues/753) - Transaction numbering (FIXED)
- [#754](https://github.com/benbjohnson/litestream/issues/754) - ltx v0.5.0 flag compatibility (CRITICAL)

## Related Documentation

- [litestream-test CLI Documentation](../README.md) - CLI tool reference
- [S3 Retention Testing Guide](../S3-RETENTION-TESTING.md) - Detailed S3 testing
- [Top-level Integration Scripts](../../../scripts/README.md) - Long-running tests

## Contributing

When adding new test scripts:

1. Follow existing naming conventions (`test-*.sh`)
2. Include clear comments explaining what is being tested
3. Use `/tmp/` for test artifacts
4. Create cleanup handlers with `trap`
5. Provide clear success/failure output
6. Update this README with script documentation
7. Add entry to Quick Reference table

---

### Overview (`packages/npm/litestream-vfs-darwin-amd64/README.md`)

# litestream-vfs-darwin-amd64

Platform package for `litestream-vfs` — macOS x64 (Intel).

This package is installed automatically by `litestream-vfs`. You should not need to install it directly.

---

### Overview (`packages/npm/litestream-vfs-darwin-arm64/README.md`)

# litestream-vfs-darwin-arm64

Platform package for `litestream-vfs` — macOS ARM64 (Apple Silicon).

This package is installed automatically by `litestream-vfs`. You should not need to install it directly.

---

### Overview (`packages/npm/litestream-vfs-linux-amd64/README.md`)

# litestream-vfs-linux-amd64

Platform package for `litestream-vfs` — Linux x64.

This package is installed automatically by `litestream-vfs`. You should not need to install it directly.

---

### Overview (`packages/npm/litestream-vfs-linux-arm64/README.md`)

# litestream-vfs-linux-arm64

Platform package for `litestream-vfs` — Linux ARM64.

This package is installed automatically by `litestream-vfs`. You should not need to install it directly.

---

### Overview (`packages/npm/litestream-vfs/README.md`)

# litestream-vfs

Litestream VFS extension for SQLite — distributed via npm.

This package bundles the [Litestream](https://litestream.io) VFS shared library.
The correct platform-specific binary is automatically installed via `optionalDependencies`.

## Installation

```bash
npm install litestream-vfs
```

## Usage

```javascript
const { getLoadablePath } = require("litestream-vfs");
const path = getLoadablePath();
// Use `path` with better-sqlite3 or other SQLite bindings
```

## Platform Support

| Platform | Architecture |
|----------|-------------|
| Linux | x86_64, aarch64 |
| macOS | x86_64, arm64 |

## License

Apache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).

---

### ARCHITECTURE (`docs/ARCHITECTURE.md`)

# Litestream Architecture - Technical Deep Dive

## Table of Contents
- [System Layers](#system-layers)
- [Core Components](#core-components)
- [IPC Server & Control Socket](#ipc-server--control-socket)
- [Distributed Leasing](#distributed-leasing)
- [Library Convenience Methods](#library-convenience-methods)
- [LTX File Format](#ltx-file-format)
- [WAL Monitoring Mechanism](#wal-monitoring-mechanism)
- [Compaction Process](#compaction-process)
- [Transaction Management](#transaction-management)
- [Concurrency Model](#concurrency-model)
- [State Management](#state-management)
- [Initialization Flow](#initialization-flow)
- [Error Handling](#error-handling)

## System Layers

Litestream follows a layered architecture with clear separation of concerns:

```mermaid
graph TB
    subgraph "Application Layer"
        CLI[CLI Commands<br/>cmd/litestream/]
        Config[Configuration<br/>config.go]
    end

    subgraph "Core Layer"
        Store[Store Manager<br/>store.go]
        DB[Database Manager<br/>db.go]
        Replica[Replica Manager<br/>replica.go]
    end

    subgraph "Storage Abstraction"
        RC[ReplicaClient Interface<br/>replica_client.go]
    end

    subgraph "Storage Implementations"
        S3[s3/replica_client.go]
        GCS[gs/replica_client.go]
        ABS[abs/replica_client.go]
        OSS[oss/replica_client.go]
        File[file/replica_client.go]
        SFTP[sftp/replica_client.go]
        NATS[nats/replica_client.go]
    end

    subgraph "External"
        SQLite[SQLite Database]
        Cloud[Cloud Storage]
    end

    CLI --> Store
    Store --> DB
    DB --> Replica
    Replica --> RC
    RC --> S3
    RC --> GCS
    RC --> ABS
    RC --> OSS
    RC --> File
    RC --> SFTP
    RC --> NATS
    DB <--> SQLite
    S3 --> Cloud
    GCS --> Cloud
    ABS --> Cloud
    OSS --> Cloud
```

### Layer Responsibilities

#### 1. Application Layer
- **CLI Commands**: User interface for operations (replicate, restore, etc.)
- **Configuration**: YAML/environment variable parsing and validation

#### 2. Core Layer
- **Store**: Multi-database coordination, compaction scheduling
- **DB**: Single database management, WAL monitoring, checkpointing
- **Replica**: Replication to single destination, position tracking

#### 3. Storage Abstraction
- **ReplicaClient Interface**: Uniform API for all storage backends

#### 4. Storage Implementations
- Backend-specific logic (authentication, retries, optimizations)

#### 5. IPC Layer
- **Server**: Unix socket HTTP server for runtime control (`server.go`)
- **Leaser**: Distributed lease interface for coordination (`leaser.go`, `s3/leaser.go`)

## Core Components

### DB Component (db.go)

The DB component is the heart of Litestream, managing a single SQLite database:

```go
type DB struct {
    // Core fields
    path     string      // Database file path
    metaPath string      // Metadata directory path
    db       *sql.DB     // SQLite connection
    f        *os.File    // Long-running file descriptor
    rtx      *sql.Tx     // Long-running read transaction
    pageSize int         // Database page size

    // Synchronization
    mu       sync.RWMutex   // Protects struct fields
    chkMu    sync.RWMutex   // Checkpoint lock
    notify   chan struct{}  // WAL change notifications

    // Lifecycle
    ctx    context.Context
    cancel func()
    wg     sync.WaitGroup

    // Configuration
    MinCheckpointPageN int              // Min pages for passive checkpoint
    TruncatePageN      int              // Pages before emergency truncate checkpoint
    CheckpointInterval time.Duration    // Time-based passive checkpoint interval
    MonitorInterval    time.Duration    // WAL monitoring frequency
    // Note: MaxCheckpointPageN removed (RESTART mode disabled due to #724)

    // Metrics
    dbSizeGauge        prometheus.Gauge
    walSizeGauge       prometheus.Gauge
    txIDGauge          prometheus.Gauge
}
```

#### Key Methods

```go
// Lifecycle
func (db *DB) Open() error
func (db *DB) Close(ctx context.Context) error

// Monitoring
func (db *DB) monitor()              // Background WAL monitoring
func (db *DB) checkWAL() (bool, error)  // Check for WAL changes

// Checkpointing
func (db *DB) Checkpoint(mode string) error
func (db *DB) autoCheckpoint() error

// Replication
func (db *DB) WALReader(pgno uint32) (io.ReadCloser, error)
func (db *DB) Sync(ctx context.Context) error

// Compaction
func (db *DB) Compact(ctx context.Context, destLevel int) (*ltx.FileInfo, error)
```

### Replica Component (replica.go)

Manages replication to a single destination:

```go
type Replica struct {
    db *DB                    // Parent database
    Client ReplicaClient      // Storage backend client

    mu  sync.RWMutex
    pos ltx.Pos              // Current replication position

    // Configuration
    SyncInterval time.Duration
    MonitorEnabled bool

    // Lifecycle
    cancel func()
    wg     sync.WaitGroup
}
```

#### Replication Position

```go
type Pos struct {
    TXID     TXID      // Transaction ID
    PageNo   uint32    // Page number within transaction
    Checksum uint64    // Running checksum
}
```

### Store Component (store.go)

Coordinates multiple databases and manages system-wide resources:

```go
type Store struct {
    mu     sync.Mutex
    dbs    []*DB
    levels CompactionLevels

    // Configuration
    SnapshotInterval           time.Duration
    SnapshotRetention          time.Duration
    L0Retention                time.Duration
    L0RetentionCheckInterval   time.Duration
    CompactionMonitorEnabled bool

    // Lifecycle
    ctx    context.Context
    cancel func()
    wg     sync.WaitGroup
}
```

## IPC Server & Control Socket

Litestream exposes a Unix socket HTTP server for runtime control (`server.go`).

### Socket Configuration

```go
type SocketConfig struct {
    Enabled     bool   `yaml:"enabled"`     // Default: false
    Path        string `yaml:"path"`        // Default: "/var/run/litestream.sock"
    Permissions uint32 `yaml:"permissions"` // Default: 0600
}
```

### HTTP Endpoints

All endpoints are served over the Unix socket via Go's `net/http` mux:

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/register` | Add a database at runtime |
| `POST` | `/unregister` | Remove a database at runtime |
| `GET` | `/txid?path=` | Get current transaction ID for a database |
| `GET` | `/list` | List all managed databases |
| `GET` | `/info` | Server info (version, PID, uptime, database count) |
| `POST` | `/start` | Start replication for a database |
| `POST` | `/stop` | Stop replication for a database |
| `GET` | `/debug/pprof/*` | Standard Go pprof endpoints |

### Request/Response Types

```go
type RegisterDatabaseRequest struct {
    Path       string `json:"path"`
    ReplicaURL string `json:"replica_url"`
}

type UnregisterDatabaseRequest struct {
    Path    string `json:"path"`
    Timeout int    `json:"timeout,omitempty"` // Seconds
}

type TXIDResponse struct {
    TXID uint64 `json:"txid"`
}
```

## Distributed Leasing

The `Leaser` interface (`leaser.go`) enables distributed coordination so multiple Litestream instances can safely share a replica destination.

### Interface

```go
type Leaser interface {
    Type() string
    AcquireLease(ctx context.Context) (*Lease, error)
    RenewLease(ctx context.Context, lease *Lease) (*Lease, error)
    ReleaseLease(ctx context.Context, lease *Lease) error
}

type Lease struct {
    Generation int64     `json:"generation"`
    ExpiresAt  time.Time `json:"expires_at"`
    Owner      string    `json:"owner,omitempty"`
    ETag       string    `json:"-"`
}
```

### S3 Implementation (`s3/leaser.go`)

- **Defaults**: `DefaultLeaseTTL = 30s`, `DefaultLeasePath = "lock.json"`
- **Owner format**: `hostname:pid` (falls back to `pid-N` if hostname unavailable)
- **Conditional writes**: Uses `If-Match`/`If-None-Match` on S3 PutObject to prevent races
- **Release**: Uses `DeleteObject` with `If-Match` to ensure only the holder can release
- **Error types**: `ErrLeaseNotHeld`, `ErrLeaseAlreadyReleased`, `LeaseExistsError{Owner, ExpiresAt}`

### Acquisition Flow

1. Read existing lease from S3 (`lock.json`)
2. If lease exists and is not expired, return `LeaseExistsError`
3. Write new lease with `If-None-Match: *` (first acquire) or `If-Match: <etag>` (expired takeover)
4. If `PreconditionFailed` (412), another instance acquired first
5. Return lease with ETag for subsequent renewal

## Library Convenience Methods

These methods on `DB` (`db.go`) simplify common operations when Litestream is used as a library:

```go
type SyncStatus struct {
    LocalTXID  ltx.TXID // Local transaction ID
    RemoteTXID ltx.TXID // Remote transaction ID
    InSync     bool     // true if LocalTXID > 0 and equal to RemoteTXID
}

func (db *DB) SyncStatus(ctx context.Context) (SyncStatus, error)
func (db *DB) SyncAndWait(ctx context.Context) error
func (db *DB) EnsureExists(ctx context.Context) error
```

- **SyncStatus**: Compares local TXID against remote replica position (performs I/O to query remote)
- **SyncAndWait**: Runs `db.Sync()` (WAL to LTX) then `db.Replica.Sync()` (LTX to remote), blocking until both complete
- **EnsureExists**: Restores database from replica if local file doesn't exist; no-op if file exists or no backup available. Must be called before `Open()`

## LTX File Format

LTX (Log Transaction) files are immutable files containing database changes:

```
+------------------+
|     Header       |  Fixed size header with metadata
+------------------+
|                  |
|   Page Frames    |  Variable number of page frames
|                  |
+------------------+
|   Page Index     |  Index for efficient page lookup
+------------------+
|     Trailer      |  Metadata and checksums
+------------------+
```

### Header Structure

```go
type Header struct {
    Magic       [4]byte  // "LTX\x00"
    Version     uint32   // Format version
    PageSize    uint32   // Database page size
    MinTXID     TXID     // Starting transaction ID
    MaxTXID     TXID     // Ending transaction ID
    Timestamp   int64    // Creation timestamp
    Checksum    uint64   // Header checksum
}
```

### Page Frame Structure

```go
type PageFrame struct {
    Header PageHeader
    Data   []byte     // Page data (pageSize bytes)
}

type PageHeader struct {
    PageNo   uint32    // Page number in database
    Size     uint32    // Size of page data
    Checksum uint64    // Page checksum
}
```

### Page Index

Binary search tree for efficient page lookup:
```go
type PageIndexElem struct {
    PageNo uint32    // Page number
    Offset int64     // Offset in file
    Size   uint32    // Size of page frame
}
```

### Trailer

```go
type Trailer struct {
    PageIndexOffset int64   // Offset to page index
    PageIndexSize   int64   // Size of page index
    PageCount       uint32  // Total pages in file
    Checksum        uint64  // Full file checksum
}
```

## WAL Monitoring Mechanism

### Monitor Loop (db.go:1499)

```go
func (db *DB) monitor() {
    ticker := time.NewTicker(db.MonitorInterval)
    defer ticker.Stop()

    for {
        select {
        case <-ticker.C:
            // Check WAL for changes
            changed, err := db.checkWAL()
            if err != nil {
                slog.Error("wal check failed", "error", err)
                continue
            }

            if changed {
                // Notify replicas of changes
                db.notifyReplicas()

                // Check if checkpoint needed
                if db.shouldCheckpoint() {
                    db.autoCheckpoint()
                }
            }

        case <-db.ctx.Done():
            return
        }
    }
}
```

### WAL Change Detection

```go
func (db *DB) checkWAL() (bool, error) {
    // Get current WAL state
    walInfo, err := db.walInfo()
    if err != nil {
        return false, err
    }

    // Compare with previous state
    db.mu.Lock()
    changed := walInfo.Size != db.prevWALSize ||
               walInfo.Checksum != db.prevWALChecksum
    db.prevWALSize = walInfo.Size
    db.prevWALChecksum = walInfo.Checksum
    db.mu.Unlock()

    return changed, nil
}
```

## Compaction Process

Compaction merges multiple LTX files to reduce storage overhead:

### Compaction Algorithm (store.go:189)

High-level compaction flow:

1. Determine whether the level is due for compaction (`Store.shouldCompact`).
2. Enumerate level-`L-1` files using `ReplicaClient.LTXFiles`, preferring local
   copies via `os.Open(db.LTXPath(...))` and falling back to
   `ReplicaClient.OpenLTXFile` only when necessary.
3. Stream the source readers through `ltx.NewCompactor`, which performs
   page-level deduplication and enforces lock-page skipping automatically.
4. Pipe the compactor output into `ReplicaClient.WriteLTXFile` to create the
   merged LTX file for level `L`.
5. Adjust the returned `ltx.FileInfo.CreatedAt` to the earliest timestamp from
   the source files so point-in-time recovery remains accurate.
6. Update the cached max file info for the level and delete old L0 files when
   promoting to level 1.

### Compaction Levels

```go
type CompactionLevel struct {
    Level    int           // Level number (0 = raw, 1+ = compacted)
    Interval time.Duration // How often to compact from previous level
}

// Default configuration
var DefaultCompactionLevels = CompactionLevels{
    {Level: 0, Interval: 0},          // Raw LTX files
    {Level: 1, Interval: 1 * Hour},   // Hourly compaction
    {Level: 2, Interval: 24 * Hour},  // Daily compaction
}
```

## Transaction Management

### Long-Running Read Transaction

Litestream maintains a long-running read transaction to ensure consistency:

```go
func (db *DB) initReadTx() error {
    // Start read transaction
    tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{
        ReadOnly: true,
    })
    if err != nil {
        return err
    }

    // Execute dummy query to start transaction
    var dummy string
    err = tx.QueryRow("SELECT ''").Scan(&dummy)
    if err != nil {
        tx.Rollback()
        return err
    }

    db.rtx = tx
    return nil
}
```

**Purpose:**
- Prevents database from being modified during replication
- Ensures consistent view of database
- Allows reading historical pages from WAL

### Checkpoint Coordination

```go
func (db *DB) Checkpoint(mode string) error {
    // Acquire checkpoint lock
    db.chkMu.Lock()
    defer db.chkMu.Unlock()

    // Close read transaction temporarily
    if db.rtx != nil {
        db.rtx.Rollback()
        db.rtx = nil
    }

    // Perform checkpoint
    _, _, err := db.db.Exec(fmt.Sprintf("PRAGMA wal_checkpoint(%s)", mode))
    if err != nil {
        return err
    }

    // Restart read transaction
    return db.initReadTx()
}
```

## Concurrency Model

### Mutex Usage Patterns

```go
// DB struct mutexes
type DB struct {
    mu     sync.RWMutex  // Protects struct fields
    chkMu  sync.RWMutex  // Checkpoint coordination
}

// Replica struct mutexes
type Replica struct {
    mu  sync.RWMutex    // Protects position
    muf sync.Mutex      // File descriptor lock
}

// Store struct mutex
type Store struct {
    mu sync.Mutex       // Protects database list
}
```

### Thundering Herd Prevention

`Store.Open()` (`store.go`) limits concurrent database opens at startup:

```go
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(50) // Max 50 concurrent DB opens
for _, db := range s.dbs {
    db := db
    g.Go(func() error { return db.Open() })
}
```

This prevents OS resource exhaustion (file descriptors, memory) when hundreds of databases are configured.

### Lock Ordering (Prevent Deadlocks)

Always acquire locks in this order:
1. Store.mu
2. DB.mu
3. DB.chkMu
4. Replica.mu

### Goroutine Management

```go
// Start background task
func (db *DB) Start() {
    db.wg.Add(1)
    go func() {
        defer db.wg.Done()
        db.monitor()
    }()
}

// Stop with timeout
func (db *DB) Close(ctx context.Context) error {
    // Signal shutdown
    db.cancel()

    // Wait for goroutines with timeout
    done := make(chan struct{})
    go func() {
        db.wg.Wait()
        close(done)
    }()

    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}
```

## State Management

### Database States

```mermaid
stateDiagram-v2
    [*] --> Closed
    Closed --> Opening: Open()
    Opening --> Open: Success
    Opening --> Closed: Error
    Open --> Monitoring: Start()
    Monitoring --> Syncing: Changes Detected
    Syncing --> Monitoring: Sync Complete
    Monitoring --> Checkpointing: Threshold Reached
    Checkpointing --> Monitoring: Checkpoint Complete
    Monitoring --> Closing: Close()
    Closing --> Closed: Cleanup Complete
```

### Replica States

```mermaid
stateDiagram-v2
    [*] --> Idle
    Idle --> Starting: Start()
    Starting --> Monitoring: Success
    Starting --> Idle: Error
    Monitoring --> Syncing: Timer/Changes
    Syncing --> Uploading: Have Changes
    Uploading --> Monitoring: Success
    Uploading --> Error: Failed
    Error --> Monitoring: Retry
    Monitoring --> Stopping: Stop()
    Stopping --> Idle: Cleanup
```

### Position Tracking

```go
type Pos struct {
    TXID     TXID      // Current transaction ID
    PageNo   uint32    // Current page number
    Checksum uint64    // Running checksum for validation
}

// Update position atomically
func (r *Replica) SetPos(pos ltx.Pos) {
    r.mu.Lock()  // MUST use Lock, not RLock!
    defer r.mu.Unlock()
    r.pos = pos
}

// Read position safely
func (r *Replica) Pos() ltx.Pos {
    r.mu.RLock()
    defer r.mu.RUnlock()
    return r.pos
}
```

## Initialization Flow

### System Startup Sequence

```mermaid
sequenceDiagram
    participant Main
    participant Store
    participant DB
    participant Replica
    participant Monitor

    Main->>Store: NewStore(config)
    Store->>Store: Validate config

    Main->>Store: Open()
    loop For each database
        Store->>DB: NewDB(path)
        Store->>DB: Open()
        DB->>DB: Open SQLite connection
        DB->>DB: Read page size
        DB->>DB: Init metadata
        DB->>DB: Start read transaction

        loop For each replica
            DB->>Replica: NewReplica()
            DB->>Replica: Start()
            Replica->>Monitor: Start monitoring
        end
    end

    Store->>Store: Start compaction monitors
    Store-->>Main: Ready
```

### Critical Initialization Steps

1. **Database Opening**
   ```go
   // Must happen in order:
   1. Open SQLite connection
   2. Read page size (PRAGMA page_size)
   3. Create metadata directory
   4. Start long-running read transaction
   5. Initialize replicas
   6. Start monitor goroutine
   ```

2. **Replica Initialization**
   ```go
   // Must happen in order:
   1. Create replica with client
   2. Load previous position from metadata
   3. Validate position against database
   4. Start sync goroutine (if monitoring enabled)
   ```

## Error Handling

### Error Categories

1. **Recoverable Errors**
   - Network timeouts
   - Temporary storage unavailability
   - Lock contention

2. **Fatal Errors**
   - Database corruption
   - Invalid configuration
   - Disk full

3. **Operational Errors**
   - Checkpoint failures
   - Compaction conflicts
   - Sync delays

### Error Propagation

```go
// Bottom-up error propagation
ReplicaClient.WriteLTXFile() error
    ↓
Replica.Sync() error
    ↓
DB.Sync() error
    ↓
Store.monitorDB() // Logs error, continues
```

### Retry Logic

```go
func (r *Replica) syncWithRetry(ctx context.Context) error {
    backoff := time.Second
    maxBackoff := time.Minute

    for attempt := 0; ; attempt++ {
        err := r.Sync(ctx)
        if err == nil {
            return nil
        }

        // Check if error is retryable
        if !isRetryable(err) {
            return err
        }

        // Check context
        if ctx.Err() != nil {
            return ctx.Err()
        }

        // Exponential backoff
        time.Sleep(backoff)
        backoff *= 2
        if backoff > maxBackoff {
            backoff = maxBackoff
        }
    }
}
```

## Performance Characteristics

### Time Complexity

| Operation | Complexity | Notes |
|-----------|------------|-------|
| WAL Monitor | O(1) | Fixed interval check |
| Page Write | O(1) | Append to LTX file |
| Compaction | O(n) | n = total pages |
| Restoration | O(n*log(m)) | n = pages, m = files |
| File List | O(k) | k = files in level |

### Space Complexity

| Component | Memory Usage | Disk Usage |
|-----------|-------------|------------|
| DB | O(1) + metrics | Original DB + WAL |
| Replica | O(1) | LTX files + metadata |
| Compaction | O(n) pages | Temporary during merge |
| Page Index | O(p) | p = pages in file |

### Optimization Points

1. **Page Index Caching**
   - Cache frequently accessed indices
   - Use estimated size for initial fetch

2. **Batch Operations**
   - Group small changes into larger LTX files
   - Batch delete operations

3. **Concurrent Operations**
   - Multiple replicas can sync in parallel
   - Compaction runs independently per level

## Security Considerations

### Access Control

- File permissions: 0600 for database files
- Directory permissions: 0700 for metadata
- No built-in authentication (rely on storage backend)

## Monitoring & Metrics

### Prometheus Metrics

```go
// Database metrics
db_size_bytes           // Current database size
wal_size_bytes         // Current WAL size
total_wal_bytes        // Total bytes written to WAL
checkpoint_count       // Number of checkpoints
sync_count            // Number of syncs
sync_error_count      // Number of sync errors

// Replica metrics
replica_lag_seconds    // Replication lag
replica_position      // Current replication position

// Validation metrics have db and level labels.
litestream_validation_checks_total                   // Checks by result: success, invalid, or error.
litestream_last_validation_success                   // Last check: 1 passed, 0 failed or not yet complete.
litestream_last_validation_success_timestamp_seconds // Unix time of last success, or 0 until a check passes.
```

### Health Checks

```go
func (db *DB) HealthCheck() error {
    // Check database connection
    if err := db.db.Ping(); err != nil {
        return fmt.Errorf("database ping failed: %w", err)
    }

    // Check replication lag
    for _, r := range db.replicas {
        lag := time.Since(r.LastSync())
        if lag > MaxAcceptableLag {
            return fmt.Errorf("replica %s lag too high: %v", r.Name(), lag)
        }
    }

    return nil
}
```

---

### CLI JSON OUTPUT (`docs/CLI_JSON_OUTPUT.md`)

# CLI JSON Output

Litestream commands that accept `-json` write a JSON document to stdout. Errors
are written to stderr by the top-level command handler and return a non-zero exit
status.

The fields below are the stable output contract for CLI consumers. New fields
may be added in future releases, so consumers should ignore unknown fields.

Runtime commands use `status` values that name the completed state. When a
command is already in its requested state, the status uses `already_<state>`,
such as `already_registered`, `already_running`, `already_stopped`, or
`already_unregistered`.

## `litestream databases -json`

Outputs an array of databases loaded from the configuration file.

```json
[
  {
    "path": "/var/lib/app.db",
    "replica": "s3"
  }
]
```

| Field | Type | Description |
| --- | --- | --- |
| `path` | string | SQLite database path. |
| `replica` | string | Replica client type configured for the database. |

## `litestream info -json`

Outputs daemon process information from the control socket.

```json
{
  "version": "v0.5.0",
  "pid": 12345,
  "uptime_seconds": 300,
  "started_at": "2026-04-24T12:00:00Z",
  "database_count": 2
}
```

| Field | Type | Description |
| --- | --- | --- |
| `version` | string | Litestream version reported by the daemon. |
| `pid` | number | Daemon process ID. |
| `uptime_seconds` | number | Daemon uptime in seconds. |
| `started_at` | string | Daemon start time in RFC 3339 format. |
| `database_count` | number | Number of databases currently managed by the daemon. |

## `litestream list -json`

Outputs databases managed by the running daemon.

```json
{
  "databases": [
    {
      "path": "/var/lib/app.db",
      "status": "replicating",
      "last_sync_at": "2026-04-24T12:00:00Z"
    }
  ]
}
```

| Field | Type | Description |
| --- | --- | --- |
| `databases` | array | Managed database summaries. |
| `databases[].path` | string | SQLite database path. |
| `databases[].status` | string | Current daemon state for the database, such as `replicating`, `open`, or `stopped`. |
| `databases[].last_sync_at` | string | Last successful replica sync time in RFC 3339 format. Omitted if the database has not synced successfully. |

## `litestream ltx -json`

Outputs LTX file metadata for the selected database or replica URL.

```json
[
  {
    "level": 0,
    "min_txid": "0000000000000001",
    "max_txid": "0000000000000004",
    "size": 8192,
    "timestamp": "2026-04-24T12:00:00Z"
  }
]
```

| Field | Type | Description |
| --- | --- | --- |
| `level` | number | LTX compaction level. |
| `min_txid` | string | Minimum transaction ID in the LTX file. |
| `max_txid` | string | Maximum transaction ID in the LTX file. |
| `size` | number | LTX file size in bytes. |
| `timestamp` | string | LTX file creation time in RFC 3339 format. |

## `litestream restore -json`

Outputs a final summary object after a restore completes. Restore logs are
written to stderr so stdout remains parseable JSON.

```json
{
  "db_path": "/var/lib/app.db",
  "replica": "file",
  "txid": "0000000000000004",
  "duration_ms": 125,
  "integrity_check": "quick"
}
```

| Field | Type | Description |
| --- | --- | --- |
| `db_path` | string | Restored database path. |
| `replica` | string | Replica client type used for the restore. |
| `txid` | string | Restored transaction ID, when available. |
| `duration_ms` | number | Restore duration in milliseconds. |
| `integrity_check` | string | Integrity check mode used for the restore: `none`, `quick`, or `full`. |

When `-dry-run` is combined with `-json`, `restore` outputs the restore plan
instead of the final restore summary.

```json
{
  "source": "file:///backups/app.db",
  "target_path": "/var/lib/app.db",
  "replica": "file",
  "min_txid": "0000000000000001",
  "max_txid": "0000000000000004",
  "files": [
    {
      "level": 9,
      "name": "0000000000000001-0000000000000004.ltx",
      "min_txid": "0000000000000001",
      "max_txid": "0000000000000004",
      "size": 8192,
      "timestamp": "2026-04-24T12:00:00Z"
    }
  ]
}
```

| Field | Type | Description |
| --- | --- | --- |
| `source` | string | Database path or replica URL passed to `restore`. |
| `target_path` | string | Database path that would be written by a restore. |
| `replica` | string | Replica client type used to build the plan. |
| `min_txid` | string | Minimum transaction ID included in the plan. |
| `max_txid` | string | Maximum transaction ID included in the plan. |
| `files` | array | LTX files that would be fetched. |
| `files[].level` | number | LTX compaction level. |
| `files[].name` | string | LTX file name. |
| `files[].min_txid` | string | Minimum transaction ID in the file. |
| `files[].max_txid` | string | Maximum transaction ID in the file. |
| `files[].size` | number | File size in bytes. |
| `files[].timestamp` | string | File creation time in RFC 3339 format. |

## `litestream status -json`

Outputs replication status for configured databases.

```json
[
  {
    "database": "/var/lib/app.db",
    "status": "ok",
    "local_txid": "0000000000000004",
    "wal_size": "32 kB"
  }
]
```

| Field | Type | Description |
| --- | --- | --- |
| `database` | string | SQLite database path. |
| `status` | string | Current local replication status: `ok`, `not initialized`, `no database`, or `error`. |
| `local_txid` | string | Latest local LTX transaction ID, or `-` if unavailable. |
| `wal_size` | string | Current WAL file size as human-readable text, or `-` if unavailable. |

---

### DOC MAINTENANCE (`docs/DOC_MAINTENANCE.md`)

# Documentation Maintenance Guide

This guide ensures documentation stays synchronized with code changes and follows the principle-based approach established in PR #787.

## Philosophy: Principles Over Examples

**Key Insight**: Code examples become outdated quickly. Documentation should focus on **stable concepts** rather than **volatile implementations**.

### What to Document

✅ **DO Document**:

- **Architectural principles** (e.g., "DB layer handles database state")
- **Interface contracts** (what methods must do, not how they do it)
- **Design patterns** (atomic file operations, eventual consistency handling)
- **Critical edge cases** (1GB lock page, timestamp preservation)
- **"Why" not "what"** (rationale behind decisions)

❌ **DON'T Document**:

- Specific function implementations that change frequently
- Exact function names without referencing actual source
- Step-by-step code that duplicates the implementation
- Version-specific details that will quickly become stale

### Documentation Principles

1. **Abstractions over Details**: Document the concept, not the specific implementation
2. **Reference over Duplication**: Point to actual source files instead of copying code
3. **Patterns over Examples**: Describe the approach, let developers read the source
4. **Contracts over Implementations**: Define what must happen, not how

## When Code Changes, Update Docs

### Interface Changes

**Trigger**: Modifying `ReplicaClient` interface or any public interface

**Required Updates**:

1. Search for interface definitions in docs:

   ```bash
   rg "type ReplicaClient interface" docs/ CLAUDE.md AGENTS.md .claude/
   ```

2. Update interface signatures (don't forget parameters!)
3. Document new parameters with clear explanations of when/why to use them
4. Update all example calls to include new parameters

**Files to Check**:

- `AGENTS.md` - Interface definitions
- `docs/REPLICA_CLIENT_GUIDE.md` - Implementation guide
- `docs/TESTING_GUIDE.md` - Test examples
- `.claude/agents/replica-client-developer.md` - Agent knowledge
- `.claude/commands/add-storage-backend.md` - Backend templates
- `.claude/commands/validate-replica.md` - Validation commands

### New Features

**Trigger**: Adding new functionality, methods, or components

**Approach**:

1. **Don't rush to document** - Wait until the feature stabilizes
2. **Document the pattern**, not the implementation:
   - What problem does it solve?
   - What's the high-level approach?
   - What are the critical constraints?
3. **Reference the source**:
   - `See implementation in file.go:lines`
   - `Reference tests in file_test.go`

### Refactoring

**Trigger**: Moving or renaming functions, restructuring code

**Required Actions**:

1. **Search for references**:

   ```bash
   # Find function name references
   rg "functionName" docs/ CLAUDE.md AGENTS.md .claude/
   ```

2. **Update or remove**:
   - If it's a reference pointer (e.g., "See `DB.init()` in db.go:123"), update it
   - If it's a code example showing implementation, consider replacing with a pattern description

3. **Verify links**: Ensure all file:line references are still valid

## Documentation Update Checklist

Use this checklist when making code changes:

- [ ] **Search docs for affected code**:

  ```bash
  # Search for function names, types, or concepts
  rg "YourFunctionName" docs/ CLAUDE.md AGENTS.md .claude/
  ```

- [ ] **Update interface definitions** if signatures changed
- [ ] **Update examples** if they won't compile anymore
- [ ] **Convert brittle examples to patterns** if refactoring made them stale
- [ ] **Update file:line references** if code moved
- [ ] **Verify contracts still hold** (update if behavior changed)
- [ ] **Run markdownlint**:

  ```bash
  markdownlint --fix docs/ CLAUDE.md AGENTS.md .claude/
  ```

## Preventing Documentation Drift

### Pre-Commit Practices

1. **Search before committing**:

   ```bash
   git diff --name-only | xargs -I {} rg "basename {}" docs/
   ```

2. **Review doc references** in your PR description
3. **Test examples compile** (if they're meant to)

### Regular Audits

**Monthly**: Spot-check one documentation file against current codebase

**Questions to ask**:

- Do interface definitions match `replica_client.go`?
- Do code examples compile?
- Are file:line references accurate?
- Have we removed outdated examples?

### When in Doubt

**Rule**: Delete outdated documentation rather than let it mislead

- Stale examples cause compilation errors
- Outdated patterns cause architectural mistakes
- Incorrect references waste developer time

**Better**: A brief pattern description + reference to source than an outdated example

## Example: Good vs Bad Documentation Updates

### ❌ Bad: Copying Implementation

```markdown
### How to initialize DB

```go
func (db *DB) init() {
    db.mu.Lock()
    defer db.mu.Unlock()
    // ... 50 lines of code copied from db.go
}
\```
```

**Problem**: This will be outdated as soon as the implementation changes.

### ✅ Good: Documenting Pattern + Reference

```markdown
### DB Initialization Pattern

**Principle**: Database initialization must complete before replication starts.

**Pattern**:

1. Acquire exclusive lock (`mu.Lock()`)
2. Verify database state consistency
3. Initialize monitoring subsystems
4. Set up replication coordination

**Critical**: Use `Lock()` not `RLock()` as initialization modifies state.

**Reference Implementation**: See `DB.init()` in db.go:150-230
```

**Benefits**: Stays accurate even if implementation details change, focuses on the "why" and "what" rather than the "how".

## Tools and Commands

### Find Documentation References

```bash
# Find all code examples in documentation
rg "^```(go|golang)" docs/ CLAUDE.md AGENTS.md .claude/

# Find file:line references
rg "\.go:\d+" docs/ CLAUDE.md AGENTS.md .claude/

# Find interface definitions
rg "type .* interface" docs/ CLAUDE.md AGENTS.md .claude/
```

### Validate Markdown

```bash
# Lint all docs
markdownlint docs/ CLAUDE.md AGENTS.md .claude/

# Auto-fix issues
markdownlint --fix docs/ CLAUDE.md AGENTS.md .claude/
```

### Check for Broken References

```bash
# List all go files mentioned in docs
rg -o "[a-z_]+\.go:\d+" docs/ CLAUDE.md AGENTS.md | sort -u

# Verify they exist and line numbers are reasonable
```

## Resources

- **PR #787**: Original principle-based documentation refactor
- **Issue #805**: Context for why accurate documentation matters
- **INNOQ Best Practices**: <https://www.innoq.com/en/articles/2022/01/principles-of-technical-documentation/>
- **Google Style Guide**: <https://google.github.io/styleguide/docguide/best_practices.html>

## Questions?

When updating documentation, ask:

1. **Is this a stable concept or a volatile implementation?**
   - Stable → Document the principle
   - Volatile → Reference the source

2. **Will this stay accurate for 6+ months?**
   - Yes → Keep it
   - No → Replace with pattern description

3. **Does this explain WHY or just WHAT?**
   - WHY → Valuable documentation
   - WHAT → Code already shows this, just reference it

4. **Would a link to source code be better?**
   - Often, yes!

---

### LTX FORMAT (`docs/LTX_FORMAT.md`)

# LTX Format Specification

LTX (Log Transaction) is Litestream's custom format for storing database changes in an immutable, append-only manner.

## Table of Contents
- [Overview](#overview)
- [File Structure](#file-structure)
- [Header Format](#header-format)
- [Page Frames](#page-frames)
- [Page Index](#page-index)
- [Trailer Format](#trailer-format)
- [File Naming Convention](#file-naming-convention)
- [Checksum Calculation](#checksum-calculation)
- [Compaction and Levels](#compaction-and-levels)
- [Reading LTX Files](#reading-ltx-files)
- [Writing LTX Files](#writing-ltx-files)
- [Relationship to SQLite WAL](#relationship-to-sqlite-wal)

## Overview

LTX files are immutable snapshots of database changes:
- **Immutable**: Once written, never modified
- **Append-only**: New changes create new files
- **Self-contained**: Each file is independent
- **Indexed**: Contains page index for efficient seeks
- **Checksummed**: Integrity verification built-in

```mermaid
graph LR
    WAL[SQLite WAL] -->|Convert| LTX[LTX File]
    LTX -->|Upload| Storage[Cloud Storage]
    Storage -->|Download| Restore[Restored DB]
```

## File Structure

```
┌─────────────────────┐
│      Header         │ Fixed size (varies by version)
├─────────────────────┤
│                     │
│    Page Frames      │ Variable number of pages
│                     │
├─────────────────────┤
│    Page Index       │ Binary search tree
├─────────────────────┤
│      Trailer        │ Fixed size metadata
└─────────────────────┘
```

### Size Calculation

```go
FileSize = HeaderSize +
           (PageCount * (PageHeaderSize + PageSize)) +
           PageIndexSize +
           TrailerSize
```

## Header Format

The LTX header contains metadata about the file:

```go
// From github.com/superfly/ltx
type Header struct {
    Version          int      // Derived from the magic string ("LTX1")
    Flags            uint32   // Reserved flag bits
    PageSize         uint32   // Database page size
    Commit           uint32   // Page count after applying file
    MinTXID          TXID
    MaxTXID          TXID
    Timestamp        int64    // Milliseconds since Unix epoch
    PreApplyChecksum Checksum // Database checksum before apply
    WALOffset        int64    // Offset within source WAL (0 for snapshots)
    WALSize          int64    // WAL byte length (0 for snapshots)
    WALSalt1         uint32
    WALSalt2         uint32
    NodeID           uint64
}

const HeaderFlagNoChecksum = uint32(1 << 1)
```

> Note: the version is implied by the magic string. Present files use
> `Magic == "LTX1"`, which corresponds to `ltx.Version == 2`.

### Binary Layout (Header)

```
Offset  Size  Field
0       4     Magic ("LTX1")
4       4     Flags
8       4     PageSize
12      4     Commit
16      8     MinTXID
24      8     MaxTXID
32      8     Timestamp
40      8     PreApplyChecksum
48      8     WALOffset
56      8     WALSize
64      4     WALSalt1
68      4     WALSalt2
72      8     NodeID
80     20     Reserved (zeros)
Total: 100 bytes
```

## Page Frames

Each page frame contains a database page with metadata:

```go
type PageFrame struct {
    Header PageHeader
    Data   []byte  // Size = PageSize from LTX header
}

type PageHeader struct {
    Pgno uint32  // Database page number (1-based)
}
```

### Binary Layout (Page Frame)

```
Offset  Size     Field
0       4        Page Number (Pgno)
4       PageSize Page Data
```

### Page Frame Constraints

1. **Sequential Writing**: Pages written in order during creation
2. **Random Access**: Can seek to any page using index
3. **Lock Page Skipping**: Page at 1GB boundary never included
4. **Deduplication**: In compacted files, only latest version of each page

## Page Index

The page index enables efficient random access to pages:

```go
type PageIndexElem struct {
    Level   int
    MinTXID TXID
    MaxTXID TXID
    Offset  int64 // Byte offset of encoded payload
    Size    int64 // Bytes occupied by encoded payload
}
```

### Binary Layout (Page Index)

```
Rather than parsing raw bytes, call `ltx.DecodePageIndex` which returns a
map of page number to `ltx.PageIndexElem` for you.
```

### Index Usage

```go
// Finding a page using the index
func findPage(index []PageIndexElem, targetPageNo uint32) (offset int64, found bool) {
    // Binary search
    idx := sort.Search(len(index), func(i int) bool {
        return index[i].PageNo >= targetPageNo
    })

    if idx < len(index) && index[idx].PageNo == targetPageNo {
        return index[idx].Offset, true
    }
    return 0, false
}
```

## Trailer Format

The trailer contains metadata and pointers:

```go
type Trailer struct {
    PostApplyChecksum Checksum // Database checksum after apply
    FileChecksum      Checksum // CRC-64 checksum of entire file
}
```

### Binary Layout (Trailer)

```
Offset  Size  Field
0       8     PostApplyChecksum
8       8     FileChecksum
Total: 16 bytes
```

### Reading Trailer

The trailer is always at the end of the file:

```go
func readTrailer(f *os.File) (*Trailer, error) {
    // Seek to trailer position
    _, err := f.Seek(-TrailerSize, io.SeekEnd)
    if err != nil {
        return nil, err
    }

    var trailer Trailer
    err = binary.Read(f, binary.BigEndian, &trailer)
    return &trailer, err
}
```

## File Naming Convention

LTX files follow a strict naming pattern:

```
Format: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx
Where:
  M = MinTXID (16 hex digits, zero-padded)
  N = MaxTXID (16 hex digits, zero-padded)

Examples:
  0000000000000001-0000000000000064.ltx  (TXID 1-100)
  0000000000000065-00000000000000c8.ltx  (TXID 101-200)
```

### Backend Storage Paths

Replica storage paths are backend-specific and are not portable by recursively
copying their raw contents:

| Backend | Path | Example for level 9 |
|---------|------|---------------------|
| `file://` | `<root>/ltx/<level>/<filename>.ltx` | `<root>/ltx/9/0000000000000001-0000000000000064.ltx` |
| `s3://` | `<prefix>/<level as four-digit lowercase hexadecimal>/<filename>.ltx` | `<prefix>/0009/0000000000000001-0000000000000064.ltx` |

These established layouts remain different for backward compatibility with
existing replicas. Neither backend writes a separate replica index or manifest
object; file discovery scans the level directory or object prefix, and the page
index is embedded in each LTX file.

Do not copy a `file://` replica directory directly into an S3 bucket. Restore
the database from the file replica locally, then create a new `s3://` replica
from the restored database.

### Parsing Filenames

```go
// From github.com/superfly/ltx
func ParseFilename(name string) (minTXID, maxTXID TXID, err error) {
    // Remove extension
    name = strings.TrimSuffix(name, ".ltx")

    // Split on hyphen
    parts := strings.Split(name, "-")
    if len(parts) != 2 {
        return 0, 0, errors.New("invalid format")
    }

    // Parse hex values
    min, err := strconv.ParseUint(parts[0], 16, 64)
    max, err := strconv.ParseUint(parts[1], 16, 64)

    return TXID(min), TXID(max), nil
}

func FormatFilename(minTXID, maxTXID TXID) string {
    return fmt.Sprintf("%016x-%016x.ltx", minTXID, maxTXID)
}
```

## Checksum Calculation

LTX uses CRC-64 ECMA checksums:

```go
import "hash/crc64"

var crcTable = crc64.MakeTable(crc64.ECMA)

func calculateChecksum(data []byte) uint64 {
    return crc64.Checksum(data, crcTable)
}

// Cumulative checksum for multiple pages
func cumulativeChecksum(pages [][]byte) uint64 {
    h := crc64.New(crcTable)
    for _, page := range pages {
        h.Write(page)
    }
    return h.Sum64()
}
```

### Verification During Read

```go
func verifyPage(header PageHeader, data []byte) error {
    if header.Checksum == 0 {
        return nil // Checksums disabled
    }

    calculated := calculateChecksum(data)
    if calculated != header.Checksum {
        return fmt.Errorf("checksum mismatch: expected %x, got %x",
            header.Checksum, calculated)
    }
    return nil
}
```

## Compaction and Levels

LTX files are organized in levels for efficient compaction:

```
Level 0: Raw files (no compaction)
         /ltx/0000/0000000000000001-0000000000000064.ltx
         /ltx/0000/0000000000000065-00000000000000c8.ltx

Level 1: Hourly compaction
         /ltx/0001/0000000000000001-0000000000000fff.ltx

Level 2: Daily compaction
         /ltx/0002/0000000000000001-000000000000ffff.ltx

Snapshots: Full database state
          /snapshots/20240101120000.ltx
```

### Compaction Process

```go
func compactLTXFiles(files []*LTXFile) (*LTXFile, error) {
    // Create page map (newer overwrites older)
    pageMap := make(map[uint32]Page)

    for _, file := range files {
        for _, page := range file.Pages {
            pageMap[page.Number] = page
        }
    }

    // Create new LTX with merged pages
    merged := &LTXFile{
        MinTXID: files[0].MinTXID,
        MaxTXID: files[len(files)-1].MaxTXID,
    }

    // Add pages in order (skip lock page)
    for pgno := uint32(1); pgno <= maxPgno; pgno++ {
        if pgno == LockPageNumber(pageSize) {
            continue // Skip 1GB lock page
        }
        if page, ok := pageMap[pgno]; ok {
            merged.Pages = append(merged.Pages, page)
        }
    }

    return merged, nil
}
```

## Reading LTX Files

### Complete File Read

```go
func ReadLTXFile(path string) (*LTXFile, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    defer f.Close()

    dec := ltx.NewDecoder(f)

    // Read and verify header
    header, err := dec.Header()
    if err != nil {
        return nil, err
    }

    // Read all pages
    var pages []Page
    for {
        var pageHeader ltx.PageHeader
        pageData := make([]byte, header.PageSize)

        err := dec.DecodePage(&pageHeader, pageData)
        if err == io.EOF {
            break
        }
        if err != nil {
            return nil, err
        }

        pages = append(pages, Page{
            Number: pageHeader.PageNo,
            Data:   pageData,
        })
    }

    return &LTXFile{
        Header: header,
        Pages:  pages,
    }, nil
}
```

### Partial Read Using Index

```go
func ReadPage(path string, pageNo uint32) ([]byte, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    defer f.Close()

    // Read trailer to find index
    trailer, err := readTrailer(f)
    if err != nil {
        return nil, err
    }

    // Read page index
    f.Seek(trailer.PageIndexOffset, io.SeekStart)
    indexData := make([]byte, trailer.PageIndexSize)
    f.Read(indexData)

    index := parsePageIndex(indexData)

    // Find page in index
    offset, found := findPage(index, pageNo)
    if !found {
        return nil, errors.New("page not found")
    }

    // Read page at offset
    f.Seek(offset, io.SeekStart)

    var pageHeader PageHeader
    binary.Read(f, binary.BigEndian, &pageHeader)

    pageData := make([]byte, pageSize)
    f.Read(pageData)

    return pageData, nil
}
```

## Writing LTX Files

### Creating New LTX File

```go
func WriteLTXFile(path string, pages []Page) error {
    f, err := os.Create(path)
    if err != nil {
        return err
    }
    defer f.Close()

    enc := ltx.NewEncoder(f)

    // Write header
    header := ltx.Header{
        Version:   ltx.Version,
        Flags:     0,
        PageSize:  4096,
        PageCount: uint32(len(pages)),
        MinTXID:   minTXID,
        MaxTXID:   maxTXID,
    }

    if err := enc.EncodeHeader(header); err != nil {
        return err
    }

    // Write pages and build index
    var index []PageIndexElem
    for _, page := range pages {
        offset := enc.Offset()

        // Skip lock page
        if page.Number == LockPageNumber(header.PageSize) {
            continue
        }

        pageHeader := ltx.PageHeader{
            PageNo:   page.Number,
            Checksum: calculateChecksum(page.Data),
        }

        if err := enc.EncodePage(pageHeader, page.Data); err != nil {
            return err
        }

        index = append(index, PageIndexElem{
            PageNo: page.Number,
            Offset: offset,
        })
    }

    // Write page index
    if err := enc.EncodePageIndex(index); err != nil {
        return err
    }

    // Write trailer
    if err := enc.EncodeTrailer(); err != nil {
        return err
    }

    return enc.Close()
}
```

## Relationship to SQLite WAL

### WAL to LTX Conversion

```mermaid
sequenceDiagram
    participant SQLite
    participant WAL
    participant Litestream
    participant LTX

    SQLite->>WAL: Write transaction
    WAL->>WAL: Append frames

    Litestream->>WAL: Monitor changes
    WAL-->>Litestream: Read frames

    Litestream->>Litestream: Convert frames
    Note over Litestream: - Skip lock page<br/>- Add checksums<br/>- Build index

    Litestream->>LTX: Write LTX file
    LTX->>Storage: Upload
```

### Key Differences

| Aspect | SQLite WAL | LTX Format |
|--------|------------|------------|
| Purpose | Temporary changes | Permanent archive |
| Mutability | Mutable (checkpoint) | Immutable |
| Structure | Sequential frames | Indexed pages |
| Checksum | Per-frame | Per-page + cumulative |
| Lock Page | Contains lock bytes | Always skipped |
| Naming | Fixed (-wal suffix) | TXID range |
| Lifetime | Until checkpoint | Forever |
| Size | Grows until checkpoint | Fixed at creation |

### Transaction ID (TXID)

```go
type TXID uint64

// TXID represents a logical transaction boundary
// Not directly from SQLite, but derived from:
// 1. WAL checkpoint sequence
// 2. Frame count
// 3. Logical grouping of changes

func (db *DB) nextTXID() TXID {
    // Increment from last known TXID
    return db.lastTXID + 1
}
```

## Best Practices

### 1. Always Skip Lock Page

```go
const PENDING_BYTE = 0x40000000

func shouldSkipPage(pageNo uint32, pageSize int) bool {
    lockPage := uint32(PENDING_BYTE/pageSize) + 1
    return pageNo == lockPage
}
```

### 2. Preserve Timestamps During Compaction

```go
// Keep earliest CreatedAt from source files
func compactWithTimestamp(files []*FileInfo) *FileInfo {
    earliest := files[0].CreatedAt
    for _, f := range files[1:] {
        if f.CreatedAt.Before(earliest) {
            earliest = f.CreatedAt
        }
    }

    return &FileInfo{
        CreatedAt: earliest, // Preserve for point-in-time recovery
    }
}
```

### 3. Verify Checksums on Read

```go
func safeReadLTX(path string) (*LTXFile, error) {
    file, err := ReadLTXFile(path)
    if err != nil {
        return nil, err
    }

    // Verify all checksums
    for _, page := range file.Pages {
        if err := verifyPage(page); err != nil {
            return nil, fmt.Errorf("corrupted page %d: %w",
                page.Number, err)
        }
    }

    return file, nil
}
```

### 4. Handle Partial Files

```go
// For eventually consistent storage
func readWithRetry(client ReplicaClient, info *FileInfo) ([]byte, error) {
    for attempts := 0; attempts < 5; attempts++ {
        data, err := client.OpenLTXFile(...)
        if err == nil {
            // Verify we got complete file
            if int64(len(data)) == info.Size {
                return data, nil
            }
        }

        time.Sleep(time.Second * time.Duration(attempts+1))
    }

    return nil, errors.New("incomplete file after retries")
}
```

## Debugging LTX Files

### Inspect LTX Files

The Litestream CLI currently exposes a single helper for listing LTX files:

```bash
litestream ltx /path/to/db.sqlite
litestream ltx s3://bucket/db
```

For low-level inspection (page payloads, checksums, etc.), use the Go API:

```go
f, err := os.Open("0000000000000001-0000000000000064.ltx")
if err != nil {
    log.Fatal(err)
}
defer f.Close()

dec := ltx.NewDecoder(f)
if err := dec.DecodeHeader(); err != nil {
    log.Fatal(err)
}
for {
    var hdr ltx.PageHeader
    data := make([]byte, dec.Header().PageSize)
    if err := dec.DecodePage(&hdr, data); err == io.EOF {
        break
    } else if err != nil {
        log.Fatal(err)
    }
    // Inspect hdr.Pgno or data here.
}
if err := dec.Close(); err != nil {
    log.Fatal(err)
}
fmt.Println("post-apply checksum:", dec.Trailer().PostApplyChecksum)
```

## Summary

LTX format provides:
1. **Immutable history** - Every change preserved
2. **Efficient storage** - Indexed, compressed via compaction
3. **Data integrity** - Checksums at multiple levels
4. **Point-in-time recovery** - Via TXID ranges
5. **Cloud-optimized** - Designed for object storage

Understanding LTX is essential for:
- Implementing replica clients
- Debugging replication issues
- Optimizing compaction
- Ensuring data integrity
- Building recovery tools

---

### PATTERNS (`docs/PATTERNS.md`)

# Litestream Code Patterns and Anti-Patterns

This document contains detailed code patterns, examples, and anti-patterns for working with Litestream. For a quick overview, see [AGENTS.md](../AGENTS.md).

## Table of Contents

- [Architectural Boundaries](#architectural-boundaries)
- [Atomic File Operations](#atomic-file-operations)
- [Error Handling](#error-handling)
- [Locking Patterns](#locking-patterns)
- [Compaction and Eventual Consistency](#compaction-and-eventual-consistency)
- [Resumable Reader Pattern](#resumable-reader-pattern)
- [Retention Bypass Pattern](#retention-bypass-pattern)
- [Conditional Write Pattern (Distributed Locking)](#conditional-write-pattern-distributed-locking)
- [Timestamp Preservation](#timestamp-preservation)
- [Common Pitfalls](#common-pitfalls)
- [Component Reference](#component-reference)

## Architectural Boundaries

### Layer Responsibilities

```text
DB Layer (db.go)          → Database state, restoration, monitoring
Replica Layer (replica.go) → Replication mechanics only
Storage Layer             → ReplicaClient implementations
```

### DO: Handle database state in DB layer

Database restoration logic belongs in the DB layer, not the Replica layer.

When the database is behind the replica (local TXID < remote TXID):

1. **Clear local L0 cache**: Remove the entire L0 directory and recreate it
2. **Fetch latest L0 file from replica**: Download the most recent L0 LTX file
3. **Write using atomic file operations**: Prevent partial/corrupted files

```go
// CORRECT - DB layer handles database state
func (db *DB) init() error {
    // DB layer handles database state
    if db.needsRestore() {
        if err := db.restore(); err != nil {
            return err
        }
    }
    // Then start replica for replication only
    return db.replica.Start()
}

func (r *Replica) Start() error {
    // Replica focuses only on replication
    return r.startSync()
}
```

Reference: `DB.checkDatabaseBehindReplica()` in db.go:670-737

### DON'T: Put database state logic in Replica layer

```go
// WRONG - Replica should only handle replication concerns
func (r *Replica) Start() error {
    // DON'T check database state here
    if needsRestore() {  // Wrong layer!
        restoreDatabase()  // Wrong layer!
    }
    // Replica should focus only on replication mechanics
}
```

## Atomic File Operations

Always use atomic writes to prevent partial/corrupted files.

### DO: Write to temp file, then rename

```go
// CORRECT - Atomic file write pattern
func writeFileAtomic(path string, data []byte) error {
    // Create temp file in same directory (for atomic rename)
    dir := filepath.Dir(path)
    tmpFile, err := os.CreateTemp(dir, ".tmp-*")
    if err != nil {
        return fmt.Errorf("create temp file: %w", err)
    }
    tmpPath := tmpFile.Name()

    // Clean up temp file on error
    defer func() {
        if tmpFile != nil {
            tmpFile.Close()
            os.Remove(tmpPath)
        }
    }()

    // Write data to temp file
    if _, err := tmpFile.Write(data); err != nil {
        return fmt.Errorf("write temp file: %w", err)
    }

    // Sync to ensure data is on disk
    if err := tmpFile.Sync(); err != nil {
        return fmt.Errorf("sync temp file: %w", err)
    }

    // Close before rename
    if err := tmpFile.Close(); err != nil {
        return fmt.Errorf("close temp file: %w", err)
    }
    tmpFile = nil // Prevent defer cleanup

    // Atomic rename (on same filesystem)
    if err := os.Rename(tmpPath, path); err != nil {
        os.Remove(tmpPath)
        return fmt.Errorf("rename to final path: %w", err)
    }

    return nil
}
```

### DON'T: Write directly to final location

```go
// WRONG - Can leave partial files on failure
func writeFileDirect(path string, data []byte) error {
    return os.WriteFile(path, data, 0644)  // Not atomic!
}
```

## Error Handling

### Decision Rule

When you handle an error, ask: "Does the caller need to know about this failure?"

- **Yes → return the error.** This is the default for virtually all cases in Litestream.
  - The result is needed for correctness
  - Failure could corrupt data or state
  - You're in a loop processing items

- **No → DEBUG log only.** This is rare. Only when ALL of these are true:
  - A valid fallback path exists that doesn't depend on the result
  - Failure cannot affect correctness
  - The operation is purely supplementary (e.g., reading an optimization hint)

When in doubt, return the error. In a disaster recovery tool, silent failures are worse than noisy ones.

### DO: Return errors immediately

```go
// CORRECT - Return error for caller to handle
func (db *DB) validatePosition() error {
    dpos, err := db.Pos()
    if err != nil {
        return err
    }
    rpos := replica.Pos()
    if dpos.TXID < rpos.TXID {
        return fmt.Errorf("database position (%v) behind replica (%v)", dpos, rpos)
    }
    return nil
}
```

### DON'T: Continue on critical errors

```go
// WRONG - Silently continuing can cause data corruption
func (db *DB) validatePosition() {
    if dpos, _ := db.Pos(); dpos.TXID < replica.Pos().TXID {
        log.Printf("warning: position mismatch")  // Don't just log!
        // Continuing here is dangerous
    }
}
```

### DON'T: Ignore errors and continue in loops

```go
// WRONG - Continuing after error can corrupt state
func (db *DB) processFiles() {
    for _, file := range files {
        if err := processFile(file); err != nil {
            log.Printf("error: %v", err)  // Just logging!
            // Continuing to next file is dangerous
        }
    }
}
```

### DO: Return errors properly in loops

```go
// CORRECT - Let caller decide how to handle errors
func (db *DB) processFiles() error {
    for _, file := range files {
        if err := processFile(file); err != nil {
            return fmt.Errorf("process file %s: %w", file, err)
        }
    }
    return nil
}
```

## Locking Patterns

### DO: Use proper lock types

```go
// CORRECT - Use Lock() for writes
r.mu.Lock()
defer r.mu.Unlock()
r.pos = pos
```

### DON'T: Use RLock for write operations

```go
// WRONG - Race condition
r.mu.RLock()  // Should be Lock() for writes
defer r.mu.RUnlock()
r.pos = pos   // Writing with RLock!
```

## Compaction and Eventual Consistency

Many storage backends (S3, R2, etc.) are eventually consistent:

- A file you just wrote might not be immediately readable
- A file might be listed but only partially available
- Reads might return stale or incomplete data

### DO: Read from local when available

```go
// CORRECT - Check local first during compaction
// db.go:1280-1294 - ALWAYS read from local disk when available
f, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))
if err == nil {
    // Use local file - it's complete and consistent
    return f, nil
}
// Only fall back to remote if local doesn't exist
return replica.Client.OpenLTXFile(...)
```

### DON'T: Read from remote during compaction

```go
// WRONG - Can get partial/corrupt data from eventually consistent storage
f, err := client.OpenLTXFile(ctx, level, minTXID, maxTXID, 0, 0)
```

## Resumable Reader Pattern

During restore, LTX file streams from S3/Tigris may sit idle while the compactor processes lower-numbered pages from the snapshot. Storage providers close these idle connections, causing "unexpected EOF" errors.

The `ResumableReader` (`internal/resumable_reader.go`) wraps `io.ReadCloser` with automatic reconnection:

### Interface

```go
type LTXFileOpener interface {
    OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)
}
```

### Two Failure Modes

1. **Non-EOF errors** (connection reset, timeout) — stream broke mid-transfer
2. **Premature EOF** — server closed cleanly, but `offset < size` (known file size)

### Key Behaviors

- Max 3 retries (`resumableReaderMaxRetries = 3`)
- Tracks current byte `offset` for range-request resume
- Returns partial reads without error so callers like `io.ReadFull` naturally retry
- Reopens from current offset using `OpenLTXFile(ctx, level, min, max, offset, 0)`

### DO: Use ResumableReader for restore streams

```go
rc, _ := client.OpenLTXFile(ctx, level, min, max, 0, fileInfo.Size)
rr := internal.NewResumableReader(ctx, client, level, min, max, fileInfo.Size, rc, logger)
defer rr.Close()
```

### DON'T: Use raw OpenLTXFile during long restore operations

```go
rc, _ := client.OpenLTXFile(ctx, level, min, max, 0, 0)
// Risk: connection may drop during idle periods in multi-file restore
io.ReadFull(rc, buf) // unexpected EOF!
```

## Retention Bypass Pattern

When using cloud provider lifecycle policies (S3 lifecycle rules, R2 auto-cleanup), Litestream's active file deletion can be disabled:

```yaml
retention:
  enabled: false
```

### Propagation Chain

1. `RetentionConfig{Enabled *bool}` in YAML config (`cmd/litestream/main.go`)
2. `Store.SetRetentionEnabled(bool)` propagates to all DBs and their compactors (`store.go`)
3. `Compactor.RetentionEnabled` guards 3 deletion points in `compactor.go`

### DO: Disable retention when cloud lifecycle handles cleanup

```go
store.SetRetentionEnabled(false) // Delegates deletion to cloud lifecycle policies
```

### DON'T: Disable retention without cloud lifecycle policies

Disabling retention without cloud lifecycle policies causes unbounded storage growth. Litestream logs a warning: "retention disabled; cloud provider lifecycle policies must handle retention".

## Conditional Write Pattern (Distributed Locking)

The S3 leaser (`s3/leaser.go`) uses S3 conditional writes (`If-Match`/`If-None-Match`) for distributed locking without an external coordination service.

### Acquire Pattern

```go
input := &s3.PutObjectInput{
    Bucket: aws.String(l.Bucket),
    Key:    aws.String(key),
    Body:   bytes.NewReader(data),
}
if etag == "" {
    input.IfNoneMatch = aws.String("*") // First acquire: only if key doesn't exist
} else {
    input.IfMatch = aws.String(etag)     // Expired takeover: only if ETag matches
}
```

### Release Pattern

```go
_, err := l.s3.DeleteObject(ctx, &s3.DeleteObjectInput{
    Bucket:  aws.String(l.Bucket),
    Key:     aws.String(key),
    IfMatch: aws.String(lease.ETag), // Only delete if we still hold the lease
})
```

### Error Handling

- **HTTP 412 (PreconditionFailed)**: Another instance acquired/renewed the lease
- **HTTP 404 (NoSuchKey)**: Lease already released

## Timestamp Preservation

During compaction, preserve the earliest CreatedAt timestamp from source files to maintain temporal granularity for point-in-time restoration.

### DO: Preserve earliest timestamp

```go
// CORRECT - Preserve temporal information
info, err := replica.Client.WriteLTXFile(ctx, level, minTXID, maxTXID, r)
if err != nil {
    return fmt.Errorf("write ltx: %w", err)
}
info.CreatedAt = oldestSourceFile.CreatedAt
```

### DON'T: Ignore CreatedAt preservation

```go
// WRONG - Loses timestamp granularity for point-in-time restores
info := &ltx.FileInfo{
    CreatedAt: time.Now(), // Don't use current time during compaction
}
```

## Common Pitfalls

### 1. Mixing architectural concerns

```go
// WRONG - Database state logic in Replica layer
func (r *Replica) Start() error {
    if db.needsRestore() {  // Wrong layer for DB state!
        r.restoreDatabase()  // Replica shouldn't manage DB state!
    }
    return r.sync()
}
```

### 2. Recreating existing functionality

```go
// WRONG - Don't reimplement what already exists
func customSnapshotTrigger() {
    // Complex custom logic to trigger snapshots
    // when db.verify() already does this!
}
```

### DO: Leverage existing mechanisms

```go
// CORRECT - Use what's already there
func triggerSnapshot() error {
    return db.verify()  // Already handles snapshot logic correctly
}
```

### 3. Skipping the lock page

The lock page at 1GB (0x40000000) must always be skipped:

```go
// db.go:951-953 - Must skip lock page during replication
lockPgno := ltx.LockPgno(pageSize)
if pgno == lockPgno {
    continue // Skip this page - it's reserved by SQLite
}
```

Lock page numbers by page size:

| Page Size | Lock Page Number |
|-----------|------------------|
| 4KB | 262145 |
| 8KB | 131073 |
| 16KB | 65537 |
| 32KB | 32769 |

## Component Reference

### DB Component (db.go)

**Responsibilities:**

- Manages SQLite database connection (via `modernc.org/sqlite` - no CGO)
- Monitors WAL for changes
- Performs checkpoints
- Maintains long-running read transaction
- Converts WAL pages to LTX format

**Key Fields:**

```go
type DB struct {
    path     string      // Database file path
    db       *sql.DB     // SQLite connection
    rtx      *sql.Tx     // Long-running read transaction
    pageSize int         // Database page size (critical for lock page)
    notify   chan struct{} // Notifies on WAL changes
}
```

**Initialization Sequence:**

1. Open database connection
2. Read page size from database
3. Initialize long-running read transaction
4. Start monitor goroutine
5. Initialize replicas

### Replica Component (replica.go)

**Responsibilities:**

- Manages replication to a single destination (one replica per DB)
- Tracks replication position (ltx.Pos)
- Handles sync intervals
- Manages encryption (if configured)

**Key Operations:**

- `Sync()`: Synchronizes pending changes
- `SetPos()`: Updates replication position (must use Lock, not RLock!)
- `Snapshot()`: Creates full database snapshot

### ReplicaClient Interface (replica_client.go)

**Required Methods:**

```go
type ReplicaClient interface {
    Type() string  // Client type identifier

    // File operations
    LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)
    OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)
    WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)
    DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error
    DeleteAll(ctx context.Context) error
}
```

**useMetadata Parameter:**

- `useMetadata=true`: Fetch accurate timestamps from backend metadata (required for point-in-time restores)
- `useMetadata=false`: Use fast timestamps for normal operations

### Compactor Component (compactor.go)

**Responsibilities:**

- Compaction and retention for LTX files
- Operates solely through `ReplicaClient` interface
- Suitable for both DB (with local file caching) and VFS (remote-only)

**Key Fields:**

```go
type Compactor struct {
    client           ReplicaClient
    VerifyCompaction bool  // Post-compaction TXID consistency check
    RetentionEnabled bool  // Default: true. Controls active file deletion

    // Local file optimization (set by DB layer)
    LocalFileOpener  func(level int, minTXID, maxTXID ltx.TXID) (io.ReadCloser, error)
    LocalFileDeleter func(level int, minTXID, maxTXID ltx.TXID) error

    // Level max-file-info caching
    CacheGetter func(level int) (*ltx.FileInfo, bool)
    CacheSetter func(level int, info *ltx.FileInfo)
}
```

### Store Component (store.go)

**Default Compaction Levels:**

```go
var defaultLevels = CompactionLevels{
    {Level: 0, Interval: 0},        // Raw LTX files (no compaction)
    {Level: 1, Interval: 30*Second},
    {Level: 2, Interval: 5*Minute},
    {Level: 3, Interval: 1*Hour},
    // Snapshots created daily (24h retention)
}
```

## Testing Patterns

### Race Condition Testing

```bash
# Always run with race detector
go test -race -v ./...

# Specific race-prone areas
go test -race -v -run TestReplica_Sync ./...
go test -race -v -run TestDB_Sync ./...
go test -race -v -run TestStore_CompactDB ./...
```

### Lock Page Testing

```bash
# Test with various page sizes
./bin/litestream-test populate -db test.db -page-size 4096 -target-size 2GB
./bin/litestream-test populate -db test.db -page-size 8192 -target-size 2GB

# Validate lock page handling
./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica
```

### Integration Testing

```bash
# Test specific backend
go test -v ./replica_client_test.go -integration s3
go test -v ./replica_client_test.go -integration gcs
go test -v ./replica_client_test.go -integration abs
go test -v ./replica_client_test.go -integration oss
go test -v ./replica_client_test.go -integration sftp
```

---

### PENDING USER DOCS (`docs/PENDING_USER_DOCS.md`)

# Pending User-Facing Documentation

This file tracks open issues on [benbjohnson/litestream.io](https://github.com/benbjohnson/litestream.io) that need user-facing documentation for features recently added to Litestream.

## Open Issues

None right now. The litestream.io issues previously listed here (#240 through #250) are all closed and their docs have shipped.

Add a row when a new Litestream feature lands that still needs user-facing docs on the site:

| Issue | Title | Related Litestream PR | Category |
|-------|-------|-----------------------|----------|

## AI Documentation Status

Internal AI documentation (this repo) was tracked in [#1106](https://github.com/benbjohnson/litestream/issues/1106), now closed.

---

### PROVIDER COMPATIBILITY (`docs/PROVIDER_COMPATIBILITY.md`)

# Storage Provider Compatibility Guide

This document details S3-compatible storage provider compatibility with Litestream,
including known limitations, required configuration, and tested configurations.

## Overview

Litestream uses the AWS SDK v2 for S3-compatible storage backends. While most providers
implement the S3 API, there are important differences in behavior that can affect
Litestream's operation.

## Provider-Specific Configuration

### AWS S3 (Default)

**Status**: Fully supported (primary target)

```yaml
replicas:
  - url: s3://bucket-name/path
    region: us-east-1
```

**Notes**:

- No special configuration required
- All features fully supported
- Checksum validation enabled by default

### Cloudflare R2

**Status**: Supported with default configuration

**Known Limitations**:

- Strict concurrent upload limit (2-3 concurrent uploads max)
- Does not support `aws-chunked` content encoding
- Does not support request/response checksums

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com
    access-key-id: your-access-key-id
    secret-access-key: your-secret-access-key
```

**Automatic Defaults** (applied when R2 endpoint detected):

- `sign-payload=true` - Signed payloads required
- `concurrency=2` - Limits concurrent multipart upload parts
- Checksums disabled automatically

**Important**: The endpoint must use `https://` scheme for R2 detection to work.

Related issues: #948, #947, #940, #941

### Backblaze B2 (S3-Compatible API)

**Status**: Supported with configuration

**Known Limitations**:

- Requires signed payloads for all requests
- Specific authentication endpoint required

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://s3.REGION.backblazeb2.com&sign-payload=true&force-path-style=true
    access-key-id: your-key-id
    secret-access-key: your-application-key
```

**Required Settings**:

- `sign-payload=true` - Required for B2 authentication
- `force-path-style=true` - Required for bucket access
- Endpoint format: `https://s3.REGION.backblazeb2.com`

Related issues: #918, #894

### DigitalOcean Spaces

**Status**: Supported with configuration

**Known Limitations**:

- Does not support `aws-chunked` content encoding
- Signature requirements differ from AWS

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://REGION.digitaloceanspaces.com&force-path-style=false
    access-key-id: your-spaces-key
    secret-access-key: your-spaces-secret
```

**Notes**:

- Use virtual-hosted style paths (force-path-style=false)
- Checksum features disabled automatically for custom endpoints

Related issues: #943

### MinIO

**Status**: Fully supported

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://your-minio-server:9000&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key
```

**Notes**:

- Works well with default settings
- Force path style recommended for single-server deployments

### Garage

**Status**: Supported with configuration when an intermediary rewrites `Accept-Encoding`

```yaml
dbs:
  - path: /path/to/database.db
    replica:
      type: s3
      bucket: bucket-name
      path: database-name
      endpoint: https://your-garage-server
      region: garage
      force-path-style: true
      sign-accept-encoding: false
      access-key-id: your-access-key
      secret-access-key: your-secret-key
```

`sign-accept-encoding` defaults to `true`. Set it to `false` when a reverse
proxy or other intermediary removes or rewrites `Accept-Encoding`, which would
otherwise invalidate the SigV4 signature. Litestream removes the header before
signing, although Go's HTTP transport may still add an unsigned header on the
wire.

Disabling this setting improves compatibility but removes `Accept-Encoding`
from SigV4 integrity protection, so changes to that header are no longer
detected during signature validation. Reported by @ahgraber in #1424.

### Scaleway Object Storage

**Status**: Supported with configuration

**Known Limitations**:

- `MissingContentLength` errors with streaming uploads
- Requires Content-Length header

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://s3.REGION.scw.cloud&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key
```

Related issues: #912

### Hetzner Object Storage

**Status**: Supported with configuration

**Known Limitations**:

- `InvalidArgument` errors with default AWS SDK settings
- Does not support `aws-chunked` content encoding
- Requires signed payloads for all requests

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://REGION.your-objectstorage.com&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key
```

### Filebase

**Status**: Supported with configuration

**Known Limitations**:

- Authentication failures with default SDK settings after SDK v2 migration

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://s3.filebase.com&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key
```

### Tigris

**Status**: Supported with configuration

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://fly.storage.tigris.dev&force-path-style=true
    access-key-id: your-access-key
    secret-access-key: your-secret-key
```

### Supabase Storage (S3-Compatible API)

**Status**: Supported (auto-detected)

**Known Limitations**:

- Requires path-style URLs (`force-path-style=true`)
- No S3 versioning support

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://PROJECT_REF.supabase.co/storage/v1/s3
    access-key-id: your-s3-access-key
    secret-access-key: your-s3-secret-key
```

**Automatic Defaults** (applied when Supabase endpoint detected):

- `sign-payload=true` - Signed payloads required
- `force-path-style=true` - Path-style URLs required

**Notes**:

- S3 access keys are generated in the Supabase dashboard under Storage > S3 Access Keys
- Endpoint format: `https://<PROJECT_REF>.supabase.co/storage/v1/s3`

Related issues: #1133

### Wasabi

**Status**: Supported

**Configuration**:

```yaml
replicas:
  - url: s3://bucket-name/path?endpoint=https://s3.REGION.wasabisys.com
    access-key-id: your-access-key
    secret-access-key: your-secret-key
```

## Google Cloud Storage (GCS)

**Status**: Fully supported (native client)

```yaml
replicas:
  - url: gcs://bucket-name/path
```

**Authentication**:

- Uses Application Default Credentials
- Set `GOOGLE_APPLICATION_CREDENTIALS` environment variable
- Or use workload identity on GCP

### S3-Compatible XML API

**Status**: Supported with HMAC keys

```yaml
replicas:
  - type: s3
    bucket: bucket-name
    path: database-name
    endpoint: https://storage.googleapis.com
    region: us-east-1
    force-path-style: true
    access-key-id: your-hmac-access-key
    secret-access-key: your-hmac-secret
```

Litestream automatically excludes `Accept-Encoding` from SigV4 signatures for
Cloud Storage endpoints under the exact `googleapis.com` domain when `storage`
is an exact segment of the first hostname label. This covers global, regional,
locational, mTLS, and legacy upload and download endpoints while rejecting
external lookalike domains.

## Azure Blob Storage (ABS)

**Status**: Fully supported (native client)

```yaml
replicas:
  - url: abs://container-name/path
    account-name: your-account-name
    account-key: your-account-key
```

**Using SAS Token** (for granular container-level access):

```yaml
replicas:
  - url: abs://container-name/path
    account-name: your-account-name
    sas-token: "sv=2023-01-03&ss=b&srt=co&sp=rwdlacx..."
```

Or via environment variable: `LITESTREAM_AZURE_SAS_TOKEN`

**Alternative Authentication**:

- SAS token: `sas-token` config or `LITESTREAM_AZURE_SAS_TOKEN` env var
- Account key: `account-key` config or `LITESTREAM_AZURE_ACCOUNT_KEY` env var
- Managed identity on Azure (via DefaultAzureCredential)

**Authentication Priority**: SAS token > Account key > Default credential chain

## Alibaba Cloud OSS

**Status**: Supported (native client)

```yaml
replicas:
  - url: oss://bucket-name/path?endpoint=oss-REGION.aliyuncs.com
    access-key-id: your-access-key-id
    access-key-secret: your-access-key-secret
```

## SFTP

**Status**: Supported

```yaml
replicas:
  - url: sftp://hostname/path
    user: username
    password: password  # or use key-path
```

## Configuration Reference

### S3 Query Parameters

Parameters with an alias accept both camelCase and hyphenated forms
(e.g., `forcePathStyle` or `force-path-style`).

| Parameter | Alias | Description | Default |
|-----------|-------|-------------|---------|
| `endpoint` | | Custom S3 endpoint URL | AWS S3 |
| `region` | | AWS region | Auto-detected |
| `forcePathStyle` | `force-path-style` | Use path-style URLs | `false` (auto for custom endpoints) |
| `skipVerify` | `skip-verify` | Skip TLS verification | `false` |
| `signPayload` | `sign-payload` | Sign request payloads | `true` |
| `signAcceptEncoding` | `sign-accept-encoding` | Include Accept-Encoding in SigV4 signatures | `true` |
| `requireContentMD5` | `require-content-md5` | Require Content-MD5 header | `true` |
| `concurrency` | | Multipart upload concurrency | `5` (`2` for R2/Tigris) |
| `partSize` | `part-size` | Multipart upload part size | `5242880` (5MB) (`16MiB` for Tigris) |
| `storageClass` | `storage-class` | Storage class for uploaded objects | None |
| `sseCustomerAlgorithm` | `sse-customer-algorithm` | SSE-C encryption algorithm | None |
| `sseCustomerKey` | `sse-customer-key` | SSE-C encryption key | None |
| `sseCustomerKeyMD5` | `sse-customer-key-md5` | SSE-C key MD5 checksum | None |
| `sseKmsKeyId` | `sse-kms-key-id` | KMS key for encryption | None |

### Provider Detection

Litestream automatically detects certain providers and applies appropriate defaults:

| Provider | Detection Pattern | Applied Settings |
|----------|-------------------|------------------|
| Hetzner | `*.your-objectstorage.com` | `sign-payload=true` |
| Cloudflare R2 | `*.r2.cloudflarestorage.com` | `sign-payload=true`, `concurrency=2` |
| Backblaze B2 | `*.backblazeb2.com` | `sign-payload=true`, `force-path-style=true` |
| DigitalOcean | `*.digitaloceanspaces.com` | `sign-payload=true` |
| Scaleway | `*.scw.cloud` | `sign-payload=true` |
| Filebase | `s3.filebase.com` | `sign-payload=true`, `force-path-style=true` |
| Tigris | `*.tigris.dev` | `sign-payload=true`, `require-content-md5=false`, `concurrency=2`, `part-size=16MiB` |
| MinIO | host with port (not cloud provider) | `sign-payload=true`, `force-path-style=true` |
| Supabase | `*.supabase.co` | `sign-payload=true`, `force-path-style=true` |
| Google Cloud Storage | Exact `googleapis.com` suffix with a first-label `storage` segment | Excludes `Accept-Encoding` from SigV4 signatures |

## Troubleshooting

### Common Errors

**`InvalidArgument: Unsupported content encoding: aws-chunked`**

- Provider doesn't support AWS SDK v2 chunked encoding
- Use a custom endpoint with automatic checksum disabling
- Or explicitly disable checksums

**`SignatureDoesNotMatch`**

- Try `sign-payload=true` in the URL
- Verify credentials are correct
- Check endpoint URL format

**`signed header accept-encoding is not present`**

- An intermediary removed or rewrote `Accept-Encoding` after request signing
- Set `sign-accept-encoding=false` in the replica URL or configuration
- This trades signature coverage of that header for intermediary compatibility

**`MissingContentLength`**

- Provider requires Content-Length header
- This is handled automatically for known providers

**`Too many concurrent uploads` or timeout errors**

- Reduce concurrency: `?concurrency=2`
- Particularly important for Cloudflare R2

**`AccessDenied` or authentication failures**

- Verify credentials
- Check IAM/bucket permissions
- For B2, ensure `sign-payload=true`

### Debug Mode

Enable verbose logging to diagnose issues:

```bash
LITESTREAM_DEBUG=1 litestream replicate ...
```

Or in configuration:

```yaml
logging:
  level: debug
```

## Testing Your Configuration

Test connectivity without starting replication:

```bash
# List any existing backups
litestream snapshots s3://bucket/path?endpoint=...

# Perform a test restore (requires existing backup)
litestream restore -o /tmp/test.db s3://bucket/path?endpoint=...
```

## Version Compatibility

- **Litestream v0.5.x**: AWS SDK v2, improved provider compatibility
- **Litestream v0.4.x**: AWS SDK v1, different authentication handling
- **Litestream v0.3.x**: Legacy format — v0.5.x can restore from v0.3.x backups via `ReplicaClientV3` interface

When upgrading from v0.3.x, v0.5.x can automatically restore from v0.3.x backups if no v0.4.x+ backup exists. The S3 backend implements `ReplicaClientV3` to read the v0.3.x `generations/{id}/snapshots/` and `generations/{id}/wal/` directory structure. See [REPLICA_CLIENT_GUIDE.md](REPLICA_CLIENT_GUIDE.md#replicaclientv3-interface-v03x-restore) for details.

### Validation Interval

Litestream supports periodic validation of replica integrity:

```yaml
validation:
  interval: "5m"  # How often to validate; 0 disables
```

This runs `Store.Validate()` on a ticker, comparing local and remote positions. Can also be set per-replica via the `validation-interval` replica config key.
and cannot restore backups created by v0.3.x. See the upgrade guide for migration
instructions.

## Reporting Issues

When reporting provider compatibility issues, please include:

1. Provider name and region
2. Litestream version (`litestream version`)
3. Full error message
4. Configuration (with credentials redacted)
5. Whether the issue is with replication, restore, or both

File issues at: [GitHub Issues](https://github.com/benbjohnson/litestream/issues)

---

### REPLICA CLIENT GUIDE (`docs/REPLICA_CLIENT_GUIDE.md`)

# ReplicaClient Implementation Guide

This guide provides comprehensive instructions for implementing new storage backends for Litestream replication.

## Table of Contents

- [Interface Contract](#interface-contract)
- [Implementation Checklist](#implementation-checklist)
- [Eventual Consistency Handling](#eventual-consistency-handling)
- [Error Handling](#error-handling)
- [Testing Requirements](#testing-requirements)
- [ReplicaClientV3 Interface (v0.3.x Restore)](#replicaclientv3-interface-v03x-restore)
- [Common Implementation Mistakes](#common-implementation-mistakes)
- [Reference Implementations](#reference-implementations)

## Interface Contract

All replica clients MUST implement the `ReplicaClient` interface defined in `replica_client.go`:

```go
type ReplicaClient interface {
    // Returns the type identifier (e.g., "s3", "gcs", "file")
    Type() string

    // Returns iterator of LTX files at given level
    // seek: Start from this TXID (0 = beginning)
    // useMetadata: When true, fetch accurate timestamps from backend metadata (required for PIT restore)
    LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)

    // Opens an LTX file for reading
    // Returns os.ErrNotExist if file doesn't exist
    OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)

    // Writes an LTX file to storage
    // SHOULD set CreatedAt based on backend metadata or upload time
    WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)

    // Deletes one or more LTX files
    DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error

    // Deletes all files for this database
    DeleteAll(ctx context.Context) error
}
```

## Implementation Checklist

### Required Features

- [ ] Implement all interface methods
- [ ] Support partial reads (offset/size in OpenLTXFile)
- [ ] Return proper error types (especially os.ErrNotExist)
- [ ] Handle context cancellation
- [ ] Preserve file timestamps (CreatedAt)
- [ ] Support concurrent operations
- [ ] Implement proper cleanup in DeleteAll

### Optional Features

- [ ] Connection pooling
- [ ] Retry logic with exponential backoff
- [ ] Request batching
- [ ] Compression
- [ ] Encryption at rest
- [ ] Bandwidth throttling

## Eventual Consistency Handling

Many cloud storage services exhibit eventual consistency, where:
- A file you just wrote might not be immediately visible
- A file might be listed but only partially readable
- Deletes might not take effect immediately

### Best Practices

#### 1. Write-After-Write Consistency

```
/* Detailed source-code truncated for AI context efficiency. */
```

#### 2. List-After-Write Consistency

```go
func (c *ReplicaClient) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {
    // List files from storage
    files, err := c.listFiles(ctx, level, useMetadata)
    if err != nil {
        return nil, err
    }

    // Sort by TXID for consistent ordering
    sort.Slice(files, func(i, j int) bool {
        if files[i].MinTXID != files[j].MinTXID {
            return files[i].MinTXID < files[j].MinTXID
        }
        return files[i].MaxTXID < files[j].MaxTXID
    })

    // Filter by seek position
    var filtered []*ltx.FileInfo
    for _, f := range files {
        if f.MinTXID >= seek {
            filtered = append(filtered, f)
        }
    }

    return ltx.NewFileInfoSliceIterator(filtered), nil
}
```

#### 3. Read-After-Write Consistency

```go
func (c *ReplicaClient) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {
    path := c.ltxPath(level, minTXID, maxTXID)

    // For eventually consistent backends, implement retry
    var lastErr error
    backoff := 100 * time.Millisecond

    for i := 0; i < 5; i++ {
        reader, err := c.openFile(ctx, path, offset, size)
        if err == nil {
            return reader, nil
        }

        // Don't retry on definitive errors
        if errors.Is(err, os.ErrNotExist) {
            return nil, err
        }

        lastErr = err
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        case <-time.After(backoff):
            backoff *= 2
        }
    }

    return nil, fmt.Errorf("open file after retries: %w", lastErr)
}
```

## Error Handling

### Standard Error Types

Always return appropriate standard errors:

```go
// File not found
return nil, os.ErrNotExist

// Permission denied
return nil, os.ErrPermission

// Context cancelled
return nil, ctx.Err()

// Custom errors should wrap standard ones
return nil, fmt.Errorf("s3 download failed: %w", err)
```

### Error Classification

```go
// Retryable errors
func isRetryable(err error) bool {
    // Network errors
    var netErr net.Error
    if errors.As(err, &netErr) && netErr.Temporary() {
        return true
    }

    // Specific HTTP status codes
    if httpErr, ok := err.(HTTPError); ok {
        switch httpErr.StatusCode {
        case 429, 500, 502, 503, 504:
            return true
        }
    }

    // Timeout errors
    if errors.Is(err, context.DeadlineExceeded) {
        return true
    }

    return false
}
```

### Logging Best Practices

```go
func (c *ReplicaClient) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {
    logger := slog.Default().With(
        "replica", c.Type(),
        "level", level,
        "minTXID", minTXID,
        "maxTXID", maxTXID,
    )

    logger.Debug("starting ltx upload")

    info, err := c.upload(ctx, level, minTXID, maxTXID, r)
    if err != nil {
        logger.Error("ltx upload failed", "error", err)
        return nil, err
    }

    logger.Info("ltx upload complete", "size", info.Size)
    return info, nil
}
```

## Testing Requirements

### Unit Tests

Every replica client MUST have comprehensive unit tests:

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Integration Tests

Integration tests run against real backends:

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Mock Client for Testing

Provide a mock implementation for testing:

```go
// mock/replica_client.go
type ReplicaClient struct {
    mu     sync.Mutex
    files  map[string]*ltx.FileInfo
    data   map[string][]byte
    errors map[string]error  // Inject errors for testing
}

func (c *ReplicaClient) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {
    c.mu.Lock()
    defer c.mu.Unlock()

    // Check for injected error
    key := fmt.Sprintf("write-%d-%d-%d", level, minTXID, maxTXID)
    if err, ok := c.errors[key]; ok {
        return nil, err
    }

    // Store data
    data, err := io.ReadAll(r)
    if err != nil {
        return nil, err
    }

    path := ltxPath(level, minTXID, maxTXID)
    c.data[path] = data

    info := &ltx.FileInfo{
        Level:     level,
        MinTXID:   minTXID,
        MaxTXID:   maxTXID,
        Size:      int64(len(data)),
        CreatedAt: time.Now(),
    }
    c.files[path] = info

    return info, nil
}
```

## ReplicaClientV3 Interface (v0.3.x Restore)

Backends that need backward-compatible restore from v0.3.x Litestream backups should implement the optional `ReplicaClientV3` interface (`v3.go`). This enables restoring databases from pre-v0.4 backup formats.

### Interface

```go
type ReplicaClientV3 interface {
    GenerationsV3(ctx context.Context) ([]string, error)
    SnapshotsV3(ctx context.Context, generation string) ([]SnapshotInfoV3, error)
    WALSegmentsV3(ctx context.Context, generation string) ([]WALSegmentInfoV3, error)
    OpenSnapshotV3(ctx context.Context, generation string, index int) (io.ReadCloser, error)
    OpenWALSegmentV3(ctx context.Context, generation string, index int, offset int64) (io.ReadCloser, error)
}
```

### Supporting Types

```go
type PosV3 struct {
    Generation string // 16-char hex string
    Index      int    // WAL index
    Offset     int64  // Offset within WAL segment
}

type SnapshotInfoV3 struct {
    Generation string
    Index      int
    Size       int64
    CreatedAt  time.Time
}

type WALSegmentInfoV3 struct {
    Generation string
    Index      int
    Offset     int64
    Size       int64
    CreatedAt  time.Time
}
```

### v0.3.x Path Structure

```text
generations/
  {generation-id}/         # 16-char hex (validated by IsGenerationIDV3)
    snapshots/
      {index:08x}.snapshot.lz4
    wal/
      {index:08x}_{offset:08x}.wal.lz4
```

### Implementation Notes

- Generation IDs are 16 hex characters, validated by `IsGenerationIDV3()` (`generationRegexV3`)
- All returned readers provide LZ4-decompressed data
- `GenerationsV3` returns IDs sorted ascending
- `SnapshotsV3` returns sorted by index
- `WALSegmentsV3` returns sorted by index, then offset
- S3 backend implements this: `var _ litestream.ReplicaClientV3 = (*ReplicaClient)(nil)`

### ResumableReader Integration

`OpenLTXFile` must support the `offset` parameter for range requests. The `ResumableReader` (`internal/resumable_reader.go`) reopens streams from the last successful byte offset on connection failures during restore. If your backend ignores `offset`, restore operations will fail on idle connection timeouts.

## Common Implementation Mistakes

### ❌ Mistake 1: Not Handling Partial Reads

```go
// WRONG - Always reads entire file
func (c *Client) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {
    return c.storage.Download(path)  // Ignores offset/size!
}
```

```go
// CORRECT - Respects offset and size
func (c *Client) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {
    if offset == 0 && size == 0 {
        // Full file
        return c.storage.Download(path)
    }

    // Partial read using Range header or equivalent
    end := offset + size - 1
    if size == 0 {
        end = 0  // Read to end
    }
    return c.storage.DownloadRange(path, offset, end)
}
```

### ❌ Mistake 2: Not Preserving CreatedAt

```go
// WRONG - Uses current time
func (c *Client) WriteLTXFile(...) (*ltx.FileInfo, error) {
    // Upload file...

    return &ltx.FileInfo{
        CreatedAt: time.Now(),  // Wrong! Loses temporal info
    }, nil
}
```

```go
// CORRECT - Preserves original timestamp
func (c *Client) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {
    // Upload file...
    uploadedSize, modTime, err := c.storage.Upload(path, r)
    if err != nil {
        return nil, err
    }

    return &ltx.FileInfo{
        Level:     level,
        MinTXID:   minTXID,
        MaxTXID:   maxTXID,
        Size:      uploadedSize,
        CreatedAt: modTime,
    }, nil
}
```

### ❌ Mistake 3: Wrong Error Types

```go
// WRONG - Generic error
func (c *Client) OpenLTXFile(...) (io.ReadCloser, error) {
    resp, err := c.get(path)
    if err != nil {
        return nil, fmt.Errorf("not found")  // Wrong type!
    }
}
```

```go
// CORRECT - Proper error type
func (c *Client) OpenLTXFile(...) (io.ReadCloser, error) {
    resp, err := c.get(path)
    if err != nil {
        if resp.StatusCode == 404 {
            return nil, os.ErrNotExist  // Correct type
        }
        return nil, fmt.Errorf("download failed: %w", err)
    }
}
```

### ❌ Mistake 4: Not Handling Context

```go
// WRONG - Ignores context
func (c *Client) WriteLTXFile(ctx context.Context, ...) (*ltx.FileInfo, error) {
    // Long operation without checking context
    for i := 0; i < 1000000; i++ {
        doWork()  // Could run forever!
    }
}
```

```go
// CORRECT - Respects context
func (c *Client) WriteLTXFile(ctx context.Context, ...) (*ltx.FileInfo, error) {
    // Check context periodically
    for i := 0; i < 1000000; i++ {
        select {
        case <-ctx.Done():
            return nil, ctx.Err()
        default:
            // Continue work
        }

        if err := doWork(ctx); err != nil {
            return nil, err
        }
    }
}
```

### ❌ Mistake 5: Blocking in Iterator

```go
// WRONG - Loads all files at once
func (c *Client) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {
    allFiles, err := c.loadAllFiles(level)  // Could be millions!
    if err != nil {
        return nil, err
    }

    return NewIterator(allFiles), nil
}
```

```go
// CORRECT - Lazy loading with pagination
func (c *Client) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {
    return &lazyIterator{
        client:      c,
        level:       level,
        seek:        seek,
        useMetadata: useMetadata,
        pageSize:    1000,
    }, nil
}

type lazyIterator struct {
    client   *Client
    level    int
    seek     ltx.TXID
    pageSize int
    current  []*ltx.FileInfo
    index    int
    done     bool
}

func (i *lazyIterator) Next() bool {
    if i.index >= len(i.current) && !i.done {
        // Load next page
        i.loadNextPage()
    }
    return i.index < len(i.current)
}
```

## Reference Implementations

### File System Client (Simplest)

See `file/replica_client.go` for the simplest implementation:
- Direct file I/O operations
- No network complexity
- Good starting reference

### S3 Client (Most Complex)

See `s3/replica_client.go` for advanced features:
- Multipart uploads for large files
- Retry logic with exponential backoff
- Request signing
- Eventual consistency handling

### Key Patterns from S3 Implementation

```go
// Path construction
func (c *ReplicaClient) ltxDir(level int) string {
    if level == SnapshotLevel {
        return path.Join(c.Path, "snapshots")
    }
    return path.Join(c.Path, "ltx", fmt.Sprintf("%04d", level))
}

// Metadata handling
func (c *ReplicaClient) WriteLTXFile(...) (*ltx.FileInfo, error) {
    // Add metadata to object
    metadata := map[string]string{
        "min-txid": fmt.Sprintf("%d", minTXID),
        "max-txid": fmt.Sprintf("%d", maxTXID),
        "level":    fmt.Sprintf("%d", level),
    }

    // Upload with metadata
    _, err := c.s3.PutObjectWithContext(ctx, &s3.PutObjectInput{
        Bucket:   &c.Bucket,
        Key:      &key,
        Body:     r,
        Metadata: metadata,
    })
}

// Error mapping
func mapS3Error(err error) error {
    if aerr, ok := err.(awserr.Error); ok {
        switch aerr.Code() {
        case s3.ErrCodeNoSuchKey:
            return os.ErrNotExist
        case s3.ErrCodeAccessDenied:
            return os.ErrPermission
        }
    }
    return err
}
```

## Performance Optimization

### Connection Pooling

```go
type ReplicaClient struct {
    pool *ConnectionPool
}

func NewReplicaClient(config Config) *ReplicaClient {
    pool := &ConnectionPool{
        MaxConnections: config.MaxConnections,
        IdleTimeout:    config.IdleTimeout,
    }

    return &ReplicaClient{
        pool: pool,
    }
}
```

### Request Batching

```go
func (c *ReplicaClient) DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error {
    // Batch deletes for efficiency
    const batchSize = 100

    for i := 0; i < len(files); i += batchSize {
        end := i + batchSize
        if end > len(files) {
            end = len(files)
        }

        batch := files[i:end]
        if err := c.deleteBatch(ctx, batch); err != nil {
            return fmt.Errorf("delete batch %d: %w", i/batchSize, err)
        }
    }

    return nil
}
```

### Caching

```go
type ReplicaClient struct {
    cache *FileInfoCache
}

func (c *ReplicaClient) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {
    // Check cache first (only cache when useMetadata=false for fast queries)
    cacheKey := fmt.Sprintf("%d-%d", level, seek)
    if !useMetadata {
        if cached, ok := c.cache.Get(cacheKey); ok {
            return ltx.NewFileInfoSliceIterator(cached), nil
        }
    }

    // Load from storage
    files, err := c.loadFiles(ctx, level, seek, useMetadata)
    if err != nil {
        return nil, err
    }

    // Cache for future requests
    c.cache.Set(cacheKey, files, 5*time.Minute)

    return ltx.NewFileInfoSliceIterator(files), nil
}
```

## Checklist for New Implementations

Before submitting a new replica client:

- [ ] All interface methods implemented
- [ ] Unit tests with >80% coverage
- [ ] Integration tests (with build tag)
- [ ] Mock client for testing
- [ ] Handles partial reads correctly
- [ ] Returns proper error types
- [ ] Preserves timestamps
- [ ] Handles context cancellation
- [ ] Documents eventual consistency behavior
- [ ] Includes retry logic for transient errors
- [ ] Logs appropriately (debug/info/error)
- [ ] README with configuration examples
- [ ] Added to main configuration parser

## Getting Help

1. Study existing implementations (start with `file/`, then `s3/`)
2. Check test files for expected behavior
3. Run integration tests against your backend
4. Use the mock client for rapid development
5. Open a GitHub issue for design feedback

---

### SQLITE INTERNALS (`docs/SQLITE_INTERNALS.md`)

# SQLite Internals for Litestream

This document explains SQLite internals critical for understanding Litestream's operation.

## Table of Contents
- [SQLite File Structure](#sqlite-file-structure)
- [Write-Ahead Log (WAL)](#write-ahead-log-wal)
- [Page Structure](#page-structure)
- [The 1GB Lock Page](#the-1gb-lock-page)
- [Transaction Management](#transaction-management)
- [Checkpoint Modes](#checkpoint-modes)
- [Important SQLite Pragmas](#important-sqlite-pragmas)
- [SQLite API Usage](#sqlite-api-usage)

## SQLite File Structure

SQLite databases consist of:
1. **Main database file** - Contains actual data in pages
2. **WAL file** (-wal suffix) - Contains uncommitted changes
3. **SHM file** (-shm suffix) - Shared memory for coordination

```
database.db         # Main database file (pages)
database.db-wal     # Write-ahead log
database.db-shm     # Shared memory file
```

## Write-Ahead Log (WAL)

### WAL Basics

WAL is SQLite's method for implementing atomic commits and rollback:
- Changes are first written to WAL
- Original database file unchanged until checkpoint
- Readers see consistent view through WAL

```mermaid
graph LR
    Write[Write Transaction] -->|Append| WAL[WAL File]
    WAL -->|Checkpoint| DB[Main Database]
    Read[Read Transaction] -->|Merge View| View[Consistent View]
    DB --> View
    WAL --> View
```

### WAL File Structure

```
+------------------+
| WAL Header       | 32 bytes
+------------------+
| Frame 1 Header   | 24 bytes
| Frame 1 Data     | Page size bytes
+------------------+
| Frame 2 Header   | 24 bytes
| Frame 2 Data     | Page size bytes
+------------------+
| ...              |
+------------------+
```

#### WAL Header (32 bytes)
```go
type WALHeader struct {
    Magic         [4]byte  // 0x377f0682 or 0x377f0683
    FileFormat    uint32   // File format version (3007000)
    PageSize      uint32   // Database page size
    Checkpoint    uint32   // Checkpoint sequence number
    Salt1         uint32   // Random salt for checksum
    Salt2         uint32   // Random salt for checksum
    Checksum1     uint32   // Header checksum
    Checksum2     uint32   // Header checksum
}
```

#### WAL Frame Header (24 bytes)
```go
type WALFrameHeader struct {
    PageNumber    uint32   // Page number in database
    DbSize        uint32   // Size of database in pages
    Salt1         uint32   // Must match header salt
    Salt2         uint32   // Must match header salt
    Checksum1     uint32   // Cumulative checksum
    Checksum2     uint32   // Cumulative checksum
}
```

### Reading WAL in Litestream

```go
// db.go - Reading WAL for replication
func (db *DB) readWAL() ([]Page, error) {
    walPath := db.path + "-wal"
    f, err := os.Open(walPath)
    if err != nil {
        return nil, err
    }
    defer f.Close()

    // Read WAL header
    var header WALHeader
    binary.Read(f, binary.BigEndian, &header)

    // Validate magic number
    magic := binary.BigEndian.Uint32(header.Magic[:])
    if magic != 0x377f0682 && magic != 0x377f0683 {
        return nil, errors.New("invalid WAL magic")
    }

    // Read frames
    var pages []Page
    for {
        var frameHeader WALFrameHeader
        err := binary.Read(f, binary.BigEndian, &frameHeader)
        if err == io.EOF {
            break
        }

        // Read page data
        pageData := make([]byte, header.PageSize)
        f.Read(pageData)

        pages = append(pages, Page{
            Number: frameHeader.PageNumber,
            Data:   pageData,
        })
    }

    return pages, nil
}
```

## Page Structure

### Database Pages

SQLite divides the database into fixed-size pages:

```
Page Size: Typically 4096 bytes (4KB)
Page Number: 1-based indexing
Page Types:
  - B-tree interior pages
  - B-tree leaf pages
  - Overflow pages
  - Freelist pages
  - Lock byte page (at 1GB)
```

### Page Layout

```
+------------------+
| Page Header      | Variable (8-12 bytes)
+------------------+
| Cell Pointers    | 2 bytes each
+------------------+
| Unallocated      |
| Space            |
+------------------+
| Cell Content     | Variable size
| Area             | (grows upward)
+------------------+
```

### Page Header Structure

```go
type PageHeader struct {
    PageType      byte     // 0x02, 0x05, 0x0a, 0x0d
    FreeBlockStart uint16  // Start of free block list
    CellCount     uint16   // Number of cells
    CellStart     uint16   // Offset to first cell
    FragmentBytes byte     // Fragmented free bytes
    // Additional fields for interior pages
    RightChild    uint32   // Only for interior pages
}
```

## The 1GB Lock Page

### Critical Concept

SQLite reserves a special page at exactly 1,073,741,824 bytes (0x40000000) for locking:

```go
const PENDING_BYTE = 0x40000000  // 1GB mark

// Page number varies by page size
func LockPageNumber(pageSize int) uint32 {
    return uint32(PENDING_BYTE/pageSize) + 1
}

// Examples:
// 4KB pages:  262145 (0x40001)
// 8KB pages:  131073 (0x20001)
// 16KB pages:  65537 (0x10001)
// 32KB pages:  32769 (0x08001)
// 64KB pages:  16385 (0x04001)
```

### Why This Matters

1. **Cannot contain data** - SQLite will never write user data here
2. **Must be skipped** - During replication/compaction
3. **Affects large databases** - Only databases >1GB
4. **Page number changes** - Different for each page size

### Implementation in Litestream

```go
// From superfly/ltx package
func LockPgno(pageSize int) uint32 {
    return uint32(PENDING_BYTE/pageSize) + 1
}

// db.go - Skipping lock page during iteration
for pgno := uint32(1); pgno <= maxPgno; pgno++ {
    if pgno == ltx.LockPgno(db.pageSize) {
        continue // Skip lock page
    }

    // Process normal page
    processPage(pgno)
}
```

### Testing Lock Page

```sql
-- Create database that spans lock page
CREATE TABLE test (id INTEGER PRIMARY KEY, data BLOB);

-- Insert data until database > 1GB
WITH RECURSIVE generate_series(value) AS (
    SELECT 1
    UNION ALL
    SELECT value+1 FROM generate_series
    LIMIT 300000
)
INSERT INTO test SELECT value, randomblob(4000) FROM generate_series;

-- Check database size
PRAGMA page_count;  -- Should be > 262145 for 4KB pages
PRAGMA page_size;   -- Typically 4096

-- Calculate if lock page is in range
-- For 4KB pages: if page_count > 262145, lock page is included
```

## Transaction Management

### SQLite Transaction Types

1. **Deferred Transaction** (default)
   ```sql
   BEGIN DEFERRED;  -- Lock acquired on first use
   ```

2. **Immediate Transaction**
   ```sql
   BEGIN IMMEDIATE; -- RESERVED lock immediately
   ```

3. **Exclusive Transaction**
   ```sql
   BEGIN EXCLUSIVE; -- EXCLUSIVE lock immediately
   ```

### Lock Types in SQLite

```mermaid
graph TD
    UNLOCKED -->|BEGIN| SHARED
    SHARED -->|Write| RESERVED
    RESERVED -->|Prepare| PENDING
    PENDING -->|Commit| EXCLUSIVE
    EXCLUSIVE -->|Done| UNLOCKED
```

1. **SHARED** - Multiple readers allowed
2. **RESERVED** - Signals intent to write
3. **PENDING** - Blocking new SHARED locks
4. **EXCLUSIVE** - Single writer, no readers

### Litestream's Long-Running Read Transaction

```go
// db.go - Maintaining read transaction for consistency
func (db *DB) initReadTx() error {
    // Start read-only transaction
    tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{
        ReadOnly: true,
    })
    if err != nil {
        return err
    }

    // Execute query to acquire SHARED lock
    var dummy string
    err = tx.QueryRow("SELECT ''").Scan(&dummy)
    if err != nil {
        tx.Rollback()
        return err
    }

    // Keep transaction open
    db.rtx = tx
    return nil
}
```

**Purpose:**
- Prevents database from being checkpointed past our read point
- Ensures consistent view of database
- Allows reading pages from WAL

## Checkpoint Modes

### PASSIVE Checkpoint (default)
```sql
PRAGMA wal_checkpoint(PASSIVE);
```
- Attempts checkpoint
- Fails if readers present
- Non-blocking

### FULL Checkpoint
```sql
PRAGMA wal_checkpoint(FULL);
```
- Waits for readers to finish
- Blocks new readers
- Ensures checkpoint completes

### RESTART Checkpoint
```sql
PRAGMA wal_checkpoint(RESTART);
```
- Like FULL, but also:
- Ensures next writer starts at beginning of WAL
- Resets WAL file

### TRUNCATE Checkpoint
```sql
PRAGMA wal_checkpoint(TRUNCATE);
```
- Like RESTART, but also:
- Truncates WAL file to zero length
- Releases disk space

### Litestream Checkpoint Strategy

```go
// db.go - Checkpoint decision logic
func (db *DB) autoCheckpoint() error {
    walSize := db.WALSize()
    pageCount := walSize / db.pageSize

    if pageCount > db.TruncatePageN {
        // Force truncation for very large WAL (emergency brake)
        return db.Checkpoint("TRUNCATE")
    } else if pageCount > db.MinCheckpointPageN {
        // Try passive checkpoint (non-blocking)
        return db.Checkpoint("PASSIVE")
    } else if db.CheckpointInterval elapsed {
        // Time-based passive checkpoint
        return db.Checkpoint("PASSIVE")
    }
    // Note: RESTART mode permanently removed due to issue #724 (write-blocking)
    }

    return nil
}
```

## Important SQLite Pragmas

### Essential Pragmas for Litestream

```sql
-- Enable WAL mode (required)
PRAGMA journal_mode = WAL;

-- Get database info
PRAGMA page_size;        -- Page size in bytes
PRAGMA page_count;       -- Total pages in database
PRAGMA freelist_count;   -- Free pages

-- WAL information
PRAGMA wal_checkpoint;           -- Perform checkpoint
PRAGMA wal_autocheckpoint;       -- Auto-checkpoint threshold
PRAGMA wal_checkpoint(PASSIVE);  -- Non-blocking checkpoint

-- Database state
PRAGMA integrity_check;  -- Verify database integrity
PRAGMA quick_check;     -- Fast integrity check

-- Lock information
PRAGMA lock_status;     -- Current locks (debug builds)

-- Performance tuning
PRAGMA synchronous = NORMAL;  -- Sync mode
PRAGMA busy_timeout = 5000;   -- Wait 5s for locks
PRAGMA cache_size = -64000;   -- 64MB cache
```

### Reading Pragmas in Go

```go
func getDatabaseInfo(db *sql.DB) (*DBInfo, error) {
    info := &DBInfo{}

    // Page size
    err := db.QueryRow("PRAGMA page_size").Scan(&info.PageSize)

    // Page count
    err = db.QueryRow("PRAGMA page_count").Scan(&info.PageCount)

    // Journal mode
    err = db.QueryRow("PRAGMA journal_mode").Scan(&info.JournalMode)

    // Calculate size
    info.Size = info.PageSize * info.PageCount

    return info, nil
}
```

## SQLite API Usage

### Direct SQLite Access

Litestream uses both database/sql and direct SQLite APIs:

```go
// Using database/sql for queries
db, err := sql.Open("sqlite3", "database.db")

// Using modernc.org/sqlite for low-level access
conn, err := sqlite.Open("database.db")

// Direct page access (requires special builds)
page := readPage(conn, pageNumber)
```

### Connection Modes

```go
// Read-only connection
db, err := sql.Open("sqlite3", "file:database.db?mode=ro")

// WAL mode connection
db, err := sql.Open("sqlite3", "database.db?_journal=WAL")

// With busy timeout
db, err := sql.Open("sqlite3", "database.db?_busy_timeout=5000")

// Multiple options
db, err := sql.Open("sqlite3", "database.db?_journal=WAL&_busy_timeout=5000&_synchronous=NORMAL")
```

### WAL File Access Pattern

```go
// Litestream's approach to reading WAL
func (db *DB) monitorWAL() {
    walPath := db.path + "-wal"

    for {
        // Check WAL file size
        stat, err := os.Stat(walPath)
        if err != nil {
            continue // WAL might not exist yet
        }

        // Compare with last known size
        if stat.Size() > db.lastWALSize {
            // New data in WAL
            db.processWALChanges()
            db.lastWALSize = stat.Size()
        }

        time.Sleep(db.MonitorInterval)
    }
}
```

## Critical SQLite Behaviors

### 1. Automatic Checkpoint
SQLite automatically checkpoints when WAL reaches 1000 pages (default):
```go
// Can interfere with Litestream's control
// Solution: Set high threshold
db.Exec("PRAGMA wal_autocheckpoint = 10000")
```

### 2. Busy Timeout
Default timeout is 0 (immediate failure):
```go
// Set reasonable timeout
db.Exec("PRAGMA busy_timeout = 5000") // 5 seconds
```

### 3. Synchronous Mode
Controls when SQLite waits for disk writes:
```go
// NORMAL is safe with WAL
db.Exec("PRAGMA synchronous = NORMAL")
```

### 4. Page Cache
SQLite maintains an in-memory page cache:
```go
// Set cache size (negative = KB, positive = pages)
db.Exec("PRAGMA cache_size = -64000") // 64MB
```

## WAL to LTX Conversion

Litestream converts WAL frames to LTX format:

```go
func walToLTX(walFrames []WALFrame) *LTXFile {
    ltx := &LTXFile{
        Header: LTXHeader{
            PageSize: walFrames[0].PageSize,
            MinTXID:  walFrames[0].TransactionID,
        },
    }

    for _, frame := range walFrames {
        // Skip lock page
        if frame.PageNumber == LockPageNumber(ltx.Header.PageSize) {
            continue
        }

        ltx.Pages = append(ltx.Pages, Page{
            Number: frame.PageNumber,
            Data:   frame.Data,
        })

        ltx.Header.MaxTXID = frame.TransactionID
    }

    return ltx
}
```

## Key Takeaways

1. **WAL is temporary** - Gets merged back via checkpoint
2. **Lock page is sacred** - Never write data at 1GB mark
3. **Page size matters** - Affects lock page number and performance
4. **Transactions provide consistency** - Long-running read prevents changes
5. **Checkpoints are critical** - Balance between WAL size and performance
6. **SQLite locks coordinate access** - Understanding prevents deadlocks
7. **Pragmas control behavior** - Must be set correctly for Litestream

This understanding is essential for:
- Debugging replication issues
- Implementing new features
- Optimizing performance
- Handling edge cases correctly

---

### TESTING GUIDE (`docs/TESTING_GUIDE.md`)

# Litestream Testing Guide

Comprehensive guide for testing Litestream components and handling edge cases.

## Table of Contents

- [Testing Philosophy](#testing-philosophy)
- [1GB Database Testing](#1gb-database-testing)
- [Race Condition Testing](#race-condition-testing)
- [Integration Testing](#integration-testing)
- [Performance Testing](#performance-testing)
- [Mock Usage Patterns](#mock-usage-patterns)
- [Test Utilities](#test-utilities)
- [Common Test Failures](#common-test-failures)

## Testing Philosophy

Litestream testing follows these principles:

1. **Test at Multiple Levels**: Unit, integration, and end-to-end
2. **Focus on Edge Cases**: Especially >1GB databases and eventual consistency
3. **Use Real SQLite**: Avoid mocking SQLite behavior
4. **Race Detection**: Always run with `-race` flag
5. **Deterministic Tests**: Use fixed seeds and timestamps where possible

## 1GB Database Testing

### The Lock Page Problem

SQLite reserves a special lock page at exactly 1GB (0x40000000 bytes). This page cannot contain data and must be skipped during replication.

### Test Requirements

#### Creating Test Databases

```bash
# Use litestream-test tool for large databases
./bin/litestream-test populate \
    -db test.db \
    -target-size 1.5GB \
    -page-size 4096
```

#### Manual Test Database Creation

```
/* Detailed source-code truncated for AI context efficiency. */
```

#### Lock Page Test Cases

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Restoration Testing

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Race Condition Testing

### Running with Race Detector

```bash
# Always run tests with race detector
go test -race -v ./...

# Run specific race-prone tests
go test -race -v -run TestReplica_Sync ./...
go test -race -v -run TestDB_Sync ./...
go test -race -v -run TestStore_CompactDB ./...
```

### Common Race Conditions

#### 1. Position Updates

```go
func TestReplica_ConcurrentPositionUpdate(t *testing.T) {
    replica := litestream.NewReplica(nil)
    var wg sync.WaitGroup
    errors := make(chan error, 100)

    // Concurrent writers
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func(n int) {
            defer wg.Done()

            pos := ltx.NewPos(ltx.TXID(n), ltx.Checksum(uint64(n)))

            // This should use proper locking
            replica.SetPos(pos)

            // Verify position
            readPos := replica.Pos()
            if readPos.TXID < ltx.TXID(n) {
                errors <- fmt.Errorf("position went backwards")
            }
        }(i)
    }

    // Concurrent readers
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()

            for j := 0; j < 100; j++ {
                _ = replica.Pos()
                time.Sleep(time.Microsecond)
            }
        }()
    }

    wg.Wait()
    close(errors)

    for err := range errors {
        t.Error(err)
    }
}
```

#### 2. WAL Monitoring

```go
func TestDB_ConcurrentWALAccess(t *testing.T) {
    db := setupTestDB(t)
    defer db.Close(context.Background())

    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    var wg sync.WaitGroup

    // Writer goroutine
    wg.Add(1)
    go func() {
        defer wg.Done()

        conn, err := sql.Open("sqlite", db.Path())
        if err != nil {
            return
        }
        defer conn.Close()

        for i := 0; i < 100; i++ {
            _, _ = conn.Exec("INSERT INTO test VALUES (?)", i)
            time.Sleep(10 * time.Millisecond)
        }
    }()

    // Monitor goroutine
    wg.Add(1)
    go func() {
        defer wg.Done()

        notifyCh := db.Notify()

        for {
            select {
            case <-ctx.Done():
                return
            case <-notifyCh:
                // Process WAL changes
                _ = db.Sync(context.Background())
            }
        }
    }()

    // Checkpoint goroutine
    wg.Add(1)
    go func() {
        defer wg.Done()

        ticker := time.NewTicker(100 * time.Millisecond)
        defer ticker.Stop()

        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
                _ = db.Checkpoint(context.Background(), litestream.CheckpointModePassive)
            }
        }
    }()

    wg.Wait()
}
```

### Test Cleanup

```go
func TestStore_Integration(t *testing.T) {
    // Setup
    tmpDir := t.TempDir()
    db := setupTestDB(t, tmpDir)

    // Use defer with error channel for cleanup
    insertErr := make(chan error, 1)

    // Cleanup function
    cleanup := func() {
        select {
        case err := <-insertErr:
            if err != nil {
                t.Errorf("insert error during test: %v", err)
            }
        default:
        }
    }
    defer cleanup()

    // Test with timeout
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    // Run test...
}
```

## Integration Testing

### Test Structure

```go
// +build integration

package litestream_test

import (
    "context"
    "os"
    "testing"
)

func TestIntegration_S3(t *testing.T) {
    if testing.Short() {
        t.Skip("skipping integration test")
    }

    // Check for credentials
    if os.Getenv("AWS_ACCESS_KEY_ID") == "" {
        t.Skip("AWS_ACCESS_KEY_ID not set")
    }

    // Run test against real S3
    runIntegrationTest(t, setupS3Client())
}

func runIntegrationTest(t *testing.T, client ReplicaClient) {
    ctx := context.Background()

    t.Run("BasicReplication", func(t *testing.T) {
        // Test basic write/read cycle
    })

    t.Run("Compaction", func(t *testing.T) {
        // Test compaction with remote storage
    })

    t.Run("EventualConsistency", func(t *testing.T) {
        // Test handling of eventual consistency
    })

    t.Run("LargeFiles", func(t *testing.T) {
        // Test with files > 100MB
    })

    t.Run("Cleanup", func(t *testing.T) {
        err := client.DeleteAll(ctx)
        require.NoError(t, err)
    })
}
```

### Environment-Based Configuration

```go
func setupS3Client() *s3.ReplicaClient {
    return &s3.ReplicaClient{
        AccessKeyID:     os.Getenv("AWS_ACCESS_KEY_ID"),
        SecretAccessKey: os.Getenv("AWS_SECRET_ACCESS_KEY"),
        Region:          getEnvOrDefault("AWS_REGION", "us-east-1"),
        Bucket:          getEnvOrDefault("TEST_S3_BUCKET", "litestream-test"),
        Path:            fmt.Sprintf("test-%d", time.Now().Unix()),
    }
}

func getEnvOrDefault(key, defaultValue string) string {
    if value := os.Getenv(key); value != "" {
        return value
    }
    return defaultValue
}
```

## Performance Testing

### Benchmarks

```go
func BenchmarkDB_Sync(b *testing.B) {
    db := setupBenchDB(b)
    defer db.Close(context.Background())

    // Prepare test data
    conn, _ := sql.Open("sqlite", db.Path())
    defer conn.Close()

    for i := 0; i < 1000; i++ {
        conn.Exec("INSERT INTO test VALUES (?)", i)
    }

    b.ResetTimer()

    for i := 0; i < b.N; i++ {
        err := db.Sync(context.Background())
        if err != nil {
            b.Fatal(err)
        }
    }

    b.ReportMetric(float64(b.N)/b.Elapsed().Seconds(), "syncs/sec")
}

func BenchmarkCompaction(b *testing.B) {
    benchmarks := []struct {
        name      string
        fileCount int
        fileSize  int
    }{
        {"Small-Many", 1000, 1024},        // Many small files
        {"Medium", 100, 10 * 1024},        // Medium files
        {"Large-Few", 10, 100 * 1024},     // Few large files
    }

    for _, bm := range benchmarks {
        b.Run(bm.name, func(b *testing.B) {
            for i := 0; i < b.N; i++ {
                b.StopTimer()
                files := generateTestFiles(bm.fileCount, bm.fileSize)
                b.StartTimer()

                _, err := compact(files)
                if err != nil {
                    b.Fatal(err)
                }
            }

            totalSize := int64(bm.fileCount * bm.fileSize)
            b.ReportMetric(float64(totalSize)/float64(b.Elapsed().Nanoseconds()), "bytes/ns")
        })
    }
}
```

### Load Testing

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Mock Usage Patterns

### Mock ReplicaClient

```go
type MockReplicaClient struct {
    mu    sync.Mutex
    files map[string]*ltx.FileInfo
    data  map[string][]byte

    // Control behavior
    FailureRate   float64
    Latency       time.Duration
    EventualDelay time.Duration
}

func (m *MockReplicaClient) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {
    // Simulate latency
    if m.Latency > 0 {
        time.Sleep(m.Latency)
    }

    // Simulate failures
    if m.FailureRate > 0 && rand.Float64() < m.FailureRate {
        return nil, errors.New("simulated failure")
    }

    // Simulate eventual consistency
    if m.EventualDelay > 0 {
        time.AfterFunc(m.EventualDelay, func() {
            m.mu.Lock()
            defer m.mu.Unlock()
            // Make file available after delay
        })
    }

    // Store file
    data, err := io.ReadAll(r)
    if err != nil {
        return nil, err
    }

    m.mu.Lock()
    defer m.mu.Unlock()

    key := fmt.Sprintf("%d-%016x-%016x", level, minTXID, maxTXID)
    info := &ltx.FileInfo{
        Level:     level,
        MinTXID:   minTXID,
        MaxTXID:   maxTXID,
        Size:      int64(len(data)),
        CreatedAt: time.Now(),
    }

    m.files[key] = info
    m.data[key] = data

    return info, nil
}
```

### Mock Database

```go
type MockDB struct {
    mu       sync.Mutex
    path     string
    replicas []*Replica
    closed   bool

    // Control behavior
    CheckpointFailures int
    SyncDelay         time.Duration
}

func (m *MockDB) Sync(ctx context.Context) error {
    if m.SyncDelay > 0 {
        select {
        case <-time.After(m.SyncDelay):
        case <-ctx.Done():
            return ctx.Err()
        }
    }

    m.mu.Lock()
    defer m.mu.Unlock()

    if m.closed {
        return errors.New("database closed")
    }

    for _, r := range m.replicas {
        if err := r.Sync(ctx); err != nil {
            return err
        }
    }

    return nil
}
```

## Test Utilities

### Helper Functions

```go
// testutil/db.go
package testutil

import (
    "database/sql"
    "testing"
    "path/filepath"
)

func NewTestDB(t testing.TB) *litestream.DB {
    t.Helper()

    path := filepath.Join(t.TempDir(), "test.db")

    // Create SQLite database
    conn, err := sql.Open("sqlite", path+"?_journal=WAL")
    require.NoError(t, err)

    _, err = conn.Exec(`
        CREATE TABLE test (
            id INTEGER PRIMARY KEY,
            data BLOB
        )
    `)
    require.NoError(t, err)
    conn.Close()

    // Open with Litestream
    db := litestream.NewDB(path)
    db.MonitorInterval = 10 * time.Millisecond  // Speed up for tests
    db.MinCheckpointPageN = 100  // Lower threshold for tests

    err = db.Open()
    require.NoError(t, err)

    t.Cleanup(func() {
        db.Close(context.Background())
    })

    return db
}

func WriteTestData(t testing.TB, db *litestream.DB, count int) {
    t.Helper()

    conn, err := sql.Open("sqlite", db.Path())
    require.NoError(t, err)
    defer conn.Close()

    tx, err := conn.Begin()
    require.NoError(t, err)

    for i := 0; i < count; i++ {
        data := make([]byte, 100)
        rand.Read(data)
        _, err = tx.Exec("INSERT INTO test (data) VALUES (?)", data)
        require.NoError(t, err)
    }

    err = tx.Commit()
    require.NoError(t, err)
}
```

### Test Fixtures

```go
// testdata/fixtures.go
package testdata

import _ "embed"

//go:embed small.db
var SmallDB []byte

//go:embed large.db
var LargeDB []byte

//go:embed corrupted.db
var CorruptedDB []byte

func ExtractFixture(name string, path string) error {
    var data []byte

    switch name {
    case "small":
        data = SmallDB
    case "large":
        data = LargeDB
    case "corrupted":
        data = CorruptedDB
    default:
        return fmt.Errorf("unknown fixture: %s", name)
    }

    return os.WriteFile(path, data, 0600)
}
```

## Common Test Failures

### 1. Database Locked Errors

```go
// Problem: Multiple connections without proper WAL mode
func TestBroken(t *testing.T) {
    db1, _ := sql.Open("sqlite", "test.db")    // Wrong! WAL disabled
    db2, _ := sql.Open("sqlite", "test.db")    // Will fail
}

// Solution: Use WAL mode
func TestFixed(t *testing.T) {
    db1, _ := sql.Open("sqlite", "test.db?_journal=WAL")
    db2, _ := sql.Open("sqlite", "test.db?_journal=WAL")
}
```

### 2. Timing Issues

```go
// Problem: Race between write and sync
func TestBroken(t *testing.T) {
    WriteData(db)
    result := ReadReplica()  // May not see data yet!
}

// Solution: Explicit sync
func TestFixed(t *testing.T) {
    WriteData(db)
    err := db.Sync(context.Background())
    require.NoError(t, err)
    result := ReadReplica()  // Now guaranteed to see data
}
```

### 3. Cleanup Issues

```go
// Problem: Goroutine outlives test
func TestBroken(t *testing.T) {
    go func() {
        time.Sleep(10 * time.Second)
        doWork()  // Test already finished!
    }()
}

// Solution: Use context and wait
func TestFixed(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    var wg sync.WaitGroup
    wg.Add(1)
    go func() {
        defer wg.Done()
        select {
        case <-ctx.Done():
            return
        case <-time.After(10 * time.Second):
            doWork()
        }
    }()

    // Test work...

    cancel()  // Signal shutdown
    wg.Wait() // Wait for goroutine
}
```

### 4. File Handle Leaks

```go
// Problem: Not closing files
func TestBroken(t *testing.T) {
    f, _ := os.Open("test.db")
    // Missing f.Close()!
}

// Solution: Always use defer
func TestFixed(t *testing.T) {
    f, err := os.Open("test.db")
    require.NoError(t, err)
    defer f.Close()
}
```

## Test Coverage

### Running Coverage

```bash
# Generate coverage report
go test -coverprofile=coverage.out ./...

# View coverage in browser
go tool cover -html=coverage.out

# Check coverage percentage
go tool cover -func=coverage.out | grep total

# Coverage by package
go test -cover ./...
```

### Coverage Requirements

- Core packages (`db.go`, `replica.go`, `store.go`): >80%
- Replica clients: >70%
- Utilities: >60%
- Mock implementations: Not required

### Improving Coverage

```go
// Use test tables for comprehensive coverage
func TestDB_Checkpoint(t *testing.T) {
    tests := []struct {
        name     string
        mode     string
        walSize  int
        wantErr  bool
    }{
        {"Passive", "PASSIVE", 100, false},
        {"Full", "FULL", 1000, false},
        {"Restart", "RESTART", 5000, false},
        {"Truncate", "TRUNCATE", 10000, false},
        {"Invalid", "INVALID", 100, true},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            db := setupTestDB(t)
            generateWAL(t, db, tt.walSize)

            err := db.Checkpoint(tt.mode)
            if tt.wantErr {
                assert.Error(t, err)
            } else {
                assert.NoError(t, err)
            }
        })
    }
}
```

---

### VFS (`docs/VFS.md`)

# Litestream VFS

The Litestream VFS (Virtual File System) is a SQLite extension that allows applications to read directly
from Litestream replica storage (S3, GCS, Azure Blob, etc.) without restoring to local disk. It also
supports write mode for remote-first SQLite databases.

## Table of Contents

- [Overview](#overview)
- [Building](#building)
- [Configuration](#configuration)
- [Usage](#usage)
- [SQL Functions](#sql-functions)
- [Time Travel](#time-travel)
- [Write Mode](#write-mode)
- [Supported Storage Backends](#supported-storage-backends)

## Overview

The VFS extension provides:

- **Direct replica reading**: Read SQLite databases directly from cloud storage
- **Automatic polling**: Background polling for new LTX files from the primary
- **Page caching**: LRU cache for frequently accessed pages (default 10MB)
- **Time travel**: Query historical database states at specific timestamps
- **Write support**: Write changes that sync back to remote storage (experimental)

### How It Works

1. The VFS loads as a SQLite extension
2. When opening a database, it reads LTX files from the configured replica URL
3. Page requests are satisfied from cached data or fetched from remote storage
4. A background goroutine polls for new LTX files at a configurable interval

## Building

### Prerequisites

- SQLite 3.31.0+ runtime
- Go 1.21+
- GCC (Linux) or Clang (macOS)
- CGO enabled

### Build Commands

**macOS (current architecture):**

```bash
make vfs
```

This creates:
- `dist/litestream-vfs.a` - Static library
- `dist/litestream-vfs.so` - Loadable SQLite extension

**Platform-specific builds:**

```bash
# macOS ARM64 (Apple Silicon)
make vfs-darwin-arm64
# Output: dist/litestream-vfs-darwin-arm64.dylib

# macOS AMD64 (Intel)
make vfs-darwin-amd64
# Output: dist/litestream-vfs-darwin-amd64.dylib

# Linux AMD64
make vfs-linux-amd64
# Output: dist/litestream-vfs-linux-amd64.so

# Linux ARM64
make vfs-linux-arm64
# Output: dist/litestream-vfs-linux-arm64.so
```

### Running Tests

```bash
make vfs-test
```

## Configuration

The VFS is configured via environment variables:

| Variable | Description | Default |
|----------|-------------|---------|
| `LITESTREAM_REPLICA_URL` | Replica storage URL (required) | - |
| `LITESTREAM_LOG_LEVEL` | Log level: `DEBUG` or `INFO` | `INFO` |
| `LITESTREAM_WRITE_ENABLED` | Enable write mode: `true` or `false` | `false` |
| `LITESTREAM_SYNC_INTERVAL` | Write sync interval (e.g., `1s`, `500ms`) | `1s` |
| `LITESTREAM_BUFFER_PATH` | Local write buffer file path | temp file |

### Replica URL Format

```
# Amazon S3
s3://bucket-name/path/to/db

# Google Cloud Storage
gs://bucket-name/path/to/db

# Azure Blob Storage
abs://container-name/path/to/db

# Alibaba OSS
oss://bucket-name/path/to/db

# Local filesystem
file:///path/to/replica

# SFTP
sftp://user@host:port/path/to/db

# NATS JetStream
nats://host:port/bucket/path

# WebDAV
webdav://host:port/path/to/db
```

## Usage

### Loading the Extension

```sql
-- Load the extension (adjust path as needed)
.load ./dist/litestream-vfs.so
```

### Opening a Database

Before loading the extension, set the replica URL:

```bash
export LITESTREAM_REPLICA_URL="s3://my-bucket/mydb"
```

Then in SQLite:

```sql
.load ./dist/litestream-vfs.so
.open file:mydb.db?vfs=litestream
SELECT * FROM my_table;
```

### Python Example

```python
import os
import sqlite3

os.environ["LITESTREAM_REPLICA_URL"] = "s3://my-bucket/mydb"

conn = sqlite3.connect(":memory:")
conn.enable_load_extension(True)
conn.load_extension("./dist/litestream-vfs.so")

# Open database using the litestream VFS
conn = sqlite3.connect("file:mydb.db?vfs=litestream")
cursor = conn.execute("SELECT * FROM users")
for row in cursor:
    print(row)
```

### Go Example

```go
import (
    "database/sql"
    "os"

    _ "github.com/mattn/go-sqlite3"
)

func main() {
    os.Setenv("LITESTREAM_REPLICA_URL", "s3://my-bucket/mydb")

    db, err := sql.Open("sqlite3", "file:mydb.db?vfs=litestream")
    if err != nil {
        panic(err)
    }
    defer db.Close()

    rows, err := db.Query("SELECT * FROM users")
    // ...
}
```

## SQL Functions

The VFS extension provides SQL functions for observability and time travel:

### `litestream_txid()`

Returns the current transaction ID as a hex string.

```sql
SELECT litestream_txid();
-- Returns: "0000000000000042"
```

### `litestream_time()`

Returns the current view timestamp (RFC3339 format) or `"latest"`.

```sql
SELECT litestream_time();
-- Returns: "2024-01-15T10:30:00.123456789Z" or "latest"
```

### `litestream_lag()`

Returns seconds since the last successful poll for new LTX files. Returns `-1` if no successful poll has occurred.

```sql
SELECT litestream_lag();
-- Returns: 2 (seconds behind primary)
```

### `litestream_set_time(timestamp)`

Sets the view time for time travel queries. See [Time Travel](#time-travel) for details.

## Time Travel

The VFS supports querying historical database states by setting a target timestamp.

### Setting a Target Time

```sql
-- View database as of a specific timestamp (RFC3339 format)
SELECT litestream_set_time('2024-01-15T10:30:00Z');

-- Relative time expressions are also supported
SELECT litestream_set_time('5 minutes ago');
SELECT litestream_set_time('yesterday');
SELECT litestream_set_time('2 hours ago');

-- Return to latest state
SELECT litestream_set_time('LATEST');
```

### Example: Comparing Historical Data

```sql
-- Check current count
SELECT COUNT(*) FROM orders;
-- Returns: 1000

-- Go back in time
SELECT litestream_set_time('1 hour ago');

-- Check historical count
SELECT COUNT(*) FROM orders;
-- Returns: 950

-- Return to present
SELECT litestream_set_time('LATEST');
```

### Time Travel Limitations

- Time travel rebuilds the page index, which may take time for large databases
- Historical data is only available if LTX files haven't been compacted away
- L0 retention settings affect how far back you can travel
- Time travel is read-only (writes are disabled while viewing historical state)

## Write Mode

Write mode allows the VFS to accept writes and sync them back to remote storage. This is experimental.

### Enabling Write Mode

```bash
export LITESTREAM_REPLICA_URL="s3://my-bucket/mydb"
export LITESTREAM_WRITE_ENABLED="true"
export LITESTREAM_SYNC_INTERVAL="1s"
```

### How Write Mode Works

1. Writes are captured to a local buffer file for durability
2. Dirty pages are tracked in memory
3. Periodically (or on close), dirty pages are packaged into an LTX file
4. The LTX file is uploaded to remote storage
5. Conflict detection prevents overwrites if the remote has newer transactions

### Write Mode Considerations

- **Connection pooling**: Multiple connections can be opened in write mode (for example, by `database/sql`)
- **Single writer**: Write contention is enforced at lock acquisition. If another connection already holds write intent, SQLite returns `SQLITE_BUSY`
- **Conflict detection**: If the remote has advanced unexpectedly, `ErrConflict` is returned
- **Buffer durability**: The local buffer file provides crash recovery for uncommitted writes
- **Sync interval**: Balance between durability (shorter) and performance (longer)
- **New databases**: Write mode can create new databases from scratch if no LTX files exist

### Creating a New Database

With write mode enabled, you can create a new database that doesn't exist yet:

```sql
.load ./dist/litestream-vfs.so
.open file:newdb.db?vfs=litestream

CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT);
INSERT INTO users (name) VALUES ('Alice');
-- Data is synced to remote storage automatically
```

## Supported Storage Backends

The VFS supports all Litestream storage backends:

| Backend | URL Scheme | Notes |
|---------|-----------|-------|
| Amazon S3 | `s3://` | Supports S3-compatible services (MinIO, DigitalOcean Spaces, etc.) |
| Google Cloud Storage | `gs://` | Requires `GOOGLE_APPLICATION_CREDENTIALS` |
| Azure Blob Storage | `abs://` | Requires `AZURE_STORAGE_ACCOUNT` and credentials |
| Alibaba OSS | `oss://` | Object Storage Service |
| Local filesystem | `file://` | Useful for testing and development |
| SFTP | `sftp://` | SSH File Transfer Protocol |
| NATS JetStream | `nats://` | Object store via NATS |
| WebDAV | `webdav://` | Web Distributed Authoring and Versioning |

### S3 Configuration

For S3 and S3-compatible services, set credentials via environment variables:

```bash
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
export AWS_REGION="us-east-1"

# For S3-compatible services:
export LITESTREAM_REPLICA_URL="s3://bucket/path?endpoint=https://custom.endpoint.com"
```

### GCS Configuration

```bash
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
export LITESTREAM_REPLICA_URL="gs://bucket/path"
```

## Troubleshooting

### Extension fails to load

Ensure the extension file matches your platform:
- macOS: `.dylib` or `.so`
- Linux: `.so`

### "no backup files available"

The VFS waits for LTX files to become available. Ensure:
1. The replica URL is correct
2. Litestream has replicated at least one transaction
3. Credentials are properly configured

### High latency reads

- Increase `CacheSize` for larger page cache
- Reduce `PollInterval` for more responsive updates
- Consider using a closer storage region

### Debug logging

```bash
export LITESTREAM_LOG_LEVEL="DEBUG"
```

This enables verbose logging of VFS operations, page fetches, and cache hits/misses.

---

--- METRICS ---
- Files Extracted: 26
- Estimated Token Budget: ~49192 tokens
- Canonical Reference: https://codewiki.google/github.com/benbjohnson/litestream