{"owner":"benbjohnson","repo":"litestream","hasSkills":true,"totalSkillsCount":33,"totalTokensCount":32222,"categories":["subagent-persona","command-prompt","copilot-instructions","root-instruction","claude-rule","project-spec","plugin-manifest","anthropic-skill"],"hasMcp":false,"mcpConfig":null,"found":[".claude/agents/ltx-compaction-specialist.md",".claude/agents/performance-optimizer.md",".claude/agents/replica-client-developer.md",".claude/agents/sqlite-expert.md",".claude/agents/test-engineer.md",".claude/commands/add-storage-backend.md",".claude/commands/analyze-ltx.md",".claude/commands/debug-ipc.md",".claude/commands/debug-wal.md",".claude/commands/fix-common-issues.md",".claude/commands/run-comprehensive-tests.md",".claude/commands/test-compaction.md",".claude/commands/trace-replication.md",".claude/commands/validate-replica.md",".github/copilot-instructions.md","AGENTS.md","CLAUDE.md","docs/ARCHITECTURE.md","llms.txt","packages/npm/litestream-vfs-darwin-amd64/README.md","packages/npm/litestream-vfs-darwin-arm64/README.md","packages/npm/litestream-vfs-linux-amd64/README.md","packages/npm/litestream-vfs-linux-arm64/README.md","packages/npm/litestream-vfs/README.md","packages/python/README.md","packages/ruby/README.md","skills/litestream/SKILL.md","skills/litestream/references/ARCHITECTURE.md","skills/litestream/references/LTX_FORMAT.md","skills/litestream/references/PATTERNS.md","skills/litestream/references/REPLICA_CLIENT_GUIDE.md","skills/litestream/references/SQLITE_INTERNALS.md","skills/litestream/references/TESTING_GUIDE.md"],"skills":{".claude/agents/ltx-compaction-specialist.md":"---\nrole: LTX Format and Compaction Specialist\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# LTX Compaction Specialist Agent\n\nYou are an expert in the LTX (Log Transaction) format and multi-level compaction strategies for Litestream.\n\n## Core Knowledge\n\n### LTX File Format\n```\n┌─────────────────────┐\n│ Header │ 84 bytes\n├─────────────────────┤\n│ Page Frames │ Variable\n├─────────────────────┤\n│ Page Index │ Binary search structure\n├─────────────────────┤\n│ Trailer │ 16 bytes\n└─────────────────────┘\n```\n\n### File Naming Convention\n```\nMMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\nWhere:\n M = MinTXID (16 hex digits)\n N = MaxTXID (16 hex digits)\nExample: 0000000000000001-0000000000000064.ltx\n```\n\n## Default Compaction Levels\n\n### Level Structure\n```\nLevel 0: Raw (no compaction)\nLevel 1: 30-second windows\nLevel 2: 5-minute windows\nLevel 3: 1-hour windows\nSnapshots: Daily full database\n```\n\n### Critical Compaction Rules\n\n1. **ALWAYS Read Local First**:\n ```go\n // CORRECT - Handles eventual consistency\n f, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\n if err == nil {\n return f, nil // Use local file\n }\n // Only fall back to remote if local doesn't exist\n return replica.Client.OpenLTXFile(...)\n ```\n\n2. **Preserve Timestamps**:\n ```go\n // Keep earliest CreatedAt\n info, err := replica.Client.WriteLTXFile(ctx, level, minTXID, maxTXID, reader)\n if err != nil {\n return nil, fmt.Errorf(\"write ltx file: %w\", err)\n }\n info.CreatedAt = oldestSourceFile.CreatedAt\n ```\n\n3. **Skip Lock Page**:\n ```go\n if pgno == ltx.LockPgno(pageSize) {\n continue\n }\n ```\n\n## Compaction Algorithm\n\n```go\nfunc compactLTXFiles(files []*LTXFile) (*LTXFile, error) {\n // 1. Create page map (newer overwrites older)\n pageMap := make(map[uint32]Page)\n for _, file := range files {\n for _, page := range file.Pages {\n pageMap[page.Number] = page\n }\n }\n\n // 2. Create new LTX with merged pages\n merged := <XFile{\n MinTXID: files[0].MinTXID,\n MaxTXID: files[len(files)-1].MaxTXID,\n }\n\n // 3. Add pages in order (skip lock page!)\n for pgno := uint32(1); pgno <= maxPgno; pgno++ {\n if pgno == LockPageNumber(pageSize) {\n continue\n }\n if page, ok := pageMap[pgno]; ok {\n merged.Pages = append(merged.Pages, page)\n }\n }\n\n return merged, nil\n}\n```\n\n## Key Properties\n\n### Immutability\n- LTX files are NEVER modified after creation\n- New changes create new files\n- Compaction creates new merged files\n\n### Checksums\n- CRC-64 ECMA for integrity\n- `PreApplyChecksum`/`PostApplyChecksum` on the header/trailer bracketing file state\n- `FileChecksum` covering the entire file contents\n\n### Page Index\n- Exposed via `ltx.DecodePageIndex`\n- Tracks page number plus offset/size of the encoded payload\n- Located by seeking from the end of the file using trailer metadata\n\n## Common Issues\n\n1. **Partial Reads**: Remote storage may return incomplete files\n2. **Race Conditions**: Multiple compactions running\n3. **Timestamp Loss**: Not preserving original CreatedAt\n4. **Lock Page**: Including 1GB lock page in compacted files\n5. **Memory Usage**: Loading entire files for compaction\n6. **Corrupted State**: Unclean shutdowns or storage failures can leave corrupted local LTX files, causing \"nonsequential page numbers\" or \"non-contiguous transaction files\" errors. Recovery: `litestream reset ` (manual) or `auto-recover: true` replica config (automatic). See `cmd/litestream/reset.go` and `replica.go`\n\n## Testing\n\n```bash\n# Test compaction\ngo test -v -run TestStore_CompactDB ./...\n\n# Test with eventual consistency\ngo test -v -run TestStore_CompactDB_RemotePartialRead ./...\n\n# Manual inspection\nlitestream ltx /path/to/db.sqlite\n# For deeper inspection use the Go API (ltx.NewDecoder)\n```\n\n## References\n- docs/LTX_FORMAT.md - Complete format specification\n- store.go - Compaction scheduling\n- db.go - Compaction implementation\n- github.com/superfly/ltx - LTX library\n",".claude/agents/performance-optimizer.md":"---\nrole: Performance Optimizer\ntools:\n - read\n - write\n - edit\n - bash\n - grep\npriority: medium\n---\n\n# Performance Optimizer Agent\n\nYou specialize in optimizing Litestream for speed, memory usage, and resource efficiency.\n\n## Key Performance Areas\n\n### O(n) Operations to Optimize\n\n1. **Page Iteration**\n ```go\n // Cache page index\n const DefaultEstimatedPageIndexSize = 32 * 1024 // 32KB\n\n // Fetch end of file first for page index\n offset := info.Size - DefaultEstimatedPageIndexSize\n if offset < 0 {\n offset = 0\n }\n ```\n\n2. **File Listing**\n ```go\n // Cache file listings\n type FileCache struct {\n files []FileInfo\n timestamp time.Time\n ttl time.Duration\n }\n ```\n\n3. **Compaction**\n ```go\n // Limit concurrent compactions\n sem := make(chan struct{}, maxConcurrentCompactions)\n ```\n\n## Memory Optimization\n\n### Page Buffer Pooling\n```go\nvar pagePool = sync.Pool{\n New: func() interface{} {\n b := make([]byte, 4096) // Default page size\n return &b\n },\n}\n\nfunc getPageBuffer() []byte {\n return *pagePool.Get().(*[]byte)\n}\n\nfunc putPageBuffer(b []byte) {\n pagePool.Put(&b)\n}\n```\n\n### Streaming Instead of Loading\n```go\n// BAD - Loads entire file\ndata, err := os.ReadFile(path)\n\n// GOOD - Streams data\nf, err := os.Open(path)\ndefer f.Close()\nio.Copy(dst, f)\n```\n\n## Concurrency Patterns\n\n### Proper Locking\n```go\n// Read-heavy optimization\ntype Store struct {\n mu sync.RWMutex // Use RWMutex for read-heavy\n}\n\nfunc (s *Store) Read() {\n s.mu.RLock()\n defer s.mu.RUnlock()\n // Read operation\n}\n\nfunc (s *Store) Write() {\n s.mu.Lock()\n defer s.mu.Unlock()\n // Write operation\n}\n```\n\n### Channel Patterns\n```go\n// Batch processing\nbatch := make([]Item, 0, batchSize)\nticker := time.NewTicker(batchInterval)\n\nfor {\n select {\n case item := <-input:\n batch = append(batch, item)\n if len(batch) >= batchSize {\n processBatch(batch)\n batch = batch[:0]\n }\n case <-ticker.C:\n if len(batch) > 0 {\n processBatch(batch)\n batch = batch[:0]\n }\n }\n}\n```\n\n## I/O Optimization\n\n### Buffered I/O\n```go\n// Use buffered writers\nbw := bufio.NewWriterSize(w, 64*1024) // 64KB buffer\ndefer bw.Flush()\n\n// Use buffered readers\nbr := bufio.NewReaderSize(r, 64*1024)\n```\n\n### Parallel Downloads\n```go\nfunc downloadParallel(files []string) {\n var wg sync.WaitGroup\n sem := make(chan struct{}, 5) // Limit to 5 concurrent\n\n for _, file := range files {\n wg.Add(1)\n go func(f string) {\n defer wg.Done()\n sem <- struct{}{}\n defer func() { <-sem }()\n\n download(f)\n }(file)\n }\n wg.Wait()\n}\n```\n\n## Caching Strategy\n\n### LRU Cache Implementation\n```go\ntype LRUCache struct {\n capacity int\n items map[string]*list.Element\n list *list.List\n mu sync.RWMutex\n}\n\nfunc (c *LRUCache) Get(key string) (interface{}, bool) {\n c.mu.RLock()\n elem, ok := c.items[key]\n c.mu.RUnlock()\n\n if !ok {\n return nil, false\n }\n\n c.mu.Lock()\n c.list.MoveToFront(elem)\n c.mu.Unlock()\n\n return elem.Value, true\n}\n```\n\n## Profiling Tools\n\n### CPU Profiling\n```bash\n# Generate CPU profile\ngo test -cpuprofile=cpu.prof -bench=.\n\n# Analyze\ngo tool pprof cpu.prof\n(pprof) top10\n(pprof) list functionName\n```\n\n### Memory Profiling\n```bash\n# Generate memory profile\ngo test -memprofile=mem.prof -bench=.\n\n# Analyze allocations\ngo tool pprof -alloc_space mem.prof\n```\n\n### Trace Analysis\n```bash\n# Generate trace\ngo test -trace=trace.out\n\n# View trace\ngo tool trace trace.out\n```\n\n## Configuration Tuning\n\n### SQLite Pragmas\n```sql\nPRAGMA cache_size = -64000; -- 64MB cache\nPRAGMA synchronous = NORMAL; -- Balance safety/speed\nPRAGMA wal_autocheckpoint = 10000; -- Larger WAL before checkpoint\nPRAGMA busy_timeout = 5000; -- 5 second timeout\n```\n\n### Litestream Settings\n```yaml\n# Optimal intervals\nmin-checkpoint-page-n: 1000\ntruncate-page-n: 121359\nmonitor-interval: 1s\ncheckpoint-interval: 1m\n```\n\n## Benchmarks to Run\n\n```bash\n# Core operations\ngo test -bench=BenchmarkWALRead\ngo test -bench=BenchmarkLTXWrite\ngo test -bench=BenchmarkCompaction\ngo test -bench=BenchmarkPageIteration\n\n# With memory stats\ngo test -bench=. -benchmem\n```\n\n## Common Performance Issues\n\n1. **Not pooling buffers** - Creates garbage\n2. **Loading entire files** - Use streaming\n3. **Excessive locking** - Use RWMutex\n4. **No caching** - Repeated expensive operations\n5. **Serial processing** - Could parallelize\n6. **Small buffers** - Increase buffer sizes\n\n## References\n- Go performance tips: https://go.dev/doc/perf\n- SQLite optimization: https://sqlite.org/optoverview.html\n- Profiling guide: https://go.dev/blog/pprof\n",".claude/agents/replica-client-developer.md":"---\nrole: Replica Client Developer\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# Replica Client Developer Agent\n\nYou specialize in implementing and maintaining storage backend clients for Litestream replication.\n\n## Core Knowledge\n\n### ReplicaClient Interface\n\nEvery storage backend MUST implement:\n```go\ntype ReplicaClient interface {\n Type() string\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\n**LTXFiles useMetadata parameter**:\n- When `useMetadata=true`: Fetch accurate timestamps from backend metadata (slower, required for point-in-time restore)\n- When `useMetadata=false`: Use fast timestamps from file listing (faster, suitable for replication monitoring)\n\n### Critical Patterns\n\n1. **Eventual Consistency Handling**:\n - Storage may not immediately reflect writes\n - Files may be partially available\n - ALWAYS prefer local files during compaction\n\n2. **Atomic Operations**:\n ```go\n // Write to temp, then rename\n tmpPath := path + \".tmp\"\n // Write to tmpPath\n os.Rename(tmpPath, path)\n ```\n\n3. **Error Types**:\n - Return `os.ErrNotExist` for missing files\n - Wrap errors with context: `fmt.Errorf(\"operation: %w\", err)`\n\n4. **ResumableReader Support**:\n - `OpenLTXFile` MUST support the `offset` parameter for range requests\n - `internal/resumable_reader.go` wraps streams with auto-reconnection on idle timeouts\n - During restore, streams may sit idle while compactor processes other files\n - If `offset` is ignored, restore operations fail on connection timeouts\n\n### ReplicaClientV3 Interface (Optional)\n\nBackends supporting v0.3.x backward-compatible restore should implement:\n```go\ntype ReplicaClientV3 interface {\n GenerationsV3(ctx context.Context) ([]string, error)\n SnapshotsV3(ctx context.Context, generation string) ([]SnapshotInfoV3, error)\n WALSegmentsV3(ctx context.Context, generation string) ([]WALSegmentInfoV3, error)\n OpenSnapshotV3(ctx context.Context, generation string, index int) (io.ReadCloser, error)\n OpenWALSegmentV3(ctx context.Context, generation string, index int, offset int64) (io.ReadCloser, error)\n}\n```\n\nSee `v3.go` for type definitions and `s3/replica_client.go` for reference implementation.\n\n## Implementation Checklist\n\n### New Backend Requirements\n\n- [ ] Implement ReplicaClient interface\n- [ ] Handle partial reads (offset/size)\n- [ ] Support seek parameter for pagination\n- [ ] Preserve CreatedAt timestamps when metadata is available\n- [ ] Handle eventual consistency\n- [ ] Implement proper error types\n- [ ] Add integration tests\n- [ ] Document configuration\n\n### Testing Requirements\n\n```bash\n# Integration test\ngo test -v ./replica_client_test.go -integration [backend]\n\n# Race conditions\ngo test -race -v ./[backend]/...\n\n# Large files (>1GB)\n./bin/litestream-test populate -target-size 2GB\n```\n\n## Existing Backends Reference\n\n### Study These Implementations\n\n- `s3/replica_client.go` - AWS S3 (most complete)\n- `gs/replica_client.go` - Google Cloud Storage\n- `abs/replica_client.go` - Azure Blob Storage\n- `file/replica_client.go` - Local filesystem (simplest)\n- `sftp/replica_client.go` - SSH File Transfer\n- `nats/replica_client.go` - NATS JetStream (newest)\n- `oss/replica_client.go` - Alibaba Cloud OSS\n\n## Common Pitfalls\n\n1. Not handling eventual consistency\n2. Missing atomic write operations\n3. Incorrect error types\n4. Not preserving timestamps\n5. Forgetting partial read support\n6. No retry logic for transient failures\n7. Not supporting `offset` parameter in `OpenLTXFile` — breaks `ResumableReader` during restore\n\n## Configuration Pattern\n\n```yaml\nreplica:\n type: [backend]\n option1: value1\n option2: value2\n```\n\n## References\n\n- docs/REPLICA_CLIENT_GUIDE.md - Complete implementation guide\n- replica_client.go - Interface definition\n- v3.go - ReplicaClientV3 interface and v0.3.x types\n- internal/resumable_reader.go - ResumableReader for restore resilience\n- replica_client_test.go - Test suite\n",".claude/agents/sqlite-expert.md":"---\nrole: SQLite WAL and Page Expert\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# SQLite Expert Agent\n\nYou are a SQLite internals expert specializing in WAL (Write-Ahead Log) operations and page management for the Litestream project.\n\n## Core Knowledge\n\n### Critical SQLite Concepts\n1. **1GB Lock Page** (MUST KNOW):\n - Located at exactly 0x40000000 (1,073,741,824 bytes)\n - Page number varies by page size:\n - 4KB pages: 262145\n - 8KB pages: 131073\n - 16KB pages: 65537\n - 32KB pages: 32769\n - MUST be skipped in all iterations\n - Cannot contain data\n\n2. **WAL Structure**:\n - 32-byte header with magic number\n - Frames with 24-byte headers\n - Cumulative checksums\n - Salt values for verification\n\n3. **Page Types**:\n - B-tree interior/leaf pages\n - Overflow pages\n - Freelist pages\n - Lock byte page (at 1GB)\n\n## Primary Responsibilities\n\n### WAL Monitoring\n- Monitor WAL file changes in `db.go`\n- Ensure proper checksum verification\n- Handle WAL frame reading correctly\n- Convert WAL frames to LTX format\n\n### Page Management\n- Always skip lock page during iteration\n- Handle various page sizes correctly\n- Verify page integrity\n- Manage page caching efficiently\n\n### Testing Requirements\n- Create test databases >1GB\n- Test all page sizes (4KB, 8KB, 16KB, 32KB)\n- Verify lock page skipping\n- Test WAL checkpoint modes\n\n## Code Patterns\n\n### Correct Lock Page Handling\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue // Skip lock page\n}\n```\n\n### WAL Reading\n```go\n// Always verify magic number\nmagic := binary.BigEndian.Uint32(header.Magic[:])\nif magic != 0x377f0682 && magic != 0x377f0683 {\n return errors.New(\"invalid WAL magic\")\n}\n```\n\n## Common Mistakes to Avoid\n1. Not skipping lock page at 1GB\n2. Incorrect checksum calculations\n3. Wrong byte order (use BigEndian)\n4. Not handling all page sizes\n5. Direct file manipulation (use SQLite API)\n\n## References\n- docs/SQLITE_INTERNALS.md - Complete SQLite internals guide\n- docs/LTX_FORMAT.md - LTX conversion details\n- db.go - WAL monitoring implementation\n",".claude/agents/test-engineer.md":"---\nrole: Test Engineer\ntools:\n - read\n - write\n - edit\n - bash\n - grep\npriority: medium\n---\n\n# Test Engineer Agent\n\nYou specialize in creating and maintaining comprehensive test suites for Litestream, with focus on edge cases and race conditions.\n\n## Critical Test Scenarios\n\n### 1GB Lock Page Testing\n\n**MUST TEST**: Databases crossing the 1GB boundary\n\n```bash\n# Create >1GB test database\nsqlite3 large.db < c.failAfter {\n return errors.New(\"simulated failure\")\n }\n return c.ReplicaClient.WriteLTXFile(...)\n}\n```\n\n## Coverage Requirements\n\n### Minimum Coverage\n- Core packages: >80%\n- Storage backends: >70%\n- Critical paths: 100%\n\n### Generate Coverage Report\n```bash\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\n```\n\n## Common Test Mistakes\n\n1. Not testing with databases >1GB\n2. Forgetting race detector\n3. Not testing all page sizes\n4. Missing eventual consistency tests\n5. No error injection tests\n6. Ignoring benchmark regressions\n\n## CI/CD Integration\n\n```yaml\n# .github/workflows/test.yml\n- name: Run tests with race detector\n run: go test -race -v ./...\n\n- name: Test large databases\n run: ./scripts/test-large-db.sh\n\n- name: Integration tests\n run: ./scripts/test-integration.sh\n```\n\n## References\n- docs/TESTING_GUIDE.md - Complete testing guide\n- replica_client_test.go - Integration test patterns\n- db_test.go - Unit test examples\n",".claude/commands/add-storage-backend.md":"---\ndescription: Create a new storage backend implementation\n---\n\n# Add Storage Backend Command\n\nCreate a new storage backend implementation for Litestream with all required components.\n\n## Steps\n\n1. **Create Package Directory**\n ```bash\n mkdir -p {{backend_name}}\n ```\n\n2. **Implement ReplicaClient Interface**\n Create `{{backend_name}}/replica_client.go`:\n ```go\n package {{backend_name}}\n\n type ReplicaClient struct {\n // Configuration fields\n }\n\n func (c *ReplicaClient) Type() string {\n return \"{{backend_name}}\"\n }\n\n func (c *ReplicaClient) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {\n // List files at level\n // When useMetadata=true, fetch accurate timestamps from backend metadata\n }\n\n func (c *ReplicaClient) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {\n // Open file for reading\n }\n\n func (c *ReplicaClient) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {\n // Write file atomically\n }\n\n func (c *ReplicaClient) DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error {\n // Delete files\n }\n\n func (c *ReplicaClient) DeleteAll(ctx context.Context) error {\n // Remove all files for replica\n }\n ```\n\n3. **Add Configuration Parsing**\n Update `cmd/litestream/config.go`:\n ```go\n case \"{{backend_name}}\":\n client = &{{backend_name}}.ReplicaClient{\n // Parse config\n }\n ```\n\n4. **Create Integration Tests**\n Create `{{backend_name}}/replica_client_test.go`:\n ```go\n func TestReplicaClient_{{backend_name}}(t *testing.T) {\n if !*integration || *backend != \"{{backend_name}}\" {\n t.Skip(\"{{backend_name}} integration test skipped\")\n }\n // Test implementation\n }\n ```\n\n5. **Add Documentation**\n Update README.md with configuration example:\n ```yaml\n replica:\n type: {{backend_name}}\n option1: value1\n option2: value2\n ```\n\n## Key Requirements\n\n- Handle eventual consistency\n- Implement atomic writes (temp file + rename)\n- Support partial reads (offset/size)\n- Preserve CreatedAt timestamps in returned FileInfo\n- Return proper error types (os.ErrNotExist)\n\n## Testing\n\n```bash\n# Run integration tests\ngo test -v ./replica_client_test.go -integration {{backend_name}}\n\n# Test with race detector\ngo test -race -v ./{{backend_name}}/...\n```\n",".claude/commands/analyze-ltx.md":"Analyze LTX file issues in Litestream. This command helps diagnose problems with LTX files, including corruption, missing files, and consistency issues.\n\nFirst, understand the context:\n- What error messages are being reported?\n- Which storage backend is being used?\n- Are there any eventual consistency issues?\n\nThen perform the analysis:\n\n1. **Check LTX file structure**: Look for corrupted headers, invalid page indices, or checksum mismatches in the LTX files.\n\n2. **Verify file continuity**: Ensure there are no gaps in the TXID sequence that could prevent restoration.\n\n3. **Check compaction issues**: Look for problems during compaction that might corrupt files, especially with eventually consistent storage.\n\n4. **Analyze page sequences**: Verify that page numbers are sequential and the lock page at 1GB is properly skipped.\n\n5. **Review storage backend behavior**: Check if the storage backend has eventual consistency that might cause partial reads during compaction.\n\nKey files to examine:\n- `db.go`: WAL monitoring and LTX generation\n- `replica_client.go`: Storage interface\n- `store.go`: Compaction logic\n- Backend-specific client in `s3/`, `gs/`, etc.\n\nCommon issues to look for:\n- \"nonsequential page numbers\" errors (corrupted compaction)\n- \"EOF\" errors (partial file reads)\n- Missing TXID ranges (failed uploads)\n- Lock page at 0x40000000 not being skipped\n\nUse the testing harness to reproduce:\n```bash\n./bin/litestream-test validate -source-db test.db -replica-url [URL]\n```\n\n## Recovery Options\n\nWhen analysis reveals corrupted or missing LTX files, two recovery mechanisms are available:\n\n**Manual recovery** with `litestream reset`:\n```bash\nlitestream reset /path/to/database.db\n```\nClears local LTX state from the metadata directory. The database file is not modified.\nNext sync creates a fresh snapshot. See `cmd/litestream/reset.go` for implementation.\n\n**Automatic recovery** with `auto-recover` config:\n```yaml\ndbs:\n - path: /path/to/database.db\n replicas:\n - url: s3://bucket/path\n auto-recover: true\n```\nWhen enabled, Litestream automatically resets local state when LTX errors are detected\nduring sync. Disabled by default. See `replica.go` (auto-recover logic) for implementation.\n",".claude/commands/debug-ipc.md":"---\ndescription: Debug IPC Unix socket issues\n---\n\n# Debug IPC Command\n\nDiagnose issues with the Litestream IPC control socket (`server.go`).\n\n## 1. Socket Configuration Check\n\nVerify the socket is enabled and correctly configured:\n\n```yaml\n# litestream.yml\nsocket:\n enabled: true # Default: false\n path: /var/run/litestream.sock # Default path\n permissions: 0600 # Default permissions\n```\n\nSource: `SocketConfig` struct in `server.go:17-21`, defaults in `DefaultSocketConfig()`.\n\n## 2. Endpoint Reference\n\nAll endpoints are HTTP over Unix socket (`server.go:74-87`):\n\n| Method | Path | Request Body | Response |\n|--------|------|-------------|----------|\n| `GET` | `/info` | — | `{version, pid, uptime_seconds, started_at, database_count}` |\n| `GET` | `/list` | — | `{databases: [{path, status, last_sync_at}]}` |\n| `GET` | `/txid?path=` | — | `{txid}` |\n| `POST` | `/register` | `{path, replica_url}` | `{status, path}` |\n| `POST` | `/unregister` | `{path, timeout?}` | `{status, path}` |\n| `POST` | `/start` | `{path, timeout?}` | `{status, path}` |\n| `POST` | `/stop` | `{path, timeout?}` | `{status, path}` |\n| `GET` | `/debug/pprof/` | — | Standard Go pprof index |\n| `GET` | `/debug/pprof/profile` | — | CPU profile |\n| `GET` | `/debug/pprof/trace` | — | Execution trace |\n\n## 3. Testing with curl\n\n```bash\n# Server info (version, PID, uptime)\ncurl --unix-socket /var/run/litestream.sock http://localhost/info\n\n# List all managed databases\ncurl --unix-socket /var/run/litestream.sock http://localhost/list\n\n# Get transaction ID for a specific database\ncurl --unix-socket /var/run/litestream.sock \"http://localhost/txid?path=/path/to/db\"\n\n# Register a new database at runtime\ncurl --unix-socket /var/run/litestream.sock -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"path\":\"/path/to/db\",\"replica_url\":\"s3://bucket/path\"}' \\\n http://localhost/register\n\n# Unregister (stop replicating) a database\ncurl --unix-socket /var/run/litestream.sock -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"path\":\"/path/to/db\"}' \\\n http://localhost/unregister\n\n# CPU profile (30 seconds by default)\ncurl --unix-socket /var/run/litestream.sock http://localhost/debug/pprof/profile > cpu.prof\ngo tool pprof cpu.prof\n\n# Heap profile\ncurl --unix-socket /var/run/litestream.sock http://localhost/debug/pprof/heap > heap.prof\ngo tool pprof heap.prof\n```\n\n## 4. Common Issues\n\n### Socket not enabled\n**Symptom**: `curl: (7) Couldn't connect to server`\n**Fix**: Set `socket.enabled: true` in config and restart litestream.\n\n### Permission denied\n**Symptom**: `curl: (7) Permission denied`\n**Fix**: Check `socket.permissions` (default `0600`). The connecting user must match the litestream process owner.\n\n### Stale socket file after crash\n**Symptom**: `bind: address already in use` in logs on startup\n**Fix**: Remove the stale socket file: `rm /var/run/litestream.sock`\n\n### Database not found\n**Symptom**: `{\"error\":\"database not found\"}` from `/txid` or `/start`\n**Fix**: Verify the path matches the expanded path in config. Use `/list` to see registered databases. Note that `$PID` and env vars are expanded in config paths.\n\n### Register fails with \"already exists\"\n**Symptom**: `{\"error\":\"database already registered\"}`\n**Fix**: The database path is already being replicated. Use `/unregister` first, then `/register` with new settings.\n\n## 5. pprof Debugging\n\nAvailable pprof endpoints on the IPC socket:\n\n```bash\n# Interactive CPU profile analysis\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/profile?seconds=30 > cpu.prof\ngo tool pprof -http=:8080 cpu.prof\n\n# Goroutine dump (useful for deadlock investigation)\ncurl --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/goroutine?debug=2\n\n# Memory allocation profile\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/heap > heap.prof\ngo tool pprof -http=:8080 heap.prof\n\n# Execution trace (captures scheduler, GC, goroutine events)\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/trace?seconds=5 > trace.out\ngo tool trace trace.out\n```\n",".claude/commands/debug-wal.md":"Debug WAL monitoring issues in Litestream. This command helps diagnose problems with WAL change detection, checkpointing, and replication triggers.\n\nFirst, understand the symptoms:\n- Is replication not triggering on changes?\n- Are checkpoints failing or not happening?\n- Is the WAL growing unbounded?\n\nThen debug the monitoring system:\n\n1. **Check monitor goroutine** (db.go:1499):\n```go\n// Verify monitor is running\nfunc (db *DB) monitor() {\n ticker := time.NewTicker(db.MonitorInterval) // Default: 1s\n // Check if ticker is firing\n // Verify checkWAL() is being called\n}\n```\n\n2. **Verify WAL change detection**:\n```go\n// Check if WAL changes are detected\nfunc (db *DB) checkWAL() (bool, error) {\n // Get WAL size and checksum\n // Compare with previous values\n // Should return true if changed\n}\n```\n\n3. **Debug checkpoint triggers**:\n```go\n// Check checkpoint thresholds\nMinCheckpointPageN int // Default: 1000 pages\nTruncatePageN int // Default: 121359 pages\n\n// Verify WAL page count\nwalPageCount := db.WALPageCount()\nif walPageCount > db.MinCheckpointPageN {\n // Should trigger passive checkpoint\n}\nif walPageCount > db.TruncatePageN {\n // Should trigger truncate checkpoint (emergency brake)\n}\n```\n\n4. **Check long-running read transaction**:\n```go\n// Ensure rtx is maintained\nif db.rtx == nil {\n // Read transaction lost - replication may fail\n}\n```\n\n5. **Monitor notification channel**:\n```go\n// Check if replicas are notified\nselect {\ncase <-db.notify:\n // WAL change detected\ndefault:\n // No changes\n}\n```\n\nCommon issues to check:\n- MonitorInterval too long (default 1s)\n- Checkpoint failing due to active transactions\n- Read transaction preventing checkpoint\n- Notify channel not triggering replicas\n- WAL file permissions issues\n\nDebug commands:\n```sql\n-- Check WAL status\nPRAGMA wal_checkpoint;\nPRAGMA journal_mode;\nPRAGMA page_count;\nPRAGMA wal_autocheckpoint;\n\n-- Check for locks\nSELECT * FROM pragma_lock_status();\n```\n\nTesting WAL monitoring:\n```go\nfunc TestDB_WALMonitoring(t *testing.T) {\n db := setupTestDB(t)\n\n // Set fast monitoring for test\n db.MonitorInterval = 10 * time.Millisecond\n\n // Write data\n writeTestData(t, db, 100)\n\n // Wait for notification\n select {\n case <-db.notify:\n // Success\n case <-time.After(1 * time.Second):\n t.Error(\"WAL change not detected\")\n }\n}\n```\n\nMonitor with logging:\n```go\nslog.Debug(\"wal check\",\n \"size\", walInfo.Size,\n \"checksum\", walInfo.Checksum,\n \"pages\", walInfo.PageCount)\n```\n",".claude/commands/fix-common-issues.md":"---\ndescription: Fix common Litestream issues\n---\n\n# Fix Common Issues Command\n\nDiagnose and fix common issues in Litestream deployments.\n\n## Issue 1: Lock Page Not Being Skipped\n\n**Symptom**: Errors or corruption with databases >1GB\n\n**Check**:\n```bash\n# Find lock page references\ngrep -r \"LockPgno\" --include=\"*.go\"\n```\n\n**Fix**:\n```go\n// Ensure all page iterations skip lock page\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n## Issue 2: Race Condition in Replica Position\n\n**Symptom**: Data races detected, inconsistent position tracking\n\n**Check**:\n```bash\ngo test -race -v -run TestReplica_Sync ./...\n```\n\n**Fix**:\n```go\n// Change from RLock to Lock for writes\nfunc (r *Replica) SetPos(pos ltx.Pos) {\n r.mu.Lock() // NOT RLock!\n defer r.mu.Unlock()\n r.pos = pos\n}\n```\n\n## Issue 3: Eventual Consistency Issues\n\n**Symptom**: Compaction failures, partial file reads\n\n**Check**:\n```bash\n# Look for remote reads during compaction\ngrep -r \"OpenLTXFile\" db.go | grep -v \"os.Open\"\n```\n\n**Fix**:\n```go\n// Always try local first\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil\n}\n// Only fall back to remote if local doesn't exist\nreturn replica.Client.OpenLTXFile(...)\n```\n\n## Issue 4: CreatedAt Timestamp Loss\n\n**Symptom**: Point-in-time recovery lacks accurate timestamps\n\n**Check**:\n```go\ninfo, err := client.WriteLTXFile(ctx, level, minTXID, maxTXID, r)\nif err != nil {\n t.Fatal(err)\n}\nif info.CreatedAt.IsZero() {\n t.Fatal(\"CreatedAt not set\")\n}\n```\n\n**Fix**:\n```go\n// Ensure storage metadata is copied into the returned FileInfo\nmodTime := resp.LastModified\ninfo.CreatedAt = modTime\n```\n\n## Issue 5: Non-Atomic File Writes\n\n**Symptom**: Partial files, corruption on crash\n\n**Check**:\n```bash\n# Find direct writes without temp files\ngrep -r \"os.Create\\|os.WriteFile\" --include=\"*.go\"\n```\n\n**Fix**:\n```go\n// Write to temp, then rename\ntmpPath := path + \".tmp\"\nif err := os.WriteFile(tmpPath, data, 0644); err != nil {\n return err\n}\nreturn os.Rename(tmpPath, path)\n```\n\n## Issue 6: WAL Checkpoint Blocking\n\n**Symptom**: WAL grows indefinitely, database locks\n\n**Check**:\n```sql\n-- Check WAL size\nPRAGMA wal_checkpoint(PASSIVE);\nSELECT page_count * page_size FROM pragma_page_count(), pragma_page_size();\n```\n\n**Fix**:\n```go\n// Release read transaction periodically\ndb.rtx.Rollback()\ndb.rtx = nil\n// Checkpoint\ndb.db.Exec(\"PRAGMA wal_checkpoint(RESTART)\")\n// Restart read transaction\ndb.initReadTx()\n```\n\n## Issue 7: Memory Leaks\n\n**Symptom**: Growing memory usage over time\n\n**Check**:\n```bash\n# Generate heap profile\ngo test -memprofile=mem.prof -run=XXX -bench=.\ngo tool pprof -top mem.prof\n```\n\n**Fix**:\n```go\n// Use sync.Pool for buffers\nvar pagePool = sync.Pool{\n New: func() interface{} {\n b := make([]byte, pageSize)\n return &b\n },\n}\n\n// Close resources properly\ndefer func() {\n if f != nil {\n f.Close()\n }\n}()\n```\n\n## Issue 8: Corrupted or Missing LTX Files\n\n**Symptom**: Sync failures with errors like \"ltx validation failed\", \"nonsequential page numbers\",\n\"non-contiguous transaction files\", or persistent sync retry backoff loops after unclean shutdowns.\n\n**Check**:\n```bash\n# Look for LTXError messages in logs\nrg -i \"ltx.*error|reset.*local|auto.recover\" /var/log/litestream.log\n\n# Check if meta directory has corrupted state (note dot prefix)\nls -la /path/to/.database.db-litestream/ltx/\n```\n\n**Fix - Manual Reset**:\n```bash\n# Clears local LTX state, forces fresh snapshot on next sync\n# Database file is NOT modified\nlitestream reset /path/to/database.db\n\n# With explicit config file\nlitestream reset -config /etc/litestream.yml /path/to/database.db\n```\n\n**Fix - Automatic Recovery**:\n```yaml\n# Add to replica config in litestream.yml\ndbs:\n - path: /path/to/database.db\n replicas:\n - url: s3://bucket/path\n auto-recover: true # Automatically resets on LTX errors\n```\n\n**When to use which**: Use `auto-recover` for unattended deployments where automatic recovery\nis preferred over manual intervention. Use manual `reset` when you want to investigate the\ncorruption first. `auto-recover` is disabled by default because resetting discards local LTX\nhistory, which may reduce point-in-time restore granularity.\n\n**Reference**: `cmd/litestream/reset.go`, `replica.go` (auto-recover logic), `db.go` (`ResetLocalState`)\n\n## Issue 9: IPC Socket Connection Failures\n\n**Symptom**: `connection refused` or `no such file or directory` when using the control socket\n\n**Check**:\n```bash\n# Verify socket exists and has correct permissions\nls -la /var/run/litestream.sock\n\n# Verify socket is enabled in config\nrg -A3 'socket:' /etc/litestream.yml\n\n# Check if litestream process is running\npgrep -a litestream\n```\n\n**Fix**:\n```yaml\n# Enable socket in litestream.yml\nsocket:\n enabled: true\n path: /var/run/litestream.sock\n permissions: 0600\n```\n\n**Stale socket**: If litestream crashed, the socket file may still exist. The process creates a new socket on startup and will fail if the stale file exists. Remove it manually:\n```bash\nrm /var/run/litestream.sock\n```\n\n**Reference**: `server.go` (`SocketConfig`, `Server.Start`)\n\n## Issue 10: Backup Files Accumulating (Retention Disabled)\n\n**Symptom**: Storage usage growing unbounded, old LTX files never deleted\n\n**Check**:\n```bash\n# Look for retention warning in logs\nrg \"retention disabled\" /var/log/litestream.log\n```\n\n**Fix**:\n```yaml\n# Option 1: Re-enable Litestream retention (default)\nretention:\n enabled: true\n\n# Option 2: Keep disabled, but configure cloud lifecycle policies\n# Example: S3 lifecycle rule to expire objects in ltx/ prefix after 30 days\n```\n\n**Reference**: `store.go` (`SetRetentionEnabled`), `compactor.go` (`RetentionEnabled`), `cmd/litestream/replicate.go:295`\n\n## Issue 11: v0.3.x Restore Not Finding Backups\n\n**Symptom**: Restore fails with \"no snapshots available\" but v0.3.x backups exist in storage\n\n**Check**:\n```bash\n# Verify v0.3.x backup structure exists\naws s3 ls s3://bucket/path/generations/ --recursive | head -20\n```\n\n**Fix**: Ensure the backend implements `ReplicaClientV3`. The S3 backend supports this automatically. The restore process checks for v0.3.x backups when no v0.4.x+ backup is found.\n\n**Reference**: `v3.go` (`ReplicaClientV3` interface), `s3/replica_client.go`\n\n## Diagnostic Commands\n\n```bash\n# Check database integrity\nsqlite3 database.db \"PRAGMA integrity_check;\"\n\n# List replicated LTX files\nlitestream ltx /path/to/db.sqlite\n\n# Check replication status\nlitestream databases\n\n# Reset corrupted local state\nlitestream reset /path/to/database.db\n\n# Test restoration\nlitestream restore -o test.db [replica-url]\n\n# IPC socket diagnostics (requires socket.enabled: true)\ncurl --unix-socket /var/run/litestream.sock http://localhost/info\ncurl --unix-socket /var/run/litestream.sock http://localhost/list\ncurl --unix-socket /var/run/litestream.sock \"http://localhost/txid?path=/path/to/db\"\n```\n\n## Prevention Checklist\n\n- [ ] Always test with databases >1GB\n- [ ] Run with race detector in CI\n- [ ] Test all page sizes (4KB, 8KB, 16KB, 32KB)\n- [ ] Verify eventual consistency handling\n- [ ] Check for proper locking (Lock vs RLock)\n- [ ] Ensure atomic file operations\n- [ ] Preserve timestamps in compaction\n",".claude/commands/run-comprehensive-tests.md":"---\ndescription: Run comprehensive test suite for Litestream\n---\n\n# Run Comprehensive Tests Command\n\nExecute a full test suite including unit tests, integration tests, race detection, and large database tests.\n\n## Quick Test Suite\n\n```bash\n# Basic tests with race detection\ngo test -race -v ./...\n\n# With coverage\ngo test -race -cover -v ./...\n```\n\n## Full Test Suite\n\n### 1. Unit Tests\n```bash\necho \"=== Running Unit Tests ===\"\ngo test -v ./... -short\n```\n\n### 2. Race Condition Tests\n```bash\necho \"=== Testing for Race Conditions ===\"\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\ngo test -race -v ./...\n```\n\n### 3. Integration Tests\n```bash\necho \"=== Running Integration Tests ===\"\n\n# S3 (requires AWS credentials)\nAWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy \\\n go test -v ./replica_client_test.go -integration s3\n\n# Google Cloud Storage (requires credentials)\nGOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json \\\n go test -v ./replica_client_test.go -integration gcs\n\n# Azure Blob Storage\nAZURE_STORAGE_ACCOUNT=xxx AZURE_STORAGE_KEY=yyy \\\n go test -v ./replica_client_test.go -integration abs\n\n# SFTP (requires SSH server)\ngo test -v ./replica_client_test.go -integration sftp\n\n# File system (always available)\ngo test -v ./replica_client_test.go -integration file\n```\n\n### 4. Large Database Tests (>1GB)\n```bash\necho \"=== Testing Large Databases ===\"\n\n# Create test database for each page size\nfor pagesize in 4096 8192 16384 32768; do\n echo \"Testing page size: $pagesize\"\n\n # Create >1GB database\n sqlite3 test-${pagesize}.db <70% for core packages\n✅ Lock page correctly skipped for all page sizes\n✅ Restoration works for databases >1GB\n✅ No memory leaks in benchmarks\n",".claude/commands/test-compaction.md":"Test Litestream compaction logic. This command helps test and debug compaction issues, especially with eventually consistent storage backends.\n\nFirst, understand the test scenario:\n- What storage backend needs testing?\n- What size database is involved?\n- Are there eventual consistency concerns?\n\nThen create comprehensive tests:\n\n1. **Test basic compaction**:\n```go\nfunc TestCompaction_Basic(t *testing.T) {\n // Create multiple LTX files at level 0\n // Run compaction to level 1\n // Verify merged file is correct\n}\n```\n\n2. **Test with eventual consistency**:\n```go\nfunc TestStore_CompactDB_RemotePartialRead(t *testing.T) {\n // Use mock client that returns partial data initially\n // Verify compaction prefers local files\n // Ensure no corruption occurs\n}\n```\n\n3. **Test lock page handling during compaction**:\n```go\nfunc TestCompaction_LockPage(t *testing.T) {\n // Create database > 1GB\n // Compact with data around lock page\n // Verify lock page is skipped (page at 0x40000000)\n}\n```\n\n4. **Test timestamp preservation**:\n```go\nfunc TestCompaction_PreserveTimestamps(t *testing.T) {\n // Compact files with different CreatedAt times\n // Verify earliest timestamp is preserved\n}\n```\n\nKey areas to test:\n- Reading from local files first (db.go:1280-1294)\n- Skipping lock page at 1GB boundary\n- Preserving CreatedAt timestamps\n- Handling partial/incomplete remote files\n- Concurrent compaction safety\n\nRun with race detector:\n```bash\ngo test -race -v -run TestStore_CompactDB ./...\n```\n\nUse the test harness for large databases:\n```bash\n./bin/litestream-test populate -db test.db -target-size 1.5GB\n./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica\n```\n",".claude/commands/trace-replication.md":"Trace the complete replication flow in Litestream. This command helps understand how changes flow from SQLite through to storage backends.\n\nFollow the replication path step by step:\n\n1. **Application writes to SQLite**:\n```sql\n-- Application performs write\nINSERT INTO table VALUES (...);\n-- SQLite appends to WAL file\n```\n\n2. **DB.monitor() syncs the shadow WAL** (db.go:1499):\n```go\nticker := time.NewTicker(db.MonitorInterval) // default 1s\nfor {\n select {\n case <-db.ctx.Done():\n return\n case <-ticker.C:\n }\n\n if err := db.Sync(db.ctx); err != nil && !errors.Is(err, context.Canceled) {\n db.Logger.Error(\"sync error\", \"error\", err)\n }\n}\n```\n\n3. **Replica.monitor() responds** (replica.go):\n```go\nticker := time.NewTicker(r.SyncInterval)\ndefer ticker.Stop()\n\nnotify := r.db.Notify()\nfor {\n select {\n case <-ctx.Done():\n return\n case <-ticker.C:\n // Enforce minimum sync interval\n case <-notify:\n // WAL changed, time to sync\n }\n\n notify = r.db.Notify()\n\n if err := r.Sync(ctx); err != nil && !errors.Is(err, context.Canceled) {\n r.Logger().Error(\"monitor error\", \"error\", err)\n }\n}\n```\n\n4. **Replica.Sync() uploads new L0 files** (replica.go):\n```go\n// Determine local database position\ndpos, err := r.db.Pos()\nif err != nil {\n return err\n}\nif dpos.IsZero() {\n return fmt.Errorf(\"no position, waiting for data\")\n}\n\n// Upload each unreplicated L0 file\nfor txID := r.Pos().TXID + 1; txID <= dpos.TXID; txID = r.Pos().TXID + 1 {\n if err := r.uploadLTXFile(ctx, 0, txID, txID); err != nil {\n return err\n }\n r.SetPos(ltx.Pos{TXID: txID})\n}\n```\n\n5. **ReplicaClient uploads to storage**:\n```go\nfunc (c *S3Client) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {\n // Stream LTX data to storage and return metadata (size, CreatedAt, checksums)\n}\n```\n\n6. **Checkpoint when thresholds are hit**:\n```go\nif walPageCount > db.MinCheckpointPageN {\n db.Checkpoint(ctx, litestream.CheckpointModePassive)\n}\nif walPageCount > db.TruncatePageN {\n db.Checkpoint(ctx, litestream.CheckpointModeTruncate)\n}\n```\n\nKey synchronization points:\n- WAL monitoring (1s intervals)\n- Replica sync (configurable, default 1s)\n- Checkpoint triggers (page thresholds)\n- Compaction (hourly/daily)\n\nTrace with logging:\n```go\n// Enable debug logging\nslog.SetLogLevel(slog.LevelDebug)\n\n// Key log points:\nslog.Debug(\"wal changed\", \"size\", walSize)\nslog.Debug(\"syncing replica\", \"pos\", r.Pos())\nslog.Debug(\"ltx uploaded\", \"txid\", maxTXID)\nslog.Debug(\"checkpoint complete\", \"mode\", mode)\n```\n\nPerformance metrics to monitor:\n- WAL growth rate\n- Sync latency\n- Upload throughput\n- Checkpoint frequency\n- Compaction duration\n\nCommon bottlenecks:\n1. Slow storage uploads\n2. Large transactions causing big LTX files\n3. Long-running read transactions blocking checkpoints\n4. Eventual consistency delays\n5. Network latency to storage\n\nTest replication flow:\n```bash\n# Start replication with verbose logging\nlitestream replicate -v\n\n# In another terminal, write to database\nsqlite3 test.db \"INSERT INTO test VALUES (1, 'data');\"\n\n# Watch logs for flow:\n# - WAL change detected\n# - Replica sync triggered\n# - LTX file uploaded\n# - Position updated\n```\n\nVerify replication:\n```bash\n# List replicated files\naws s3 ls s3://bucket/path/ltx/0000/\n\n# Restore and verify\nlitestream restore -o restored.db s3://bucket/path\nsqlite3 restored.db \"SELECT * FROM test;\"\n```\n",".claude/commands/validate-replica.md":"Validate a ReplicaClient implementation in Litestream. This command helps ensure a replica client correctly implements the interface and handles edge cases.\n\nFirst, identify what needs validation:\n- Which replica client implementation?\n- What storage backend specifics?\n- Any known issues or concerns?\n\nThen validate the implementation:\n\n1. **Interface compliance check**:\n\n ```go\n // Ensure all methods are implemented\n var _ litestream.ReplicaClient = (*YourClient)(nil)\n ```\n\n1. **Verify error types**:\n\n ```go\n // OpenLTXFile must return os.ErrNotExist for missing files\n _, err := client.OpenLTXFile(ctx, 0, 999, 999, 0, 0)\n if !errors.Is(err, os.ErrNotExist) {\n t.Errorf(\"Expected os.ErrNotExist, got %v\", err)\n }\n ```\n\n1. **Test partial reads**:\n\n ```go\n // Must support offset and size parameters\n rc, err := client.OpenLTXFile(ctx, 0, 1, 100, 50, 25)\n data, _ := io.ReadAll(rc)\n if len(data) != 25 {\n t.Errorf(\"Expected 25 bytes, got %d\", len(data))\n }\n ```\n\n1. **Verify timestamp preservation**:\n\n ```go\n // CreatedAt should reflect remote object metadata (or upload time)\n start := time.Now()\n info, _ := client.WriteLTXFile(ctx, 0, 1, 100, reader)\n if info.CreatedAt.IsZero() || info.CreatedAt.Before(start.Add(-time.Second)) {\n t.Error(\"unexpected CreatedAt timestamp\")\n }\n ```\n\n1. **Test eventual consistency handling**:\n - Implement retry logic for transient failures\n - Handle partial file availability\n - Verify write-after-write consistency\n\n1. **Validate cleanup**:\n\n ```go\n // DeleteAll must remove everything\n err := client.DeleteAll(ctx)\n files, _ := client.LTXFiles(ctx, 0, 0, false)\n if files.Next() {\n t.Error(\"Files remain after DeleteAll\")\n }\n ```\n\nKey validation points:\n- Proper error types (os.ErrNotExist, os.ErrPermission)\n- Context cancellation handling\n- Concurrent operation safety\n- Iterator doesn't load all files at once\n- Proper path construction for storage backend\n\nRun integration tests:\n```bash\ngo test -v ./[backend]/replica_client_test.go -integration\n```\n",".github/copilot-instructions.md":"../AGENTS.md","AGENTS.md":"# AGENTS.md - Litestream AI Agent Guide\n\nLitestream is a disaster recovery tool for SQLite that runs as a background process, monitors the WAL, converts changes to immutable LTX files, and replicates them to cloud storage. It uses `modernc.org/sqlite` (pure Go, no CGO required).\n\n## Before You Start\n\n1. Read [AI_PR_GUIDE.md](AI_PR_GUIDE.md) for contribution requirements\n2. Check [CONTRIBUTING.md](CONTRIBUTING.md) for what we accept (bug fixes welcome, features need discussion)\n3. Review recent PRs for current patterns\n\n## Critical Rules\n\n- **Lock page at 1GB**: SQLite reserves page at 0x40000000. Always skip it. See [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md)\n- **LTX files are immutable**: Never modify after creation. See [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md)\n- **Single replica per database**: Each DB replicates to exactly one destination\n- **Use `litestream ltx`**: Not `litestream wal` (deprecated)\n- **Use `litestream reset`**: Clears corrupted local LTX state for a database. See `cmd/litestream/reset.go`\n- **`auto-recover` config**: Replica option that automatically resets local state on LTX errors. Disabled by default. See `replica.go`\n- **Retention enabled by default**: `Store.RetentionEnabled` is `true` by default. Disable only when cloud lifecycle policies handle cleanup. See `store.go`\n- **IPC socket disabled by default**: Control socket is off by default. Enable with `socket.enabled: true` in config. See `server.go`\n- **`$PID` config expansion**: Config files support `$PID` to expand to the current process ID, plus standard `$ENV_VAR` expansion. See `cmd/litestream/main.go`\n- **`litestream ltx -level`**: Use `-level 0`–`9` or `-level all` to inspect specific compaction levels. See `cmd/litestream/ltx.go`\n- **Return errors, don't log them**: Always return errors to callers. Never `log.Printf(err)` and continue — this silently hides failures in a disaster recovery tool. Only use DEBUG log for best-effort operations where failure doesn't affect correctness and a valid fallback exists (e.g., reading SHM mxFrame optimization hint). See [docs/PATTERNS.md](docs/PATTERNS.md#error-handling)\n\n## Layer Boundaries\n\n| Layer | File | Responsibility |\n|-------|------|----------------|\n| DB | `db.go` | Database state, restoration, WAL monitoring, library API (`SyncStatus`, `SyncAndWait`, `EnsureExists`) |\n| Replica | `replica.go` | Replication mechanics only |\n| Storage | `**/replica_client.go` | Backend implementations (includes `ReplicaClientV3` for v0.3.x restore) |\n| IPC | `server.go` | Unix socket control API (register/unregister, /txid, pprof) |\n| Leasing | `leaser.go`, `s3/leaser.go` | Distributed lease acquisition via conditional writes |\n\nDatabase state logic belongs in DB layer, not Replica layer.\n\n## Quick Reference\n\n**Build:**\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\n```\n\n**Code quality:**\n\n```bash\npre-commit run --all-files\n```\n\n## Documentation\n\n| Document | When to Read |\n|----------|--------------|\n| [docs/PATTERNS.md](docs/PATTERNS.md) | Code patterns and anti-patterns |\n| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Deep component details |\n| [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) | WAL format, lock page |\n| [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md) | Replication format |\n| [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md) | Test strategies |\n| [docs/REPLICA_CLIENT_GUIDE.md](docs/REPLICA_CLIENT_GUIDE.md) | Adding storage backends |\n| [docs/PROVIDER_COMPATIBILITY.md](docs/PROVIDER_COMPATIBILITY.md) | Provider-specific S3/cloud configs |\n\n## Checklist\n\nBefore submitting changes:\n\n- [ ] Read relevant docs above\n- [ ] Follow patterns in [docs/PATTERNS.md](docs/PATTERNS.md)\n- [ ] Test with race detector (`go test -race`)\n- [ ] Run `pre-commit run --all-files`\n- [ ] For page iteration: test with >1GB databases\n- [ ] Show investigation evidence in PR (see [AI_PR_GUIDE.md](AI_PR_GUIDE.md))\n","CLAUDE.md":"# CLAUDE.md - Claude Code Configuration\n\nClaude-specific optimizations for Litestream. See [AGENTS.md](AGENTS.md) for project documentation.\n\n## Critical Coding Rules\n\n- **Always return errors, never log and continue.** The only exception is DEBUG logging for best-effort operations that don't affect correctness. This is the #1 cause of PR review feedback. See [docs/PATTERNS.md](docs/PATTERNS.md#error-handling) for the full decision framework.\n\n## Context Window\n\nWith Claude's large context window, load documentation as needed:\n\n- Start with [AGENTS.md](AGENTS.md) for overview and checklist\n- Load [docs/PATTERNS.md](docs/PATTERNS.md) when writing code\n- Load [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) for WAL/page work\n- Load [docs/PROVIDER_COMPATIBILITY.md](docs/PROVIDER_COMPATIBILITY.md) for storage backend configs\n\n## Claude-Specific Resources\n\n### Specialized Agents (.claude/agents/)\n\n- `sqlite-expert.md` - SQLite WAL and page management\n- `replica-client-developer.md` - Storage backend implementation\n- `ltx-compaction-specialist.md` - LTX format and compaction\n- `test-engineer.md` - Testing strategies\n- `performance-optimizer.md` - Performance optimization\n\n### Commands (.claude/commands/)\n\n- `/analyze-ltx` - Analyze LTX file structure\n- `/debug-ipc` - Debug IPC Unix socket issues\n- `/debug-wal` - Debug WAL replication issues\n- `/test-compaction` - Test compaction scenarios\n- `/trace-replication` - Trace replication flow\n- `/validate-replica` - Validate replica client\n- `/add-storage-backend` - Create new storage backend\n- `/fix-common-issues` - Diagnose common problems\n- `/run-comprehensive-tests` - Execute full test suite\n\n## Quick Commands\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\npre-commit run --all-files\n```\n","docs/ARCHITECTURE.md":"# Litestream Architecture - Technical Deep Dive\n\n## Table of Contents\n- [System Layers](#system-layers)\n- [Core Components](#core-components)\n- [IPC Server & Control Socket](#ipc-server--control-socket)\n- [Distributed Leasing](#distributed-leasing)\n- [Library Convenience Methods](#library-convenience-methods)\n- [LTX File Format](#ltx-file-format)\n- [WAL Monitoring Mechanism](#wal-monitoring-mechanism)\n- [Compaction Process](#compaction-process)\n- [Transaction Management](#transaction-management)\n- [Concurrency Model](#concurrency-model)\n- [State Management](#state-management)\n- [Initialization Flow](#initialization-flow)\n- [Error Handling](#error-handling)\n\n## System Layers\n\nLitestream follows a layered architecture with clear separation of concerns:\n\n```mermaid\ngraph TB\n subgraph \"Application Layer\"\n CLI[CLI Commands
cmd/litestream/]\n Config[Configuration
config.go]\n end\n\n subgraph \"Core Layer\"\n Store[Store Manager
store.go]\n DB[Database Manager
db.go]\n Replica[Replica Manager
replica.go]\n end\n\n subgraph \"Storage Abstraction\"\n RC[ReplicaClient Interface
replica_client.go]\n end\n\n subgraph \"Storage Implementations\"\n S3[s3/replica_client.go]\n GCS[gs/replica_client.go]\n ABS[abs/replica_client.go]\n OSS[oss/replica_client.go]\n File[file/replica_client.go]\n SFTP[sftp/replica_client.go]\n NATS[nats/replica_client.go]\n end\n\n subgraph \"External\"\n SQLite[SQLite Database]\n Cloud[Cloud Storage]\n end\n\n CLI --> Store\n Store --> DB\n DB --> Replica\n Replica --> RC\n RC --> S3\n RC --> GCS\n RC --> ABS\n RC --> OSS\n RC --> File\n RC --> SFTP\n RC --> NATS\n DB <--> SQLite\n S3 --> Cloud\n GCS --> Cloud\n ABS --> Cloud\n OSS --> Cloud\n```\n\n### Layer Responsibilities\n\n#### 1. Application Layer\n- **CLI Commands**: User interface for operations (replicate, restore, etc.)\n- **Configuration**: YAML/environment variable parsing and validation\n\n#### 2. Core Layer\n- **Store**: Multi-database coordination, compaction scheduling\n- **DB**: Single database management, WAL monitoring, checkpointing\n- **Replica**: Replication to single destination, position tracking\n\n#### 3. Storage Abstraction\n- **ReplicaClient Interface**: Uniform API for all storage backends\n\n#### 4. Storage Implementations\n- Backend-specific logic (authentication, retries, optimizations)\n\n#### 5. IPC Layer\n- **Server**: Unix socket HTTP server for runtime control (`server.go`)\n- **Leaser**: Distributed lease interface for coordination (`leaser.go`, `s3/leaser.go`)\n\n## Core Components\n\n### DB Component (db.go)\n\nThe DB component is the heart of Litestream, managing a single SQLite database:\n\n```go\ntype DB struct {\n // Core fields\n path string // Database file path\n metaPath string // Metadata directory path\n db *sql.DB // SQLite connection\n f *os.File // Long-running file descriptor\n rtx *sql.Tx // Long-running read transaction\n pageSize int // Database page size\n\n // Synchronization\n mu sync.RWMutex // Protects struct fields\n chkMu sync.RWMutex // Checkpoint lock\n notify chan struct{} // WAL change notifications\n\n // Lifecycle\n ctx context.Context\n cancel func()\n wg sync.WaitGroup\n\n // Configuration\n MinCheckpointPageN int // Min pages for passive checkpoint\n TruncatePageN int // Pages before emergency truncate checkpoint\n CheckpointInterval time.Duration // Time-based passive checkpoint interval\n MonitorInterval time.Duration // WAL monitoring frequency\n // Note: MaxCheckpointPageN removed (RESTART mode disabled due to #724)\n\n // Metrics\n dbSizeGauge prometheus.Gauge\n walSizeGauge prometheus.Gauge\n txIDGauge prometheus.Gauge\n}\n```\n\n#### Key Methods\n\n```go\n// Lifecycle\nfunc (db *DB) Open() error\nfunc (db *DB) Close(ctx context.Context) error\n\n// Monitoring\nfunc (db *DB) monitor() // Background WAL monitoring\nfunc (db *DB) checkWAL() (bool, error) // Check for WAL changes\n\n// Checkpointing\nfunc (db *DB) Checkpoint(mode string) error\nfunc (db *DB) autoCheckpoint() error\n\n// Replication\nfunc (db *DB) WALReader(pgno uint32) (io.ReadCloser, error)\nfunc (db *DB) Sync(ctx context.Context) error\n\n// Compaction\nfunc (db *DB) Compact(ctx context.Context, destLevel int) (*ltx.FileInfo, error)\n```\n\n### Replica Component (replica.go)\n\nManages replication to a single destination:\n\n```go\ntype Replica struct {\n db *DB // Parent database\n Client ReplicaClient // Storage backend client\n\n mu sync.RWMutex\n pos ltx.Pos // Current replication position\n\n // Configuration\n SyncInterval time.Duration\n MonitorEnabled bool\n\n // Lifecycle\n cancel func()\n wg sync.WaitGroup\n}\n```\n\n#### Replication Position\n\n```go\ntype Pos struct {\n TXID TXID // Transaction ID\n PageNo uint32 // Page number within transaction\n Checksum uint64 // Running checksum\n}\n```\n\n### Store Component (store.go)\n\nCoordinates multiple databases and manages system-wide resources:\n\n```go\ntype Store struct {\n mu sync.Mutex\n dbs []*DB\n levels CompactionLevels\n\n // Configuration\n SnapshotInterval time.Duration\n SnapshotRetention time.Duration\n L0Retention time.Duration\n L0RetentionCheckInterval time.Duration\n CompactionMonitorEnabled bool\n\n // Lifecycle\n ctx context.Context\n cancel func()\n wg sync.WaitGroup\n}\n```\n\n## IPC Server & Control Socket\n\nLitestream exposes a Unix socket HTTP server for runtime control (`server.go`).\n\n### Socket Configuration\n\n```go\ntype SocketConfig struct {\n Enabled bool `yaml:\"enabled\"` // Default: false\n Path string `yaml:\"path\"` // Default: \"/var/run/litestream.sock\"\n Permissions uint32 `yaml:\"permissions\"` // Default: 0600\n}\n```\n\n### HTTP Endpoints\n\nAll endpoints are served over the Unix socket via Go's `net/http` mux:\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `POST` | `/register` | Add a database at runtime |\n| `POST` | `/unregister` | Remove a database at runtime |\n| `GET` | `/txid?path=` | Get current transaction ID for a database |\n| `GET` | `/list` | List all managed databases |\n| `GET` | `/info` | Server info (version, PID, uptime, database count) |\n| `POST` | `/start` | Start replication for a database |\n| `POST` | `/stop` | Stop replication for a database |\n| `GET` | `/debug/pprof/*` | Standard Go pprof endpoints |\n\n### Request/Response Types\n\n```go\ntype RegisterDatabaseRequest struct {\n Path string `json:\"path\"`\n ReplicaURL string `json:\"replica_url\"`\n}\n\ntype UnregisterDatabaseRequest struct {\n Path string `json:\"path\"`\n Timeout int `json:\"timeout,omitempty\"` // Seconds\n}\n\ntype TXIDResponse struct {\n TXID uint64 `json:\"txid\"`\n}\n```\n\n## Distributed Leasing\n\nThe `Leaser` interface (`leaser.go`) enables distributed coordination so multiple Litestream instances can safely share a replica destination.\n\n### Interface\n\n```go\ntype Leaser interface {\n Type() string\n AcquireLease(ctx context.Context) (*Lease, error)\n RenewLease(ctx context.Context, lease *Lease) (*Lease, error)\n ReleaseLease(ctx context.Context, lease *Lease) error\n}\n\ntype Lease struct {\n Generation int64 `json:\"generation\"`\n ExpiresAt time.Time `json:\"expires_at\"`\n Owner string `json:\"owner,omitempty\"`\n ETag string `json:\"-\"`\n}\n```\n\n### S3 Implementation (`s3/leaser.go`)\n\n- **Defaults**: `DefaultLeaseTTL = 30s`, `DefaultLeasePath = \"lock.json\"`\n- **Owner format**: `hostname:pid` (falls back to `pid-N` if hostname unavailable)\n- **Conditional writes**: Uses `If-Match`/`If-None-Match` on S3 PutObject to prevent races\n- **Release**: Uses `DeleteObject` with `If-Match` to ensure only the holder can release\n- **Error types**: `ErrLeaseNotHeld`, `ErrLeaseAlreadyReleased`, `LeaseExistsError{Owner, ExpiresAt}`\n\n### Acquisition Flow\n\n1. Read existing lease from S3 (`lock.json`)\n2. If lease exists and is not expired, return `LeaseExistsError`\n3. Write new lease with `If-None-Match: *` (first acquire) or `If-Match: ` (expired takeover)\n4. If `PreconditionFailed` (412), another instance acquired first\n5. Return lease with ETag for subsequent renewal\n\n## Library Convenience Methods\n\nThese methods on `DB` (`db.go`) simplify common operations when Litestream is used as a library:\n\n```go\ntype SyncStatus struct {\n LocalTXID ltx.TXID // Local transaction ID\n RemoteTXID ltx.TXID // Remote transaction ID\n InSync bool // true if LocalTXID > 0 and equal to RemoteTXID\n}\n\nfunc (db *DB) SyncStatus(ctx context.Context) (SyncStatus, error)\nfunc (db *DB) SyncAndWait(ctx context.Context) error\nfunc (db *DB) EnsureExists(ctx context.Context) error\n```\n\n- **SyncStatus**: Compares local TXID against remote replica position (performs I/O to query remote)\n- **SyncAndWait**: Runs `db.Sync()` (WAL to LTX) then `db.Replica.Sync()` (LTX to remote), blocking until both complete\n- **EnsureExists**: Restores database from replica if local file doesn't exist; no-op if file exists or no backup available. Must be called before `Open()`\n\n## LTX File Format\n\nLTX (Log Transaction) files are immutable files containing database changes:\n\n```\n+------------------+\n| Header | Fixed size header with metadata\n+------------------+\n| |\n| Page Frames | Variable number of page frames\n| |\n+------------------+\n| Page Index | Index for efficient page lookup\n+------------------+\n| Trailer | Metadata and checksums\n+------------------+\n```\n\n### Header Structure\n\n```go\ntype Header struct {\n Magic [4]byte // \"LTX\\x00\"\n Version uint32 // Format version\n PageSize uint32 // Database page size\n MinTXID TXID // Starting transaction ID\n MaxTXID TXID // Ending transaction ID\n Timestamp int64 // Creation timestamp\n Checksum uint64 // Header checksum\n}\n```\n\n### Page Frame Structure\n\n```go\ntype PageFrame struct {\n Header PageHeader\n Data []byte // Page data (pageSize bytes)\n}\n\ntype PageHeader struct {\n PageNo uint32 // Page number in database\n Size uint32 // Size of page data\n Checksum uint64 // Page checksum\n}\n```\n\n### Page Index\n\nBinary search tree for efficient page lookup:\n```go\ntype PageIndexElem struct {\n PageNo uint32 // Page number\n Offset int64 // Offset in file\n Size uint32 // Size of page frame\n}\n```\n\n### Trailer\n\n```go\ntype Trailer struct {\n PageIndexOffset int64 // Offset to page index\n PageIndexSize int64 // Size of page index\n PageCount uint32 // Total pages in file\n Checksum uint64 // Full file checksum\n}\n```\n\n## WAL Monitoring Mechanism\n\n### Monitor Loop (db.go:1499)\n\n```go\nfunc (db *DB) monitor() {\n ticker := time.NewTicker(db.MonitorInterval)\n defer ticker.Stop()\n\n for {\n select {\n case <-ticker.C:\n // Check WAL for changes\n changed, err := db.checkWAL()\n if err != nil {\n slog.Error(\"wal check failed\", \"error\", err)\n continue\n }\n\n if changed {\n // Notify replicas of changes\n db.notifyReplicas()\n\n // Check if checkpoint needed\n if db.shouldCheckpoint() {\n db.autoCheckpoint()\n }\n }\n\n case <-db.ctx.Done():\n return\n }\n }\n}\n```\n\n### WAL Change Detection\n\n```go\nfunc (db *DB) checkWAL() (bool, error) {\n // Get current WAL state\n walInfo, err := db.walInfo()\n if err != nil {\n return false, err\n }\n\n // Compare with previous state\n db.mu.Lock()\n changed := walInfo.Size != db.prevWALSize ||\n walInfo.Checksum != db.prevWALChecksum\n db.prevWALSize = walInfo.Size\n db.prevWALChecksum = walInfo.Checksum\n db.mu.Unlock()\n\n return changed, nil\n}\n```\n\n## Compaction Process\n\nCompaction merges multiple LTX files to reduce storage overhead:\n\n### Compaction Algorithm (store.go:189)\n\nHigh-level compaction flow:\n\n1. Determine whether the level is due for compaction (`Store.shouldCompact`).\n2. Enumerate level-`L-1` files using `ReplicaClient.LTXFiles`, preferring local\n copies via `os.Open(db.LTXPath(...))` and falling back to\n `ReplicaClient.OpenLTXFile` only when necessary.\n3. Stream the source readers through `ltx.NewCompactor`, which performs\n page-level deduplication and enforces lock-page skipping automatically.\n4. Pipe the compactor output into `ReplicaClient.WriteLTXFile` to create the\n merged LTX file for level `L`.\n5. Adjust the returned `ltx.FileInfo.CreatedAt` to the earliest timestamp from\n the source files so point-in-time recovery remains accurate.\n6. Update the cached max file info for the level and delete old L0 files when\n promoting to level 1.\n\n### Compaction Levels\n\n```go\ntype CompactionLevel struct {\n Level int // Level number (0 = raw, 1+ = compacted)\n Interval time.Duration // How often to compact from previous level\n}\n\n// Default configuration\nvar DefaultCompactionLevels = CompactionLevels{\n {Level: 0, Interval: 0}, // Raw LTX files\n {Level: 1, Interval: 1 * Hour}, // Hourly compaction\n {Level: 2, Interval: 24 * Hour}, // Daily compaction\n}\n```\n\n## Transaction Management\n\n### Long-Running Read Transaction\n\nLitestream maintains a long-running read transaction to ensure consistency:\n\n```go\nfunc (db *DB) initReadTx() error {\n // Start read transaction\n tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{\n ReadOnly: true,\n })\n if err != nil {\n return err\n }\n\n // Execute dummy query to start transaction\n var dummy string\n err = tx.QueryRow(\"SELECT ''\").Scan(&dummy)\n if err != nil {\n tx.Rollback()\n return err\n }\n\n db.rtx = tx\n return nil\n}\n```\n\n**Purpose:**\n- Prevents database from being modified during replication\n- Ensures consistent view of database\n- Allows reading historical pages from WAL\n\n### Checkpoint Coordination\n\n```go\nfunc (db *DB) Checkpoint(mode string) error {\n // Acquire checkpoint lock\n db.chkMu.Lock()\n defer db.chkMu.Unlock()\n\n // Close read transaction temporarily\n if db.rtx != nil {\n db.rtx.Rollback()\n db.rtx = nil\n }\n\n // Perform checkpoint\n _, _, err := db.db.Exec(fmt.Sprintf(\"PRAGMA wal_checkpoint(%s)\", mode))\n if err != nil {\n return err\n }\n\n // Restart read transaction\n return db.initReadTx()\n}\n```\n\n## Concurrency Model\n\n### Mutex Usage Patterns\n\n```go\n// DB struct mutexes\ntype DB struct {\n mu sync.RWMutex // Protects struct fields\n chkMu sync.RWMutex // Checkpoint coordination\n}\n\n// Replica struct mutexes\ntype Replica struct {\n mu sync.RWMutex // Protects position\n muf sync.Mutex // File descriptor lock\n}\n\n// Store struct mutex\ntype Store struct {\n mu sync.Mutex // Protects database list\n}\n```\n\n### Thundering Herd Prevention\n\n`Store.Open()` (`store.go`) limits concurrent database opens at startup:\n\n```go\ng, ctx := errgroup.WithContext(ctx)\ng.SetLimit(50) // Max 50 concurrent DB opens\nfor _, db := range s.dbs {\n db := db\n g.Go(func() error { return db.Open() })\n}\n```\n\nThis prevents OS resource exhaustion (file descriptors, memory) when hundreds of databases are configured.\n\n### Lock Ordering (Prevent Deadlocks)\n\nAlways acquire locks in this order:\n1. Store.mu\n2. DB.mu\n3. DB.chkMu\n4. Replica.mu\n\n### Goroutine Management\n\n```go\n// Start background task\nfunc (db *DB) Start() {\n db.wg.Add(1)\n go func() {\n defer db.wg.Done()\n db.monitor()\n }()\n}\n\n// Stop with timeout\nfunc (db *DB) Close(ctx context.Context) error {\n // Signal shutdown\n db.cancel()\n\n // Wait for goroutines with timeout\n done := make(chan struct{})\n go func() {\n db.wg.Wait()\n close(done)\n }()\n\n select {\n case <-done:\n return nil\n case <-ctx.Done():\n return ctx.Err()\n }\n}\n```\n\n## State Management\n\n### Database States\n\n```mermaid\nstateDiagram-v2\n [*] --> Closed\n Closed --> Opening: Open()\n Opening --> Open: Success\n Opening --> Closed: Error\n Open --> Monitoring: Start()\n Monitoring --> Syncing: Changes Detected\n Syncing --> Monitoring: Sync Complete\n Monitoring --> Checkpointing: Threshold Reached\n Checkpointing --> Monitoring: Checkpoint Complete\n Monitoring --> Closing: Close()\n Closing --> Closed: Cleanup Complete\n```\n\n### Replica States\n\n```mermaid\nstateDiagram-v2\n [*] --> Idle\n Idle --> Starting: Start()\n Starting --> Monitoring: Success\n Starting --> Idle: Error\n Monitoring --> Syncing: Timer/Changes\n Syncing --> Uploading: Have Changes\n Uploading --> Monitoring: Success\n Uploading --> Error: Failed\n Error --> Monitoring: Retry\n Monitoring --> Stopping: Stop()\n Stopping --> Idle: Cleanup\n```\n\n### Position Tracking\n\n```go\ntype Pos struct {\n TXID TXID // Current transaction ID\n PageNo uint32 // Current page number\n Checksum uint64 // Running checksum for validation\n}\n\n// Update position atomically\nfunc (r *Replica) SetPos(pos ltx.Pos) {\n r.mu.Lock() // MUST use Lock, not RLock!\n defer r.mu.Unlock()\n r.pos = pos\n}\n\n// Read position safely\nfunc (r *Replica) Pos() ltx.Pos {\n r.mu.RLock()\n defer r.mu.RUnlock()\n return r.pos\n}\n```\n\n## Initialization Flow\n\n### System Startup Sequence\n\n```mermaid\nsequenceDiagram\n participant Main\n participant Store\n participant DB\n participant Replica\n participant Monitor\n\n Main->>Store: NewStore(config)\n Store->>Store: Validate config\n\n Main->>Store: Open()\n loop For each database\n Store->>DB: NewDB(path)\n Store->>DB: Open()\n DB->>DB: Open SQLite connection\n DB->>DB: Read page size\n DB->>DB: Init metadata\n DB->>DB: Start read transaction\n\n loop For each replica\n DB->>Replica: NewReplica()\n DB->>Replica: Start()\n Replica->>Monitor: Start monitoring\n end\n end\n\n Store->>Store: Start compaction monitors\n Store-->>Main: Ready\n```\n\n### Critical Initialization Steps\n\n1. **Database Opening**\n ```go\n // Must happen in order:\n 1. Open SQLite connection\n 2. Read page size (PRAGMA page_size)\n 3. Create metadata directory\n 4. Start long-running read transaction\n 5. Initialize replicas\n 6. Start monitor goroutine\n ```\n\n2. **Replica Initialization**\n ```go\n // Must happen in order:\n 1. Create replica with client\n 2. Load previous position from metadata\n 3. Validate position against database\n 4. Start sync goroutine (if monitoring enabled)\n ```\n\n## Error Handling\n\n### Error Categories\n\n1. **Recoverable Errors**\n - Network timeouts\n - Temporary storage unavailability\n - Lock contention\n\n2. **Fatal Errors**\n - Database corruption\n - Invalid configuration\n - Disk full\n\n3. **Operational Errors**\n - Checkpoint failures\n - Compaction conflicts\n - Sync delays\n\n### Error Propagation\n\n```go\n// Bottom-up error propagation\nReplicaClient.WriteLTXFile() error\n ↓\nReplica.Sync() error\n ↓\nDB.Sync() error\n ↓\nStore.monitorDB() // Logs error, continues\n```\n\n### Retry Logic\n\n```go\nfunc (r *Replica) syncWithRetry(ctx context.Context) error {\n backoff := time.Second\n maxBackoff := time.Minute\n\n for attempt := 0; ; attempt++ {\n err := r.Sync(ctx)\n if err == nil {\n return nil\n }\n\n // Check if error is retryable\n if !isRetryable(err) {\n return err\n }\n\n // Check context\n if ctx.Err() != nil {\n return ctx.Err()\n }\n\n // Exponential backoff\n time.Sleep(backoff)\n backoff *= 2\n if backoff > maxBackoff {\n backoff = maxBackoff\n }\n }\n}\n```\n\n## Performance Characteristics\n\n### Time Complexity\n\n| Operation | Complexity | Notes |\n|-----------|------------|-------|\n| WAL Monitor | O(1) | Fixed interval check |\n| Page Write | O(1) | Append to LTX file |\n| Compaction | O(n) | n = total pages |\n| Restoration | O(n*log(m)) | n = pages, m = files |\n| File List | O(k) | k = files in level |\n\n### Space Complexity\n\n| Component | Memory Usage | Disk Usage |\n|-----------|-------------|------------|\n| DB | O(1) + metrics | Original DB + WAL |\n| Replica | O(1) | LTX files + metadata |\n| Compaction | O(n) pages | Temporary during merge |\n| Page Index | O(p) | p = pages in file |\n\n### Optimization Points\n\n1. **Page Index Caching**\n - Cache frequently accessed indices\n - Use estimated size for initial fetch\n\n2. **Batch Operations**\n - Group small changes into larger LTX files\n - Batch delete operations\n\n3. **Concurrent Operations**\n - Multiple replicas can sync in parallel\n - Compaction runs independently per level\n\n## Security Considerations\n\n### Access Control\n\n- File permissions: 0600 for database files\n- Directory permissions: 0700 for metadata\n- No built-in authentication (rely on storage backend)\n\n## Monitoring & Metrics\n\n### Prometheus Metrics\n\n```go\n// Database metrics\ndb_size_bytes // Current database size\nwal_size_bytes // Current WAL size\ntotal_wal_bytes // Total bytes written to WAL\ncheckpoint_count // Number of checkpoints\nsync_count // Number of syncs\nsync_error_count // Number of sync errors\n\n// Replica metrics\nreplica_lag_seconds // Replication lag\nreplica_position // Current replication position\n\n// Validation metrics have db and level labels.\nlitestream_validation_checks_total // Checks by result: success, invalid, or error.\nlitestream_last_validation_success // Last check: 1 passed, 0 failed or not yet complete.\nlitestream_last_validation_success_timestamp_seconds // Unix time of last success, or 0 until a check passes.\n```\n\n### Health Checks\n\n```go\nfunc (db *DB) HealthCheck() error {\n // Check database connection\n if err := db.db.Ping(); err != nil {\n return fmt.Errorf(\"database ping failed: %w\", err)\n }\n\n // Check replication lag\n for _, r := range db.replicas {\n lag := time.Since(r.LastSync())\n if lag > MaxAcceptableLag {\n return fmt.Errorf(\"replica %s lag too high: %v\", r.Name(), lag)\n }\n }\n\n return nil\n}\n```\n","llms.txt":"# Litestream\n\nDisaster recovery tool for SQLite. Replicates WAL changes to S3, GCS, Azure, SFTP, or local filesystem.\n\n## Quick Start for AI Contributors\n\n1. Read [AI_PR_GUIDE.md](AI_PR_GUIDE.md) - PR quality requirements\n2. Read [AGENTS.md](AGENTS.md) - Project overview and checklist\n3. Check [CONTRIBUTING.md](CONTRIBUTING.md) - What we accept\n4. Show investigation evidence in PRs\n\n## PR Checklist\n\n- [ ] Evidence of problem (logs, file patterns)\n- [ ] Clear scope (what PR does/doesn't do)\n- [ ] Runnable test commands\n- [ ] Race detector tested (`go test -race`)\n\n## Documentation\n\n| Document | Purpose |\n|----------|---------|\n| [AGENTS.md](AGENTS.md) | Project overview, critical rules |\n| [AI_PR_GUIDE.md](AI_PR_GUIDE.md) | PR templates, common mistakes |\n| [docs/PATTERNS.md](docs/PATTERNS.md) | Code patterns and anti-patterns |\n| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Component details |\n| [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) | WAL format, 1GB lock page |\n| [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md) | Replication format |\n| [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md) | Test strategies |\n| [docs/REPLICA_CLIENT_GUIDE.md](docs/REPLICA_CLIENT_GUIDE.md) | Storage backends |\n\n## Core Files\n\n| File | Purpose |\n|------|---------|\n| `db.go` | Database monitoring, WAL, checkpoints |\n| `replica.go` | Replication management |\n| `store.go` | Multi-database coordination |\n| `replica_client.go` | Storage backend interface |\n\n## Storage Backends\n\n- `s3/replica_client.go` - AWS S3\n- `gs/replica_client.go` - Google Cloud Storage\n- `abs/replica_client.go` - Azure Blob Storage\n- `sftp/replica_client.go` - SFTP\n- `file/replica_client.go` - Local filesystem\n- `nats/replica_client.go` - NATS JetStream\n\n## Critical Concepts\n\n- **Lock page at 1GB** - Always skip page at 0x40000000\n- **LTX files are immutable** - Never modify after creation\n- **Single replica per DB** - One destination per database\n- **Layer boundaries** - DB handles state, Replica handles replication\n\n## Build\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\npre-commit run --all-files\n```\n","packages/npm/litestream-vfs-darwin-amd64/README.md":"# litestream-vfs-darwin-amd64\n\nPlatform package for `litestream-vfs` — macOS x64 (Intel).\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","packages/npm/litestream-vfs-darwin-arm64/README.md":"# litestream-vfs-darwin-arm64\n\nPlatform package for `litestream-vfs` — macOS ARM64 (Apple Silicon).\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","packages/npm/litestream-vfs-linux-amd64/README.md":"# litestream-vfs-linux-amd64\n\nPlatform package for `litestream-vfs` — Linux x64.\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","packages/npm/litestream-vfs-linux-arm64/README.md":"# litestream-vfs-linux-arm64\n\nPlatform package for `litestream-vfs` — Linux ARM64.\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","packages/npm/litestream-vfs/README.md":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed via npm.\n\nThis package bundles the [Litestream](https://litestream.io) VFS shared library.\nThe correct platform-specific binary is automatically installed via `optionalDependencies`.\n\n## Installation\n\n```bash\nnpm install litestream-vfs\n```\n\n## Usage\n\n```javascript\nconst { getLoadablePath } = require(\"litestream-vfs\");\nconst path = getLoadablePath();\n// Use `path` with better-sqlite3 or other SQLite bindings\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","packages/python/README.md":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed as a Python wheel.\n\nThis package bundles the [Litestream](https://litestream.io) VFS shared library\nso you can load it directly into a Python `sqlite3` connection.\n\n## Installation\n\n```bash\npip install litestream-vfs\n```\n\n## Usage\n\n```python\nimport sqlite3\nimport litestream_vfs\n\nconn = sqlite3.connect(\":memory:\")\nlitestream_vfs.load(conn)\n```\n\nTo get the path to the shared library (for use with other SQLite bindings):\n\n```python\npath = litestream_vfs.loadable_path()\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","packages/ruby/README.md":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed as a Ruby gem.\n\nThis gem bundles the [Litestream](https://litestream.io) VFS shared library\nso you can load it directly into a SQLite connection.\n\n## Installation\n\n```bash\ngem install litestream-vfs\n```\n\nOr add to your Gemfile:\n\n```ruby\ngem \"litestream-vfs\"\n```\n\n## Usage\n\n```ruby\nrequire \"litestream_vfs\"\n\ndb = SQLite3::Database.new(\":memory:\")\nLitestreamVfs.load(db)\n```\n\nTo get the path to the shared library:\n\n```ruby\npath = LitestreamVfs.loadable_path\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","skills/litestream/SKILL.md":"---\nname: litestream\ndescription: >-\n Expert knowledge for contributing to Litestream, a standalone disaster recovery\n tool for SQLite. Provides architectural understanding, code patterns, critical\n rules, and debugging procedures for WAL monitoring, LTX replication format,\n storage backend implementation, multi-level compaction, and SQLite page\n management. Use when working with Litestream source code, writing storage\n backends, debugging replication issues, implementing compaction logic, or\n handling SQLite WAL operations.\nlicense: Apache-2.0\nmetadata:\n author: benbjohnson\n version: \"1.0\"\n repository: https://github.com/benbjohnson/litestream\n---\n\n# Litestream Agent Skill\n\nLitestream is a standalone disaster recovery tool for SQLite. It runs as a\nbackground process, monitors the SQLite WAL (Write-Ahead Log), converts changes\nto immutable LTX files, and replicates them to cloud storage. It uses\n`modernc.org/sqlite` (pure Go, no CGO required).\n\n## Quick Start\n\n```bash\n# Build\ngo build -o bin/litestream ./cmd/litestream\n\n# Test (always use race detector)\ngo test -race -v ./...\n\n# Code quality\npre-commit run --all-files\n```\n\n## Critical Rules\n\nThese invariants must never be violated:\n\n### 1. Lock Page at 1GB\n\nSQLite reserves a page at byte offset 0x40000000 (1 GB). Always skip it during\nreplication and compaction. The page number varies by page size:\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n### 2. LTX Files Are Immutable\n\nOnce an LTX file is written, it must never be modified. New changes create new\nfiles. This guarantees point-in-time recovery integrity.\n\n### 3. Single Replica per Database\n\nEach database replicates to exactly one destination. The Replica component\nmanages replication mechanics; database state belongs in the DB layer.\n\n### 4. Read Local Before Remote During Compaction\n\nCloud storage is eventually consistent. Always read from local disk first:\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil // Use local copy\n}\nreturn replica.Client.OpenLTXFile(...) // Fall back to remote\n```\n\n### 5. Preserve Timestamps During Compaction\n\nSet the compacted file's `CreatedAt` to the earliest source file timestamp to\nmaintain temporal granularity for point-in-time restoration.\n\n```go\ninfo.CreatedAt = oldestSourceFile.CreatedAt\n```\n\n### 6. Use Lock() Not RLock() for Writes\n\n```go\n// CORRECT\nr.mu.Lock()\ndefer r.mu.Unlock()\nr.pos = pos\n\n// WRONG - race condition\nr.mu.RLock()\ndefer r.mu.RUnlock()\nr.pos = pos\n```\n\n### 7. Atomic File Operations\n\nAlways write to a temp file then rename. Never write directly to the final path.\n\n```go\ntmpFile, err := os.CreateTemp(dir, \".tmp-*\")\n// ... write data, sync ...\nos.Rename(tmpFile.Name(), finalPath)\n```\n\n## Architecture\n\n### System Layers\n\n| Layer | File(s) | Responsibility |\n|---------|--------------------------|-------------------------------------------|\n| App | `cmd/litestream/` | CLI commands, YAML/env config |\n| Store | `store.go` | Multi-DB coordination, compaction |\n| DB | `db.go` | Single DB management, WAL monitoring |\n| Replica | `replica.go` | Replication to one destination |\n| Storage | `*/replica_client.go` | Backend implementations (S3, GCS, etc.) |\n\nDatabase state logic belongs in the DB layer, not the Replica layer.\n\n### ReplicaClient Interface\n\nAll storage backends implement this interface from `replica_client.go`:\n\n```go\ntype ReplicaClient interface {\n Type() string\n Init(ctx context.Context) error\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\nKey contract details:\n- `OpenLTXFile` must return `os.ErrNotExist` when file is missing\n- `WriteLTXFile` must set `CreatedAt` from backend metadata or upload time\n- `LTXFiles` with `useMetadata=true` fetches accurate timestamps (for PIT restore)\n- `LTXFiles` with `useMetadata=false` uses fast timestamps (normal operations)\n\n### Lock Ordering\n\nAlways acquire locks in this order to prevent deadlocks:\n\n1. `Store.mu`\n2. `DB.mu`\n3. `DB.chkMu`\n4. `Replica.mu`\n\n### Core Components\n\n**DB** (`db.go`): Manages SQLite connection, WAL monitoring, checkpointing, and\nlong-running read transaction for consistency. Key fields: `path`, `db`, `rtx`\n(read transaction), `pageSize`, `notify` channel.\n\n**Replica** (`replica.go`): Tracks replication position (`ltx.Pos` with TXID,\nPageNo, Checksum). One replica per database.\n\n**Store** (`store.go`): Coordinates multiple databases and schedules compaction\nacross levels.\n\n## LTX File Format\n\nLTX (Log Transaction) files are immutable, checksummed archives of database\nchanges. Structure:\n\n```\n+------------------+\n| Header | 100 bytes (magic \"LTX1\", page size, TXID range, timestamp)\n+------------------+\n| Page Frames | 4-byte pgno + pageSize bytes data, per page\n+------------------+\n| Page Index | Binary search index for page lookup\n+------------------+\n| Trailer | 16 bytes (post-apply checksum, file checksum)\n+------------------+\n```\n\n### Naming Convention\n\n```\nFormat: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\nExample: 0000000000000001-0000000000000064.ltx (TXID 1-100)\n```\n\n### Compaction Levels\n\n```\nLevel 0: /ltx/0000/ Raw LTX files (no compaction)\nLevel 1: /ltx/0001/ Compacted periodically\nLevel 2: /ltx/0002/ Compacted less frequently\n```\n\nDefault compaction levels: L0 (raw), L1 (30s), L2 (5min), L3 (1h), plus daily\nsnapshots. Compaction merges files by deduplicating pages (latest version wins)\nand always skips the lock page.\n\n## Code Patterns\n\n### DO\n\n- Return errors immediately; let callers decide handling\n- Use `fmt.Errorf(\"context: %w\", err)` for error wrapping\n- Handle database state in the DB layer, not Replica\n- Use `db.verify()` to trigger snapshots (don't reimplement)\n- Test with race detector: `go test -race`\n- Use lazy iterators for `LTXFiles` (paginate, don't load all at once)\n\n### DON'T\n\n- Write data at the 1 GB lock page boundary\n- Modify LTX files after creation\n- Put database state logic in the Replica layer\n- Use `RLock()` when writing shared state\n- Write directly to final file paths (use temp + rename)\n- Ignore context cancellation in long operations\n- Return generic errors instead of `os.ErrNotExist` for missing files\n\n## Specialized Knowledge Areas\n\nLoad reference files on demand based on the task:\n\n| Task | Reference File |\n|-----------------------------------|-----------------------------------------|\n| Understanding system design | `references/ARCHITECTURE.md` |\n| Writing or reviewing code | `references/PATTERNS.md` |\n| Working with LTX files | `references/LTX_FORMAT.md` |\n| WAL monitoring or page operations | `references/SQLITE_INTERNALS.md` |\n| Implementing storage backends | `references/REPLICA_CLIENT_GUIDE.md` |\n| Writing or debugging tests | `references/TESTING_GUIDE.md` |\n\n## Common Debugging Procedures\n\n### Replication Not Working\n\n1. Verify WAL mode: `PRAGMA journal_mode` must return `wal`\n2. Check monitor interval and that the monitor goroutine is running\n3. Confirm `db.notify` channel is being signaled on WAL changes\n4. Check replica position: `replica.Pos()` should advance with writes\n5. Look for `os.ErrNotExist` from `OpenLTXFile` (file not replicated yet)\n\n### Large Database Issues (>1 GB)\n\n1. Verify lock page is being skipped: check `ltx.LockPgno(pageSize)`\n2. Test with multiple page sizes (4K, 8K, 16K, 32K)\n3. Run with databases both smaller and larger than 1 GB\n4. Ensure page iteration loops include the `continue` guard for lock page\n\n### Compaction Problems\n\n1. Confirm local L0 files exist before compaction reads them\n2. Check that `CreatedAt` timestamps are preserved (earliest source)\n3. Verify compaction level intervals in `Store.levels`\n4. Look for eventual consistency issues if reading from remote storage\n\n### Storage Backend Issues\n\n1. Return `os.ErrNotExist` for missing files (not generic errors)\n2. Support partial reads via `offset`/`size` in `OpenLTXFile`\n3. Handle context cancellation in all methods\n4. Test concurrent operations with `-race` flag\n5. For eventually consistent backends, add retry logic with backoff\n\n### Corrupted or Missing LTX Files\n\n1. Check logs for `LTXError` messages - they include context (Op, Path, Level, TXID) and recovery hints\n2. Common error messages: \"nonsequential page numbers\", \"non-contiguous transaction files\", \"ltx validation failed\"\n3. Manual fix: `litestream reset ` clears local LTX state and forces fresh snapshot on next sync (database file is not modified)\n4. Automatic fix: set `auto-recover: true` on the replica config to auto-reset on LTX errors (disabled by default)\n5. Reference: `cmd/litestream/reset.go`, `replica.go` (auto-recover logic), `db.go` (`ResetLocalState`)\n\n## Contribution Guidelines\n\n### What's Accepted\n\n- Bug fixes and patches (welcome)\n- Documentation improvements\n- Small code improvements and performance optimizations\n- Security vulnerability reports (report privately)\n\n### Discuss First\n\n- Feature requests: open an issue before implementing\n- Large changes: discuss approach in an issue first\n\n### Pre-Submit Checklist\n\n- [ ] Read relevant docs from the reference table above\n- [ ] Follow patterns in `references/PATTERNS.md`\n- [ ] Run `go test -race -v ./...`\n- [ ] Run `pre-commit run --all-files`\n- [ ] For page iteration: test with >1 GB databases\n- [ ] Show investigation evidence in PR (see CONTRIBUTING.md)\n\n## Testing\n\n```bash\n# Full test suite with race detection\ngo test -race -v ./...\n\n# Specific areas\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\n\n# Coverage\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\n```\n\nKey testing areas:\n- Lock page handling with >1 GB databases and multiple page sizes\n- Race conditions in position updates, WAL monitoring, and checkpointing\n- Eventual consistency in storage backend operations\n- Atomic file operations and cleanup on error paths\n\n## Environment Validation\n\nRun `scripts/validate-setup.sh` to verify your development environment is\ncorrectly configured for Litestream development.\n","skills/litestream/references/ARCHITECTURE.md":"# Litestream Architecture Reference\n\nCondensed architectural reference for agents working on Litestream.\n\n## System Layers\n\n```\nApplication Layer cmd/litestream/ CLI commands, YAML/env config\nCore Layer store.go Multi-DB coordination, compaction scheduling\n db.go Single DB management, WAL monitoring, checkpoints\n replica.go Replication to single destination, position tracking\nStorage Abstraction replica_client.go ReplicaClient interface\nStorage Backends s3/, gs/, abs/, Backend-specific implementations\n oss/, file/, sftp/,\n nats/, webdav/\n```\n\nDatabase state logic belongs in the DB layer. The Replica layer handles\nreplication mechanics only.\n\n## Core Components\n\n### DB (db.go)\n\nThe heart of Litestream. Manages a single SQLite database.\n\nKey fields:\n- `path` / `metaPath`: Database and metadata paths\n- `db *sql.DB`: SQLite connection (via modernc.org/sqlite, pure Go)\n- `f *os.File`: Long-running file descriptor\n- `rtx *sql.Tx`: Long-running read transaction (prevents checkpoint past read point)\n- `pageSize int`: Database page size (critical for lock page calculation)\n- `mu sync.RWMutex`: Protects struct fields\n- `chkMu sync.RWMutex`: Checkpoint coordination lock\n- `notify chan struct{}`: WAL change notifications\n\nKey methods:\n- `Open()` / `Close()`: Lifecycle management\n- `monitor()`: Background WAL monitoring loop\n- `checkWAL()`: Detect WAL changes by comparing size/checksum\n- `Checkpoint(mode)`: Run SQLite checkpoint (PASSIVE, FULL, TRUNCATE)\n- `autoCheckpoint()`: Threshold-based checkpoint decisions\n- `Sync()`: Synchronize pending changes to replicas\n- `Compact()`: Merge LTX files at a given level\n\nConfiguration:\n- `MinCheckpointPageN`: Minimum pages for passive checkpoint\n- `TruncatePageN`: Emergency truncate checkpoint threshold\n- `CheckpointInterval`: Time-based passive checkpoint interval\n- `MonitorInterval`: WAL monitoring frequency\n- Note: RESTART checkpoint mode permanently removed (issue #724)\n\n### Replica (replica.go)\n\nManages replication to a single destination.\n\nKey fields:\n- `db *DB`: Parent database\n- `Client ReplicaClient`: Storage backend\n- `pos ltx.Pos`: Current replication position (TXID + PageNo + Checksum)\n- `mu sync.RWMutex`: Protects position\n\nPosition tracking: `SetPos()` MUST use `Lock()` not `RLock()`.\n\n### Store (store.go)\n\nCoordinates multiple databases and manages compaction.\n\nKey fields:\n- `dbs []*DB`: Managed databases\n- `levels CompactionLevels`: Compaction level configuration\n\nDefault compaction levels: L0 (raw), L1 (30s), L2 (5min), L3 (1h),\nplus daily snapshots.\n\n## WAL Monitoring\n\nThe monitor loop in `db.go` runs on a ticker:\n\n1. `checkWAL()` compares current WAL size/checksum to previous state\n2. If changed, notify replicas via `notifyReplicas()`\n3. Check if checkpoint needed via `shouldCheckpoint()`\n4. Run `autoCheckpoint()` if thresholds are met\n\nCheckpoint strategy:\n- WAL pages > `TruncatePageN` → TRUNCATE checkpoint (emergency)\n- WAL pages > `MinCheckpointPageN` → PASSIVE checkpoint\n- `CheckpointInterval` elapsed → PASSIVE checkpoint\n\n## Compaction Algorithm\n\nHigh-level flow (in `store.go`):\n\n1. Determine if level is due for compaction (`shouldCompact`)\n2. Enumerate level L-1 files via `ReplicaClient.LTXFiles`\n3. Prefer local copies (`os.Open(db.LTXPath(...))`) over remote reads\n4. Stream through `ltx.NewCompactor` (page deduplication, lock page skipping)\n5. Write merged file via `ReplicaClient.WriteLTXFile` for level L\n6. Set `CreatedAt` to earliest source file timestamp\n7. Update cached max file info; delete old L0 files when promoting to L1\n\n## State Management\n\n### Database States\n\nClosed → Opening → Open → Monitoring ↔ Syncing/Checkpointing → Closing → Closed\n\n### Replica States\n\nIdle → Starting → Monitoring ↔ Syncing/Uploading → Stopping → Idle\n\nError states retry with exponential backoff (1s initial, 1min max).\n\n## Initialization Flow\n\n1. `Store.Open()` iterates databases\n2. For each DB: open SQLite connection → read page size → create metadata dir →\n start long-running read transaction → initialize replicas → start monitor\n3. For each Replica: create with client → load previous position from metadata →\n validate against database → start sync goroutine\n4. Store starts compaction monitors\n\n## Error Handling\n\n### Error Categories\n\n- **Recoverable**: Network timeouts, temporary storage unavailability, lock contention\n- **Fatal**: Database corruption, invalid configuration, disk full\n- **Operational**: Checkpoint failures, compaction conflicts, sync delays\n\n### Error Propagation\n\n```\nReplicaClient.WriteLTXFile() error\n → Replica.Sync() error\n → DB.Sync() error\n → Store.monitorDB() // logs error, continues\n```\n\n## Lock Ordering\n\nAlways acquire in this order to prevent deadlocks:\n\n1. `Store.mu`\n2. `DB.mu`\n3. `DB.chkMu`\n4. `Replica.mu`\n\n## Goroutine Management\n\n- Use `sync.WaitGroup` for lifecycle tracking\n- Signal shutdown via `context.Cancel()`\n- Wait with timeout in `Close()` methods\n- Every goroutine must have a `defer wg.Done()`\n\n## Performance Characteristics\n\n| Operation | Time | Space |\n|-------------|------------|----------------|\n| WAL Monitor | O(1) | O(1) + metrics |\n| Page Write | O(1) | Original DB + WAL |\n| Compaction | O(n pages) | Temporary during merge |\n| Restoration | O(n·log m) | n=pages, m=files |\n| File List | O(k files) | Per-level |\n","skills/litestream/references/LTX_FORMAT.md":"# LTX Format Reference\n\nCondensed reference for agents working with Litestream's LTX file format.\n\n## Overview\n\nLTX (Log Transaction) is Litestream's custom format for storing database changes.\nLTX files are:\n- **Immutable**: Once written, never modified\n- **Self-contained**: Each file is independent\n- **Indexed**: Contains page index for efficient seeks\n- **Checksummed**: CRC-64 ECMA integrity verification\n\n## File Structure\n\n```\n+---------------------+\n| Header | 100 bytes\n+---------------------+\n| Page Frames | Variable (4-byte pgno + pageSize data per page)\n+---------------------+\n| Page Index | Binary search index\n+---------------------+\n| Trailer | 16 bytes\n+---------------------+\n\nFileSize = HeaderSize + (PageCount * (4 + PageSize)) + PageIndexSize + TrailerSize\n```\n\n## Header (100 bytes)\n\n```\nOffset Size Field\n0 4 Magic (\"LTX1\")\n4 4 Flags (bit 1 = NoChecksum)\n8 4 PageSize\n12 4 Commit (page count after applying)\n16 8 MinTXID\n24 8 MaxTXID\n32 8 Timestamp (ms since Unix epoch)\n40 8 PreApplyChecksum\n48 8 WALOffset (0 for snapshots)\n56 8 WALSize (0 for snapshots)\n64 4 WALSalt1\n68 4 WALSalt2\n72 8 NodeID\n80 20 Reserved (zeros)\n```\n\nThe version is implied by the magic string. `\"LTX1\"` → `ltx.Version == 2`.\n\n## Page Frames\n\nEach frame contains one database page:\n\n```\nOffset Size Field\n0 4 Page Number (1-based)\n4 PageSize Page Data\n```\n\nConstraints:\n- Pages written in sequential order during creation\n- Lock page at 1 GB boundary is never included\n- In compacted files, only the latest version of each page\n\n## Page Index\n\nBinary search index for efficient random access to pages. Use\n`ltx.DecodePageIndex()` to parse into a map of page number to\n`ltx.PageIndexElem`:\n\n```go\ntype PageIndexElem struct {\n Level int\n MinTXID TXID\n MaxTXID TXID\n Offset int64 // Byte offset of encoded payload\n Size int64 // Bytes occupied by encoded payload\n}\n```\n\n## Trailer (16 bytes)\n\n```\nOffset Size Field\n0 8 PostApplyChecksum (database checksum after applying file)\n8 8 FileChecksum (CRC-64 of entire file)\n```\n\nThe trailer is always at the end of the file. Read it by seeking\n`-TrailerSize` from `io.SeekEnd`.\n\n## File Naming Convention\n\n```\nFormat: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\n MinTXID (16 hex) MaxTXID (16 hex)\n\nExample: 0000000000000001-0000000000000064.ltx (TXID 1-100)\n 0000000000000065-00000000000000c8.ltx (TXID 101-200)\n```\n\nParse with `ltx.ParseFilename()`, format with `ltx.FormatFilename()`.\n\n## Compaction Levels\n\nLTX files are organized in levels for efficient storage:\n\n```\nLevel 0: /ltx/0000/ Raw LTX files (no compaction)\nLevel 1: /ltx/0001/ Compacted (default: every 30s)\nLevel 2: /ltx/0002/ Compacted (default: every 5min)\nLevel 3: /ltx/0003/ Compacted (default: every 1h)\nSnapshots: Full database state (daily)\n```\n\n### Compaction Process\n\n1. Enumerate level L-1 files\n2. Build page map (newer pages overwrite older)\n3. Write merged file skipping lock page\n4. Preserve earliest `CreatedAt` from source files\n5. Delete old L0 files when promoting to L1\n\n```go\n// Page deduplication: latest version wins\nfor _, file := range files {\n for _, page := range file.Pages {\n pageMap[page.Number] = page\n }\n}\n```\n\n## Checksums\n\nLTX uses CRC-64 ECMA checksums (`hash/crc64` with `crc64.ECMA` table):\n\n- **PreApplyChecksum**: Database state before applying this file\n- **PostApplyChecksum**: Database state after applying this file\n- **FileChecksum**: Integrity of the entire LTX file\n\n## Reading LTX Files\n\n```go\ndec := ltx.NewDecoder(reader)\nheader, err := dec.Header()\nfor {\n var hdr ltx.PageHeader\n data := make([]byte, header.PageSize)\n if err := dec.DecodePage(&hdr, data); err == io.EOF {\n break\n }\n // Process hdr.Pgno and data\n}\ntrailer := dec.Trailer()\n```\n\n## Writing LTX Files\n\n```go\nenc := ltx.NewEncoder(writer)\nenc.EncodeHeader(header)\nfor _, page := range pages {\n if page.Number == ltx.LockPgno(pageSize) {\n continue // Skip lock page\n }\n enc.EncodePage(pageHeader, pageData)\n}\nenc.EncodePageIndex(index)\nenc.EncodeTrailer()\nenc.Close()\n```\n\n## CLI Inspection\n\n```bash\nlitestream ltx /path/to/db.sqlite\nlitestream ltx s3://bucket/db\n```\n\nFor low-level inspection, use the Go API with `ltx.NewDecoder`.\n\n## WAL to LTX Conversion\n\nSQLite WAL frames are converted to LTX format during replication:\n1. Read WAL frames from the WAL file\n2. Skip the lock page\n3. Add checksums and build page index\n4. Write as immutable LTX file\n5. Upload to storage backend\n","skills/litestream/references/PATTERNS.md":"# Litestream Code Patterns Reference\n\nCondensed code patterns and anti-patterns for agents writing Litestream code.\n\n## Architectural Boundaries\n\n```\nDB Layer (db.go) → Database state, restoration, monitoring\nReplica Layer (replica.go) → Replication mechanics only\nStorage Layer → ReplicaClient implementations\n```\n\n### DO: Handle database state in DB layer\n\n```go\nfunc (db *DB) init() error {\n if db.needsRestore() {\n if err := db.restore(); err != nil {\n return err\n }\n }\n return db.replica.Start() // Replica focuses only on replication\n}\n```\n\n### DON'T: Put database state logic in Replica layer\n\n```go\n// WRONG\nfunc (r *Replica) Start() error {\n if needsRestore() { // Wrong layer!\n restoreDatabase()\n }\n}\n```\n\n## Atomic File Operations\n\n### DO: Write to temp file, then rename\n\n```go\nfunc writeFileAtomic(path string, data []byte) error {\n dir := filepath.Dir(path)\n tmpFile, err := os.CreateTemp(dir, \".tmp-*\")\n if err != nil {\n return fmt.Errorf(\"create temp file: %w\", err)\n }\n tmpPath := tmpFile.Name()\n\n defer func() {\n if tmpFile != nil {\n tmpFile.Close()\n os.Remove(tmpPath)\n }\n }()\n\n if _, err := tmpFile.Write(data); err != nil {\n return fmt.Errorf(\"write temp file: %w\", err)\n }\n if err := tmpFile.Sync(); err != nil {\n return fmt.Errorf(\"sync temp file: %w\", err)\n }\n if err := tmpFile.Close(); err != nil {\n return fmt.Errorf(\"close temp file: %w\", err)\n }\n tmpFile = nil\n\n if err := os.Rename(tmpPath, path); err != nil {\n os.Remove(tmpPath)\n return fmt.Errorf(\"rename to final path: %w\", err)\n }\n return nil\n}\n```\n\n### DON'T: Write directly to final location\n\n```go\n// WRONG - Can leave partial files on failure\nos.WriteFile(path, data, 0644)\n```\n\n## Error Handling\n\n### DO: Return errors immediately\n\n```go\nfunc (db *DB) validatePosition() error {\n dpos, err := db.Pos()\n if err != nil {\n return err\n }\n rpos := replica.Pos()\n if dpos.TXID < rpos.TXID {\n return fmt.Errorf(\"database position (%v) behind replica (%v)\", dpos, rpos)\n }\n return nil\n}\n```\n\n### DON'T: Log and continue on critical errors\n\n```go\n// WRONG\nif err := processFile(file); err != nil {\n log.Printf(\"error: %v\", err) // Just logging!\n // Continuing is dangerous\n}\n```\n\n### DO: Return errors from loops\n\n```go\nfunc (db *DB) processFiles() error {\n for _, file := range files {\n if err := processFile(file); err != nil {\n return fmt.Errorf(\"process file %s: %w\", file, err)\n }\n }\n return nil\n}\n```\n\n## Locking Patterns\n\n### DO: Use Lock() for writes\n\n```go\nr.mu.Lock()\ndefer r.mu.Unlock()\nr.pos = pos\n```\n\n### DON'T: Use RLock for write operations\n\n```go\n// WRONG - Race condition\nr.mu.RLock()\ndefer r.mu.RUnlock()\nr.pos = pos // Writing with RLock!\n```\n\n### Lock Ordering\n\nAlways acquire in order: Store.mu → DB.mu → DB.chkMu → Replica.mu\n\n## Compaction Patterns\n\n### DO: Read from local when available\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil // Local copy is complete and consistent\n}\nreturn replica.Client.OpenLTXFile(...) // Fall back to remote\n```\n\n### DON'T: Read from remote during compaction\n\n```go\n// WRONG - Can get partial/corrupt data from eventually consistent storage\nf, err := client.OpenLTXFile(ctx, level, minTXID, maxTXID, 0, 0)\n```\n\n### DO: Preserve earliest timestamp\n\n```go\ninfo.CreatedAt = oldestSourceFile.CreatedAt\n```\n\n### DON'T: Use current time during compaction\n\n```go\n// WRONG - Loses timestamp granularity for point-in-time restores\ninfo := <x.FileInfo{CreatedAt: time.Now()}\n```\n\n## Lock Page Handling\n\nThe lock page at 1 GB (0x40000000) must always be skipped:\n\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n\n## Common Pitfalls\n\n1. **Mixing architectural concerns**: DB state logic in Replica layer\n2. **Recreating existing functionality**: Use `db.verify()` for snapshots\n3. **Ignoring lock page**: Must skip during replication and compaction\n4. **Generic error types**: Return `os.ErrNotExist` for missing files\n5. **Blocking iterators**: Use lazy pagination for `LTXFiles`\n6. **Ignoring context**: Check `ctx.Done()` in long operations\n","skills/litestream/references/REPLICA_CLIENT_GUIDE.md":"# ReplicaClient Implementation Reference\n\nCondensed reference for agents implementing or modifying Litestream storage backends.\n\n## Interface Contract\n\nFrom `replica_client.go`:\n\n```go\ntype ReplicaClient interface {\n Type() string\n Init(ctx context.Context) error\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\n### Method Contracts\n\n**Type()**: Return identifier string (e.g., \"s3\", \"gcs\", \"file\").\n\n**Init()**: Initialize connection. Must be idempotent (no-op if already initialized).\n\n**LTXFiles()**: Return iterator sorted by MinTXID. `seek` starts from given TXID.\n`useMetadata=true` fetches accurate timestamps (required for point-in-time restore).\n`useMetadata=false` uses fast timestamps for normal operations.\n\n**OpenLTXFile()**: Open file for reading. Must support partial reads via `offset`/`size`.\nReturn `os.ErrNotExist` if file is missing.\n\n**WriteLTXFile()**: Write file to storage. Set `CreatedAt` from backend\nmetadata or upload time.\n\n**DeleteLTXFiles()**: Delete one or more files. Batch if possible.\n\n**DeleteAll()**: Delete all files for this database.\n\n## Implementation Checklist\n\n### Required\n\n- [ ] All interface methods implemented\n- [ ] `Init()` is idempotent\n- [ ] Partial reads supported (`offset`/`size` in `OpenLTXFile`)\n- [ ] `os.ErrNotExist` returned for missing files\n- [ ] Context cancellation handled in all methods\n- [ ] `CreatedAt` timestamps preserved in `WriteLTXFile`\n- [ ] Concurrent operations supported\n- [ ] Proper cleanup in `DeleteAll`\n\n### Optional\n\n- [ ] Connection pooling\n- [ ] Retry logic with exponential backoff\n- [ ] Request batching for deletes\n- [ ] Compression / encryption at rest\n- [ ] Bandwidth throttling\n\n## Eventual Consistency\n\nMany cloud backends (S3, R2, etc.) are eventually consistent:\n- Recently written files may not be immediately visible\n- Listed files may be only partially readable\n- Deletes may not take effect immediately\n\n### Pattern: Read Local First\n\nDuring compaction, always prefer local files:\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil\n}\nreturn replica.Client.OpenLTXFile(...)\n```\n\n### Pattern: Retry with Backoff\n\n```go\nbackoff := 100 * time.Millisecond\nfor i := 0; i < 5; i++ {\n reader, err := c.openFile(ctx, path, offset, size)\n if err == nil {\n return reader, nil\n }\n if errors.Is(err, os.ErrNotExist) {\n return nil, err // Don't retry definitive errors\n }\n select {\n case <-ctx.Done():\n return nil, ctx.Err()\n case <-time.After(backoff):\n backoff *= 2\n }\n}\n```\n\n### Pattern: Lazy Iterator\n\n```go\nfunc (c *Client) LTXFiles(...) (ltx.FileIterator, error) {\n return &lazyIterator{\n client: c,\n level: level,\n seek: seek,\n pageSize: 1000, // Paginate, don't load all at once\n }, nil\n}\n```\n\n## Error Handling\n\n### Standard Error Types\n\n```go\n// File not found\nreturn nil, os.ErrNotExist\n\n// Permission denied\nreturn nil, os.ErrPermission\n\n// Context cancelled\nreturn nil, ctx.Err()\n\n// Wrapped errors\nreturn nil, fmt.Errorf(\"s3 download failed: %w\", err)\n```\n\n### Retryable Error Classification\n\n```go\nfunc isRetryable(err error) bool {\n var netErr net.Error\n if errors.As(err, &netErr) && netErr.Temporary() {\n return true\n }\n // HTTP 429, 500, 502, 503, 504\n if errors.Is(err, context.DeadlineExceeded) {\n return true\n }\n return false\n}\n```\n\n## Common Mistakes\n\n### 1. Not Handling Partial Reads\n\n```go\n// WRONG - Ignores offset/size\nreturn c.storage.Download(path)\n\n// CORRECT\nif offset == 0 && size == 0 {\n return c.storage.Download(path)\n}\nreturn c.storage.DownloadRange(path, offset, offset+size-1)\n```\n\n### 2. Not Preserving CreatedAt\n\n```go\n// WRONG\nreturn <x.FileInfo{CreatedAt: time.Now()}\n\n// CORRECT - Use backend metadata\nreturn <x.FileInfo{CreatedAt: modTime}\n```\n\n### 3. Wrong Error Types\n\n```go\n// WRONG\nreturn nil, fmt.Errorf(\"not found\")\n\n// CORRECT\nif resp.StatusCode == 404 {\n return nil, os.ErrNotExist\n}\n```\n\n### 4. Ignoring Context\n\n```go\n// WRONG - Could run forever\nfor i := 0; i < 1000000; i++ {\n doWork()\n}\n\n// CORRECT\nselect {\ncase <-ctx.Done():\n return nil, ctx.Err()\ndefault:\n}\n```\n\n### 5. Loading All Files at Once\n\n```go\n// WRONG - Could be millions of files\nallFiles, _ := c.loadAllFiles(level)\n\n// CORRECT - Lazy pagination\nreturn &lazyIterator{pageSize: 1000}, nil\n```\n\n## Path Construction\n\n```go\nfunc (c *Client) ltxDir(level int) string {\n if level == SnapshotLevel {\n return path.Join(c.Path, \"snapshots\")\n }\n return path.Join(c.Path, \"ltx\", fmt.Sprintf(\"%04d\", level))\n}\n```\n\n## Reference Implementations\n\n- **Simplest**: `file/replica_client.go` (direct file I/O, no network)\n- **Most complete**: `s3/replica_client.go` (multipart uploads, retries, signing)\n\nStart with the file backend for understanding, then study S3 for advanced\npatterns.\n\n## Testing Requirements\n\n- Unit tests with >80% coverage\n- Integration tests with build tag\n- Test partial reads, concurrent operations, missing files\n- Run with `-race` flag\n- Verify `os.ErrNotExist` for missing files\n- Test context cancellation\n","skills/litestream/references/SQLITE_INTERNALS.md":"# SQLite Internals Reference\n\nCondensed reference for agents working with SQLite internals in Litestream.\n\n## SQLite File Structure\n\n```\ndatabase.db Main database file (pages)\ndatabase.db-wal Write-ahead log\ndatabase.db-shm Shared memory file (coordination)\n```\n\n## Write-Ahead Log (WAL)\n\nWAL is SQLite's method for atomic commits:\n- Changes written to WAL first, database unchanged until checkpoint\n- Readers merge WAL + database for consistent view\n- Litestream monitors WAL and converts frames to LTX format\n\n### WAL File Structure\n\n```\n+------------------+\n| WAL Header | 32 bytes\n+------------------+\n| Frame 1 Header | 24 bytes\n| Frame 1 Data | PageSize bytes\n+------------------+\n| Frame 2 Header | 24 bytes\n| Frame 2 Data | PageSize bytes\n+------------------+\n| ... |\n```\n\n### WAL Header (32 bytes)\n\n```\nMagic (4B) | FileFormat (4B) | PageSize (4B) | Checkpoint (4B) |\nSalt1 (4B) | Salt2 (4B) | Checksum1 (4B) | Checksum2 (4B)\n```\n\nMagic: `0x377f0682` or `0x377f0683`\n\n### WAL Frame Header (24 bytes)\n\n```\nPageNumber (4B) | DbSize (4B) | Salt1 (4B) | Salt2 (4B) |\nChecksum1 (4B) | Checksum2 (4B)\n```\n\n## The 1 GB Lock Page\n\nSQLite reserves a page at exactly 0x40000000 bytes (1 GB) for locking. This is\nthe single most important SQLite detail for Litestream development.\n\n```go\nconst PENDING_BYTE = 0x40000000\n\nfunc LockPgno(pageSize int) uint32 {\n return uint32(PENDING_BYTE/pageSize) + 1\n}\n```\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n| 64 KB | 16385 |\n\nRules:\n- Cannot contain data; SQLite never writes user data here\n- Must be skipped during replication and compaction\n- Only affects databases > 1 GB\n- Page number changes with page size\n\nImplementation in Litestream:\n\n```go\nfor pgno := uint32(1); pgno <= maxPgno; pgno++ {\n if pgno == ltx.LockPgno(db.pageSize) {\n continue\n }\n processPage(pgno)\n}\n```\n\n## Page Structure\n\nSQLite divides databases into fixed-size pages (typically 4096 bytes):\n- Page numbers are 1-based\n- Types: B-tree interior, B-tree leaf, overflow, freelist, lock byte page\n\n## Transaction Types\n\n1. **Deferred** (default): Lock acquired on first use\n2. **Immediate**: RESERVED lock acquired immediately\n3. **Exclusive**: EXCLUSIVE lock acquired immediately\n\n### Lock Hierarchy\n\nUNLOCKED → SHARED → RESERVED → PENDING → EXCLUSIVE\n\n- SHARED: Multiple readers allowed\n- RESERVED: Signals intent to write\n- PENDING: Blocks new SHARED locks\n- EXCLUSIVE: Single writer, no readers\n\n## Long-Running Read Transaction\n\nLitestream maintains a read transaction for consistency:\n\n```go\nfunc (db *DB) initReadTx() error {\n tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{ReadOnly: true})\n if err != nil {\n return err\n }\n var dummy string\n err = tx.QueryRow(\"SELECT ''\").Scan(&dummy)\n if err != nil {\n tx.Rollback()\n return err\n }\n db.rtx = tx\n return nil\n}\n```\n\nPurpose:\n- Prevents checkpoint past our read point\n- Ensures consistent database view\n- Allows reading historical pages from WAL\n\n## Checkpoint Modes\n\n| Mode | Behavior |\n|----------|-----------------------------------------------|\n| PASSIVE | Non-blocking; fails if readers present |\n| FULL | Waits for readers; blocks new readers |\n| RESTART | Like FULL + resets WAL start (removed in #724) |\n| TRUNCATE | Like RESTART + truncates WAL to zero |\n\n### Litestream Checkpoint Strategy\n\n```\nWAL pages > TruncatePageN → TRUNCATE (emergency)\nWAL pages > MinCheckpointPageN → PASSIVE\nCheckpointInterval elapsed → PASSIVE\n```\n\nNote: RESTART mode permanently removed due to issue #724 (write-blocking).\n\n## Important SQLite Pragmas\n\n```sql\nPRAGMA journal_mode = WAL; -- Required for Litestream\nPRAGMA page_size; -- Get page size\nPRAGMA page_count; -- Total pages in database\nPRAGMA freelist_count; -- Free pages\nPRAGMA wal_checkpoint(PASSIVE); -- Non-blocking checkpoint\nPRAGMA wal_autocheckpoint = 10000; -- High threshold (prevent interference)\nPRAGMA busy_timeout = 5000; -- Wait 5s for locks\nPRAGMA synchronous = NORMAL; -- Safe with WAL\nPRAGMA cache_size = -64000; -- 64 MB cache\nPRAGMA integrity_check; -- Verify database integrity\n```\n\n## Critical SQLite Behaviors\n\n1. **Automatic checkpoint**: Default at 1000 WAL pages; set high threshold to\n prevent interference with Litestream's control\n2. **Busy timeout**: Default is 0 (immediate failure); set a reasonable timeout\n3. **Synchronous mode**: NORMAL is safe with WAL mode\n4. **Page cache**: In-memory cache; configure with negative KB or positive pages\n\n## WAL to LTX Conversion\n\nLitestream converts WAL frames to LTX:\n\n1. Read WAL header and validate magic number\n2. Iterate frames, reading page number and data\n3. Skip lock page (`pgno == LockPgno(pageSize)`)\n4. Calculate checksums\n5. Write as immutable LTX file with page index\n","skills/litestream/references/TESTING_GUIDE.md":"# Litestream Testing Reference\n\nCondensed reference for agents writing and debugging Litestream tests.\n\n## Testing Philosophy\n\n1. Test at multiple levels: unit, integration, end-to-end\n2. Focus on edge cases: >1 GB databases, eventual consistency\n3. Use real SQLite: avoid mocking SQLite behavior\n4. Always run with `-race` flag\n5. Use fixed seeds and timestamps for deterministic tests\n\n## Essential Commands\n\n```bash\n# Full test suite with race detection\ngo test -race -v ./...\n\n# Specific test areas\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\n\n# Coverage\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\ngo tool cover -func=coverage.out | grep total\n\n# Integration tests (backend-specific)\ngo test -v ./replica_client_test.go -integration s3\ngo test -v ./replica_client_test.go -integration gcs\n```\n\n## Lock Page Testing\n\nThe lock page at 1 GB must be skipped. Test with databases spanning this\nboundary using multiple page sizes:\n\n```bash\n./bin/litestream-test populate -db test.db -page-size 4096 -target-size 2GB\n./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica\n```\n\nTest all page size variants:\n\n| Page Size | Lock Page | Test Target |\n|-----------|-----------|-------------|\n| 4 KB | 262145 | >1.0 GB |\n| 8 KB | 131073 | >1.0 GB |\n| 16 KB | 65537 | >1.0 GB |\n| 32 KB | 32769 | >1.0 GB |\n\nVerify lock page is not present in any LTX file after replication.\n\n## Race Condition Testing\n\nRace-prone areas (always run with `-race`):\n\n1. **Position updates**: Concurrent `SetPos()` + `Pos()` on Replica\n2. **WAL monitoring**: Concurrent writes + monitor + checkpoint\n3. **Compaction**: Concurrent compaction at different levels\n\n### Pattern: Concurrent Test\n\n```go\nfunc TestConcurrent(t *testing.T) {\n ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\n defer cancel()\n\n var wg sync.WaitGroup\n // Start writer, monitor, checkpoint goroutines\n // Use ctx for shutdown, wg for cleanup\n wg.Wait()\n}\n```\n\n## Common Test Failures\n\n### 1. Database Locked\n\n```go\n// WRONG - WAL mode not enabled\ndb, _ := sql.Open(\"sqlite\", \"test.db\")\n\n// CORRECT\ndb, _ := sql.Open(\"sqlite\", \"test.db?_journal=WAL\")\n```\n\n### 2. Timing Issues\n\n```go\n// WRONG - Data may not be replicated yet\nWriteData(db)\nresult := ReadReplica()\n\n// CORRECT - Explicit sync\nWriteData(db)\ndb.Sync(context.Background())\nresult := ReadReplica()\n```\n\n### 3. Goroutine Leaks\n\n```go\n// WRONG - Goroutine outlives test\ngo func() {\n time.Sleep(10 * time.Second)\n doWork()\n}()\n\n// CORRECT - Use context + WaitGroup\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\nvar wg sync.WaitGroup\nwg.Add(1)\ngo func() {\n defer wg.Done()\n select {\n case <-ctx.Done():\n return\n case <-time.After(10 * time.Second):\n doWork()\n }\n}()\ncancel()\nwg.Wait()\n```\n\n### 4. File Handle Leaks\n\n```go\n// Always use defer for cleanup\nf, err := os.Open(\"test.db\")\nrequire.NoError(t, err)\ndefer f.Close()\n```\n\n## Test Helper Patterns\n\n```go\nfunc NewTestDB(t testing.TB) *litestream.DB {\n t.Helper()\n path := filepath.Join(t.TempDir(), \"test.db\")\n conn, err := sql.Open(\"sqlite\", path+\"?_journal=WAL\")\n require.NoError(t, err)\n _, err = conn.Exec(\"CREATE TABLE test (id INTEGER PRIMARY KEY, data BLOB)\")\n require.NoError(t, err)\n conn.Close()\n\n db := litestream.NewDB(path)\n db.MonitorInterval = 10 * time.Millisecond\n db.MinCheckpointPageN = 100\n err = db.Open()\n require.NoError(t, err)\n t.Cleanup(func() { db.Close(context.Background()) })\n return db\n}\n```\n\n## Coverage Requirements\n\n| Package | Target |\n|--------------------|--------|\n| Core (db, replica) | >80% |\n| Replica clients | >70% |\n| Utilities | >60% |\n\n## Table-Driven Tests\n\n```go\nfunc TestCheckpoint(t *testing.T) {\n tests := []struct {\n name string\n mode string\n wantErr bool\n }{\n {\"Passive\", \"PASSIVE\", false},\n {\"Full\", \"FULL\", false},\n {\"Truncate\", \"TRUNCATE\", false},\n {\"Invalid\", \"INVALID\", true},\n }\n for _, tt := range tests {\n t.Run(tt.name, func(t *testing.T) {\n // ...\n })\n }\n}\n```\n"},"files":{".claude/agents/ltx-compaction-specialist.md":"---\nrole: LTX Format and Compaction Specialist\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# LTX Compaction Specialist Agent\n\nYou are an expert in the LTX (Log Transaction) format and multi-level compaction strategies for Litestream.\n\n## Core Knowledge\n\n### LTX File Format\n```\n┌─────────────────────┐\n│ Header │ 84 bytes\n├─────────────────────┤\n│ Page Frames │ Variable\n├─────────────────────┤\n│ Page Index │ Binary search structure\n├─────────────────────┤\n│ Trailer │ 16 bytes\n└─────────────────────┘\n```\n\n### File Naming Convention\n```\nMMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\nWhere:\n M = MinTXID (16 hex digits)\n N = MaxTXID (16 hex digits)\nExample: 0000000000000001-0000000000000064.ltx\n```\n\n## Default Compaction Levels\n\n### Level Structure\n```\nLevel 0: Raw (no compaction)\nLevel 1: 30-second windows\nLevel 2: 5-minute windows\nLevel 3: 1-hour windows\nSnapshots: Daily full database\n```\n\n### Critical Compaction Rules\n\n1. **ALWAYS Read Local First**:\n ```go\n // CORRECT - Handles eventual consistency\n f, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\n if err == nil {\n return f, nil // Use local file\n }\n // Only fall back to remote if local doesn't exist\n return replica.Client.OpenLTXFile(...)\n ```\n\n2. **Preserve Timestamps**:\n ```go\n // Keep earliest CreatedAt\n info, err := replica.Client.WriteLTXFile(ctx, level, minTXID, maxTXID, reader)\n if err != nil {\n return nil, fmt.Errorf(\"write ltx file: %w\", err)\n }\n info.CreatedAt = oldestSourceFile.CreatedAt\n ```\n\n3. **Skip Lock Page**:\n ```go\n if pgno == ltx.LockPgno(pageSize) {\n continue\n }\n ```\n\n## Compaction Algorithm\n\n```go\nfunc compactLTXFiles(files []*LTXFile) (*LTXFile, error) {\n // 1. Create page map (newer overwrites older)\n pageMap := make(map[uint32]Page)\n for _, file := range files {\n for _, page := range file.Pages {\n pageMap[page.Number] = page\n }\n }\n\n // 2. Create new LTX with merged pages\n merged := <XFile{\n MinTXID: files[0].MinTXID,\n MaxTXID: files[len(files)-1].MaxTXID,\n }\n\n // 3. Add pages in order (skip lock page!)\n for pgno := uint32(1); pgno <= maxPgno; pgno++ {\n if pgno == LockPageNumber(pageSize) {\n continue\n }\n if page, ok := pageMap[pgno]; ok {\n merged.Pages = append(merged.Pages, page)\n }\n }\n\n return merged, nil\n}\n```\n\n## Key Properties\n\n### Immutability\n- LTX files are NEVER modified after creation\n- New changes create new files\n- Compaction creates new merged files\n\n### Checksums\n- CRC-64 ECMA for integrity\n- `PreApplyChecksum`/`PostApplyChecksum` on the header/trailer bracketing file state\n- `FileChecksum` covering the entire file contents\n\n### Page Index\n- Exposed via `ltx.DecodePageIndex`\n- Tracks page number plus offset/size of the encoded payload\n- Located by seeking from the end of the file using trailer metadata\n\n## Common Issues\n\n1. **Partial Reads**: Remote storage may return incomplete files\n2. **Race Conditions**: Multiple compactions running\n3. **Timestamp Loss**: Not preserving original CreatedAt\n4. **Lock Page**: Including 1GB lock page in compacted files\n5. **Memory Usage**: Loading entire files for compaction\n6. **Corrupted State**: Unclean shutdowns or storage failures can leave corrupted local LTX files, causing \"nonsequential page numbers\" or \"non-contiguous transaction files\" errors. Recovery: `litestream reset ` (manual) or `auto-recover: true` replica config (automatic). See `cmd/litestream/reset.go` and `replica.go`\n\n## Testing\n\n```bash\n# Test compaction\ngo test -v -run TestStore_CompactDB ./...\n\n# Test with eventual consistency\ngo test -v -run TestStore_CompactDB_RemotePartialRead ./...\n\n# Manual inspection\nlitestream ltx /path/to/db.sqlite\n# For deeper inspection use the Go API (ltx.NewDecoder)\n```\n\n## References\n- docs/LTX_FORMAT.md - Complete format specification\n- store.go - Compaction scheduling\n- db.go - Compaction implementation\n- github.com/superfly/ltx - LTX library\n",".claude/agents/performance-optimizer.md":"---\nrole: Performance Optimizer\ntools:\n - read\n - write\n - edit\n - bash\n - grep\npriority: medium\n---\n\n# Performance Optimizer Agent\n\nYou specialize in optimizing Litestream for speed, memory usage, and resource efficiency.\n\n## Key Performance Areas\n\n### O(n) Operations to Optimize\n\n1. **Page Iteration**\n ```go\n // Cache page index\n const DefaultEstimatedPageIndexSize = 32 * 1024 // 32KB\n\n // Fetch end of file first for page index\n offset := info.Size - DefaultEstimatedPageIndexSize\n if offset < 0 {\n offset = 0\n }\n ```\n\n2. **File Listing**\n ```go\n // Cache file listings\n type FileCache struct {\n files []FileInfo\n timestamp time.Time\n ttl time.Duration\n }\n ```\n\n3. **Compaction**\n ```go\n // Limit concurrent compactions\n sem := make(chan struct{}, maxConcurrentCompactions)\n ```\n\n## Memory Optimization\n\n### Page Buffer Pooling\n```go\nvar pagePool = sync.Pool{\n New: func() interface{} {\n b := make([]byte, 4096) // Default page size\n return &b\n },\n}\n\nfunc getPageBuffer() []byte {\n return *pagePool.Get().(*[]byte)\n}\n\nfunc putPageBuffer(b []byte) {\n pagePool.Put(&b)\n}\n```\n\n### Streaming Instead of Loading\n```go\n// BAD - Loads entire file\ndata, err := os.ReadFile(path)\n\n// GOOD - Streams data\nf, err := os.Open(path)\ndefer f.Close()\nio.Copy(dst, f)\n```\n\n## Concurrency Patterns\n\n### Proper Locking\n```go\n// Read-heavy optimization\ntype Store struct {\n mu sync.RWMutex // Use RWMutex for read-heavy\n}\n\nfunc (s *Store) Read() {\n s.mu.RLock()\n defer s.mu.RUnlock()\n // Read operation\n}\n\nfunc (s *Store) Write() {\n s.mu.Lock()\n defer s.mu.Unlock()\n // Write operation\n}\n```\n\n### Channel Patterns\n```go\n// Batch processing\nbatch := make([]Item, 0, batchSize)\nticker := time.NewTicker(batchInterval)\n\nfor {\n select {\n case item := <-input:\n batch = append(batch, item)\n if len(batch) >= batchSize {\n processBatch(batch)\n batch = batch[:0]\n }\n case <-ticker.C:\n if len(batch) > 0 {\n processBatch(batch)\n batch = batch[:0]\n }\n }\n}\n```\n\n## I/O Optimization\n\n### Buffered I/O\n```go\n// Use buffered writers\nbw := bufio.NewWriterSize(w, 64*1024) // 64KB buffer\ndefer bw.Flush()\n\n// Use buffered readers\nbr := bufio.NewReaderSize(r, 64*1024)\n```\n\n### Parallel Downloads\n```go\nfunc downloadParallel(files []string) {\n var wg sync.WaitGroup\n sem := make(chan struct{}, 5) // Limit to 5 concurrent\n\n for _, file := range files {\n wg.Add(1)\n go func(f string) {\n defer wg.Done()\n sem <- struct{}{}\n defer func() { <-sem }()\n\n download(f)\n }(file)\n }\n wg.Wait()\n}\n```\n\n## Caching Strategy\n\n### LRU Cache Implementation\n```go\ntype LRUCache struct {\n capacity int\n items map[string]*list.Element\n list *list.List\n mu sync.RWMutex\n}\n\nfunc (c *LRUCache) Get(key string) (interface{}, bool) {\n c.mu.RLock()\n elem, ok := c.items[key]\n c.mu.RUnlock()\n\n if !ok {\n return nil, false\n }\n\n c.mu.Lock()\n c.list.MoveToFront(elem)\n c.mu.Unlock()\n\n return elem.Value, true\n}\n```\n\n## Profiling Tools\n\n### CPU Profiling\n```bash\n# Generate CPU profile\ngo test -cpuprofile=cpu.prof -bench=.\n\n# Analyze\ngo tool pprof cpu.prof\n(pprof) top10\n(pprof) list functionName\n```\n\n### Memory Profiling\n```bash\n# Generate memory profile\ngo test -memprofile=mem.prof -bench=.\n\n# Analyze allocations\ngo tool pprof -alloc_space mem.prof\n```\n\n### Trace Analysis\n```bash\n# Generate trace\ngo test -trace=trace.out\n\n# View trace\ngo tool trace trace.out\n```\n\n## Configuration Tuning\n\n### SQLite Pragmas\n```sql\nPRAGMA cache_size = -64000; -- 64MB cache\nPRAGMA synchronous = NORMAL; -- Balance safety/speed\nPRAGMA wal_autocheckpoint = 10000; -- Larger WAL before checkpoint\nPRAGMA busy_timeout = 5000; -- 5 second timeout\n```\n\n### Litestream Settings\n```yaml\n# Optimal intervals\nmin-checkpoint-page-n: 1000\ntruncate-page-n: 121359\nmonitor-interval: 1s\ncheckpoint-interval: 1m\n```\n\n## Benchmarks to Run\n\n```bash\n# Core operations\ngo test -bench=BenchmarkWALRead\ngo test -bench=BenchmarkLTXWrite\ngo test -bench=BenchmarkCompaction\ngo test -bench=BenchmarkPageIteration\n\n# With memory stats\ngo test -bench=. -benchmem\n```\n\n## Common Performance Issues\n\n1. **Not pooling buffers** - Creates garbage\n2. **Loading entire files** - Use streaming\n3. **Excessive locking** - Use RWMutex\n4. **No caching** - Repeated expensive operations\n5. **Serial processing** - Could parallelize\n6. **Small buffers** - Increase buffer sizes\n\n## References\n- Go performance tips: https://go.dev/doc/perf\n- SQLite optimization: https://sqlite.org/optoverview.html\n- Profiling guide: https://go.dev/blog/pprof\n",".claude/agents/replica-client-developer.md":"---\nrole: Replica Client Developer\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# Replica Client Developer Agent\n\nYou specialize in implementing and maintaining storage backend clients for Litestream replication.\n\n## Core Knowledge\n\n### ReplicaClient Interface\n\nEvery storage backend MUST implement:\n```go\ntype ReplicaClient interface {\n Type() string\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\n**LTXFiles useMetadata parameter**:\n- When `useMetadata=true`: Fetch accurate timestamps from backend metadata (slower, required for point-in-time restore)\n- When `useMetadata=false`: Use fast timestamps from file listing (faster, suitable for replication monitoring)\n\n### Critical Patterns\n\n1. **Eventual Consistency Handling**:\n - Storage may not immediately reflect writes\n - Files may be partially available\n - ALWAYS prefer local files during compaction\n\n2. **Atomic Operations**:\n ```go\n // Write to temp, then rename\n tmpPath := path + \".tmp\"\n // Write to tmpPath\n os.Rename(tmpPath, path)\n ```\n\n3. **Error Types**:\n - Return `os.ErrNotExist` for missing files\n - Wrap errors with context: `fmt.Errorf(\"operation: %w\", err)`\n\n4. **ResumableReader Support**:\n - `OpenLTXFile` MUST support the `offset` parameter for range requests\n - `internal/resumable_reader.go` wraps streams with auto-reconnection on idle timeouts\n - During restore, streams may sit idle while compactor processes other files\n - If `offset` is ignored, restore operations fail on connection timeouts\n\n### ReplicaClientV3 Interface (Optional)\n\nBackends supporting v0.3.x backward-compatible restore should implement:\n```go\ntype ReplicaClientV3 interface {\n GenerationsV3(ctx context.Context) ([]string, error)\n SnapshotsV3(ctx context.Context, generation string) ([]SnapshotInfoV3, error)\n WALSegmentsV3(ctx context.Context, generation string) ([]WALSegmentInfoV3, error)\n OpenSnapshotV3(ctx context.Context, generation string, index int) (io.ReadCloser, error)\n OpenWALSegmentV3(ctx context.Context, generation string, index int, offset int64) (io.ReadCloser, error)\n}\n```\n\nSee `v3.go` for type definitions and `s3/replica_client.go` for reference implementation.\n\n## Implementation Checklist\n\n### New Backend Requirements\n\n- [ ] Implement ReplicaClient interface\n- [ ] Handle partial reads (offset/size)\n- [ ] Support seek parameter for pagination\n- [ ] Preserve CreatedAt timestamps when metadata is available\n- [ ] Handle eventual consistency\n- [ ] Implement proper error types\n- [ ] Add integration tests\n- [ ] Document configuration\n\n### Testing Requirements\n\n```bash\n# Integration test\ngo test -v ./replica_client_test.go -integration [backend]\n\n# Race conditions\ngo test -race -v ./[backend]/...\n\n# Large files (>1GB)\n./bin/litestream-test populate -target-size 2GB\n```\n\n## Existing Backends Reference\n\n### Study These Implementations\n\n- `s3/replica_client.go` - AWS S3 (most complete)\n- `gs/replica_client.go` - Google Cloud Storage\n- `abs/replica_client.go` - Azure Blob Storage\n- `file/replica_client.go` - Local filesystem (simplest)\n- `sftp/replica_client.go` - SSH File Transfer\n- `nats/replica_client.go` - NATS JetStream (newest)\n- `oss/replica_client.go` - Alibaba Cloud OSS\n\n## Common Pitfalls\n\n1. Not handling eventual consistency\n2. Missing atomic write operations\n3. Incorrect error types\n4. Not preserving timestamps\n5. Forgetting partial read support\n6. No retry logic for transient failures\n7. Not supporting `offset` parameter in `OpenLTXFile` — breaks `ResumableReader` during restore\n\n## Configuration Pattern\n\n```yaml\nreplica:\n type: [backend]\n option1: value1\n option2: value2\n```\n\n## References\n\n- docs/REPLICA_CLIENT_GUIDE.md - Complete implementation guide\n- replica_client.go - Interface definition\n- v3.go - ReplicaClientV3 interface and v0.3.x types\n- internal/resumable_reader.go - ResumableReader for restore resilience\n- replica_client_test.go - Test suite\n",".claude/agents/sqlite-expert.md":"---\nrole: SQLite WAL and Page Expert\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# SQLite Expert Agent\n\nYou are a SQLite internals expert specializing in WAL (Write-Ahead Log) operations and page management for the Litestream project.\n\n## Core Knowledge\n\n### Critical SQLite Concepts\n1. **1GB Lock Page** (MUST KNOW):\n - Located at exactly 0x40000000 (1,073,741,824 bytes)\n - Page number varies by page size:\n - 4KB pages: 262145\n - 8KB pages: 131073\n - 16KB pages: 65537\n - 32KB pages: 32769\n - MUST be skipped in all iterations\n - Cannot contain data\n\n2. **WAL Structure**:\n - 32-byte header with magic number\n - Frames with 24-byte headers\n - Cumulative checksums\n - Salt values for verification\n\n3. **Page Types**:\n - B-tree interior/leaf pages\n - Overflow pages\n - Freelist pages\n - Lock byte page (at 1GB)\n\n## Primary Responsibilities\n\n### WAL Monitoring\n- Monitor WAL file changes in `db.go`\n- Ensure proper checksum verification\n- Handle WAL frame reading correctly\n- Convert WAL frames to LTX format\n\n### Page Management\n- Always skip lock page during iteration\n- Handle various page sizes correctly\n- Verify page integrity\n- Manage page caching efficiently\n\n### Testing Requirements\n- Create test databases >1GB\n- Test all page sizes (4KB, 8KB, 16KB, 32KB)\n- Verify lock page skipping\n- Test WAL checkpoint modes\n\n## Code Patterns\n\n### Correct Lock Page Handling\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue // Skip lock page\n}\n```\n\n### WAL Reading\n```go\n// Always verify magic number\nmagic := binary.BigEndian.Uint32(header.Magic[:])\nif magic != 0x377f0682 && magic != 0x377f0683 {\n return errors.New(\"invalid WAL magic\")\n}\n```\n\n## Common Mistakes to Avoid\n1. Not skipping lock page at 1GB\n2. Incorrect checksum calculations\n3. Wrong byte order (use BigEndian)\n4. Not handling all page sizes\n5. Direct file manipulation (use SQLite API)\n\n## References\n- docs/SQLITE_INTERNALS.md - Complete SQLite internals guide\n- docs/LTX_FORMAT.md - LTX conversion details\n- db.go - WAL monitoring implementation\n",".claude/agents/test-engineer.md":"---\nrole: Test Engineer\ntools:\n - read\n - write\n - edit\n - bash\n - grep\npriority: medium\n---\n\n# Test Engineer Agent\n\nYou specialize in creating and maintaining comprehensive test suites for Litestream, with focus on edge cases and race conditions.\n\n## Critical Test Scenarios\n\n### 1GB Lock Page Testing\n\n**MUST TEST**: Databases crossing the 1GB boundary\n\n```bash\n# Create >1GB test database\nsqlite3 large.db < c.failAfter {\n return errors.New(\"simulated failure\")\n }\n return c.ReplicaClient.WriteLTXFile(...)\n}\n```\n\n## Coverage Requirements\n\n### Minimum Coverage\n- Core packages: >80%\n- Storage backends: >70%\n- Critical paths: 100%\n\n### Generate Coverage Report\n```bash\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\n```\n\n## Common Test Mistakes\n\n1. Not testing with databases >1GB\n2. Forgetting race detector\n3. Not testing all page sizes\n4. Missing eventual consistency tests\n5. No error injection tests\n6. Ignoring benchmark regressions\n\n## CI/CD Integration\n\n```yaml\n# .github/workflows/test.yml\n- name: Run tests with race detector\n run: go test -race -v ./...\n\n- name: Test large databases\n run: ./scripts/test-large-db.sh\n\n- name: Integration tests\n run: ./scripts/test-integration.sh\n```\n\n## References\n- docs/TESTING_GUIDE.md - Complete testing guide\n- replica_client_test.go - Integration test patterns\n- db_test.go - Unit test examples\n",".claude/commands/add-storage-backend.md":"---\ndescription: Create a new storage backend implementation\n---\n\n# Add Storage Backend Command\n\nCreate a new storage backend implementation for Litestream with all required components.\n\n## Steps\n\n1. **Create Package Directory**\n ```bash\n mkdir -p {{backend_name}}\n ```\n\n2. **Implement ReplicaClient Interface**\n Create `{{backend_name}}/replica_client.go`:\n ```go\n package {{backend_name}}\n\n type ReplicaClient struct {\n // Configuration fields\n }\n\n func (c *ReplicaClient) Type() string {\n return \"{{backend_name}}\"\n }\n\n func (c *ReplicaClient) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {\n // List files at level\n // When useMetadata=true, fetch accurate timestamps from backend metadata\n }\n\n func (c *ReplicaClient) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {\n // Open file for reading\n }\n\n func (c *ReplicaClient) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {\n // Write file atomically\n }\n\n func (c *ReplicaClient) DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error {\n // Delete files\n }\n\n func (c *ReplicaClient) DeleteAll(ctx context.Context) error {\n // Remove all files for replica\n }\n ```\n\n3. **Add Configuration Parsing**\n Update `cmd/litestream/config.go`:\n ```go\n case \"{{backend_name}}\":\n client = &{{backend_name}}.ReplicaClient{\n // Parse config\n }\n ```\n\n4. **Create Integration Tests**\n Create `{{backend_name}}/replica_client_test.go`:\n ```go\n func TestReplicaClient_{{backend_name}}(t *testing.T) {\n if !*integration || *backend != \"{{backend_name}}\" {\n t.Skip(\"{{backend_name}} integration test skipped\")\n }\n // Test implementation\n }\n ```\n\n5. **Add Documentation**\n Update README.md with configuration example:\n ```yaml\n replica:\n type: {{backend_name}}\n option1: value1\n option2: value2\n ```\n\n## Key Requirements\n\n- Handle eventual consistency\n- Implement atomic writes (temp file + rename)\n- Support partial reads (offset/size)\n- Preserve CreatedAt timestamps in returned FileInfo\n- Return proper error types (os.ErrNotExist)\n\n## Testing\n\n```bash\n# Run integration tests\ngo test -v ./replica_client_test.go -integration {{backend_name}}\n\n# Test with race detector\ngo test -race -v ./{{backend_name}}/...\n```\n",".claude/commands/analyze-ltx.md":"Analyze LTX file issues in Litestream. This command helps diagnose problems with LTX files, including corruption, missing files, and consistency issues.\n\nFirst, understand the context:\n- What error messages are being reported?\n- Which storage backend is being used?\n- Are there any eventual consistency issues?\n\nThen perform the analysis:\n\n1. **Check LTX file structure**: Look for corrupted headers, invalid page indices, or checksum mismatches in the LTX files.\n\n2. **Verify file continuity**: Ensure there are no gaps in the TXID sequence that could prevent restoration.\n\n3. **Check compaction issues**: Look for problems during compaction that might corrupt files, especially with eventually consistent storage.\n\n4. **Analyze page sequences**: Verify that page numbers are sequential and the lock page at 1GB is properly skipped.\n\n5. **Review storage backend behavior**: Check if the storage backend has eventual consistency that might cause partial reads during compaction.\n\nKey files to examine:\n- `db.go`: WAL monitoring and LTX generation\n- `replica_client.go`: Storage interface\n- `store.go`: Compaction logic\n- Backend-specific client in `s3/`, `gs/`, etc.\n\nCommon issues to look for:\n- \"nonsequential page numbers\" errors (corrupted compaction)\n- \"EOF\" errors (partial file reads)\n- Missing TXID ranges (failed uploads)\n- Lock page at 0x40000000 not being skipped\n\nUse the testing harness to reproduce:\n```bash\n./bin/litestream-test validate -source-db test.db -replica-url [URL]\n```\n\n## Recovery Options\n\nWhen analysis reveals corrupted or missing LTX files, two recovery mechanisms are available:\n\n**Manual recovery** with `litestream reset`:\n```bash\nlitestream reset /path/to/database.db\n```\nClears local LTX state from the metadata directory. The database file is not modified.\nNext sync creates a fresh snapshot. See `cmd/litestream/reset.go` for implementation.\n\n**Automatic recovery** with `auto-recover` config:\n```yaml\ndbs:\n - path: /path/to/database.db\n replicas:\n - url: s3://bucket/path\n auto-recover: true\n```\nWhen enabled, Litestream automatically resets local state when LTX errors are detected\nduring sync. Disabled by default. See `replica.go` (auto-recover logic) for implementation.\n",".claude/commands/debug-ipc.md":"---\ndescription: Debug IPC Unix socket issues\n---\n\n# Debug IPC Command\n\nDiagnose issues with the Litestream IPC control socket (`server.go`).\n\n## 1. Socket Configuration Check\n\nVerify the socket is enabled and correctly configured:\n\n```yaml\n# litestream.yml\nsocket:\n enabled: true # Default: false\n path: /var/run/litestream.sock # Default path\n permissions: 0600 # Default permissions\n```\n\nSource: `SocketConfig` struct in `server.go:17-21`, defaults in `DefaultSocketConfig()`.\n\n## 2. Endpoint Reference\n\nAll endpoints are HTTP over Unix socket (`server.go:74-87`):\n\n| Method | Path | Request Body | Response |\n|--------|------|-------------|----------|\n| `GET` | `/info` | — | `{version, pid, uptime_seconds, started_at, database_count}` |\n| `GET` | `/list` | — | `{databases: [{path, status, last_sync_at}]}` |\n| `GET` | `/txid?path=` | — | `{txid}` |\n| `POST` | `/register` | `{path, replica_url}` | `{status, path}` |\n| `POST` | `/unregister` | `{path, timeout?}` | `{status, path}` |\n| `POST` | `/start` | `{path, timeout?}` | `{status, path}` |\n| `POST` | `/stop` | `{path, timeout?}` | `{status, path}` |\n| `GET` | `/debug/pprof/` | — | Standard Go pprof index |\n| `GET` | `/debug/pprof/profile` | — | CPU profile |\n| `GET` | `/debug/pprof/trace` | — | Execution trace |\n\n## 3. Testing with curl\n\n```bash\n# Server info (version, PID, uptime)\ncurl --unix-socket /var/run/litestream.sock http://localhost/info\n\n# List all managed databases\ncurl --unix-socket /var/run/litestream.sock http://localhost/list\n\n# Get transaction ID for a specific database\ncurl --unix-socket /var/run/litestream.sock \"http://localhost/txid?path=/path/to/db\"\n\n# Register a new database at runtime\ncurl --unix-socket /var/run/litestream.sock -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"path\":\"/path/to/db\",\"replica_url\":\"s3://bucket/path\"}' \\\n http://localhost/register\n\n# Unregister (stop replicating) a database\ncurl --unix-socket /var/run/litestream.sock -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"path\":\"/path/to/db\"}' \\\n http://localhost/unregister\n\n# CPU profile (30 seconds by default)\ncurl --unix-socket /var/run/litestream.sock http://localhost/debug/pprof/profile > cpu.prof\ngo tool pprof cpu.prof\n\n# Heap profile\ncurl --unix-socket /var/run/litestream.sock http://localhost/debug/pprof/heap > heap.prof\ngo tool pprof heap.prof\n```\n\n## 4. Common Issues\n\n### Socket not enabled\n**Symptom**: `curl: (7) Couldn't connect to server`\n**Fix**: Set `socket.enabled: true` in config and restart litestream.\n\n### Permission denied\n**Symptom**: `curl: (7) Permission denied`\n**Fix**: Check `socket.permissions` (default `0600`). The connecting user must match the litestream process owner.\n\n### Stale socket file after crash\n**Symptom**: `bind: address already in use` in logs on startup\n**Fix**: Remove the stale socket file: `rm /var/run/litestream.sock`\n\n### Database not found\n**Symptom**: `{\"error\":\"database not found\"}` from `/txid` or `/start`\n**Fix**: Verify the path matches the expanded path in config. Use `/list` to see registered databases. Note that `$PID` and env vars are expanded in config paths.\n\n### Register fails with \"already exists\"\n**Symptom**: `{\"error\":\"database already registered\"}`\n**Fix**: The database path is already being replicated. Use `/unregister` first, then `/register` with new settings.\n\n## 5. pprof Debugging\n\nAvailable pprof endpoints on the IPC socket:\n\n```bash\n# Interactive CPU profile analysis\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/profile?seconds=30 > cpu.prof\ngo tool pprof -http=:8080 cpu.prof\n\n# Goroutine dump (useful for deadlock investigation)\ncurl --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/goroutine?debug=2\n\n# Memory allocation profile\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/heap > heap.prof\ngo tool pprof -http=:8080 heap.prof\n\n# Execution trace (captures scheduler, GC, goroutine events)\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/trace?seconds=5 > trace.out\ngo tool trace trace.out\n```\n",".claude/commands/debug-wal.md":"Debug WAL monitoring issues in Litestream. This command helps diagnose problems with WAL change detection, checkpointing, and replication triggers.\n\nFirst, understand the symptoms:\n- Is replication not triggering on changes?\n- Are checkpoints failing or not happening?\n- Is the WAL growing unbounded?\n\nThen debug the monitoring system:\n\n1. **Check monitor goroutine** (db.go:1499):\n```go\n// Verify monitor is running\nfunc (db *DB) monitor() {\n ticker := time.NewTicker(db.MonitorInterval) // Default: 1s\n // Check if ticker is firing\n // Verify checkWAL() is being called\n}\n```\n\n2. **Verify WAL change detection**:\n```go\n// Check if WAL changes are detected\nfunc (db *DB) checkWAL() (bool, error) {\n // Get WAL size and checksum\n // Compare with previous values\n // Should return true if changed\n}\n```\n\n3. **Debug checkpoint triggers**:\n```go\n// Check checkpoint thresholds\nMinCheckpointPageN int // Default: 1000 pages\nTruncatePageN int // Default: 121359 pages\n\n// Verify WAL page count\nwalPageCount := db.WALPageCount()\nif walPageCount > db.MinCheckpointPageN {\n // Should trigger passive checkpoint\n}\nif walPageCount > db.TruncatePageN {\n // Should trigger truncate checkpoint (emergency brake)\n}\n```\n\n4. **Check long-running read transaction**:\n```go\n// Ensure rtx is maintained\nif db.rtx == nil {\n // Read transaction lost - replication may fail\n}\n```\n\n5. **Monitor notification channel**:\n```go\n// Check if replicas are notified\nselect {\ncase <-db.notify:\n // WAL change detected\ndefault:\n // No changes\n}\n```\n\nCommon issues to check:\n- MonitorInterval too long (default 1s)\n- Checkpoint failing due to active transactions\n- Read transaction preventing checkpoint\n- Notify channel not triggering replicas\n- WAL file permissions issues\n\nDebug commands:\n```sql\n-- Check WAL status\nPRAGMA wal_checkpoint;\nPRAGMA journal_mode;\nPRAGMA page_count;\nPRAGMA wal_autocheckpoint;\n\n-- Check for locks\nSELECT * FROM pragma_lock_status();\n```\n\nTesting WAL monitoring:\n```go\nfunc TestDB_WALMonitoring(t *testing.T) {\n db := setupTestDB(t)\n\n // Set fast monitoring for test\n db.MonitorInterval = 10 * time.Millisecond\n\n // Write data\n writeTestData(t, db, 100)\n\n // Wait for notification\n select {\n case <-db.notify:\n // Success\n case <-time.After(1 * time.Second):\n t.Error(\"WAL change not detected\")\n }\n}\n```\n\nMonitor with logging:\n```go\nslog.Debug(\"wal check\",\n \"size\", walInfo.Size,\n \"checksum\", walInfo.Checksum,\n \"pages\", walInfo.PageCount)\n```\n",".claude/commands/fix-common-issues.md":"---\ndescription: Fix common Litestream issues\n---\n\n# Fix Common Issues Command\n\nDiagnose and fix common issues in Litestream deployments.\n\n## Issue 1: Lock Page Not Being Skipped\n\n**Symptom**: Errors or corruption with databases >1GB\n\n**Check**:\n```bash\n# Find lock page references\ngrep -r \"LockPgno\" --include=\"*.go\"\n```\n\n**Fix**:\n```go\n// Ensure all page iterations skip lock page\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n## Issue 2: Race Condition in Replica Position\n\n**Symptom**: Data races detected, inconsistent position tracking\n\n**Check**:\n```bash\ngo test -race -v -run TestReplica_Sync ./...\n```\n\n**Fix**:\n```go\n// Change from RLock to Lock for writes\nfunc (r *Replica) SetPos(pos ltx.Pos) {\n r.mu.Lock() // NOT RLock!\n defer r.mu.Unlock()\n r.pos = pos\n}\n```\n\n## Issue 3: Eventual Consistency Issues\n\n**Symptom**: Compaction failures, partial file reads\n\n**Check**:\n```bash\n# Look for remote reads during compaction\ngrep -r \"OpenLTXFile\" db.go | grep -v \"os.Open\"\n```\n\n**Fix**:\n```go\n// Always try local first\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil\n}\n// Only fall back to remote if local doesn't exist\nreturn replica.Client.OpenLTXFile(...)\n```\n\n## Issue 4: CreatedAt Timestamp Loss\n\n**Symptom**: Point-in-time recovery lacks accurate timestamps\n\n**Check**:\n```go\ninfo, err := client.WriteLTXFile(ctx, level, minTXID, maxTXID, r)\nif err != nil {\n t.Fatal(err)\n}\nif info.CreatedAt.IsZero() {\n t.Fatal(\"CreatedAt not set\")\n}\n```\n\n**Fix**:\n```go\n// Ensure storage metadata is copied into the returned FileInfo\nmodTime := resp.LastModified\ninfo.CreatedAt = modTime\n```\n\n## Issue 5: Non-Atomic File Writes\n\n**Symptom**: Partial files, corruption on crash\n\n**Check**:\n```bash\n# Find direct writes without temp files\ngrep -r \"os.Create\\|os.WriteFile\" --include=\"*.go\"\n```\n\n**Fix**:\n```go\n// Write to temp, then rename\ntmpPath := path + \".tmp\"\nif err := os.WriteFile(tmpPath, data, 0644); err != nil {\n return err\n}\nreturn os.Rename(tmpPath, path)\n```\n\n## Issue 6: WAL Checkpoint Blocking\n\n**Symptom**: WAL grows indefinitely, database locks\n\n**Check**:\n```sql\n-- Check WAL size\nPRAGMA wal_checkpoint(PASSIVE);\nSELECT page_count * page_size FROM pragma_page_count(), pragma_page_size();\n```\n\n**Fix**:\n```go\n// Release read transaction periodically\ndb.rtx.Rollback()\ndb.rtx = nil\n// Checkpoint\ndb.db.Exec(\"PRAGMA wal_checkpoint(RESTART)\")\n// Restart read transaction\ndb.initReadTx()\n```\n\n## Issue 7: Memory Leaks\n\n**Symptom**: Growing memory usage over time\n\n**Check**:\n```bash\n# Generate heap profile\ngo test -memprofile=mem.prof -run=XXX -bench=.\ngo tool pprof -top mem.prof\n```\n\n**Fix**:\n```go\n// Use sync.Pool for buffers\nvar pagePool = sync.Pool{\n New: func() interface{} {\n b := make([]byte, pageSize)\n return &b\n },\n}\n\n// Close resources properly\ndefer func() {\n if f != nil {\n f.Close()\n }\n}()\n```\n\n## Issue 8: Corrupted or Missing LTX Files\n\n**Symptom**: Sync failures with errors like \"ltx validation failed\", \"nonsequential page numbers\",\n\"non-contiguous transaction files\", or persistent sync retry backoff loops after unclean shutdowns.\n\n**Check**:\n```bash\n# Look for LTXError messages in logs\nrg -i \"ltx.*error|reset.*local|auto.recover\" /var/log/litestream.log\n\n# Check if meta directory has corrupted state (note dot prefix)\nls -la /path/to/.database.db-litestream/ltx/\n```\n\n**Fix - Manual Reset**:\n```bash\n# Clears local LTX state, forces fresh snapshot on next sync\n# Database file is NOT modified\nlitestream reset /path/to/database.db\n\n# With explicit config file\nlitestream reset -config /etc/litestream.yml /path/to/database.db\n```\n\n**Fix - Automatic Recovery**:\n```yaml\n# Add to replica config in litestream.yml\ndbs:\n - path: /path/to/database.db\n replicas:\n - url: s3://bucket/path\n auto-recover: true # Automatically resets on LTX errors\n```\n\n**When to use which**: Use `auto-recover` for unattended deployments where automatic recovery\nis preferred over manual intervention. Use manual `reset` when you want to investigate the\ncorruption first. `auto-recover` is disabled by default because resetting discards local LTX\nhistory, which may reduce point-in-time restore granularity.\n\n**Reference**: `cmd/litestream/reset.go`, `replica.go` (auto-recover logic), `db.go` (`ResetLocalState`)\n\n## Issue 9: IPC Socket Connection Failures\n\n**Symptom**: `connection refused` or `no such file or directory` when using the control socket\n\n**Check**:\n```bash\n# Verify socket exists and has correct permissions\nls -la /var/run/litestream.sock\n\n# Verify socket is enabled in config\nrg -A3 'socket:' /etc/litestream.yml\n\n# Check if litestream process is running\npgrep -a litestream\n```\n\n**Fix**:\n```yaml\n# Enable socket in litestream.yml\nsocket:\n enabled: true\n path: /var/run/litestream.sock\n permissions: 0600\n```\n\n**Stale socket**: If litestream crashed, the socket file may still exist. The process creates a new socket on startup and will fail if the stale file exists. Remove it manually:\n```bash\nrm /var/run/litestream.sock\n```\n\n**Reference**: `server.go` (`SocketConfig`, `Server.Start`)\n\n## Issue 10: Backup Files Accumulating (Retention Disabled)\n\n**Symptom**: Storage usage growing unbounded, old LTX files never deleted\n\n**Check**:\n```bash\n# Look for retention warning in logs\nrg \"retention disabled\" /var/log/litestream.log\n```\n\n**Fix**:\n```yaml\n# Option 1: Re-enable Litestream retention (default)\nretention:\n enabled: true\n\n# Option 2: Keep disabled, but configure cloud lifecycle policies\n# Example: S3 lifecycle rule to expire objects in ltx/ prefix after 30 days\n```\n\n**Reference**: `store.go` (`SetRetentionEnabled`), `compactor.go` (`RetentionEnabled`), `cmd/litestream/replicate.go:295`\n\n## Issue 11: v0.3.x Restore Not Finding Backups\n\n**Symptom**: Restore fails with \"no snapshots available\" but v0.3.x backups exist in storage\n\n**Check**:\n```bash\n# Verify v0.3.x backup structure exists\naws s3 ls s3://bucket/path/generations/ --recursive | head -20\n```\n\n**Fix**: Ensure the backend implements `ReplicaClientV3`. The S3 backend supports this automatically. The restore process checks for v0.3.x backups when no v0.4.x+ backup is found.\n\n**Reference**: `v3.go` (`ReplicaClientV3` interface), `s3/replica_client.go`\n\n## Diagnostic Commands\n\n```bash\n# Check database integrity\nsqlite3 database.db \"PRAGMA integrity_check;\"\n\n# List replicated LTX files\nlitestream ltx /path/to/db.sqlite\n\n# Check replication status\nlitestream databases\n\n# Reset corrupted local state\nlitestream reset /path/to/database.db\n\n# Test restoration\nlitestream restore -o test.db [replica-url]\n\n# IPC socket diagnostics (requires socket.enabled: true)\ncurl --unix-socket /var/run/litestream.sock http://localhost/info\ncurl --unix-socket /var/run/litestream.sock http://localhost/list\ncurl --unix-socket /var/run/litestream.sock \"http://localhost/txid?path=/path/to/db\"\n```\n\n## Prevention Checklist\n\n- [ ] Always test with databases >1GB\n- [ ] Run with race detector in CI\n- [ ] Test all page sizes (4KB, 8KB, 16KB, 32KB)\n- [ ] Verify eventual consistency handling\n- [ ] Check for proper locking (Lock vs RLock)\n- [ ] Ensure atomic file operations\n- [ ] Preserve timestamps in compaction\n",".claude/commands/run-comprehensive-tests.md":"---\ndescription: Run comprehensive test suite for Litestream\n---\n\n# Run Comprehensive Tests Command\n\nExecute a full test suite including unit tests, integration tests, race detection, and large database tests.\n\n## Quick Test Suite\n\n```bash\n# Basic tests with race detection\ngo test -race -v ./...\n\n# With coverage\ngo test -race -cover -v ./...\n```\n\n## Full Test Suite\n\n### 1. Unit Tests\n```bash\necho \"=== Running Unit Tests ===\"\ngo test -v ./... -short\n```\n\n### 2. Race Condition Tests\n```bash\necho \"=== Testing for Race Conditions ===\"\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\ngo test -race -v ./...\n```\n\n### 3. Integration Tests\n```bash\necho \"=== Running Integration Tests ===\"\n\n# S3 (requires AWS credentials)\nAWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy \\\n go test -v ./replica_client_test.go -integration s3\n\n# Google Cloud Storage (requires credentials)\nGOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json \\\n go test -v ./replica_client_test.go -integration gcs\n\n# Azure Blob Storage\nAZURE_STORAGE_ACCOUNT=xxx AZURE_STORAGE_KEY=yyy \\\n go test -v ./replica_client_test.go -integration abs\n\n# SFTP (requires SSH server)\ngo test -v ./replica_client_test.go -integration sftp\n\n# File system (always available)\ngo test -v ./replica_client_test.go -integration file\n```\n\n### 4. Large Database Tests (>1GB)\n```bash\necho \"=== Testing Large Databases ===\"\n\n# Create test database for each page size\nfor pagesize in 4096 8192 16384 32768; do\n echo \"Testing page size: $pagesize\"\n\n # Create >1GB database\n sqlite3 test-${pagesize}.db <70% for core packages\n✅ Lock page correctly skipped for all page sizes\n✅ Restoration works for databases >1GB\n✅ No memory leaks in benchmarks\n",".claude/commands/test-compaction.md":"Test Litestream compaction logic. This command helps test and debug compaction issues, especially with eventually consistent storage backends.\n\nFirst, understand the test scenario:\n- What storage backend needs testing?\n- What size database is involved?\n- Are there eventual consistency concerns?\n\nThen create comprehensive tests:\n\n1. **Test basic compaction**:\n```go\nfunc TestCompaction_Basic(t *testing.T) {\n // Create multiple LTX files at level 0\n // Run compaction to level 1\n // Verify merged file is correct\n}\n```\n\n2. **Test with eventual consistency**:\n```go\nfunc TestStore_CompactDB_RemotePartialRead(t *testing.T) {\n // Use mock client that returns partial data initially\n // Verify compaction prefers local files\n // Ensure no corruption occurs\n}\n```\n\n3. **Test lock page handling during compaction**:\n```go\nfunc TestCompaction_LockPage(t *testing.T) {\n // Create database > 1GB\n // Compact with data around lock page\n // Verify lock page is skipped (page at 0x40000000)\n}\n```\n\n4. **Test timestamp preservation**:\n```go\nfunc TestCompaction_PreserveTimestamps(t *testing.T) {\n // Compact files with different CreatedAt times\n // Verify earliest timestamp is preserved\n}\n```\n\nKey areas to test:\n- Reading from local files first (db.go:1280-1294)\n- Skipping lock page at 1GB boundary\n- Preserving CreatedAt timestamps\n- Handling partial/incomplete remote files\n- Concurrent compaction safety\n\nRun with race detector:\n```bash\ngo test -race -v -run TestStore_CompactDB ./...\n```\n\nUse the test harness for large databases:\n```bash\n./bin/litestream-test populate -db test.db -target-size 1.5GB\n./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica\n```\n",".claude/commands/trace-replication.md":"Trace the complete replication flow in Litestream. This command helps understand how changes flow from SQLite through to storage backends.\n\nFollow the replication path step by step:\n\n1. **Application writes to SQLite**:\n```sql\n-- Application performs write\nINSERT INTO table VALUES (...);\n-- SQLite appends to WAL file\n```\n\n2. **DB.monitor() syncs the shadow WAL** (db.go:1499):\n```go\nticker := time.NewTicker(db.MonitorInterval) // default 1s\nfor {\n select {\n case <-db.ctx.Done():\n return\n case <-ticker.C:\n }\n\n if err := db.Sync(db.ctx); err != nil && !errors.Is(err, context.Canceled) {\n db.Logger.Error(\"sync error\", \"error\", err)\n }\n}\n```\n\n3. **Replica.monitor() responds** (replica.go):\n```go\nticker := time.NewTicker(r.SyncInterval)\ndefer ticker.Stop()\n\nnotify := r.db.Notify()\nfor {\n select {\n case <-ctx.Done():\n return\n case <-ticker.C:\n // Enforce minimum sync interval\n case <-notify:\n // WAL changed, time to sync\n }\n\n notify = r.db.Notify()\n\n if err := r.Sync(ctx); err != nil && !errors.Is(err, context.Canceled) {\n r.Logger().Error(\"monitor error\", \"error\", err)\n }\n}\n```\n\n4. **Replica.Sync() uploads new L0 files** (replica.go):\n```go\n// Determine local database position\ndpos, err := r.db.Pos()\nif err != nil {\n return err\n}\nif dpos.IsZero() {\n return fmt.Errorf(\"no position, waiting for data\")\n}\n\n// Upload each unreplicated L0 file\nfor txID := r.Pos().TXID + 1; txID <= dpos.TXID; txID = r.Pos().TXID + 1 {\n if err := r.uploadLTXFile(ctx, 0, txID, txID); err != nil {\n return err\n }\n r.SetPos(ltx.Pos{TXID: txID})\n}\n```\n\n5. **ReplicaClient uploads to storage**:\n```go\nfunc (c *S3Client) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {\n // Stream LTX data to storage and return metadata (size, CreatedAt, checksums)\n}\n```\n\n6. **Checkpoint when thresholds are hit**:\n```go\nif walPageCount > db.MinCheckpointPageN {\n db.Checkpoint(ctx, litestream.CheckpointModePassive)\n}\nif walPageCount > db.TruncatePageN {\n db.Checkpoint(ctx, litestream.CheckpointModeTruncate)\n}\n```\n\nKey synchronization points:\n- WAL monitoring (1s intervals)\n- Replica sync (configurable, default 1s)\n- Checkpoint triggers (page thresholds)\n- Compaction (hourly/daily)\n\nTrace with logging:\n```go\n// Enable debug logging\nslog.SetLogLevel(slog.LevelDebug)\n\n// Key log points:\nslog.Debug(\"wal changed\", \"size\", walSize)\nslog.Debug(\"syncing replica\", \"pos\", r.Pos())\nslog.Debug(\"ltx uploaded\", \"txid\", maxTXID)\nslog.Debug(\"checkpoint complete\", \"mode\", mode)\n```\n\nPerformance metrics to monitor:\n- WAL growth rate\n- Sync latency\n- Upload throughput\n- Checkpoint frequency\n- Compaction duration\n\nCommon bottlenecks:\n1. Slow storage uploads\n2. Large transactions causing big LTX files\n3. Long-running read transactions blocking checkpoints\n4. Eventual consistency delays\n5. Network latency to storage\n\nTest replication flow:\n```bash\n# Start replication with verbose logging\nlitestream replicate -v\n\n# In another terminal, write to database\nsqlite3 test.db \"INSERT INTO test VALUES (1, 'data');\"\n\n# Watch logs for flow:\n# - WAL change detected\n# - Replica sync triggered\n# - LTX file uploaded\n# - Position updated\n```\n\nVerify replication:\n```bash\n# List replicated files\naws s3 ls s3://bucket/path/ltx/0000/\n\n# Restore and verify\nlitestream restore -o restored.db s3://bucket/path\nsqlite3 restored.db \"SELECT * FROM test;\"\n```\n",".claude/commands/validate-replica.md":"Validate a ReplicaClient implementation in Litestream. This command helps ensure a replica client correctly implements the interface and handles edge cases.\n\nFirst, identify what needs validation:\n- Which replica client implementation?\n- What storage backend specifics?\n- Any known issues or concerns?\n\nThen validate the implementation:\n\n1. **Interface compliance check**:\n\n ```go\n // Ensure all methods are implemented\n var _ litestream.ReplicaClient = (*YourClient)(nil)\n ```\n\n1. **Verify error types**:\n\n ```go\n // OpenLTXFile must return os.ErrNotExist for missing files\n _, err := client.OpenLTXFile(ctx, 0, 999, 999, 0, 0)\n if !errors.Is(err, os.ErrNotExist) {\n t.Errorf(\"Expected os.ErrNotExist, got %v\", err)\n }\n ```\n\n1. **Test partial reads**:\n\n ```go\n // Must support offset and size parameters\n rc, err := client.OpenLTXFile(ctx, 0, 1, 100, 50, 25)\n data, _ := io.ReadAll(rc)\n if len(data) != 25 {\n t.Errorf(\"Expected 25 bytes, got %d\", len(data))\n }\n ```\n\n1. **Verify timestamp preservation**:\n\n ```go\n // CreatedAt should reflect remote object metadata (or upload time)\n start := time.Now()\n info, _ := client.WriteLTXFile(ctx, 0, 1, 100, reader)\n if info.CreatedAt.IsZero() || info.CreatedAt.Before(start.Add(-time.Second)) {\n t.Error(\"unexpected CreatedAt timestamp\")\n }\n ```\n\n1. **Test eventual consistency handling**:\n - Implement retry logic for transient failures\n - Handle partial file availability\n - Verify write-after-write consistency\n\n1. **Validate cleanup**:\n\n ```go\n // DeleteAll must remove everything\n err := client.DeleteAll(ctx)\n files, _ := client.LTXFiles(ctx, 0, 0, false)\n if files.Next() {\n t.Error(\"Files remain after DeleteAll\")\n }\n ```\n\nKey validation points:\n- Proper error types (os.ErrNotExist, os.ErrPermission)\n- Context cancellation handling\n- Concurrent operation safety\n- Iterator doesn't load all files at once\n- Proper path construction for storage backend\n\nRun integration tests:\n```bash\ngo test -v ./[backend]/replica_client_test.go -integration\n```\n",".github/copilot-instructions.md":"../AGENTS.md","AGENTS.md":"# AGENTS.md - Litestream AI Agent Guide\n\nLitestream is a disaster recovery tool for SQLite that runs as a background process, monitors the WAL, converts changes to immutable LTX files, and replicates them to cloud storage. It uses `modernc.org/sqlite` (pure Go, no CGO required).\n\n## Before You Start\n\n1. Read [AI_PR_GUIDE.md](AI_PR_GUIDE.md) for contribution requirements\n2. Check [CONTRIBUTING.md](CONTRIBUTING.md) for what we accept (bug fixes welcome, features need discussion)\n3. Review recent PRs for current patterns\n\n## Critical Rules\n\n- **Lock page at 1GB**: SQLite reserves page at 0x40000000. Always skip it. See [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md)\n- **LTX files are immutable**: Never modify after creation. See [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md)\n- **Single replica per database**: Each DB replicates to exactly one destination\n- **Use `litestream ltx`**: Not `litestream wal` (deprecated)\n- **Use `litestream reset`**: Clears corrupted local LTX state for a database. See `cmd/litestream/reset.go`\n- **`auto-recover` config**: Replica option that automatically resets local state on LTX errors. Disabled by default. See `replica.go`\n- **Retention enabled by default**: `Store.RetentionEnabled` is `true` by default. Disable only when cloud lifecycle policies handle cleanup. See `store.go`\n- **IPC socket disabled by default**: Control socket is off by default. Enable with `socket.enabled: true` in config. See `server.go`\n- **`$PID` config expansion**: Config files support `$PID` to expand to the current process ID, plus standard `$ENV_VAR` expansion. See `cmd/litestream/main.go`\n- **`litestream ltx -level`**: Use `-level 0`–`9` or `-level all` to inspect specific compaction levels. See `cmd/litestream/ltx.go`\n- **Return errors, don't log them**: Always return errors to callers. Never `log.Printf(err)` and continue — this silently hides failures in a disaster recovery tool. Only use DEBUG log for best-effort operations where failure doesn't affect correctness and a valid fallback exists (e.g., reading SHM mxFrame optimization hint). See [docs/PATTERNS.md](docs/PATTERNS.md#error-handling)\n\n## Layer Boundaries\n\n| Layer | File | Responsibility |\n|-------|------|----------------|\n| DB | `db.go` | Database state, restoration, WAL monitoring, library API (`SyncStatus`, `SyncAndWait`, `EnsureExists`) |\n| Replica | `replica.go` | Replication mechanics only |\n| Storage | `**/replica_client.go` | Backend implementations (includes `ReplicaClientV3` for v0.3.x restore) |\n| IPC | `server.go` | Unix socket control API (register/unregister, /txid, pprof) |\n| Leasing | `leaser.go`, `s3/leaser.go` | Distributed lease acquisition via conditional writes |\n\nDatabase state logic belongs in DB layer, not Replica layer.\n\n## Quick Reference\n\n**Build:**\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\n```\n\n**Code quality:**\n\n```bash\npre-commit run --all-files\n```\n\n## Documentation\n\n| Document | When to Read |\n|----------|--------------|\n| [docs/PATTERNS.md](docs/PATTERNS.md) | Code patterns and anti-patterns |\n| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Deep component details |\n| [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) | WAL format, lock page |\n| [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md) | Replication format |\n| [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md) | Test strategies |\n| [docs/REPLICA_CLIENT_GUIDE.md](docs/REPLICA_CLIENT_GUIDE.md) | Adding storage backends |\n| [docs/PROVIDER_COMPATIBILITY.md](docs/PROVIDER_COMPATIBILITY.md) | Provider-specific S3/cloud configs |\n\n## Checklist\n\nBefore submitting changes:\n\n- [ ] Read relevant docs above\n- [ ] Follow patterns in [docs/PATTERNS.md](docs/PATTERNS.md)\n- [ ] Test with race detector (`go test -race`)\n- [ ] Run `pre-commit run --all-files`\n- [ ] For page iteration: test with >1GB databases\n- [ ] Show investigation evidence in PR (see [AI_PR_GUIDE.md](AI_PR_GUIDE.md))\n","CLAUDE.md":"# CLAUDE.md - Claude Code Configuration\n\nClaude-specific optimizations for Litestream. See [AGENTS.md](AGENTS.md) for project documentation.\n\n## Critical Coding Rules\n\n- **Always return errors, never log and continue.** The only exception is DEBUG logging for best-effort operations that don't affect correctness. This is the #1 cause of PR review feedback. See [docs/PATTERNS.md](docs/PATTERNS.md#error-handling) for the full decision framework.\n\n## Context Window\n\nWith Claude's large context window, load documentation as needed:\n\n- Start with [AGENTS.md](AGENTS.md) for overview and checklist\n- Load [docs/PATTERNS.md](docs/PATTERNS.md) when writing code\n- Load [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) for WAL/page work\n- Load [docs/PROVIDER_COMPATIBILITY.md](docs/PROVIDER_COMPATIBILITY.md) for storage backend configs\n\n## Claude-Specific Resources\n\n### Specialized Agents (.claude/agents/)\n\n- `sqlite-expert.md` - SQLite WAL and page management\n- `replica-client-developer.md` - Storage backend implementation\n- `ltx-compaction-specialist.md` - LTX format and compaction\n- `test-engineer.md` - Testing strategies\n- `performance-optimizer.md` - Performance optimization\n\n### Commands (.claude/commands/)\n\n- `/analyze-ltx` - Analyze LTX file structure\n- `/debug-ipc` - Debug IPC Unix socket issues\n- `/debug-wal` - Debug WAL replication issues\n- `/test-compaction` - Test compaction scenarios\n- `/trace-replication` - Trace replication flow\n- `/validate-replica` - Validate replica client\n- `/add-storage-backend` - Create new storage backend\n- `/fix-common-issues` - Diagnose common problems\n- `/run-comprehensive-tests` - Execute full test suite\n\n## Quick Commands\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\npre-commit run --all-files\n```\n","docs/ARCHITECTURE.md":"# Litestream Architecture - Technical Deep Dive\n\n## Table of Contents\n- [System Layers](#system-layers)\n- [Core Components](#core-components)\n- [IPC Server & Control Socket](#ipc-server--control-socket)\n- [Distributed Leasing](#distributed-leasing)\n- [Library Convenience Methods](#library-convenience-methods)\n- [LTX File Format](#ltx-file-format)\n- [WAL Monitoring Mechanism](#wal-monitoring-mechanism)\n- [Compaction Process](#compaction-process)\n- [Transaction Management](#transaction-management)\n- [Concurrency Model](#concurrency-model)\n- [State Management](#state-management)\n- [Initialization Flow](#initialization-flow)\n- [Error Handling](#error-handling)\n\n## System Layers\n\nLitestream follows a layered architecture with clear separation of concerns:\n\n```mermaid\ngraph TB\n subgraph \"Application Layer\"\n CLI[CLI Commands
cmd/litestream/]\n Config[Configuration
config.go]\n end\n\n subgraph \"Core Layer\"\n Store[Store Manager
store.go]\n DB[Database Manager
db.go]\n Replica[Replica Manager
replica.go]\n end\n\n subgraph \"Storage Abstraction\"\n RC[ReplicaClient Interface
replica_client.go]\n end\n\n subgraph \"Storage Implementations\"\n S3[s3/replica_client.go]\n GCS[gs/replica_client.go]\n ABS[abs/replica_client.go]\n OSS[oss/replica_client.go]\n File[file/replica_client.go]\n SFTP[sftp/replica_client.go]\n NATS[nats/replica_client.go]\n end\n\n subgraph \"External\"\n SQLite[SQLite Database]\n Cloud[Cloud Storage]\n end\n\n CLI --> Store\n Store --> DB\n DB --> Replica\n Replica --> RC\n RC --> S3\n RC --> GCS\n RC --> ABS\n RC --> OSS\n RC --> File\n RC --> SFTP\n RC --> NATS\n DB <--> SQLite\n S3 --> Cloud\n GCS --> Cloud\n ABS --> Cloud\n OSS --> Cloud\n```\n\n### Layer Responsibilities\n\n#### 1. Application Layer\n- **CLI Commands**: User interface for operations (replicate, restore, etc.)\n- **Configuration**: YAML/environment variable parsing and validation\n\n#### 2. Core Layer\n- **Store**: Multi-database coordination, compaction scheduling\n- **DB**: Single database management, WAL monitoring, checkpointing\n- **Replica**: Replication to single destination, position tracking\n\n#### 3. Storage Abstraction\n- **ReplicaClient Interface**: Uniform API for all storage backends\n\n#### 4. Storage Implementations\n- Backend-specific logic (authentication, retries, optimizations)\n\n#### 5. IPC Layer\n- **Server**: Unix socket HTTP server for runtime control (`server.go`)\n- **Leaser**: Distributed lease interface for coordination (`leaser.go`, `s3/leaser.go`)\n\n## Core Components\n\n### DB Component (db.go)\n\nThe DB component is the heart of Litestream, managing a single SQLite database:\n\n```go\ntype DB struct {\n // Core fields\n path string // Database file path\n metaPath string // Metadata directory path\n db *sql.DB // SQLite connection\n f *os.File // Long-running file descriptor\n rtx *sql.Tx // Long-running read transaction\n pageSize int // Database page size\n\n // Synchronization\n mu sync.RWMutex // Protects struct fields\n chkMu sync.RWMutex // Checkpoint lock\n notify chan struct{} // WAL change notifications\n\n // Lifecycle\n ctx context.Context\n cancel func()\n wg sync.WaitGroup\n\n // Configuration\n MinCheckpointPageN int // Min pages for passive checkpoint\n TruncatePageN int // Pages before emergency truncate checkpoint\n CheckpointInterval time.Duration // Time-based passive checkpoint interval\n MonitorInterval time.Duration // WAL monitoring frequency\n // Note: MaxCheckpointPageN removed (RESTART mode disabled due to #724)\n\n // Metrics\n dbSizeGauge prometheus.Gauge\n walSizeGauge prometheus.Gauge\n txIDGauge prometheus.Gauge\n}\n```\n\n#### Key Methods\n\n```go\n// Lifecycle\nfunc (db *DB) Open() error\nfunc (db *DB) Close(ctx context.Context) error\n\n// Monitoring\nfunc (db *DB) monitor() // Background WAL monitoring\nfunc (db *DB) checkWAL() (bool, error) // Check for WAL changes\n\n// Checkpointing\nfunc (db *DB) Checkpoint(mode string) error\nfunc (db *DB) autoCheckpoint() error\n\n// Replication\nfunc (db *DB) WALReader(pgno uint32) (io.ReadCloser, error)\nfunc (db *DB) Sync(ctx context.Context) error\n\n// Compaction\nfunc (db *DB) Compact(ctx context.Context, destLevel int) (*ltx.FileInfo, error)\n```\n\n### Replica Component (replica.go)\n\nManages replication to a single destination:\n\n```go\ntype Replica struct {\n db *DB // Parent database\n Client ReplicaClient // Storage backend client\n\n mu sync.RWMutex\n pos ltx.Pos // Current replication position\n\n // Configuration\n SyncInterval time.Duration\n MonitorEnabled bool\n\n // Lifecycle\n cancel func()\n wg sync.WaitGroup\n}\n```\n\n#### Replication Position\n\n```go\ntype Pos struct {\n TXID TXID // Transaction ID\n PageNo uint32 // Page number within transaction\n Checksum uint64 // Running checksum\n}\n```\n\n### Store Component (store.go)\n\nCoordinates multiple databases and manages system-wide resources:\n\n```go\ntype Store struct {\n mu sync.Mutex\n dbs []*DB\n levels CompactionLevels\n\n // Configuration\n SnapshotInterval time.Duration\n SnapshotRetention time.Duration\n L0Retention time.Duration\n L0RetentionCheckInterval time.Duration\n CompactionMonitorEnabled bool\n\n // Lifecycle\n ctx context.Context\n cancel func()\n wg sync.WaitGroup\n}\n```\n\n## IPC Server & Control Socket\n\nLitestream exposes a Unix socket HTTP server for runtime control (`server.go`).\n\n### Socket Configuration\n\n```go\ntype SocketConfig struct {\n Enabled bool `yaml:\"enabled\"` // Default: false\n Path string `yaml:\"path\"` // Default: \"/var/run/litestream.sock\"\n Permissions uint32 `yaml:\"permissions\"` // Default: 0600\n}\n```\n\n### HTTP Endpoints\n\nAll endpoints are served over the Unix socket via Go's `net/http` mux:\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `POST` | `/register` | Add a database at runtime |\n| `POST` | `/unregister` | Remove a database at runtime |\n| `GET` | `/txid?path=` | Get current transaction ID for a database |\n| `GET` | `/list` | List all managed databases |\n| `GET` | `/info` | Server info (version, PID, uptime, database count) |\n| `POST` | `/start` | Start replication for a database |\n| `POST` | `/stop` | Stop replication for a database |\n| `GET` | `/debug/pprof/*` | Standard Go pprof endpoints |\n\n### Request/Response Types\n\n```go\ntype RegisterDatabaseRequest struct {\n Path string `json:\"path\"`\n ReplicaURL string `json:\"replica_url\"`\n}\n\ntype UnregisterDatabaseRequest struct {\n Path string `json:\"path\"`\n Timeout int `json:\"timeout,omitempty\"` // Seconds\n}\n\ntype TXIDResponse struct {\n TXID uint64 `json:\"txid\"`\n}\n```\n\n## Distributed Leasing\n\nThe `Leaser` interface (`leaser.go`) enables distributed coordination so multiple Litestream instances can safely share a replica destination.\n\n### Interface\n\n```go\ntype Leaser interface {\n Type() string\n AcquireLease(ctx context.Context) (*Lease, error)\n RenewLease(ctx context.Context, lease *Lease) (*Lease, error)\n ReleaseLease(ctx context.Context, lease *Lease) error\n}\n\ntype Lease struct {\n Generation int64 `json:\"generation\"`\n ExpiresAt time.Time `json:\"expires_at\"`\n Owner string `json:\"owner,omitempty\"`\n ETag string `json:\"-\"`\n}\n```\n\n### S3 Implementation (`s3/leaser.go`)\n\n- **Defaults**: `DefaultLeaseTTL = 30s`, `DefaultLeasePath = \"lock.json\"`\n- **Owner format**: `hostname:pid` (falls back to `pid-N` if hostname unavailable)\n- **Conditional writes**: Uses `If-Match`/`If-None-Match` on S3 PutObject to prevent races\n- **Release**: Uses `DeleteObject` with `If-Match` to ensure only the holder can release\n- **Error types**: `ErrLeaseNotHeld`, `ErrLeaseAlreadyReleased`, `LeaseExistsError{Owner, ExpiresAt}`\n\n### Acquisition Flow\n\n1. Read existing lease from S3 (`lock.json`)\n2. If lease exists and is not expired, return `LeaseExistsError`\n3. Write new lease with `If-None-Match: *` (first acquire) or `If-Match: ` (expired takeover)\n4. If `PreconditionFailed` (412), another instance acquired first\n5. Return lease with ETag for subsequent renewal\n\n## Library Convenience Methods\n\nThese methods on `DB` (`db.go`) simplify common operations when Litestream is used as a library:\n\n```go\ntype SyncStatus struct {\n LocalTXID ltx.TXID // Local transaction ID\n RemoteTXID ltx.TXID // Remote transaction ID\n InSync bool // true if LocalTXID > 0 and equal to RemoteTXID\n}\n\nfunc (db *DB) SyncStatus(ctx context.Context) (SyncStatus, error)\nfunc (db *DB) SyncAndWait(ctx context.Context) error\nfunc (db *DB) EnsureExists(ctx context.Context) error\n```\n\n- **SyncStatus**: Compares local TXID against remote replica position (performs I/O to query remote)\n- **SyncAndWait**: Runs `db.Sync()` (WAL to LTX) then `db.Replica.Sync()` (LTX to remote), blocking until both complete\n- **EnsureExists**: Restores database from replica if local file doesn't exist; no-op if file exists or no backup available. Must be called before `Open()`\n\n## LTX File Format\n\nLTX (Log Transaction) files are immutable files containing database changes:\n\n```\n+------------------+\n| Header | Fixed size header with metadata\n+------------------+\n| |\n| Page Frames | Variable number of page frames\n| |\n+------------------+\n| Page Index | Index for efficient page lookup\n+------------------+\n| Trailer | Metadata and checksums\n+------------------+\n```\n\n### Header Structure\n\n```go\ntype Header struct {\n Magic [4]byte // \"LTX\\x00\"\n Version uint32 // Format version\n PageSize uint32 // Database page size\n MinTXID TXID // Starting transaction ID\n MaxTXID TXID // Ending transaction ID\n Timestamp int64 // Creation timestamp\n Checksum uint64 // Header checksum\n}\n```\n\n### Page Frame Structure\n\n```go\ntype PageFrame struct {\n Header PageHeader\n Data []byte // Page data (pageSize bytes)\n}\n\ntype PageHeader struct {\n PageNo uint32 // Page number in database\n Size uint32 // Size of page data\n Checksum uint64 // Page checksum\n}\n```\n\n### Page Index\n\nBinary search tree for efficient page lookup:\n```go\ntype PageIndexElem struct {\n PageNo uint32 // Page number\n Offset int64 // Offset in file\n Size uint32 // Size of page frame\n}\n```\n\n### Trailer\n\n```go\ntype Trailer struct {\n PageIndexOffset int64 // Offset to page index\n PageIndexSize int64 // Size of page index\n PageCount uint32 // Total pages in file\n Checksum uint64 // Full file checksum\n}\n```\n\n## WAL Monitoring Mechanism\n\n### Monitor Loop (db.go:1499)\n\n```go\nfunc (db *DB) monitor() {\n ticker := time.NewTicker(db.MonitorInterval)\n defer ticker.Stop()\n\n for {\n select {\n case <-ticker.C:\n // Check WAL for changes\n changed, err := db.checkWAL()\n if err != nil {\n slog.Error(\"wal check failed\", \"error\", err)\n continue\n }\n\n if changed {\n // Notify replicas of changes\n db.notifyReplicas()\n\n // Check if checkpoint needed\n if db.shouldCheckpoint() {\n db.autoCheckpoint()\n }\n }\n\n case <-db.ctx.Done():\n return\n }\n }\n}\n```\n\n### WAL Change Detection\n\n```go\nfunc (db *DB) checkWAL() (bool, error) {\n // Get current WAL state\n walInfo, err := db.walInfo()\n if err != nil {\n return false, err\n }\n\n // Compare with previous state\n db.mu.Lock()\n changed := walInfo.Size != db.prevWALSize ||\n walInfo.Checksum != db.prevWALChecksum\n db.prevWALSize = walInfo.Size\n db.prevWALChecksum = walInfo.Checksum\n db.mu.Unlock()\n\n return changed, nil\n}\n```\n\n## Compaction Process\n\nCompaction merges multiple LTX files to reduce storage overhead:\n\n### Compaction Algorithm (store.go:189)\n\nHigh-level compaction flow:\n\n1. Determine whether the level is due for compaction (`Store.shouldCompact`).\n2. Enumerate level-`L-1` files using `ReplicaClient.LTXFiles`, preferring local\n copies via `os.Open(db.LTXPath(...))` and falling back to\n `ReplicaClient.OpenLTXFile` only when necessary.\n3. Stream the source readers through `ltx.NewCompactor`, which performs\n page-level deduplication and enforces lock-page skipping automatically.\n4. Pipe the compactor output into `ReplicaClient.WriteLTXFile` to create the\n merged LTX file for level `L`.\n5. Adjust the returned `ltx.FileInfo.CreatedAt` to the earliest timestamp from\n the source files so point-in-time recovery remains accurate.\n6. Update the cached max file info for the level and delete old L0 files when\n promoting to level 1.\n\n### Compaction Levels\n\n```go\ntype CompactionLevel struct {\n Level int // Level number (0 = raw, 1+ = compacted)\n Interval time.Duration // How often to compact from previous level\n}\n\n// Default configuration\nvar DefaultCompactionLevels = CompactionLevels{\n {Level: 0, Interval: 0}, // Raw LTX files\n {Level: 1, Interval: 1 * Hour}, // Hourly compaction\n {Level: 2, Interval: 24 * Hour}, // Daily compaction\n}\n```\n\n## Transaction Management\n\n### Long-Running Read Transaction\n\nLitestream maintains a long-running read transaction to ensure consistency:\n\n```go\nfunc (db *DB) initReadTx() error {\n // Start read transaction\n tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{\n ReadOnly: true,\n })\n if err != nil {\n return err\n }\n\n // Execute dummy query to start transaction\n var dummy string\n err = tx.QueryRow(\"SELECT ''\").Scan(&dummy)\n if err != nil {\n tx.Rollback()\n return err\n }\n\n db.rtx = tx\n return nil\n}\n```\n\n**Purpose:**\n- Prevents database from being modified during replication\n- Ensures consistent view of database\n- Allows reading historical pages from WAL\n\n### Checkpoint Coordination\n\n```go\nfunc (db *DB) Checkpoint(mode string) error {\n // Acquire checkpoint lock\n db.chkMu.Lock()\n defer db.chkMu.Unlock()\n\n // Close read transaction temporarily\n if db.rtx != nil {\n db.rtx.Rollback()\n db.rtx = nil\n }\n\n // Perform checkpoint\n _, _, err := db.db.Exec(fmt.Sprintf(\"PRAGMA wal_checkpoint(%s)\", mode))\n if err != nil {\n return err\n }\n\n // Restart read transaction\n return db.initReadTx()\n}\n```\n\n## Concurrency Model\n\n### Mutex Usage Patterns\n\n```go\n// DB struct mutexes\ntype DB struct {\n mu sync.RWMutex // Protects struct fields\n chkMu sync.RWMutex // Checkpoint coordination\n}\n\n// Replica struct mutexes\ntype Replica struct {\n mu sync.RWMutex // Protects position\n muf sync.Mutex // File descriptor lock\n}\n\n// Store struct mutex\ntype Store struct {\n mu sync.Mutex // Protects database list\n}\n```\n\n### Thundering Herd Prevention\n\n`Store.Open()` (`store.go`) limits concurrent database opens at startup:\n\n```go\ng, ctx := errgroup.WithContext(ctx)\ng.SetLimit(50) // Max 50 concurrent DB opens\nfor _, db := range s.dbs {\n db := db\n g.Go(func() error { return db.Open() })\n}\n```\n\nThis prevents OS resource exhaustion (file descriptors, memory) when hundreds of databases are configured.\n\n### Lock Ordering (Prevent Deadlocks)\n\nAlways acquire locks in this order:\n1. Store.mu\n2. DB.mu\n3. DB.chkMu\n4. Replica.mu\n\n### Goroutine Management\n\n```go\n// Start background task\nfunc (db *DB) Start() {\n db.wg.Add(1)\n go func() {\n defer db.wg.Done()\n db.monitor()\n }()\n}\n\n// Stop with timeout\nfunc (db *DB) Close(ctx context.Context) error {\n // Signal shutdown\n db.cancel()\n\n // Wait for goroutines with timeout\n done := make(chan struct{})\n go func() {\n db.wg.Wait()\n close(done)\n }()\n\n select {\n case <-done:\n return nil\n case <-ctx.Done():\n return ctx.Err()\n }\n}\n```\n\n## State Management\n\n### Database States\n\n```mermaid\nstateDiagram-v2\n [*] --> Closed\n Closed --> Opening: Open()\n Opening --> Open: Success\n Opening --> Closed: Error\n Open --> Monitoring: Start()\n Monitoring --> Syncing: Changes Detected\n Syncing --> Monitoring: Sync Complete\n Monitoring --> Checkpointing: Threshold Reached\n Checkpointing --> Monitoring: Checkpoint Complete\n Monitoring --> Closing: Close()\n Closing --> Closed: Cleanup Complete\n```\n\n### Replica States\n\n```mermaid\nstateDiagram-v2\n [*] --> Idle\n Idle --> Starting: Start()\n Starting --> Monitoring: Success\n Starting --> Idle: Error\n Monitoring --> Syncing: Timer/Changes\n Syncing --> Uploading: Have Changes\n Uploading --> Monitoring: Success\n Uploading --> Error: Failed\n Error --> Monitoring: Retry\n Monitoring --> Stopping: Stop()\n Stopping --> Idle: Cleanup\n```\n\n### Position Tracking\n\n```go\ntype Pos struct {\n TXID TXID // Current transaction ID\n PageNo uint32 // Current page number\n Checksum uint64 // Running checksum for validation\n}\n\n// Update position atomically\nfunc (r *Replica) SetPos(pos ltx.Pos) {\n r.mu.Lock() // MUST use Lock, not RLock!\n defer r.mu.Unlock()\n r.pos = pos\n}\n\n// Read position safely\nfunc (r *Replica) Pos() ltx.Pos {\n r.mu.RLock()\n defer r.mu.RUnlock()\n return r.pos\n}\n```\n\n## Initialization Flow\n\n### System Startup Sequence\n\n```mermaid\nsequenceDiagram\n participant Main\n participant Store\n participant DB\n participant Replica\n participant Monitor\n\n Main->>Store: NewStore(config)\n Store->>Store: Validate config\n\n Main->>Store: Open()\n loop For each database\n Store->>DB: NewDB(path)\n Store->>DB: Open()\n DB->>DB: Open SQLite connection\n DB->>DB: Read page size\n DB->>DB: Init metadata\n DB->>DB: Start read transaction\n\n loop For each replica\n DB->>Replica: NewReplica()\n DB->>Replica: Start()\n Replica->>Monitor: Start monitoring\n end\n end\n\n Store->>Store: Start compaction monitors\n Store-->>Main: Ready\n```\n\n### Critical Initialization Steps\n\n1. **Database Opening**\n ```go\n // Must happen in order:\n 1. Open SQLite connection\n 2. Read page size (PRAGMA page_size)\n 3. Create metadata directory\n 4. Start long-running read transaction\n 5. Initialize replicas\n 6. Start monitor goroutine\n ```\n\n2. **Replica Initialization**\n ```go\n // Must happen in order:\n 1. Create replica with client\n 2. Load previous position from metadata\n 3. Validate position against database\n 4. Start sync goroutine (if monitoring enabled)\n ```\n\n## Error Handling\n\n### Error Categories\n\n1. **Recoverable Errors**\n - Network timeouts\n - Temporary storage unavailability\n - Lock contention\n\n2. **Fatal Errors**\n - Database corruption\n - Invalid configuration\n - Disk full\n\n3. **Operational Errors**\n - Checkpoint failures\n - Compaction conflicts\n - Sync delays\n\n### Error Propagation\n\n```go\n// Bottom-up error propagation\nReplicaClient.WriteLTXFile() error\n ↓\nReplica.Sync() error\n ↓\nDB.Sync() error\n ↓\nStore.monitorDB() // Logs error, continues\n```\n\n### Retry Logic\n\n```go\nfunc (r *Replica) syncWithRetry(ctx context.Context) error {\n backoff := time.Second\n maxBackoff := time.Minute\n\n for attempt := 0; ; attempt++ {\n err := r.Sync(ctx)\n if err == nil {\n return nil\n }\n\n // Check if error is retryable\n if !isRetryable(err) {\n return err\n }\n\n // Check context\n if ctx.Err() != nil {\n return ctx.Err()\n }\n\n // Exponential backoff\n time.Sleep(backoff)\n backoff *= 2\n if backoff > maxBackoff {\n backoff = maxBackoff\n }\n }\n}\n```\n\n## Performance Characteristics\n\n### Time Complexity\n\n| Operation | Complexity | Notes |\n|-----------|------------|-------|\n| WAL Monitor | O(1) | Fixed interval check |\n| Page Write | O(1) | Append to LTX file |\n| Compaction | O(n) | n = total pages |\n| Restoration | O(n*log(m)) | n = pages, m = files |\n| File List | O(k) | k = files in level |\n\n### Space Complexity\n\n| Component | Memory Usage | Disk Usage |\n|-----------|-------------|------------|\n| DB | O(1) + metrics | Original DB + WAL |\n| Replica | O(1) | LTX files + metadata |\n| Compaction | O(n) pages | Temporary during merge |\n| Page Index | O(p) | p = pages in file |\n\n### Optimization Points\n\n1. **Page Index Caching**\n - Cache frequently accessed indices\n - Use estimated size for initial fetch\n\n2. **Batch Operations**\n - Group small changes into larger LTX files\n - Batch delete operations\n\n3. **Concurrent Operations**\n - Multiple replicas can sync in parallel\n - Compaction runs independently per level\n\n## Security Considerations\n\n### Access Control\n\n- File permissions: 0600 for database files\n- Directory permissions: 0700 for metadata\n- No built-in authentication (rely on storage backend)\n\n## Monitoring & Metrics\n\n### Prometheus Metrics\n\n```go\n// Database metrics\ndb_size_bytes // Current database size\nwal_size_bytes // Current WAL size\ntotal_wal_bytes // Total bytes written to WAL\ncheckpoint_count // Number of checkpoints\nsync_count // Number of syncs\nsync_error_count // Number of sync errors\n\n// Replica metrics\nreplica_lag_seconds // Replication lag\nreplica_position // Current replication position\n\n// Validation metrics have db and level labels.\nlitestream_validation_checks_total // Checks by result: success, invalid, or error.\nlitestream_last_validation_success // Last check: 1 passed, 0 failed or not yet complete.\nlitestream_last_validation_success_timestamp_seconds // Unix time of last success, or 0 until a check passes.\n```\n\n### Health Checks\n\n```go\nfunc (db *DB) HealthCheck() error {\n // Check database connection\n if err := db.db.Ping(); err != nil {\n return fmt.Errorf(\"database ping failed: %w\", err)\n }\n\n // Check replication lag\n for _, r := range db.replicas {\n lag := time.Since(r.LastSync())\n if lag > MaxAcceptableLag {\n return fmt.Errorf(\"replica %s lag too high: %v\", r.Name(), lag)\n }\n }\n\n return nil\n}\n```\n","llms.txt":"# Litestream\n\nDisaster recovery tool for SQLite. Replicates WAL changes to S3, GCS, Azure, SFTP, or local filesystem.\n\n## Quick Start for AI Contributors\n\n1. Read [AI_PR_GUIDE.md](AI_PR_GUIDE.md) - PR quality requirements\n2. Read [AGENTS.md](AGENTS.md) - Project overview and checklist\n3. Check [CONTRIBUTING.md](CONTRIBUTING.md) - What we accept\n4. Show investigation evidence in PRs\n\n## PR Checklist\n\n- [ ] Evidence of problem (logs, file patterns)\n- [ ] Clear scope (what PR does/doesn't do)\n- [ ] Runnable test commands\n- [ ] Race detector tested (`go test -race`)\n\n## Documentation\n\n| Document | Purpose |\n|----------|---------|\n| [AGENTS.md](AGENTS.md) | Project overview, critical rules |\n| [AI_PR_GUIDE.md](AI_PR_GUIDE.md) | PR templates, common mistakes |\n| [docs/PATTERNS.md](docs/PATTERNS.md) | Code patterns and anti-patterns |\n| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Component details |\n| [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) | WAL format, 1GB lock page |\n| [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md) | Replication format |\n| [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md) | Test strategies |\n| [docs/REPLICA_CLIENT_GUIDE.md](docs/REPLICA_CLIENT_GUIDE.md) | Storage backends |\n\n## Core Files\n\n| File | Purpose |\n|------|---------|\n| `db.go` | Database monitoring, WAL, checkpoints |\n| `replica.go` | Replication management |\n| `store.go` | Multi-database coordination |\n| `replica_client.go` | Storage backend interface |\n\n## Storage Backends\n\n- `s3/replica_client.go` - AWS S3\n- `gs/replica_client.go` - Google Cloud Storage\n- `abs/replica_client.go` - Azure Blob Storage\n- `sftp/replica_client.go` - SFTP\n- `file/replica_client.go` - Local filesystem\n- `nats/replica_client.go` - NATS JetStream\n\n## Critical Concepts\n\n- **Lock page at 1GB** - Always skip page at 0x40000000\n- **LTX files are immutable** - Never modify after creation\n- **Single replica per DB** - One destination per database\n- **Layer boundaries** - DB handles state, Replica handles replication\n\n## Build\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\npre-commit run --all-files\n```\n","packages/npm/litestream-vfs-darwin-amd64/README.md":"# litestream-vfs-darwin-amd64\n\nPlatform package for `litestream-vfs` — macOS x64 (Intel).\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","packages/npm/litestream-vfs-darwin-arm64/README.md":"# litestream-vfs-darwin-arm64\n\nPlatform package for `litestream-vfs` — macOS ARM64 (Apple Silicon).\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","packages/npm/litestream-vfs-linux-amd64/README.md":"# litestream-vfs-linux-amd64\n\nPlatform package for `litestream-vfs` — Linux x64.\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","packages/npm/litestream-vfs-linux-arm64/README.md":"# litestream-vfs-linux-arm64\n\nPlatform package for `litestream-vfs` — Linux ARM64.\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","packages/npm/litestream-vfs/README.md":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed via npm.\n\nThis package bundles the [Litestream](https://litestream.io) VFS shared library.\nThe correct platform-specific binary is automatically installed via `optionalDependencies`.\n\n## Installation\n\n```bash\nnpm install litestream-vfs\n```\n\n## Usage\n\n```javascript\nconst { getLoadablePath } = require(\"litestream-vfs\");\nconst path = getLoadablePath();\n// Use `path` with better-sqlite3 or other SQLite bindings\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","packages/python/README.md":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed as a Python wheel.\n\nThis package bundles the [Litestream](https://litestream.io) VFS shared library\nso you can load it directly into a Python `sqlite3` connection.\n\n## Installation\n\n```bash\npip install litestream-vfs\n```\n\n## Usage\n\n```python\nimport sqlite3\nimport litestream_vfs\n\nconn = sqlite3.connect(\":memory:\")\nlitestream_vfs.load(conn)\n```\n\nTo get the path to the shared library (for use with other SQLite bindings):\n\n```python\npath = litestream_vfs.loadable_path()\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","packages/ruby/README.md":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed as a Ruby gem.\n\nThis gem bundles the [Litestream](https://litestream.io) VFS shared library\nso you can load it directly into a SQLite connection.\n\n## Installation\n\n```bash\ngem install litestream-vfs\n```\n\nOr add to your Gemfile:\n\n```ruby\ngem \"litestream-vfs\"\n```\n\n## Usage\n\n```ruby\nrequire \"litestream_vfs\"\n\ndb = SQLite3::Database.new(\":memory:\")\nLitestreamVfs.load(db)\n```\n\nTo get the path to the shared library:\n\n```ruby\npath = LitestreamVfs.loadable_path\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","skills/litestream/SKILL.md":"---\nname: litestream\ndescription: >-\n Expert knowledge for contributing to Litestream, a standalone disaster recovery\n tool for SQLite. Provides architectural understanding, code patterns, critical\n rules, and debugging procedures for WAL monitoring, LTX replication format,\n storage backend implementation, multi-level compaction, and SQLite page\n management. Use when working with Litestream source code, writing storage\n backends, debugging replication issues, implementing compaction logic, or\n handling SQLite WAL operations.\nlicense: Apache-2.0\nmetadata:\n author: benbjohnson\n version: \"1.0\"\n repository: https://github.com/benbjohnson/litestream\n---\n\n# Litestream Agent Skill\n\nLitestream is a standalone disaster recovery tool for SQLite. It runs as a\nbackground process, monitors the SQLite WAL (Write-Ahead Log), converts changes\nto immutable LTX files, and replicates them to cloud storage. It uses\n`modernc.org/sqlite` (pure Go, no CGO required).\n\n## Quick Start\n\n```bash\n# Build\ngo build -o bin/litestream ./cmd/litestream\n\n# Test (always use race detector)\ngo test -race -v ./...\n\n# Code quality\npre-commit run --all-files\n```\n\n## Critical Rules\n\nThese invariants must never be violated:\n\n### 1. Lock Page at 1GB\n\nSQLite reserves a page at byte offset 0x40000000 (1 GB). Always skip it during\nreplication and compaction. The page number varies by page size:\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n### 2. LTX Files Are Immutable\n\nOnce an LTX file is written, it must never be modified. New changes create new\nfiles. This guarantees point-in-time recovery integrity.\n\n### 3. Single Replica per Database\n\nEach database replicates to exactly one destination. The Replica component\nmanages replication mechanics; database state belongs in the DB layer.\n\n### 4. Read Local Before Remote During Compaction\n\nCloud storage is eventually consistent. Always read from local disk first:\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil // Use local copy\n}\nreturn replica.Client.OpenLTXFile(...) // Fall back to remote\n```\n\n### 5. Preserve Timestamps During Compaction\n\nSet the compacted file's `CreatedAt` to the earliest source file timestamp to\nmaintain temporal granularity for point-in-time restoration.\n\n```go\ninfo.CreatedAt = oldestSourceFile.CreatedAt\n```\n\n### 6. Use Lock() Not RLock() for Writes\n\n```go\n// CORRECT\nr.mu.Lock()\ndefer r.mu.Unlock()\nr.pos = pos\n\n// WRONG - race condition\nr.mu.RLock()\ndefer r.mu.RUnlock()\nr.pos = pos\n```\n\n### 7. Atomic File Operations\n\nAlways write to a temp file then rename. Never write directly to the final path.\n\n```go\ntmpFile, err := os.CreateTemp(dir, \".tmp-*\")\n// ... write data, sync ...\nos.Rename(tmpFile.Name(), finalPath)\n```\n\n## Architecture\n\n### System Layers\n\n| Layer | File(s) | Responsibility |\n|---------|--------------------------|-------------------------------------------|\n| App | `cmd/litestream/` | CLI commands, YAML/env config |\n| Store | `store.go` | Multi-DB coordination, compaction |\n| DB | `db.go` | Single DB management, WAL monitoring |\n| Replica | `replica.go` | Replication to one destination |\n| Storage | `*/replica_client.go` | Backend implementations (S3, GCS, etc.) |\n\nDatabase state logic belongs in the DB layer, not the Replica layer.\n\n### ReplicaClient Interface\n\nAll storage backends implement this interface from `replica_client.go`:\n\n```go\ntype ReplicaClient interface {\n Type() string\n Init(ctx context.Context) error\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\nKey contract details:\n- `OpenLTXFile` must return `os.ErrNotExist` when file is missing\n- `WriteLTXFile` must set `CreatedAt` from backend metadata or upload time\n- `LTXFiles` with `useMetadata=true` fetches accurate timestamps (for PIT restore)\n- `LTXFiles` with `useMetadata=false` uses fast timestamps (normal operations)\n\n### Lock Ordering\n\nAlways acquire locks in this order to prevent deadlocks:\n\n1. `Store.mu`\n2. `DB.mu`\n3. `DB.chkMu`\n4. `Replica.mu`\n\n### Core Components\n\n**DB** (`db.go`): Manages SQLite connection, WAL monitoring, checkpointing, and\nlong-running read transaction for consistency. Key fields: `path`, `db`, `rtx`\n(read transaction), `pageSize`, `notify` channel.\n\n**Replica** (`replica.go`): Tracks replication position (`ltx.Pos` with TXID,\nPageNo, Checksum). One replica per database.\n\n**Store** (`store.go`): Coordinates multiple databases and schedules compaction\nacross levels.\n\n## LTX File Format\n\nLTX (Log Transaction) files are immutable, checksummed archives of database\nchanges. Structure:\n\n```\n+------------------+\n| Header | 100 bytes (magic \"LTX1\", page size, TXID range, timestamp)\n+------------------+\n| Page Frames | 4-byte pgno + pageSize bytes data, per page\n+------------------+\n| Page Index | Binary search index for page lookup\n+------------------+\n| Trailer | 16 bytes (post-apply checksum, file checksum)\n+------------------+\n```\n\n### Naming Convention\n\n```\nFormat: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\nExample: 0000000000000001-0000000000000064.ltx (TXID 1-100)\n```\n\n### Compaction Levels\n\n```\nLevel 0: /ltx/0000/ Raw LTX files (no compaction)\nLevel 1: /ltx/0001/ Compacted periodically\nLevel 2: /ltx/0002/ Compacted less frequently\n```\n\nDefault compaction levels: L0 (raw), L1 (30s), L2 (5min), L3 (1h), plus daily\nsnapshots. Compaction merges files by deduplicating pages (latest version wins)\nand always skips the lock page.\n\n## Code Patterns\n\n### DO\n\n- Return errors immediately; let callers decide handling\n- Use `fmt.Errorf(\"context: %w\", err)` for error wrapping\n- Handle database state in the DB layer, not Replica\n- Use `db.verify()` to trigger snapshots (don't reimplement)\n- Test with race detector: `go test -race`\n- Use lazy iterators for `LTXFiles` (paginate, don't load all at once)\n\n### DON'T\n\n- Write data at the 1 GB lock page boundary\n- Modify LTX files after creation\n- Put database state logic in the Replica layer\n- Use `RLock()` when writing shared state\n- Write directly to final file paths (use temp + rename)\n- Ignore context cancellation in long operations\n- Return generic errors instead of `os.ErrNotExist` for missing files\n\n## Specialized Knowledge Areas\n\nLoad reference files on demand based on the task:\n\n| Task | Reference File |\n|-----------------------------------|-----------------------------------------|\n| Understanding system design | `references/ARCHITECTURE.md` |\n| Writing or reviewing code | `references/PATTERNS.md` |\n| Working with LTX files | `references/LTX_FORMAT.md` |\n| WAL monitoring or page operations | `references/SQLITE_INTERNALS.md` |\n| Implementing storage backends | `references/REPLICA_CLIENT_GUIDE.md` |\n| Writing or debugging tests | `references/TESTING_GUIDE.md` |\n\n## Common Debugging Procedures\n\n### Replication Not Working\n\n1. Verify WAL mode: `PRAGMA journal_mode` must return `wal`\n2. Check monitor interval and that the monitor goroutine is running\n3. Confirm `db.notify` channel is being signaled on WAL changes\n4. Check replica position: `replica.Pos()` should advance with writes\n5. Look for `os.ErrNotExist` from `OpenLTXFile` (file not replicated yet)\n\n### Large Database Issues (>1 GB)\n\n1. Verify lock page is being skipped: check `ltx.LockPgno(pageSize)`\n2. Test with multiple page sizes (4K, 8K, 16K, 32K)\n3. Run with databases both smaller and larger than 1 GB\n4. Ensure page iteration loops include the `continue` guard for lock page\n\n### Compaction Problems\n\n1. Confirm local L0 files exist before compaction reads them\n2. Check that `CreatedAt` timestamps are preserved (earliest source)\n3. Verify compaction level intervals in `Store.levels`\n4. Look for eventual consistency issues if reading from remote storage\n\n### Storage Backend Issues\n\n1. Return `os.ErrNotExist` for missing files (not generic errors)\n2. Support partial reads via `offset`/`size` in `OpenLTXFile`\n3. Handle context cancellation in all methods\n4. Test concurrent operations with `-race` flag\n5. For eventually consistent backends, add retry logic with backoff\n\n### Corrupted or Missing LTX Files\n\n1. Check logs for `LTXError` messages - they include context (Op, Path, Level, TXID) and recovery hints\n2. Common error messages: \"nonsequential page numbers\", \"non-contiguous transaction files\", \"ltx validation failed\"\n3. Manual fix: `litestream reset ` clears local LTX state and forces fresh snapshot on next sync (database file is not modified)\n4. Automatic fix: set `auto-recover: true` on the replica config to auto-reset on LTX errors (disabled by default)\n5. Reference: `cmd/litestream/reset.go`, `replica.go` (auto-recover logic), `db.go` (`ResetLocalState`)\n\n## Contribution Guidelines\n\n### What's Accepted\n\n- Bug fixes and patches (welcome)\n- Documentation improvements\n- Small code improvements and performance optimizations\n- Security vulnerability reports (report privately)\n\n### Discuss First\n\n- Feature requests: open an issue before implementing\n- Large changes: discuss approach in an issue first\n\n### Pre-Submit Checklist\n\n- [ ] Read relevant docs from the reference table above\n- [ ] Follow patterns in `references/PATTERNS.md`\n- [ ] Run `go test -race -v ./...`\n- [ ] Run `pre-commit run --all-files`\n- [ ] For page iteration: test with >1 GB databases\n- [ ] Show investigation evidence in PR (see CONTRIBUTING.md)\n\n## Testing\n\n```bash\n# Full test suite with race detection\ngo test -race -v ./...\n\n# Specific areas\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\n\n# Coverage\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\n```\n\nKey testing areas:\n- Lock page handling with >1 GB databases and multiple page sizes\n- Race conditions in position updates, WAL monitoring, and checkpointing\n- Eventual consistency in storage backend operations\n- Atomic file operations and cleanup on error paths\n\n## Environment Validation\n\nRun `scripts/validate-setup.sh` to verify your development environment is\ncorrectly configured for Litestream development.\n","skills/litestream/references/ARCHITECTURE.md":"# Litestream Architecture Reference\n\nCondensed architectural reference for agents working on Litestream.\n\n## System Layers\n\n```\nApplication Layer cmd/litestream/ CLI commands, YAML/env config\nCore Layer store.go Multi-DB coordination, compaction scheduling\n db.go Single DB management, WAL monitoring, checkpoints\n replica.go Replication to single destination, position tracking\nStorage Abstraction replica_client.go ReplicaClient interface\nStorage Backends s3/, gs/, abs/, Backend-specific implementations\n oss/, file/, sftp/,\n nats/, webdav/\n```\n\nDatabase state logic belongs in the DB layer. The Replica layer handles\nreplication mechanics only.\n\n## Core Components\n\n### DB (db.go)\n\nThe heart of Litestream. Manages a single SQLite database.\n\nKey fields:\n- `path` / `metaPath`: Database and metadata paths\n- `db *sql.DB`: SQLite connection (via modernc.org/sqlite, pure Go)\n- `f *os.File`: Long-running file descriptor\n- `rtx *sql.Tx`: Long-running read transaction (prevents checkpoint past read point)\n- `pageSize int`: Database page size (critical for lock page calculation)\n- `mu sync.RWMutex`: Protects struct fields\n- `chkMu sync.RWMutex`: Checkpoint coordination lock\n- `notify chan struct{}`: WAL change notifications\n\nKey methods:\n- `Open()` / `Close()`: Lifecycle management\n- `monitor()`: Background WAL monitoring loop\n- `checkWAL()`: Detect WAL changes by comparing size/checksum\n- `Checkpoint(mode)`: Run SQLite checkpoint (PASSIVE, FULL, TRUNCATE)\n- `autoCheckpoint()`: Threshold-based checkpoint decisions\n- `Sync()`: Synchronize pending changes to replicas\n- `Compact()`: Merge LTX files at a given level\n\nConfiguration:\n- `MinCheckpointPageN`: Minimum pages for passive checkpoint\n- `TruncatePageN`: Emergency truncate checkpoint threshold\n- `CheckpointInterval`: Time-based passive checkpoint interval\n- `MonitorInterval`: WAL monitoring frequency\n- Note: RESTART checkpoint mode permanently removed (issue #724)\n\n### Replica (replica.go)\n\nManages replication to a single destination.\n\nKey fields:\n- `db *DB`: Parent database\n- `Client ReplicaClient`: Storage backend\n- `pos ltx.Pos`: Current replication position (TXID + PageNo + Checksum)\n- `mu sync.RWMutex`: Protects position\n\nPosition tracking: `SetPos()` MUST use `Lock()` not `RLock()`.\n\n### Store (store.go)\n\nCoordinates multiple databases and manages compaction.\n\nKey fields:\n- `dbs []*DB`: Managed databases\n- `levels CompactionLevels`: Compaction level configuration\n\nDefault compaction levels: L0 (raw), L1 (30s), L2 (5min), L3 (1h),\nplus daily snapshots.\n\n## WAL Monitoring\n\nThe monitor loop in `db.go` runs on a ticker:\n\n1. `checkWAL()` compares current WAL size/checksum to previous state\n2. If changed, notify replicas via `notifyReplicas()`\n3. Check if checkpoint needed via `shouldCheckpoint()`\n4. Run `autoCheckpoint()` if thresholds are met\n\nCheckpoint strategy:\n- WAL pages > `TruncatePageN` → TRUNCATE checkpoint (emergency)\n- WAL pages > `MinCheckpointPageN` → PASSIVE checkpoint\n- `CheckpointInterval` elapsed → PASSIVE checkpoint\n\n## Compaction Algorithm\n\nHigh-level flow (in `store.go`):\n\n1. Determine if level is due for compaction (`shouldCompact`)\n2. Enumerate level L-1 files via `ReplicaClient.LTXFiles`\n3. Prefer local copies (`os.Open(db.LTXPath(...))`) over remote reads\n4. Stream through `ltx.NewCompactor` (page deduplication, lock page skipping)\n5. Write merged file via `ReplicaClient.WriteLTXFile` for level L\n6. Set `CreatedAt` to earliest source file timestamp\n7. Update cached max file info; delete old L0 files when promoting to L1\n\n## State Management\n\n### Database States\n\nClosed → Opening → Open → Monitoring ↔ Syncing/Checkpointing → Closing → Closed\n\n### Replica States\n\nIdle → Starting → Monitoring ↔ Syncing/Uploading → Stopping → Idle\n\nError states retry with exponential backoff (1s initial, 1min max).\n\n## Initialization Flow\n\n1. `Store.Open()` iterates databases\n2. For each DB: open SQLite connection → read page size → create metadata dir →\n start long-running read transaction → initialize replicas → start monitor\n3. For each Replica: create with client → load previous position from metadata →\n validate against database → start sync goroutine\n4. Store starts compaction monitors\n\n## Error Handling\n\n### Error Categories\n\n- **Recoverable**: Network timeouts, temporary storage unavailability, lock contention\n- **Fatal**: Database corruption, invalid configuration, disk full\n- **Operational**: Checkpoint failures, compaction conflicts, sync delays\n\n### Error Propagation\n\n```\nReplicaClient.WriteLTXFile() error\n → Replica.Sync() error\n → DB.Sync() error\n → Store.monitorDB() // logs error, continues\n```\n\n## Lock Ordering\n\nAlways acquire in this order to prevent deadlocks:\n\n1. `Store.mu`\n2. `DB.mu`\n3. `DB.chkMu`\n4. `Replica.mu`\n\n## Goroutine Management\n\n- Use `sync.WaitGroup` for lifecycle tracking\n- Signal shutdown via `context.Cancel()`\n- Wait with timeout in `Close()` methods\n- Every goroutine must have a `defer wg.Done()`\n\n## Performance Characteristics\n\n| Operation | Time | Space |\n|-------------|------------|----------------|\n| WAL Monitor | O(1) | O(1) + metrics |\n| Page Write | O(1) | Original DB + WAL |\n| Compaction | O(n pages) | Temporary during merge |\n| Restoration | O(n·log m) | n=pages, m=files |\n| File List | O(k files) | Per-level |\n","skills/litestream/references/LTX_FORMAT.md":"# LTX Format Reference\n\nCondensed reference for agents working with Litestream's LTX file format.\n\n## Overview\n\nLTX (Log Transaction) is Litestream's custom format for storing database changes.\nLTX files are:\n- **Immutable**: Once written, never modified\n- **Self-contained**: Each file is independent\n- **Indexed**: Contains page index for efficient seeks\n- **Checksummed**: CRC-64 ECMA integrity verification\n\n## File Structure\n\n```\n+---------------------+\n| Header | 100 bytes\n+---------------------+\n| Page Frames | Variable (4-byte pgno + pageSize data per page)\n+---------------------+\n| Page Index | Binary search index\n+---------------------+\n| Trailer | 16 bytes\n+---------------------+\n\nFileSize = HeaderSize + (PageCount * (4 + PageSize)) + PageIndexSize + TrailerSize\n```\n\n## Header (100 bytes)\n\n```\nOffset Size Field\n0 4 Magic (\"LTX1\")\n4 4 Flags (bit 1 = NoChecksum)\n8 4 PageSize\n12 4 Commit (page count after applying)\n16 8 MinTXID\n24 8 MaxTXID\n32 8 Timestamp (ms since Unix epoch)\n40 8 PreApplyChecksum\n48 8 WALOffset (0 for snapshots)\n56 8 WALSize (0 for snapshots)\n64 4 WALSalt1\n68 4 WALSalt2\n72 8 NodeID\n80 20 Reserved (zeros)\n```\n\nThe version is implied by the magic string. `\"LTX1\"` → `ltx.Version == 2`.\n\n## Page Frames\n\nEach frame contains one database page:\n\n```\nOffset Size Field\n0 4 Page Number (1-based)\n4 PageSize Page Data\n```\n\nConstraints:\n- Pages written in sequential order during creation\n- Lock page at 1 GB boundary is never included\n- In compacted files, only the latest version of each page\n\n## Page Index\n\nBinary search index for efficient random access to pages. Use\n`ltx.DecodePageIndex()` to parse into a map of page number to\n`ltx.PageIndexElem`:\n\n```go\ntype PageIndexElem struct {\n Level int\n MinTXID TXID\n MaxTXID TXID\n Offset int64 // Byte offset of encoded payload\n Size int64 // Bytes occupied by encoded payload\n}\n```\n\n## Trailer (16 bytes)\n\n```\nOffset Size Field\n0 8 PostApplyChecksum (database checksum after applying file)\n8 8 FileChecksum (CRC-64 of entire file)\n```\n\nThe trailer is always at the end of the file. Read it by seeking\n`-TrailerSize` from `io.SeekEnd`.\n\n## File Naming Convention\n\n```\nFormat: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\n MinTXID (16 hex) MaxTXID (16 hex)\n\nExample: 0000000000000001-0000000000000064.ltx (TXID 1-100)\n 0000000000000065-00000000000000c8.ltx (TXID 101-200)\n```\n\nParse with `ltx.ParseFilename()`, format with `ltx.FormatFilename()`.\n\n## Compaction Levels\n\nLTX files are organized in levels for efficient storage:\n\n```\nLevel 0: /ltx/0000/ Raw LTX files (no compaction)\nLevel 1: /ltx/0001/ Compacted (default: every 30s)\nLevel 2: /ltx/0002/ Compacted (default: every 5min)\nLevel 3: /ltx/0003/ Compacted (default: every 1h)\nSnapshots: Full database state (daily)\n```\n\n### Compaction Process\n\n1. Enumerate level L-1 files\n2. Build page map (newer pages overwrite older)\n3. Write merged file skipping lock page\n4. Preserve earliest `CreatedAt` from source files\n5. Delete old L0 files when promoting to L1\n\n```go\n// Page deduplication: latest version wins\nfor _, file := range files {\n for _, page := range file.Pages {\n pageMap[page.Number] = page\n }\n}\n```\n\n## Checksums\n\nLTX uses CRC-64 ECMA checksums (`hash/crc64` with `crc64.ECMA` table):\n\n- **PreApplyChecksum**: Database state before applying this file\n- **PostApplyChecksum**: Database state after applying this file\n- **FileChecksum**: Integrity of the entire LTX file\n\n## Reading LTX Files\n\n```go\ndec := ltx.NewDecoder(reader)\nheader, err := dec.Header()\nfor {\n var hdr ltx.PageHeader\n data := make([]byte, header.PageSize)\n if err := dec.DecodePage(&hdr, data); err == io.EOF {\n break\n }\n // Process hdr.Pgno and data\n}\ntrailer := dec.Trailer()\n```\n\n## Writing LTX Files\n\n```go\nenc := ltx.NewEncoder(writer)\nenc.EncodeHeader(header)\nfor _, page := range pages {\n if page.Number == ltx.LockPgno(pageSize) {\n continue // Skip lock page\n }\n enc.EncodePage(pageHeader, pageData)\n}\nenc.EncodePageIndex(index)\nenc.EncodeTrailer()\nenc.Close()\n```\n\n## CLI Inspection\n\n```bash\nlitestream ltx /path/to/db.sqlite\nlitestream ltx s3://bucket/db\n```\n\nFor low-level inspection, use the Go API with `ltx.NewDecoder`.\n\n## WAL to LTX Conversion\n\nSQLite WAL frames are converted to LTX format during replication:\n1. Read WAL frames from the WAL file\n2. Skip the lock page\n3. Add checksums and build page index\n4. Write as immutable LTX file\n5. Upload to storage backend\n","skills/litestream/references/PATTERNS.md":"# Litestream Code Patterns Reference\n\nCondensed code patterns and anti-patterns for agents writing Litestream code.\n\n## Architectural Boundaries\n\n```\nDB Layer (db.go) → Database state, restoration, monitoring\nReplica Layer (replica.go) → Replication mechanics only\nStorage Layer → ReplicaClient implementations\n```\n\n### DO: Handle database state in DB layer\n\n```go\nfunc (db *DB) init() error {\n if db.needsRestore() {\n if err := db.restore(); err != nil {\n return err\n }\n }\n return db.replica.Start() // Replica focuses only on replication\n}\n```\n\n### DON'T: Put database state logic in Replica layer\n\n```go\n// WRONG\nfunc (r *Replica) Start() error {\n if needsRestore() { // Wrong layer!\n restoreDatabase()\n }\n}\n```\n\n## Atomic File Operations\n\n### DO: Write to temp file, then rename\n\n```go\nfunc writeFileAtomic(path string, data []byte) error {\n dir := filepath.Dir(path)\n tmpFile, err := os.CreateTemp(dir, \".tmp-*\")\n if err != nil {\n return fmt.Errorf(\"create temp file: %w\", err)\n }\n tmpPath := tmpFile.Name()\n\n defer func() {\n if tmpFile != nil {\n tmpFile.Close()\n os.Remove(tmpPath)\n }\n }()\n\n if _, err := tmpFile.Write(data); err != nil {\n return fmt.Errorf(\"write temp file: %w\", err)\n }\n if err := tmpFile.Sync(); err != nil {\n return fmt.Errorf(\"sync temp file: %w\", err)\n }\n if err := tmpFile.Close(); err != nil {\n return fmt.Errorf(\"close temp file: %w\", err)\n }\n tmpFile = nil\n\n if err := os.Rename(tmpPath, path); err != nil {\n os.Remove(tmpPath)\n return fmt.Errorf(\"rename to final path: %w\", err)\n }\n return nil\n}\n```\n\n### DON'T: Write directly to final location\n\n```go\n// WRONG - Can leave partial files on failure\nos.WriteFile(path, data, 0644)\n```\n\n## Error Handling\n\n### DO: Return errors immediately\n\n```go\nfunc (db *DB) validatePosition() error {\n dpos, err := db.Pos()\n if err != nil {\n return err\n }\n rpos := replica.Pos()\n if dpos.TXID < rpos.TXID {\n return fmt.Errorf(\"database position (%v) behind replica (%v)\", dpos, rpos)\n }\n return nil\n}\n```\n\n### DON'T: Log and continue on critical errors\n\n```go\n// WRONG\nif err := processFile(file); err != nil {\n log.Printf(\"error: %v\", err) // Just logging!\n // Continuing is dangerous\n}\n```\n\n### DO: Return errors from loops\n\n```go\nfunc (db *DB) processFiles() error {\n for _, file := range files {\n if err := processFile(file); err != nil {\n return fmt.Errorf(\"process file %s: %w\", file, err)\n }\n }\n return nil\n}\n```\n\n## Locking Patterns\n\n### DO: Use Lock() for writes\n\n```go\nr.mu.Lock()\ndefer r.mu.Unlock()\nr.pos = pos\n```\n\n### DON'T: Use RLock for write operations\n\n```go\n// WRONG - Race condition\nr.mu.RLock()\ndefer r.mu.RUnlock()\nr.pos = pos // Writing with RLock!\n```\n\n### Lock Ordering\n\nAlways acquire in order: Store.mu → DB.mu → DB.chkMu → Replica.mu\n\n## Compaction Patterns\n\n### DO: Read from local when available\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil // Local copy is complete and consistent\n}\nreturn replica.Client.OpenLTXFile(...) // Fall back to remote\n```\n\n### DON'T: Read from remote during compaction\n\n```go\n// WRONG - Can get partial/corrupt data from eventually consistent storage\nf, err := client.OpenLTXFile(ctx, level, minTXID, maxTXID, 0, 0)\n```\n\n### DO: Preserve earliest timestamp\n\n```go\ninfo.CreatedAt = oldestSourceFile.CreatedAt\n```\n\n### DON'T: Use current time during compaction\n\n```go\n// WRONG - Loses timestamp granularity for point-in-time restores\ninfo := <x.FileInfo{CreatedAt: time.Now()}\n```\n\n## Lock Page Handling\n\nThe lock page at 1 GB (0x40000000) must always be skipped:\n\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n\n## Common Pitfalls\n\n1. **Mixing architectural concerns**: DB state logic in Replica layer\n2. **Recreating existing functionality**: Use `db.verify()` for snapshots\n3. **Ignoring lock page**: Must skip during replication and compaction\n4. **Generic error types**: Return `os.ErrNotExist` for missing files\n5. **Blocking iterators**: Use lazy pagination for `LTXFiles`\n6. **Ignoring context**: Check `ctx.Done()` in long operations\n","skills/litestream/references/REPLICA_CLIENT_GUIDE.md":"# ReplicaClient Implementation Reference\n\nCondensed reference for agents implementing or modifying Litestream storage backends.\n\n## Interface Contract\n\nFrom `replica_client.go`:\n\n```go\ntype ReplicaClient interface {\n Type() string\n Init(ctx context.Context) error\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\n### Method Contracts\n\n**Type()**: Return identifier string (e.g., \"s3\", \"gcs\", \"file\").\n\n**Init()**: Initialize connection. Must be idempotent (no-op if already initialized).\n\n**LTXFiles()**: Return iterator sorted by MinTXID. `seek` starts from given TXID.\n`useMetadata=true` fetches accurate timestamps (required for point-in-time restore).\n`useMetadata=false` uses fast timestamps for normal operations.\n\n**OpenLTXFile()**: Open file for reading. Must support partial reads via `offset`/`size`.\nReturn `os.ErrNotExist` if file is missing.\n\n**WriteLTXFile()**: Write file to storage. Set `CreatedAt` from backend\nmetadata or upload time.\n\n**DeleteLTXFiles()**: Delete one or more files. Batch if possible.\n\n**DeleteAll()**: Delete all files for this database.\n\n## Implementation Checklist\n\n### Required\n\n- [ ] All interface methods implemented\n- [ ] `Init()` is idempotent\n- [ ] Partial reads supported (`offset`/`size` in `OpenLTXFile`)\n- [ ] `os.ErrNotExist` returned for missing files\n- [ ] Context cancellation handled in all methods\n- [ ] `CreatedAt` timestamps preserved in `WriteLTXFile`\n- [ ] Concurrent operations supported\n- [ ] Proper cleanup in `DeleteAll`\n\n### Optional\n\n- [ ] Connection pooling\n- [ ] Retry logic with exponential backoff\n- [ ] Request batching for deletes\n- [ ] Compression / encryption at rest\n- [ ] Bandwidth throttling\n\n## Eventual Consistency\n\nMany cloud backends (S3, R2, etc.) are eventually consistent:\n- Recently written files may not be immediately visible\n- Listed files may be only partially readable\n- Deletes may not take effect immediately\n\n### Pattern: Read Local First\n\nDuring compaction, always prefer local files:\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil\n}\nreturn replica.Client.OpenLTXFile(...)\n```\n\n### Pattern: Retry with Backoff\n\n```go\nbackoff := 100 * time.Millisecond\nfor i := 0; i < 5; i++ {\n reader, err := c.openFile(ctx, path, offset, size)\n if err == nil {\n return reader, nil\n }\n if errors.Is(err, os.ErrNotExist) {\n return nil, err // Don't retry definitive errors\n }\n select {\n case <-ctx.Done():\n return nil, ctx.Err()\n case <-time.After(backoff):\n backoff *= 2\n }\n}\n```\n\n### Pattern: Lazy Iterator\n\n```go\nfunc (c *Client) LTXFiles(...) (ltx.FileIterator, error) {\n return &lazyIterator{\n client: c,\n level: level,\n seek: seek,\n pageSize: 1000, // Paginate, don't load all at once\n }, nil\n}\n```\n\n## Error Handling\n\n### Standard Error Types\n\n```go\n// File not found\nreturn nil, os.ErrNotExist\n\n// Permission denied\nreturn nil, os.ErrPermission\n\n// Context cancelled\nreturn nil, ctx.Err()\n\n// Wrapped errors\nreturn nil, fmt.Errorf(\"s3 download failed: %w\", err)\n```\n\n### Retryable Error Classification\n\n```go\nfunc isRetryable(err error) bool {\n var netErr net.Error\n if errors.As(err, &netErr) && netErr.Temporary() {\n return true\n }\n // HTTP 429, 500, 502, 503, 504\n if errors.Is(err, context.DeadlineExceeded) {\n return true\n }\n return false\n}\n```\n\n## Common Mistakes\n\n### 1. Not Handling Partial Reads\n\n```go\n// WRONG - Ignores offset/size\nreturn c.storage.Download(path)\n\n// CORRECT\nif offset == 0 && size == 0 {\n return c.storage.Download(path)\n}\nreturn c.storage.DownloadRange(path, offset, offset+size-1)\n```\n\n### 2. Not Preserving CreatedAt\n\n```go\n// WRONG\nreturn <x.FileInfo{CreatedAt: time.Now()}\n\n// CORRECT - Use backend metadata\nreturn <x.FileInfo{CreatedAt: modTime}\n```\n\n### 3. Wrong Error Types\n\n```go\n// WRONG\nreturn nil, fmt.Errorf(\"not found\")\n\n// CORRECT\nif resp.StatusCode == 404 {\n return nil, os.ErrNotExist\n}\n```\n\n### 4. Ignoring Context\n\n```go\n// WRONG - Could run forever\nfor i := 0; i < 1000000; i++ {\n doWork()\n}\n\n// CORRECT\nselect {\ncase <-ctx.Done():\n return nil, ctx.Err()\ndefault:\n}\n```\n\n### 5. Loading All Files at Once\n\n```go\n// WRONG - Could be millions of files\nallFiles, _ := c.loadAllFiles(level)\n\n// CORRECT - Lazy pagination\nreturn &lazyIterator{pageSize: 1000}, nil\n```\n\n## Path Construction\n\n```go\nfunc (c *Client) ltxDir(level int) string {\n if level == SnapshotLevel {\n return path.Join(c.Path, \"snapshots\")\n }\n return path.Join(c.Path, \"ltx\", fmt.Sprintf(\"%04d\", level))\n}\n```\n\n## Reference Implementations\n\n- **Simplest**: `file/replica_client.go` (direct file I/O, no network)\n- **Most complete**: `s3/replica_client.go` (multipart uploads, retries, signing)\n\nStart with the file backend for understanding, then study S3 for advanced\npatterns.\n\n## Testing Requirements\n\n- Unit tests with >80% coverage\n- Integration tests with build tag\n- Test partial reads, concurrent operations, missing files\n- Run with `-race` flag\n- Verify `os.ErrNotExist` for missing files\n- Test context cancellation\n","skills/litestream/references/SQLITE_INTERNALS.md":"# SQLite Internals Reference\n\nCondensed reference for agents working with SQLite internals in Litestream.\n\n## SQLite File Structure\n\n```\ndatabase.db Main database file (pages)\ndatabase.db-wal Write-ahead log\ndatabase.db-shm Shared memory file (coordination)\n```\n\n## Write-Ahead Log (WAL)\n\nWAL is SQLite's method for atomic commits:\n- Changes written to WAL first, database unchanged until checkpoint\n- Readers merge WAL + database for consistent view\n- Litestream monitors WAL and converts frames to LTX format\n\n### WAL File Structure\n\n```\n+------------------+\n| WAL Header | 32 bytes\n+------------------+\n| Frame 1 Header | 24 bytes\n| Frame 1 Data | PageSize bytes\n+------------------+\n| Frame 2 Header | 24 bytes\n| Frame 2 Data | PageSize bytes\n+------------------+\n| ... |\n```\n\n### WAL Header (32 bytes)\n\n```\nMagic (4B) | FileFormat (4B) | PageSize (4B) | Checkpoint (4B) |\nSalt1 (4B) | Salt2 (4B) | Checksum1 (4B) | Checksum2 (4B)\n```\n\nMagic: `0x377f0682` or `0x377f0683`\n\n### WAL Frame Header (24 bytes)\n\n```\nPageNumber (4B) | DbSize (4B) | Salt1 (4B) | Salt2 (4B) |\nChecksum1 (4B) | Checksum2 (4B)\n```\n\n## The 1 GB Lock Page\n\nSQLite reserves a page at exactly 0x40000000 bytes (1 GB) for locking. This is\nthe single most important SQLite detail for Litestream development.\n\n```go\nconst PENDING_BYTE = 0x40000000\n\nfunc LockPgno(pageSize int) uint32 {\n return uint32(PENDING_BYTE/pageSize) + 1\n}\n```\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n| 64 KB | 16385 |\n\nRules:\n- Cannot contain data; SQLite never writes user data here\n- Must be skipped during replication and compaction\n- Only affects databases > 1 GB\n- Page number changes with page size\n\nImplementation in Litestream:\n\n```go\nfor pgno := uint32(1); pgno <= maxPgno; pgno++ {\n if pgno == ltx.LockPgno(db.pageSize) {\n continue\n }\n processPage(pgno)\n}\n```\n\n## Page Structure\n\nSQLite divides databases into fixed-size pages (typically 4096 bytes):\n- Page numbers are 1-based\n- Types: B-tree interior, B-tree leaf, overflow, freelist, lock byte page\n\n## Transaction Types\n\n1. **Deferred** (default): Lock acquired on first use\n2. **Immediate**: RESERVED lock acquired immediately\n3. **Exclusive**: EXCLUSIVE lock acquired immediately\n\n### Lock Hierarchy\n\nUNLOCKED → SHARED → RESERVED → PENDING → EXCLUSIVE\n\n- SHARED: Multiple readers allowed\n- RESERVED: Signals intent to write\n- PENDING: Blocks new SHARED locks\n- EXCLUSIVE: Single writer, no readers\n\n## Long-Running Read Transaction\n\nLitestream maintains a read transaction for consistency:\n\n```go\nfunc (db *DB) initReadTx() error {\n tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{ReadOnly: true})\n if err != nil {\n return err\n }\n var dummy string\n err = tx.QueryRow(\"SELECT ''\").Scan(&dummy)\n if err != nil {\n tx.Rollback()\n return err\n }\n db.rtx = tx\n return nil\n}\n```\n\nPurpose:\n- Prevents checkpoint past our read point\n- Ensures consistent database view\n- Allows reading historical pages from WAL\n\n## Checkpoint Modes\n\n| Mode | Behavior |\n|----------|-----------------------------------------------|\n| PASSIVE | Non-blocking; fails if readers present |\n| FULL | Waits for readers; blocks new readers |\n| RESTART | Like FULL + resets WAL start (removed in #724) |\n| TRUNCATE | Like RESTART + truncates WAL to zero |\n\n### Litestream Checkpoint Strategy\n\n```\nWAL pages > TruncatePageN → TRUNCATE (emergency)\nWAL pages > MinCheckpointPageN → PASSIVE\nCheckpointInterval elapsed → PASSIVE\n```\n\nNote: RESTART mode permanently removed due to issue #724 (write-blocking).\n\n## Important SQLite Pragmas\n\n```sql\nPRAGMA journal_mode = WAL; -- Required for Litestream\nPRAGMA page_size; -- Get page size\nPRAGMA page_count; -- Total pages in database\nPRAGMA freelist_count; -- Free pages\nPRAGMA wal_checkpoint(PASSIVE); -- Non-blocking checkpoint\nPRAGMA wal_autocheckpoint = 10000; -- High threshold (prevent interference)\nPRAGMA busy_timeout = 5000; -- Wait 5s for locks\nPRAGMA synchronous = NORMAL; -- Safe with WAL\nPRAGMA cache_size = -64000; -- 64 MB cache\nPRAGMA integrity_check; -- Verify database integrity\n```\n\n## Critical SQLite Behaviors\n\n1. **Automatic checkpoint**: Default at 1000 WAL pages; set high threshold to\n prevent interference with Litestream's control\n2. **Busy timeout**: Default is 0 (immediate failure); set a reasonable timeout\n3. **Synchronous mode**: NORMAL is safe with WAL mode\n4. **Page cache**: In-memory cache; configure with negative KB or positive pages\n\n## WAL to LTX Conversion\n\nLitestream converts WAL frames to LTX:\n\n1. Read WAL header and validate magic number\n2. Iterate frames, reading page number and data\n3. Skip lock page (`pgno == LockPgno(pageSize)`)\n4. Calculate checksums\n5. Write as immutable LTX file with page index\n","skills/litestream/references/TESTING_GUIDE.md":"# Litestream Testing Reference\n\nCondensed reference for agents writing and debugging Litestream tests.\n\n## Testing Philosophy\n\n1. Test at multiple levels: unit, integration, end-to-end\n2. Focus on edge cases: >1 GB databases, eventual consistency\n3. Use real SQLite: avoid mocking SQLite behavior\n4. Always run with `-race` flag\n5. Use fixed seeds and timestamps for deterministic tests\n\n## Essential Commands\n\n```bash\n# Full test suite with race detection\ngo test -race -v ./...\n\n# Specific test areas\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\n\n# Coverage\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\ngo tool cover -func=coverage.out | grep total\n\n# Integration tests (backend-specific)\ngo test -v ./replica_client_test.go -integration s3\ngo test -v ./replica_client_test.go -integration gcs\n```\n\n## Lock Page Testing\n\nThe lock page at 1 GB must be skipped. Test with databases spanning this\nboundary using multiple page sizes:\n\n```bash\n./bin/litestream-test populate -db test.db -page-size 4096 -target-size 2GB\n./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica\n```\n\nTest all page size variants:\n\n| Page Size | Lock Page | Test Target |\n|-----------|-----------|-------------|\n| 4 KB | 262145 | >1.0 GB |\n| 8 KB | 131073 | >1.0 GB |\n| 16 KB | 65537 | >1.0 GB |\n| 32 KB | 32769 | >1.0 GB |\n\nVerify lock page is not present in any LTX file after replication.\n\n## Race Condition Testing\n\nRace-prone areas (always run with `-race`):\n\n1. **Position updates**: Concurrent `SetPos()` + `Pos()` on Replica\n2. **WAL monitoring**: Concurrent writes + monitor + checkpoint\n3. **Compaction**: Concurrent compaction at different levels\n\n### Pattern: Concurrent Test\n\n```go\nfunc TestConcurrent(t *testing.T) {\n ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\n defer cancel()\n\n var wg sync.WaitGroup\n // Start writer, monitor, checkpoint goroutines\n // Use ctx for shutdown, wg for cleanup\n wg.Wait()\n}\n```\n\n## Common Test Failures\n\n### 1. Database Locked\n\n```go\n// WRONG - WAL mode not enabled\ndb, _ := sql.Open(\"sqlite\", \"test.db\")\n\n// CORRECT\ndb, _ := sql.Open(\"sqlite\", \"test.db?_journal=WAL\")\n```\n\n### 2. Timing Issues\n\n```go\n// WRONG - Data may not be replicated yet\nWriteData(db)\nresult := ReadReplica()\n\n// CORRECT - Explicit sync\nWriteData(db)\ndb.Sync(context.Background())\nresult := ReadReplica()\n```\n\n### 3. Goroutine Leaks\n\n```go\n// WRONG - Goroutine outlives test\ngo func() {\n time.Sleep(10 * time.Second)\n doWork()\n}()\n\n// CORRECT - Use context + WaitGroup\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\nvar wg sync.WaitGroup\nwg.Add(1)\ngo func() {\n defer wg.Done()\n select {\n case <-ctx.Done():\n return\n case <-time.After(10 * time.Second):\n doWork()\n }\n}()\ncancel()\nwg.Wait()\n```\n\n### 4. File Handle Leaks\n\n```go\n// Always use defer for cleanup\nf, err := os.Open(\"test.db\")\nrequire.NoError(t, err)\ndefer f.Close()\n```\n\n## Test Helper Patterns\n\n```go\nfunc NewTestDB(t testing.TB) *litestream.DB {\n t.Helper()\n path := filepath.Join(t.TempDir(), \"test.db\")\n conn, err := sql.Open(\"sqlite\", path+\"?_journal=WAL\")\n require.NoError(t, err)\n _, err = conn.Exec(\"CREATE TABLE test (id INTEGER PRIMARY KEY, data BLOB)\")\n require.NoError(t, err)\n conn.Close()\n\n db := litestream.NewDB(path)\n db.MonitorInterval = 10 * time.Millisecond\n db.MinCheckpointPageN = 100\n err = db.Open()\n require.NoError(t, err)\n t.Cleanup(func() { db.Close(context.Background()) })\n return db\n}\n```\n\n## Coverage Requirements\n\n| Package | Target |\n|--------------------|--------|\n| Core (db, replica) | >80% |\n| Replica clients | >70% |\n| Utilities | >60% |\n\n## Table-Driven Tests\n\n```go\nfunc TestCheckpoint(t *testing.T) {\n tests := []struct {\n name string\n mode string\n wantErr bool\n }{\n {\"Passive\", \"PASSIVE\", false},\n {\"Full\", \"FULL\", false},\n {\"Truncate\", \"TRUNCATE\", false},\n {\"Invalid\", \"INVALID\", true},\n }\n for _, tt := range tests {\n t.Run(tt.name, func(t *testing.T) {\n // ...\n })\n }\n}\n```\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md - Litestream AI Agent Guide\n\nLitestream is a disaster recovery tool for SQLite that runs as a background process, monitors the WAL, converts changes to immutable LTX files, and replicates them to cloud storage. It uses `modernc.org/sqlite` (pure Go, no CGO required).\n\n## Before You Start\n\n1. Read [AI_PR_GUIDE.md](AI_PR_GUIDE.md) for contribution requirements\n2. Check [CONTRIBUTING.md](CONTRIBUTING.md) for what we accept (bug fixes welcome, features need discussion)\n3. Review recent PRs for current patterns\n\n## Critical Rules\n\n- **Lock page at 1GB**: SQLite reserves page at 0x40000000. Always skip it. See [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md)\n- **LTX files are immutable**: Never modify after creation. See [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md)\n- **Single replica per database**: Each DB replicates to exactly one destination\n- **Use `litestream ltx`**: Not `litestream wal` (deprecated)\n- **Use `litestream reset`**: Clears corrupted local LTX state for a database. See `cmd/litestream/reset.go`\n- **`auto-recover` config**: Replica option that automatically resets local state on LTX errors. Disabled by default. See `replica.go`\n- **Retention enabled by default**: `Store.RetentionEnabled` is `true` by default. Disable only when cloud lifecycle policies handle cleanup. See `store.go`\n- **IPC socket disabled by default**: Control socket is off by default. Enable with `socket.enabled: true` in config. See `server.go`\n- **`$PID` config expansion**: Config files support `$PID` to expand to the current process ID, plus standard `$ENV_VAR` expansion. See `cmd/litestream/main.go`\n- **`litestream ltx -level`**: Use `-level 0`–`9` or `-level all` to inspect specific compaction levels. See `cmd/litestream/ltx.go`\n- **Return errors, don't log them**: Always return errors to callers. Never `log.Printf(err)` and continue — this silently hides failures in a disaster recovery tool. Only use DEBUG log for best-effort operations where failure doesn't affect correctness and a valid fallback exists (e.g., reading SHM mxFrame optimization hint). See [docs/PATTERNS.md](docs/PATTERNS.md#error-handling)\n\n## Layer Boundaries\n\n| Layer | File | Responsibility |\n|-------|------|----------------|\n| DB | `db.go` | Database state, restoration, WAL monitoring, library API (`SyncStatus`, `SyncAndWait`, `EnsureExists`) |\n| Replica | `replica.go` | Replication mechanics only |\n| Storage | `**/replica_client.go` | Backend implementations (includes `ReplicaClientV3` for v0.3.x restore) |\n| IPC | `server.go` | Unix socket control API (register/unregister, /txid, pprof) |\n| Leasing | `leaser.go`, `s3/leaser.go` | Distributed lease acquisition via conditional writes |\n\nDatabase state logic belongs in DB layer, not Replica layer.\n\n## Quick Reference\n\n**Build:**\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\n```\n\n**Code quality:**\n\n```bash\npre-commit run --all-files\n```\n\n## Documentation\n\n| Document | When to Read |\n|----------|--------------|\n| [docs/PATTERNS.md](docs/PATTERNS.md) | Code patterns and anti-patterns |\n| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Deep component details |\n| [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) | WAL format, lock page |\n| [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md) | Replication format |\n| [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md) | Test strategies |\n| [docs/REPLICA_CLIENT_GUIDE.md](docs/REPLICA_CLIENT_GUIDE.md) | Adding storage backends |\n| [docs/PROVIDER_COMPATIBILITY.md](docs/PROVIDER_COMPATIBILITY.md) | Provider-specific S3/cloud configs |\n\n## Checklist\n\nBefore submitting changes:\n\n- [ ] Read relevant docs above\n- [ ] Follow patterns in [docs/PATTERNS.md](docs/PATTERNS.md)\n- [ ] Test with race detector (`go test -race`)\n- [ ] Run `pre-commit run --all-files`\n- [ ] For page iteration: test with >1GB databases\n- [ ] Show investigation evidence in PR (see [AI_PR_GUIDE.md](AI_PR_GUIDE.md))\n","isInternal":false,"tokens":1022,"sizeBytes":3926},{"name":"llms.txt","path":"llms.txt","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/llms.txt","title":"LLM Index & Context Digest","category":"root-instruction","format":"text","content":"# Litestream\n\nDisaster recovery tool for SQLite. Replicates WAL changes to S3, GCS, Azure, SFTP, or local filesystem.\n\n## Quick Start for AI Contributors\n\n1. Read [AI_PR_GUIDE.md](AI_PR_GUIDE.md) - PR quality requirements\n2. Read [AGENTS.md](AGENTS.md) - Project overview and checklist\n3. Check [CONTRIBUTING.md](CONTRIBUTING.md) - What we accept\n4. Show investigation evidence in PRs\n\n## PR Checklist\n\n- [ ] Evidence of problem (logs, file patterns)\n- [ ] Clear scope (what PR does/doesn't do)\n- [ ] Runnable test commands\n- [ ] Race detector tested (`go test -race`)\n\n## Documentation\n\n| Document | Purpose |\n|----------|---------|\n| [AGENTS.md](AGENTS.md) | Project overview, critical rules |\n| [AI_PR_GUIDE.md](AI_PR_GUIDE.md) | PR templates, common mistakes |\n| [docs/PATTERNS.md](docs/PATTERNS.md) | Code patterns and anti-patterns |\n| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Component details |\n| [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) | WAL format, 1GB lock page |\n| [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md) | Replication format |\n| [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md) | Test strategies |\n| [docs/REPLICA_CLIENT_GUIDE.md](docs/REPLICA_CLIENT_GUIDE.md) | Storage backends |\n\n## Core Files\n\n| File | Purpose |\n|------|---------|\n| `db.go` | Database monitoring, WAL, checkpoints |\n| `replica.go` | Replication management |\n| `store.go` | Multi-database coordination |\n| `replica_client.go` | Storage backend interface |\n\n## Storage Backends\n\n- `s3/replica_client.go` - AWS S3\n- `gs/replica_client.go` - Google Cloud Storage\n- `abs/replica_client.go` - Azure Blob Storage\n- `sftp/replica_client.go` - SFTP\n- `file/replica_client.go` - Local filesystem\n- `nats/replica_client.go` - NATS JetStream\n\n## Critical Concepts\n\n- **Lock page at 1GB** - Always skip page at 0x40000000\n- **LTX files are immutable** - Never modify after creation\n- **Single replica per DB** - One destination per database\n- **Layer boundaries** - DB handles state, Replica handles replication\n\n## Build\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\npre-commit run --all-files\n```\n","isInternal":false,"tokens":605,"sizeBytes":2119},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md - Claude Code Configuration\n\nClaude-specific optimizations for Litestream. See [AGENTS.md](AGENTS.md) for project documentation.\n\n## Critical Coding Rules\n\n- **Always return errors, never log and continue.** The only exception is DEBUG logging for best-effort operations that don't affect correctness. This is the #1 cause of PR review feedback. See [docs/PATTERNS.md](docs/PATTERNS.md#error-handling) for the full decision framework.\n\n## Context Window\n\nWith Claude's large context window, load documentation as needed:\n\n- Start with [AGENTS.md](AGENTS.md) for overview and checklist\n- Load [docs/PATTERNS.md](docs/PATTERNS.md) when writing code\n- Load [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) for WAL/page work\n- Load [docs/PROVIDER_COMPATIBILITY.md](docs/PROVIDER_COMPATIBILITY.md) for storage backend configs\n\n## Claude-Specific Resources\n\n### Specialized Agents (.claude/agents/)\n\n- `sqlite-expert.md` - SQLite WAL and page management\n- `replica-client-developer.md` - Storage backend implementation\n- `ltx-compaction-specialist.md` - LTX format and compaction\n- `test-engineer.md` - Testing strategies\n- `performance-optimizer.md` - Performance optimization\n\n### Commands (.claude/commands/)\n\n- `/analyze-ltx` - Analyze LTX file structure\n- `/debug-ipc` - Debug IPC Unix socket issues\n- `/debug-wal` - Debug WAL replication issues\n- `/test-compaction` - Test compaction scenarios\n- `/trace-replication` - Trace replication flow\n- `/validate-replica` - Validate replica client\n- `/add-storage-backend` - Create new storage backend\n- `/fix-common-issues` - Diagnose common problems\n- `/run-comprehensive-tests` - Execute full test suite\n\n## Quick Commands\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\npre-commit run --all-files\n```\n","isInternal":false,"tokens":449,"sizeBytes":1791},{"name":"copilot-instructions.md","path":".github/copilot-instructions.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.github/copilot-instructions.md","title":"GitHub Copilot Instructions","category":"copilot-instructions","format":"markdown","content":"../AGENTS.md","isInternal":false,"tokens":4,"sizeBytes":12},{"name":"ARCHITECTURE.md","path":"skills/litestream/references/ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/skills/litestream/references/ARCHITECTURE.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Litestream Architecture Reference\n\nCondensed architectural reference for agents working on Litestream.\n\n## System Layers\n\n```\nApplication Layer cmd/litestream/ CLI commands, YAML/env config\nCore Layer store.go Multi-DB coordination, compaction scheduling\n db.go Single DB management, WAL monitoring, checkpoints\n replica.go Replication to single destination, position tracking\nStorage Abstraction replica_client.go ReplicaClient interface\nStorage Backends s3/, gs/, abs/, Backend-specific implementations\n oss/, file/, sftp/,\n nats/, webdav/\n```\n\nDatabase state logic belongs in the DB layer. The Replica layer handles\nreplication mechanics only.\n\n## Core Components\n\n### DB (db.go)\n\nThe heart of Litestream. Manages a single SQLite database.\n\nKey fields:\n- `path` / `metaPath`: Database and metadata paths\n- `db *sql.DB`: SQLite connection (via modernc.org/sqlite, pure Go)\n- `f *os.File`: Long-running file descriptor\n- `rtx *sql.Tx`: Long-running read transaction (prevents checkpoint past read point)\n- `pageSize int`: Database page size (critical for lock page calculation)\n- `mu sync.RWMutex`: Protects struct fields\n- `chkMu sync.RWMutex`: Checkpoint coordination lock\n- `notify chan struct{}`: WAL change notifications\n\nKey methods:\n- `Open()` / `Close()`: Lifecycle management\n- `monitor()`: Background WAL monitoring loop\n- `checkWAL()`: Detect WAL changes by comparing size/checksum\n- `Checkpoint(mode)`: Run SQLite checkpoint (PASSIVE, FULL, TRUNCATE)\n- `autoCheckpoint()`: Threshold-based checkpoint decisions\n- `Sync()`: Synchronize pending changes to replicas\n- `Compact()`: Merge LTX files at a given level\n\nConfiguration:\n- `MinCheckpointPageN`: Minimum pages for passive checkpoint\n- `TruncatePageN`: Emergency truncate checkpoint threshold\n- `CheckpointInterval`: Time-based passive checkpoint interval\n- `MonitorInterval`: WAL monitoring frequency\n- Note: RESTART checkpoint mode permanently removed (issue #724)\n\n### Replica (replica.go)\n\nManages replication to a single destination.\n\nKey fields:\n- `db *DB`: Parent database\n- `Client ReplicaClient`: Storage backend\n- `pos ltx.Pos`: Current replication position (TXID + PageNo + Checksum)\n- `mu sync.RWMutex`: Protects position\n\nPosition tracking: `SetPos()` MUST use `Lock()` not `RLock()`.\n\n### Store (store.go)\n\nCoordinates multiple databases and manages compaction.\n\nKey fields:\n- `dbs []*DB`: Managed databases\n- `levels CompactionLevels`: Compaction level configuration\n\nDefault compaction levels: L0 (raw), L1 (30s), L2 (5min), L3 (1h),\nplus daily snapshots.\n\n## WAL Monitoring\n\nThe monitor loop in `db.go` runs on a ticker:\n\n1. `checkWAL()` compares current WAL size/checksum to previous state\n2. If changed, notify replicas via `notifyReplicas()`\n3. Check if checkpoint needed via `shouldCheckpoint()`\n4. Run `autoCheckpoint()` if thresholds are met\n\nCheckpoint strategy:\n- WAL pages > `TruncatePageN` → TRUNCATE checkpoint (emergency)\n- WAL pages > `MinCheckpointPageN` → PASSIVE checkpoint\n- `CheckpointInterval` elapsed → PASSIVE checkpoint\n\n## Compaction Algorithm\n\nHigh-level flow (in `store.go`):\n\n1. Determine if level is due for compaction (`shouldCompact`)\n2. Enumerate level L-1 files via `ReplicaClient.LTXFiles`\n3. Prefer local copies (`os.Open(db.LTXPath(...))`) over remote reads\n4. Stream through `ltx.NewCompactor` (page deduplication, lock page skipping)\n5. Write merged file via `ReplicaClient.WriteLTXFile` for level L\n6. Set `CreatedAt` to earliest source file timestamp\n7. Update cached max file info; delete old L0 files when promoting to L1\n\n## State Management\n\n### Database States\n\nClosed → Opening → Open → Monitoring ↔ Syncing/Checkpointing → Closing → Closed\n\n### Replica States\n\nIdle → Starting → Monitoring ↔ Syncing/Uploading → Stopping → Idle\n\nError states retry with exponential backoff (1s initial, 1min max).\n\n## Initialization Flow\n\n1. `Store.Open()` iterates databases\n2. For each DB: open SQLite connection → read page size → create metadata dir →\n start long-running read transaction → initialize replicas → start monitor\n3. For each Replica: create with client → load previous position from metadata →\n validate against database → start sync goroutine\n4. Store starts compaction monitors\n\n## Error Handling\n\n### Error Categories\n\n- **Recoverable**: Network timeouts, temporary storage unavailability, lock contention\n- **Fatal**: Database corruption, invalid configuration, disk full\n- **Operational**: Checkpoint failures, compaction conflicts, sync delays\n\n### Error Propagation\n\n```\nReplicaClient.WriteLTXFile() error\n → Replica.Sync() error\n → DB.Sync() error\n → Store.monitorDB() // logs error, continues\n```\n\n## Lock Ordering\n\nAlways acquire in this order to prevent deadlocks:\n\n1. `Store.mu`\n2. `DB.mu`\n3. `DB.chkMu`\n4. `Replica.mu`\n\n## Goroutine Management\n\n- Use `sync.WaitGroup` for lifecycle tracking\n- Signal shutdown via `context.Cancel()`\n- Wait with timeout in `Close()` methods\n- Every goroutine must have a `defer wg.Done()`\n\n## Performance Characteristics\n\n| Operation | Time | Space |\n|-------------|------------|----------------|\n| WAL Monitor | O(1) | O(1) + metrics |\n| Page Write | O(1) | Original DB + WAL |\n| Compaction | O(n pages) | Temporary during merge |\n| Restoration | O(n·log m) | n=pages, m=files |\n| File List | O(k files) | Per-level |\n","isInternal":false,"tokens":1296,"sizeBytes":5559},{"name":"LTX_FORMAT.md","path":"skills/litestream/references/LTX_FORMAT.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/skills/litestream/references/LTX_FORMAT.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# LTX Format Reference\n\nCondensed reference for agents working with Litestream's LTX file format.\n\n## Overview\n\nLTX (Log Transaction) is Litestream's custom format for storing database changes.\nLTX files are:\n- **Immutable**: Once written, never modified\n- **Self-contained**: Each file is independent\n- **Indexed**: Contains page index for efficient seeks\n- **Checksummed**: CRC-64 ECMA integrity verification\n\n## File Structure\n\n```\n+---------------------+\n| Header | 100 bytes\n+---------------------+\n| Page Frames | Variable (4-byte pgno + pageSize data per page)\n+---------------------+\n| Page Index | Binary search index\n+---------------------+\n| Trailer | 16 bytes\n+---------------------+\n\nFileSize = HeaderSize + (PageCount * (4 + PageSize)) + PageIndexSize + TrailerSize\n```\n\n## Header (100 bytes)\n\n```\nOffset Size Field\n0 4 Magic (\"LTX1\")\n4 4 Flags (bit 1 = NoChecksum)\n8 4 PageSize\n12 4 Commit (page count after applying)\n16 8 MinTXID\n24 8 MaxTXID\n32 8 Timestamp (ms since Unix epoch)\n40 8 PreApplyChecksum\n48 8 WALOffset (0 for snapshots)\n56 8 WALSize (0 for snapshots)\n64 4 WALSalt1\n68 4 WALSalt2\n72 8 NodeID\n80 20 Reserved (zeros)\n```\n\nThe version is implied by the magic string. `\"LTX1\"` → `ltx.Version == 2`.\n\n## Page Frames\n\nEach frame contains one database page:\n\n```\nOffset Size Field\n0 4 Page Number (1-based)\n4 PageSize Page Data\n```\n\nConstraints:\n- Pages written in sequential order during creation\n- Lock page at 1 GB boundary is never included\n- In compacted files, only the latest version of each page\n\n## Page Index\n\nBinary search index for efficient random access to pages. Use\n`ltx.DecodePageIndex()` to parse into a map of page number to\n`ltx.PageIndexElem`:\n\n```go\ntype PageIndexElem struct {\n Level int\n MinTXID TXID\n MaxTXID TXID\n Offset int64 // Byte offset of encoded payload\n Size int64 // Bytes occupied by encoded payload\n}\n```\n\n## Trailer (16 bytes)\n\n```\nOffset Size Field\n0 8 PostApplyChecksum (database checksum after applying file)\n8 8 FileChecksum (CRC-64 of entire file)\n```\n\nThe trailer is always at the end of the file. Read it by seeking\n`-TrailerSize` from `io.SeekEnd`.\n\n## File Naming Convention\n\n```\nFormat: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\n MinTXID (16 hex) MaxTXID (16 hex)\n\nExample: 0000000000000001-0000000000000064.ltx (TXID 1-100)\n 0000000000000065-00000000000000c8.ltx (TXID 101-200)\n```\n\nParse with `ltx.ParseFilename()`, format with `ltx.FormatFilename()`.\n\n## Compaction Levels\n\nLTX files are organized in levels for efficient storage:\n\n```\nLevel 0: /ltx/0000/ Raw LTX files (no compaction)\nLevel 1: /ltx/0001/ Compacted (default: every 30s)\nLevel 2: /ltx/0002/ Compacted (default: every 5min)\nLevel 3: /ltx/0003/ Compacted (default: every 1h)\nSnapshots: Full database state (daily)\n```\n\n### Compaction Process\n\n1. Enumerate level L-1 files\n2. Build page map (newer pages overwrite older)\n3. Write merged file skipping lock page\n4. Preserve earliest `CreatedAt` from source files\n5. Delete old L0 files when promoting to L1\n\n```go\n// Page deduplication: latest version wins\nfor _, file := range files {\n for _, page := range file.Pages {\n pageMap[page.Number] = page\n }\n}\n```\n\n## Checksums\n\nLTX uses CRC-64 ECMA checksums (`hash/crc64` with `crc64.ECMA` table):\n\n- **PreApplyChecksum**: Database state before applying this file\n- **PostApplyChecksum**: Database state after applying this file\n- **FileChecksum**: Integrity of the entire LTX file\n\n## Reading LTX Files\n\n```go\ndec := ltx.NewDecoder(reader)\nheader, err := dec.Header()\nfor {\n var hdr ltx.PageHeader\n data := make([]byte, header.PageSize)\n if err := dec.DecodePage(&hdr, data); err == io.EOF {\n break\n }\n // Process hdr.Pgno and data\n}\ntrailer := dec.Trailer()\n```\n\n## Writing LTX Files\n\n```go\nenc := ltx.NewEncoder(writer)\nenc.EncodeHeader(header)\nfor _, page := range pages {\n if page.Number == ltx.LockPgno(pageSize) {\n continue // Skip lock page\n }\n enc.EncodePage(pageHeader, pageData)\n}\nenc.EncodePageIndex(index)\nenc.EncodeTrailer()\nenc.Close()\n```\n\n## CLI Inspection\n\n```bash\nlitestream ltx /path/to/db.sqlite\nlitestream ltx s3://bucket/db\n```\n\nFor low-level inspection, use the Go API with `ltx.NewDecoder`.\n\n## WAL to LTX Conversion\n\nSQLite WAL frames are converted to LTX format during replication:\n1. Read WAL frames from the WAL file\n2. Skip the lock page\n3. Add checksums and build page index\n4. Write as immutable LTX file\n5. Upload to storage backend\n","isInternal":false,"tokens":1303,"sizeBytes":4730},{"name":"PATTERNS.md","path":"skills/litestream/references/PATTERNS.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/skills/litestream/references/PATTERNS.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Litestream Code Patterns Reference\n\nCondensed code patterns and anti-patterns for agents writing Litestream code.\n\n## Architectural Boundaries\n\n```\nDB Layer (db.go) → Database state, restoration, monitoring\nReplica Layer (replica.go) → Replication mechanics only\nStorage Layer → ReplicaClient implementations\n```\n\n### DO: Handle database state in DB layer\n\n```go\nfunc (db *DB) init() error {\n if db.needsRestore() {\n if err := db.restore(); err != nil {\n return err\n }\n }\n return db.replica.Start() // Replica focuses only on replication\n}\n```\n\n### DON'T: Put database state logic in Replica layer\n\n```go\n// WRONG\nfunc (r *Replica) Start() error {\n if needsRestore() { // Wrong layer!\n restoreDatabase()\n }\n}\n```\n\n## Atomic File Operations\n\n### DO: Write to temp file, then rename\n\n```go\nfunc writeFileAtomic(path string, data []byte) error {\n dir := filepath.Dir(path)\n tmpFile, err := os.CreateTemp(dir, \".tmp-*\")\n if err != nil {\n return fmt.Errorf(\"create temp file: %w\", err)\n }\n tmpPath := tmpFile.Name()\n\n defer func() {\n if tmpFile != nil {\n tmpFile.Close()\n os.Remove(tmpPath)\n }\n }()\n\n if _, err := tmpFile.Write(data); err != nil {\n return fmt.Errorf(\"write temp file: %w\", err)\n }\n if err := tmpFile.Sync(); err != nil {\n return fmt.Errorf(\"sync temp file: %w\", err)\n }\n if err := tmpFile.Close(); err != nil {\n return fmt.Errorf(\"close temp file: %w\", err)\n }\n tmpFile = nil\n\n if err := os.Rename(tmpPath, path); err != nil {\n os.Remove(tmpPath)\n return fmt.Errorf(\"rename to final path: %w\", err)\n }\n return nil\n}\n```\n\n### DON'T: Write directly to final location\n\n```go\n// WRONG - Can leave partial files on failure\nos.WriteFile(path, data, 0644)\n```\n\n## Error Handling\n\n### DO: Return errors immediately\n\n```go\nfunc (db *DB) validatePosition() error {\n dpos, err := db.Pos()\n if err != nil {\n return err\n }\n rpos := replica.Pos()\n if dpos.TXID < rpos.TXID {\n return fmt.Errorf(\"database position (%v) behind replica (%v)\", dpos, rpos)\n }\n return nil\n}\n```\n\n### DON'T: Log and continue on critical errors\n\n```go\n// WRONG\nif err := processFile(file); err != nil {\n log.Printf(\"error: %v\", err) // Just logging!\n // Continuing is dangerous\n}\n```\n\n### DO: Return errors from loops\n\n```go\nfunc (db *DB) processFiles() error {\n for _, file := range files {\n if err := processFile(file); err != nil {\n return fmt.Errorf(\"process file %s: %w\", file, err)\n }\n }\n return nil\n}\n```\n\n## Locking Patterns\n\n### DO: Use Lock() for writes\n\n```go\nr.mu.Lock()\ndefer r.mu.Unlock()\nr.pos = pos\n```\n\n### DON'T: Use RLock for write operations\n\n```go\n// WRONG - Race condition\nr.mu.RLock()\ndefer r.mu.RUnlock()\nr.pos = pos // Writing with RLock!\n```\n\n### Lock Ordering\n\nAlways acquire in order: Store.mu → DB.mu → DB.chkMu → Replica.mu\n\n## Compaction Patterns\n\n### DO: Read from local when available\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil // Local copy is complete and consistent\n}\nreturn replica.Client.OpenLTXFile(...) // Fall back to remote\n```\n\n### DON'T: Read from remote during compaction\n\n```go\n// WRONG - Can get partial/corrupt data from eventually consistent storage\nf, err := client.OpenLTXFile(ctx, level, minTXID, maxTXID, 0, 0)\n```\n\n### DO: Preserve earliest timestamp\n\n```go\ninfo.CreatedAt = oldestSourceFile.CreatedAt\n```\n\n### DON'T: Use current time during compaction\n\n```go\n// WRONG - Loses timestamp granularity for point-in-time restores\ninfo := <x.FileInfo{CreatedAt: time.Now()}\n```\n\n## Lock Page Handling\n\nThe lock page at 1 GB (0x40000000) must always be skipped:\n\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n\n## Common Pitfalls\n\n1. **Mixing architectural concerns**: DB state logic in Replica layer\n2. **Recreating existing functionality**: Use `db.verify()` for snapshots\n3. **Ignoring lock page**: Must skip during replication and compaction\n4. **Generic error types**: Return `os.ErrNotExist` for missing files\n5. **Blocking iterators**: Use lazy pagination for `LTXFiles`\n6. **Ignoring context**: Check `ctx.Done()` in long operations\n","isInternal":false,"tokens":1195,"sizeBytes":4556},{"name":"REPLICA_CLIENT_GUIDE.md","path":"skills/litestream/references/REPLICA_CLIENT_GUIDE.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/skills/litestream/references/REPLICA_CLIENT_GUIDE.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# ReplicaClient Implementation Reference\n\nCondensed reference for agents implementing or modifying Litestream storage backends.\n\n## Interface Contract\n\nFrom `replica_client.go`:\n\n```go\ntype ReplicaClient interface {\n Type() string\n Init(ctx context.Context) error\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\n### Method Contracts\n\n**Type()**: Return identifier string (e.g., \"s3\", \"gcs\", \"file\").\n\n**Init()**: Initialize connection. Must be idempotent (no-op if already initialized).\n\n**LTXFiles()**: Return iterator sorted by MinTXID. `seek` starts from given TXID.\n`useMetadata=true` fetches accurate timestamps (required for point-in-time restore).\n`useMetadata=false` uses fast timestamps for normal operations.\n\n**OpenLTXFile()**: Open file for reading. Must support partial reads via `offset`/`size`.\nReturn `os.ErrNotExist` if file is missing.\n\n**WriteLTXFile()**: Write file to storage. Set `CreatedAt` from backend\nmetadata or upload time.\n\n**DeleteLTXFiles()**: Delete one or more files. Batch if possible.\n\n**DeleteAll()**: Delete all files for this database.\n\n## Implementation Checklist\n\n### Required\n\n- [ ] All interface methods implemented\n- [ ] `Init()` is idempotent\n- [ ] Partial reads supported (`offset`/`size` in `OpenLTXFile`)\n- [ ] `os.ErrNotExist` returned for missing files\n- [ ] Context cancellation handled in all methods\n- [ ] `CreatedAt` timestamps preserved in `WriteLTXFile`\n- [ ] Concurrent operations supported\n- [ ] Proper cleanup in `DeleteAll`\n\n### Optional\n\n- [ ] Connection pooling\n- [ ] Retry logic with exponential backoff\n- [ ] Request batching for deletes\n- [ ] Compression / encryption at rest\n- [ ] Bandwidth throttling\n\n## Eventual Consistency\n\nMany cloud backends (S3, R2, etc.) are eventually consistent:\n- Recently written files may not be immediately visible\n- Listed files may be only partially readable\n- Deletes may not take effect immediately\n\n### Pattern: Read Local First\n\nDuring compaction, always prefer local files:\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil\n}\nreturn replica.Client.OpenLTXFile(...)\n```\n\n### Pattern: Retry with Backoff\n\n```go\nbackoff := 100 * time.Millisecond\nfor i := 0; i < 5; i++ {\n reader, err := c.openFile(ctx, path, offset, size)\n if err == nil {\n return reader, nil\n }\n if errors.Is(err, os.ErrNotExist) {\n return nil, err // Don't retry definitive errors\n }\n select {\n case <-ctx.Done():\n return nil, ctx.Err()\n case <-time.After(backoff):\n backoff *= 2\n }\n}\n```\n\n### Pattern: Lazy Iterator\n\n```go\nfunc (c *Client) LTXFiles(...) (ltx.FileIterator, error) {\n return &lazyIterator{\n client: c,\n level: level,\n seek: seek,\n pageSize: 1000, // Paginate, don't load all at once\n }, nil\n}\n```\n\n## Error Handling\n\n### Standard Error Types\n\n```go\n// File not found\nreturn nil, os.ErrNotExist\n\n// Permission denied\nreturn nil, os.ErrPermission\n\n// Context cancelled\nreturn nil, ctx.Err()\n\n// Wrapped errors\nreturn nil, fmt.Errorf(\"s3 download failed: %w\", err)\n```\n\n### Retryable Error Classification\n\n```go\nfunc isRetryable(err error) bool {\n var netErr net.Error\n if errors.As(err, &netErr) && netErr.Temporary() {\n return true\n }\n // HTTP 429, 500, 502, 503, 504\n if errors.Is(err, context.DeadlineExceeded) {\n return true\n }\n return false\n}\n```\n\n## Common Mistakes\n\n### 1. Not Handling Partial Reads\n\n```go\n// WRONG - Ignores offset/size\nreturn c.storage.Download(path)\n\n// CORRECT\nif offset == 0 && size == 0 {\n return c.storage.Download(path)\n}\nreturn c.storage.DownloadRange(path, offset, offset+size-1)\n```\n\n### 2. Not Preserving CreatedAt\n\n```go\n// WRONG\nreturn <x.FileInfo{CreatedAt: time.Now()}\n\n// CORRECT - Use backend metadata\nreturn <x.FileInfo{CreatedAt: modTime}\n```\n\n### 3. Wrong Error Types\n\n```go\n// WRONG\nreturn nil, fmt.Errorf(\"not found\")\n\n// CORRECT\nif resp.StatusCode == 404 {\n return nil, os.ErrNotExist\n}\n```\n\n### 4. Ignoring Context\n\n```go\n// WRONG - Could run forever\nfor i := 0; i < 1000000; i++ {\n doWork()\n}\n\n// CORRECT\nselect {\ncase <-ctx.Done():\n return nil, ctx.Err()\ndefault:\n}\n```\n\n### 5. Loading All Files at Once\n\n```go\n// WRONG - Could be millions of files\nallFiles, _ := c.loadAllFiles(level)\n\n// CORRECT - Lazy pagination\nreturn &lazyIterator{pageSize: 1000}, nil\n```\n\n## Path Construction\n\n```go\nfunc (c *Client) ltxDir(level int) string {\n if level == SnapshotLevel {\n return path.Join(c.Path, \"snapshots\")\n }\n return path.Join(c.Path, \"ltx\", fmt.Sprintf(\"%04d\", level))\n}\n```\n\n## Reference Implementations\n\n- **Simplest**: `file/replica_client.go` (direct file I/O, no network)\n- **Most complete**: `s3/replica_client.go` (multipart uploads, retries, signing)\n\nStart with the file backend for understanding, then study S3 for advanced\npatterns.\n\n## Testing Requirements\n\n- Unit tests with >80% coverage\n- Integration tests with build tag\n- Test partial reads, concurrent operations, missing files\n- Run with `-race` flag\n- Verify `os.ErrNotExist` for missing files\n- Test context cancellation\n","isInternal":false,"tokens":1466,"sizeBytes":5555},{"name":"SQLITE_INTERNALS.md","path":"skills/litestream/references/SQLITE_INTERNALS.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/skills/litestream/references/SQLITE_INTERNALS.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# SQLite Internals Reference\n\nCondensed reference for agents working with SQLite internals in Litestream.\n\n## SQLite File Structure\n\n```\ndatabase.db Main database file (pages)\ndatabase.db-wal Write-ahead log\ndatabase.db-shm Shared memory file (coordination)\n```\n\n## Write-Ahead Log (WAL)\n\nWAL is SQLite's method for atomic commits:\n- Changes written to WAL first, database unchanged until checkpoint\n- Readers merge WAL + database for consistent view\n- Litestream monitors WAL and converts frames to LTX format\n\n### WAL File Structure\n\n```\n+------------------+\n| WAL Header | 32 bytes\n+------------------+\n| Frame 1 Header | 24 bytes\n| Frame 1 Data | PageSize bytes\n+------------------+\n| Frame 2 Header | 24 bytes\n| Frame 2 Data | PageSize bytes\n+------------------+\n| ... |\n```\n\n### WAL Header (32 bytes)\n\n```\nMagic (4B) | FileFormat (4B) | PageSize (4B) | Checkpoint (4B) |\nSalt1 (4B) | Salt2 (4B) | Checksum1 (4B) | Checksum2 (4B)\n```\n\nMagic: `0x377f0682` or `0x377f0683`\n\n### WAL Frame Header (24 bytes)\n\n```\nPageNumber (4B) | DbSize (4B) | Salt1 (4B) | Salt2 (4B) |\nChecksum1 (4B) | Checksum2 (4B)\n```\n\n## The 1 GB Lock Page\n\nSQLite reserves a page at exactly 0x40000000 bytes (1 GB) for locking. This is\nthe single most important SQLite detail for Litestream development.\n\n```go\nconst PENDING_BYTE = 0x40000000\n\nfunc LockPgno(pageSize int) uint32 {\n return uint32(PENDING_BYTE/pageSize) + 1\n}\n```\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n| 64 KB | 16385 |\n\nRules:\n- Cannot contain data; SQLite never writes user data here\n- Must be skipped during replication and compaction\n- Only affects databases > 1 GB\n- Page number changes with page size\n\nImplementation in Litestream:\n\n```go\nfor pgno := uint32(1); pgno <= maxPgno; pgno++ {\n if pgno == ltx.LockPgno(db.pageSize) {\n continue\n }\n processPage(pgno)\n}\n```\n\n## Page Structure\n\nSQLite divides databases into fixed-size pages (typically 4096 bytes):\n- Page numbers are 1-based\n- Types: B-tree interior, B-tree leaf, overflow, freelist, lock byte page\n\n## Transaction Types\n\n1. **Deferred** (default): Lock acquired on first use\n2. **Immediate**: RESERVED lock acquired immediately\n3. **Exclusive**: EXCLUSIVE lock acquired immediately\n\n### Lock Hierarchy\n\nUNLOCKED → SHARED → RESERVED → PENDING → EXCLUSIVE\n\n- SHARED: Multiple readers allowed\n- RESERVED: Signals intent to write\n- PENDING: Blocks new SHARED locks\n- EXCLUSIVE: Single writer, no readers\n\n## Long-Running Read Transaction\n\nLitestream maintains a read transaction for consistency:\n\n```go\nfunc (db *DB) initReadTx() error {\n tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{ReadOnly: true})\n if err != nil {\n return err\n }\n var dummy string\n err = tx.QueryRow(\"SELECT ''\").Scan(&dummy)\n if err != nil {\n tx.Rollback()\n return err\n }\n db.rtx = tx\n return nil\n}\n```\n\nPurpose:\n- Prevents checkpoint past our read point\n- Ensures consistent database view\n- Allows reading historical pages from WAL\n\n## Checkpoint Modes\n\n| Mode | Behavior |\n|----------|-----------------------------------------------|\n| PASSIVE | Non-blocking; fails if readers present |\n| FULL | Waits for readers; blocks new readers |\n| RESTART | Like FULL + resets WAL start (removed in #724) |\n| TRUNCATE | Like RESTART + truncates WAL to zero |\n\n### Litestream Checkpoint Strategy\n\n```\nWAL pages > TruncatePageN → TRUNCATE (emergency)\nWAL pages > MinCheckpointPageN → PASSIVE\nCheckpointInterval elapsed → PASSIVE\n```\n\nNote: RESTART mode permanently removed due to issue #724 (write-blocking).\n\n## Important SQLite Pragmas\n\n```sql\nPRAGMA journal_mode = WAL; -- Required for Litestream\nPRAGMA page_size; -- Get page size\nPRAGMA page_count; -- Total pages in database\nPRAGMA freelist_count; -- Free pages\nPRAGMA wal_checkpoint(PASSIVE); -- Non-blocking checkpoint\nPRAGMA wal_autocheckpoint = 10000; -- High threshold (prevent interference)\nPRAGMA busy_timeout = 5000; -- Wait 5s for locks\nPRAGMA synchronous = NORMAL; -- Safe with WAL\nPRAGMA cache_size = -64000; -- 64 MB cache\nPRAGMA integrity_check; -- Verify database integrity\n```\n\n## Critical SQLite Behaviors\n\n1. **Automatic checkpoint**: Default at 1000 WAL pages; set high threshold to\n prevent interference with Litestream's control\n2. **Busy timeout**: Default is 0 (immediate failure); set a reasonable timeout\n3. **Synchronous mode**: NORMAL is safe with WAL mode\n4. **Page cache**: In-memory cache; configure with negative KB or positive pages\n\n## WAL to LTX Conversion\n\nLitestream converts WAL frames to LTX:\n\n1. Read WAL header and validate magic number\n2. Iterate frames, reading page number and data\n3. Skip lock page (`pgno == LockPgno(pageSize)`)\n4. Calculate checksums\n5. Write as immutable LTX file with page index\n","isInternal":false,"tokens":1324,"sizeBytes":5151},{"name":"TESTING_GUIDE.md","path":"skills/litestream/references/TESTING_GUIDE.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/skills/litestream/references/TESTING_GUIDE.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Litestream Testing Reference\n\nCondensed reference for agents writing and debugging Litestream tests.\n\n## Testing Philosophy\n\n1. Test at multiple levels: unit, integration, end-to-end\n2. Focus on edge cases: >1 GB databases, eventual consistency\n3. Use real SQLite: avoid mocking SQLite behavior\n4. Always run with `-race` flag\n5. Use fixed seeds and timestamps for deterministic tests\n\n## Essential Commands\n\n```bash\n# Full test suite with race detection\ngo test -race -v ./...\n\n# Specific test areas\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\n\n# Coverage\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\ngo tool cover -func=coverage.out | grep total\n\n# Integration tests (backend-specific)\ngo test -v ./replica_client_test.go -integration s3\ngo test -v ./replica_client_test.go -integration gcs\n```\n\n## Lock Page Testing\n\nThe lock page at 1 GB must be skipped. Test with databases spanning this\nboundary using multiple page sizes:\n\n```bash\n./bin/litestream-test populate -db test.db -page-size 4096 -target-size 2GB\n./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica\n```\n\nTest all page size variants:\n\n| Page Size | Lock Page | Test Target |\n|-----------|-----------|-------------|\n| 4 KB | 262145 | >1.0 GB |\n| 8 KB | 131073 | >1.0 GB |\n| 16 KB | 65537 | >1.0 GB |\n| 32 KB | 32769 | >1.0 GB |\n\nVerify lock page is not present in any LTX file after replication.\n\n## Race Condition Testing\n\nRace-prone areas (always run with `-race`):\n\n1. **Position updates**: Concurrent `SetPos()` + `Pos()` on Replica\n2. **WAL monitoring**: Concurrent writes + monitor + checkpoint\n3. **Compaction**: Concurrent compaction at different levels\n\n### Pattern: Concurrent Test\n\n```go\nfunc TestConcurrent(t *testing.T) {\n ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\n defer cancel()\n\n var wg sync.WaitGroup\n // Start writer, monitor, checkpoint goroutines\n // Use ctx for shutdown, wg for cleanup\n wg.Wait()\n}\n```\n\n## Common Test Failures\n\n### 1. Database Locked\n\n```go\n// WRONG - WAL mode not enabled\ndb, _ := sql.Open(\"sqlite\", \"test.db\")\n\n// CORRECT\ndb, _ := sql.Open(\"sqlite\", \"test.db?_journal=WAL\")\n```\n\n### 2. Timing Issues\n\n```go\n// WRONG - Data may not be replicated yet\nWriteData(db)\nresult := ReadReplica()\n\n// CORRECT - Explicit sync\nWriteData(db)\ndb.Sync(context.Background())\nresult := ReadReplica()\n```\n\n### 3. Goroutine Leaks\n\n```go\n// WRONG - Goroutine outlives test\ngo func() {\n time.Sleep(10 * time.Second)\n doWork()\n}()\n\n// CORRECT - Use context + WaitGroup\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\nvar wg sync.WaitGroup\nwg.Add(1)\ngo func() {\n defer wg.Done()\n select {\n case <-ctx.Done():\n return\n case <-time.After(10 * time.Second):\n doWork()\n }\n}()\ncancel()\nwg.Wait()\n```\n\n### 4. File Handle Leaks\n\n```go\n// Always use defer for cleanup\nf, err := os.Open(\"test.db\")\nrequire.NoError(t, err)\ndefer f.Close()\n```\n\n## Test Helper Patterns\n\n```go\nfunc NewTestDB(t testing.TB) *litestream.DB {\n t.Helper()\n path := filepath.Join(t.TempDir(), \"test.db\")\n conn, err := sql.Open(\"sqlite\", path+\"?_journal=WAL\")\n require.NoError(t, err)\n _, err = conn.Exec(\"CREATE TABLE test (id INTEGER PRIMARY KEY, data BLOB)\")\n require.NoError(t, err)\n conn.Close()\n\n db := litestream.NewDB(path)\n db.MonitorInterval = 10 * time.Millisecond\n db.MinCheckpointPageN = 100\n err = db.Open()\n require.NoError(t, err)\n t.Cleanup(func() { db.Close(context.Background()) })\n return db\n}\n```\n\n## Coverage Requirements\n\n| Package | Target |\n|--------------------|--------|\n| Core (db, replica) | >80% |\n| Replica clients | >70% |\n| Utilities | >60% |\n\n## Table-Driven Tests\n\n```go\nfunc TestCheckpoint(t *testing.T) {\n tests := []struct {\n name string\n mode string\n wantErr bool\n }{\n {\"Passive\", \"PASSIVE\", false},\n {\"Full\", \"FULL\", false},\n {\"Truncate\", \"TRUNCATE\", false},\n {\"Invalid\", \"INVALID\", true},\n }\n for _, tt := range tests {\n t.Run(tt.name, func(t *testing.T) {\n // ...\n })\n }\n}\n```\n","isInternal":false,"tokens":1158,"sizeBytes":4339},{"name":"SKILL.md","path":"skills/litestream/SKILL.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/skills/litestream/SKILL.md","title":"litestream","category":"anthropic-skill","format":"markdown","content":"---\nname: litestream\ndescription: >-\n Expert knowledge for contributing to Litestream, a standalone disaster recovery\n tool for SQLite. Provides architectural understanding, code patterns, critical\n rules, and debugging procedures for WAL monitoring, LTX replication format,\n storage backend implementation, multi-level compaction, and SQLite page\n management. Use when working with Litestream source code, writing storage\n backends, debugging replication issues, implementing compaction logic, or\n handling SQLite WAL operations.\nlicense: Apache-2.0\nmetadata:\n author: benbjohnson\n version: \"1.0\"\n repository: https://github.com/benbjohnson/litestream\n---\n\n# Litestream Agent Skill\n\nLitestream is a standalone disaster recovery tool for SQLite. It runs as a\nbackground process, monitors the SQLite WAL (Write-Ahead Log), converts changes\nto immutable LTX files, and replicates them to cloud storage. It uses\n`modernc.org/sqlite` (pure Go, no CGO required).\n\n## Quick Start\n\n```bash\n# Build\ngo build -o bin/litestream ./cmd/litestream\n\n# Test (always use race detector)\ngo test -race -v ./...\n\n# Code quality\npre-commit run --all-files\n```\n\n## Critical Rules\n\nThese invariants must never be violated:\n\n### 1. Lock Page at 1GB\n\nSQLite reserves a page at byte offset 0x40000000 (1 GB). Always skip it during\nreplication and compaction. The page number varies by page size:\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n### 2. LTX Files Are Immutable\n\nOnce an LTX file is written, it must never be modified. New changes create new\nfiles. This guarantees point-in-time recovery integrity.\n\n### 3. Single Replica per Database\n\nEach database replicates to exactly one destination. The Replica component\nmanages replication mechanics; database state belongs in the DB layer.\n\n### 4. Read Local Before Remote During Compaction\n\nCloud storage is eventually consistent. Always read from local disk first:\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil // Use local copy\n}\nreturn replica.Client.OpenLTXFile(...) // Fall back to remote\n```\n\n### 5. Preserve Timestamps During Compaction\n\nSet the compacted file's `CreatedAt` to the earliest source file timestamp to\nmaintain temporal granularity for point-in-time restoration.\n\n```go\ninfo.CreatedAt = oldestSourceFile.CreatedAt\n```\n\n### 6. Use Lock() Not RLock() for Writes\n\n```go\n// CORRECT\nr.mu.Lock()\ndefer r.mu.Unlock()\nr.pos = pos\n\n// WRONG - race condition\nr.mu.RLock()\ndefer r.mu.RUnlock()\nr.pos = pos\n```\n\n### 7. Atomic File Operations\n\nAlways write to a temp file then rename. Never write directly to the final path.\n\n```go\ntmpFile, err := os.CreateTemp(dir, \".tmp-*\")\n// ... write data, sync ...\nos.Rename(tmpFile.Name(), finalPath)\n```\n\n## Architecture\n\n### System Layers\n\n| Layer | File(s) | Responsibility |\n|---------|--------------------------|-------------------------------------------|\n| App | `cmd/litestream/` | CLI commands, YAML/env config |\n| Store | `store.go` | Multi-DB coordination, compaction |\n| DB | `db.go` | Single DB management, WAL monitoring |\n| Replica | `replica.go` | Replication to one destination |\n| Storage | `*/replica_client.go` | Backend implementations (S3, GCS, etc.) |\n\nDatabase state logic belongs in the DB layer, not the Replica layer.\n\n### ReplicaClient Interface\n\nAll storage backends implement this interface from `replica_client.go`:\n\n```go\ntype ReplicaClient interface {\n Type() string\n Init(ctx context.Context) error\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\nKey contract details:\n- `OpenLTXFile` must return `os.ErrNotExist` when file is missing\n- `WriteLTXFile` must set `CreatedAt` from backend metadata or upload time\n- `LTXFiles` with `useMetadata=true` fetches accurate timestamps (for PIT restore)\n- `LTXFiles` with `useMetadata=false` uses fast timestamps (normal operations)\n\n### Lock Ordering\n\nAlways acquire locks in this order to prevent deadlocks:\n\n1. `Store.mu`\n2. `DB.mu`\n3. `DB.chkMu`\n4. `Replica.mu`\n\n### Core Components\n\n**DB** (`db.go`): Manages SQLite connection, WAL monitoring, checkpointing, and\nlong-running read transaction for consistency. Key fields: `path`, `db`, `rtx`\n(read transaction), `pageSize`, `notify` channel.\n\n**Replica** (`replica.go`): Tracks replication position (`ltx.Pos` with TXID,\nPageNo, Checksum). One replica per database.\n\n**Store** (`store.go`): Coordinates multiple databases and schedules compaction\nacross levels.\n\n## LTX File Format\n\nLTX (Log Transaction) files are immutable, checksummed archives of database\nchanges. Structure:\n\n```\n+------------------+\n| Header | 100 bytes (magic \"LTX1\", page size, TXID range, timestamp)\n+------------------+\n| Page Frames | 4-byte pgno + pageSize bytes data, per page\n+------------------+\n| Page Index | Binary search index for page lookup\n+------------------+\n| Trailer | 16 bytes (post-apply checksum, file checksum)\n+------------------+\n```\n\n### Naming Convention\n\n```\nFormat: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\nExample: 0000000000000001-0000000000000064.ltx (TXID 1-100)\n```\n\n### Compaction Levels\n\n```\nLevel 0: /ltx/0000/ Raw LTX files (no compaction)\nLevel 1: /ltx/0001/ Compacted periodically\nLevel 2: /ltx/0002/ Compacted less frequently\n```\n\nDefault compaction levels: L0 (raw), L1 (30s), L2 (5min), L3 (1h), plus daily\nsnapshots. Compaction merges files by deduplicating pages (latest version wins)\nand always skips the lock page.\n\n## Code Patterns\n\n### DO\n\n- Return errors immediately; let callers decide handling\n- Use `fmt.Errorf(\"context: %w\", err)` for error wrapping\n- Handle database state in the DB layer, not Replica\n- Use `db.verify()` to trigger snapshots (don't reimplement)\n- Test with race detector: `go test -race`\n- Use lazy iterators for `LTXFiles` (paginate, don't load all at once)\n\n### DON'T\n\n- Write data at the 1 GB lock page boundary\n- Modify LTX files after creation\n- Put database state logic in the Replica layer\n- Use `RLock()` when writing shared state\n- Write directly to final file paths (use temp + rename)\n- Ignore context cancellation in long operations\n- Return generic errors instead of `os.ErrNotExist` for missing files\n\n## Specialized Knowledge Areas\n\nLoad reference files on demand based on the task:\n\n| Task | Reference File |\n|-----------------------------------|-----------------------------------------|\n| Understanding system design | `references/ARCHITECTURE.md` |\n| Writing or reviewing code | `references/PATTERNS.md` |\n| Working with LTX files | `references/LTX_FORMAT.md` |\n| WAL monitoring or page operations | `references/SQLITE_INTERNALS.md` |\n| Implementing storage backends | `references/REPLICA_CLIENT_GUIDE.md` |\n| Writing or debugging tests | `references/TESTING_GUIDE.md` |\n\n## Common Debugging Procedures\n\n### Replication Not Working\n\n1. Verify WAL mode: `PRAGMA journal_mode` must return `wal`\n2. Check monitor interval and that the monitor goroutine is running\n3. Confirm `db.notify` channel is being signaled on WAL changes\n4. Check replica position: `replica.Pos()` should advance with writes\n5. Look for `os.ErrNotExist` from `OpenLTXFile` (file not replicated yet)\n\n### Large Database Issues (>1 GB)\n\n1. Verify lock page is being skipped: check `ltx.LockPgno(pageSize)`\n2. Test with multiple page sizes (4K, 8K, 16K, 32K)\n3. Run with databases both smaller and larger than 1 GB\n4. Ensure page iteration loops include the `continue` guard for lock page\n\n### Compaction Problems\n\n1. Confirm local L0 files exist before compaction reads them\n2. Check that `CreatedAt` timestamps are preserved (earliest source)\n3. Verify compaction level intervals in `Store.levels`\n4. Look for eventual consistency issues if reading from remote storage\n\n### Storage Backend Issues\n\n1. Return `os.ErrNotExist` for missing files (not generic errors)\n2. Support partial reads via `offset`/`size` in `OpenLTXFile`\n3. Handle context cancellation in all methods\n4. Test concurrent operations with `-race` flag\n5. For eventually consistent backends, add retry logic with backoff\n\n### Corrupted or Missing LTX Files\n\n1. Check logs for `LTXError` messages - they include context (Op, Path, Level, TXID) and recovery hints\n2. Common error messages: \"nonsequential page numbers\", \"non-contiguous transaction files\", \"ltx validation failed\"\n3. Manual fix: `litestream reset ` clears local LTX state and forces fresh snapshot on next sync (database file is not modified)\n4. Automatic fix: set `auto-recover: true` on the replica config to auto-reset on LTX errors (disabled by default)\n5. Reference: `cmd/litestream/reset.go`, `replica.go` (auto-recover logic), `db.go` (`ResetLocalState`)\n\n## Contribution Guidelines\n\n### What's Accepted\n\n- Bug fixes and patches (welcome)\n- Documentation improvements\n- Small code improvements and performance optimizations\n- Security vulnerability reports (report privately)\n\n### Discuss First\n\n- Feature requests: open an issue before implementing\n- Large changes: discuss approach in an issue first\n\n### Pre-Submit Checklist\n\n- [ ] Read relevant docs from the reference table above\n- [ ] Follow patterns in `references/PATTERNS.md`\n- [ ] Run `go test -race -v ./...`\n- [ ] Run `pre-commit run --all-files`\n- [ ] For page iteration: test with >1 GB databases\n- [ ] Show investigation evidence in PR (see CONTRIBUTING.md)\n\n## Testing\n\n```bash\n# Full test suite with race detection\ngo test -race -v ./...\n\n# Specific areas\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\n\n# Coverage\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\n```\n\nKey testing areas:\n- Lock page handling with >1 GB databases and multiple page sizes\n- Race conditions in position updates, WAL monitoring, and checkpointing\n- Eventual consistency in storage backend operations\n- Atomic file operations and cleanup on error paths\n\n## Environment Validation\n\nRun `scripts/validate-setup.sh` to verify your development environment is\ncorrectly configured for Litestream development.\n","frontmatter":{"name":"litestream","description":"Expert knowledge for contributing to Litestream, a standalone disaster recovery tool for SQLite. Provides architectural understanding, code patterns, critical rules, and debugging procedures for WAL monitoring, LTX replication format, storage backend implementation, multi-level compaction, and SQLite page management. Use when working with Litestream source code, writing storage backends, debugging replication issues, implementing compaction logic, or handling SQLite WAL operations.","license":"Apache-2.0","metadata":{"author":"benbjohnson","version":"1.0","repository":"https://github.com/benbjohnson/litestream"}},"isInternal":false,"tokens":2704,"sizeBytes":10998},{"name":"ltx-compaction-specialist.md","path":".claude/agents/ltx-compaction-specialist.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/agents/ltx-compaction-specialist.md","title":"Subagent: ltx-compaction-specialist","category":"subagent-persona","format":"markdown","content":"---\nrole: LTX Format and Compaction Specialist\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# LTX Compaction Specialist Agent\n\nYou are an expert in the LTX (Log Transaction) format and multi-level compaction strategies for Litestream.\n\n## Core Knowledge\n\n### LTX File Format\n```\n┌─────────────────────┐\n│ Header │ 84 bytes\n├─────────────────────┤\n│ Page Frames │ Variable\n├─────────────────────┤\n│ Page Index │ Binary search structure\n├─────────────────────┤\n│ Trailer │ 16 bytes\n└─────────────────────┘\n```\n\n### File Naming Convention\n```\nMMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\nWhere:\n M = MinTXID (16 hex digits)\n N = MaxTXID (16 hex digits)\nExample: 0000000000000001-0000000000000064.ltx\n```\n\n## Default Compaction Levels\n\n### Level Structure\n```\nLevel 0: Raw (no compaction)\nLevel 1: 30-second windows\nLevel 2: 5-minute windows\nLevel 3: 1-hour windows\nSnapshots: Daily full database\n```\n\n### Critical Compaction Rules\n\n1. **ALWAYS Read Local First**:\n ```go\n // CORRECT - Handles eventual consistency\n f, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\n if err == nil {\n return f, nil // Use local file\n }\n // Only fall back to remote if local doesn't exist\n return replica.Client.OpenLTXFile(...)\n ```\n\n2. **Preserve Timestamps**:\n ```go\n // Keep earliest CreatedAt\n info, err := replica.Client.WriteLTXFile(ctx, level, minTXID, maxTXID, reader)\n if err != nil {\n return nil, fmt.Errorf(\"write ltx file: %w\", err)\n }\n info.CreatedAt = oldestSourceFile.CreatedAt\n ```\n\n3. **Skip Lock Page**:\n ```go\n if pgno == ltx.LockPgno(pageSize) {\n continue\n }\n ```\n\n## Compaction Algorithm\n\n```go\nfunc compactLTXFiles(files []*LTXFile) (*LTXFile, error) {\n // 1. Create page map (newer overwrites older)\n pageMap := make(map[uint32]Page)\n for _, file := range files {\n for _, page := range file.Pages {\n pageMap[page.Number] = page\n }\n }\n\n // 2. Create new LTX with merged pages\n merged := <XFile{\n MinTXID: files[0].MinTXID,\n MaxTXID: files[len(files)-1].MaxTXID,\n }\n\n // 3. Add pages in order (skip lock page!)\n for pgno := uint32(1); pgno <= maxPgno; pgno++ {\n if pgno == LockPageNumber(pageSize) {\n continue\n }\n if page, ok := pageMap[pgno]; ok {\n merged.Pages = append(merged.Pages, page)\n }\n }\n\n return merged, nil\n}\n```\n\n## Key Properties\n\n### Immutability\n- LTX files are NEVER modified after creation\n- New changes create new files\n- Compaction creates new merged files\n\n### Checksums\n- CRC-64 ECMA for integrity\n- `PreApplyChecksum`/`PostApplyChecksum` on the header/trailer bracketing file state\n- `FileChecksum` covering the entire file contents\n\n### Page Index\n- Exposed via `ltx.DecodePageIndex`\n- Tracks page number plus offset/size of the encoded payload\n- Located by seeking from the end of the file using trailer metadata\n\n## Common Issues\n\n1. **Partial Reads**: Remote storage may return incomplete files\n2. **Race Conditions**: Multiple compactions running\n3. **Timestamp Loss**: Not preserving original CreatedAt\n4. **Lock Page**: Including 1GB lock page in compacted files\n5. **Memory Usage**: Loading entire files for compaction\n6. **Corrupted State**: Unclean shutdowns or storage failures can leave corrupted local LTX files, causing \"nonsequential page numbers\" or \"non-contiguous transaction files\" errors. Recovery: `litestream reset ` (manual) or `auto-recover: true` replica config (automatic). See `cmd/litestream/reset.go` and `replica.go`\n\n## Testing\n\n```bash\n# Test compaction\ngo test -v -run TestStore_CompactDB ./...\n\n# Test with eventual consistency\ngo test -v -run TestStore_CompactDB_RemotePartialRead ./...\n\n# Manual inspection\nlitestream ltx /path/to/db.sqlite\n# For deeper inspection use the Go API (ltx.NewDecoder)\n```\n\n## References\n- docs/LTX_FORMAT.md - Complete format specification\n- store.go - Compaction scheduling\n- db.go - Compaction implementation\n- github.com/superfly/ltx - LTX library\n","frontmatter":{"role":"LTX Format and Compaction Specialist","tools":["read","write","edit","grep","bash"],"priority":"high"},"isInternal":false,"tokens":1127,"sizeBytes":4359},{"name":"performance-optimizer.md","path":".claude/agents/performance-optimizer.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/agents/performance-optimizer.md","title":"Subagent: performance-optimizer","category":"subagent-persona","format":"markdown","content":"---\nrole: Performance Optimizer\ntools:\n - read\n - write\n - edit\n - bash\n - grep\npriority: medium\n---\n\n# Performance Optimizer Agent\n\nYou specialize in optimizing Litestream for speed, memory usage, and resource efficiency.\n\n## Key Performance Areas\n\n### O(n) Operations to Optimize\n\n1. **Page Iteration**\n ```go\n // Cache page index\n const DefaultEstimatedPageIndexSize = 32 * 1024 // 32KB\n\n // Fetch end of file first for page index\n offset := info.Size - DefaultEstimatedPageIndexSize\n if offset < 0 {\n offset = 0\n }\n ```\n\n2. **File Listing**\n ```go\n // Cache file listings\n type FileCache struct {\n files []FileInfo\n timestamp time.Time\n ttl time.Duration\n }\n ```\n\n3. **Compaction**\n ```go\n // Limit concurrent compactions\n sem := make(chan struct{}, maxConcurrentCompactions)\n ```\n\n## Memory Optimization\n\n### Page Buffer Pooling\n```go\nvar pagePool = sync.Pool{\n New: func() interface{} {\n b := make([]byte, 4096) // Default page size\n return &b\n },\n}\n\nfunc getPageBuffer() []byte {\n return *pagePool.Get().(*[]byte)\n}\n\nfunc putPageBuffer(b []byte) {\n pagePool.Put(&b)\n}\n```\n\n### Streaming Instead of Loading\n```go\n// BAD - Loads entire file\ndata, err := os.ReadFile(path)\n\n// GOOD - Streams data\nf, err := os.Open(path)\ndefer f.Close()\nio.Copy(dst, f)\n```\n\n## Concurrency Patterns\n\n### Proper Locking\n```go\n// Read-heavy optimization\ntype Store struct {\n mu sync.RWMutex // Use RWMutex for read-heavy\n}\n\nfunc (s *Store) Read() {\n s.mu.RLock()\n defer s.mu.RUnlock()\n // Read operation\n}\n\nfunc (s *Store) Write() {\n s.mu.Lock()\n defer s.mu.Unlock()\n // Write operation\n}\n```\n\n### Channel Patterns\n```go\n// Batch processing\nbatch := make([]Item, 0, batchSize)\nticker := time.NewTicker(batchInterval)\n\nfor {\n select {\n case item := <-input:\n batch = append(batch, item)\n if len(batch) >= batchSize {\n processBatch(batch)\n batch = batch[:0]\n }\n case <-ticker.C:\n if len(batch) > 0 {\n processBatch(batch)\n batch = batch[:0]\n }\n }\n}\n```\n\n## I/O Optimization\n\n### Buffered I/O\n```go\n// Use buffered writers\nbw := bufio.NewWriterSize(w, 64*1024) // 64KB buffer\ndefer bw.Flush()\n\n// Use buffered readers\nbr := bufio.NewReaderSize(r, 64*1024)\n```\n\n### Parallel Downloads\n```go\nfunc downloadParallel(files []string) {\n var wg sync.WaitGroup\n sem := make(chan struct{}, 5) // Limit to 5 concurrent\n\n for _, file := range files {\n wg.Add(1)\n go func(f string) {\n defer wg.Done()\n sem <- struct{}{}\n defer func() { <-sem }()\n\n download(f)\n }(file)\n }\n wg.Wait()\n}\n```\n\n## Caching Strategy\n\n### LRU Cache Implementation\n```go\ntype LRUCache struct {\n capacity int\n items map[string]*list.Element\n list *list.List\n mu sync.RWMutex\n}\n\nfunc (c *LRUCache) Get(key string) (interface{}, bool) {\n c.mu.RLock()\n elem, ok := c.items[key]\n c.mu.RUnlock()\n\n if !ok {\n return nil, false\n }\n\n c.mu.Lock()\n c.list.MoveToFront(elem)\n c.mu.Unlock()\n\n return elem.Value, true\n}\n```\n\n## Profiling Tools\n\n### CPU Profiling\n```bash\n# Generate CPU profile\ngo test -cpuprofile=cpu.prof -bench=.\n\n# Analyze\ngo tool pprof cpu.prof\n(pprof) top10\n(pprof) list functionName\n```\n\n### Memory Profiling\n```bash\n# Generate memory profile\ngo test -memprofile=mem.prof -bench=.\n\n# Analyze allocations\ngo tool pprof -alloc_space mem.prof\n```\n\n### Trace Analysis\n```bash\n# Generate trace\ngo test -trace=trace.out\n\n# View trace\ngo tool trace trace.out\n```\n\n## Configuration Tuning\n\n### SQLite Pragmas\n```sql\nPRAGMA cache_size = -64000; -- 64MB cache\nPRAGMA synchronous = NORMAL; -- Balance safety/speed\nPRAGMA wal_autocheckpoint = 10000; -- Larger WAL before checkpoint\nPRAGMA busy_timeout = 5000; -- 5 second timeout\n```\n\n### Litestream Settings\n```yaml\n# Optimal intervals\nmin-checkpoint-page-n: 1000\ntruncate-page-n: 121359\nmonitor-interval: 1s\ncheckpoint-interval: 1m\n```\n\n## Benchmarks to Run\n\n```bash\n# Core operations\ngo test -bench=BenchmarkWALRead\ngo test -bench=BenchmarkLTXWrite\ngo test -bench=BenchmarkCompaction\ngo test -bench=BenchmarkPageIteration\n\n# With memory stats\ngo test -bench=. -benchmem\n```\n\n## Common Performance Issues\n\n1. **Not pooling buffers** - Creates garbage\n2. **Loading entire files** - Use streaming\n3. **Excessive locking** - Use RWMutex\n4. **No caching** - Repeated expensive operations\n5. **Serial processing** - Could parallelize\n6. **Small buffers** - Increase buffer sizes\n\n## References\n- Go performance tips: https://go.dev/doc/perf\n- SQLite optimization: https://sqlite.org/optoverview.html\n- Profiling guide: https://go.dev/blog/pprof\n","frontmatter":{"role":"Performance Optimizer","tools":["read","write","edit","bash","grep"],"priority":"medium"},"isInternal":false,"tokens":1291,"sizeBytes":4803},{"name":"replica-client-developer.md","path":".claude/agents/replica-client-developer.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/agents/replica-client-developer.md","title":"Subagent: replica-client-developer","category":"subagent-persona","format":"markdown","content":"---\nrole: Replica Client Developer\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# Replica Client Developer Agent\n\nYou specialize in implementing and maintaining storage backend clients for Litestream replication.\n\n## Core Knowledge\n\n### ReplicaClient Interface\n\nEvery storage backend MUST implement:\n```go\ntype ReplicaClient interface {\n Type() string\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\n**LTXFiles useMetadata parameter**:\n- When `useMetadata=true`: Fetch accurate timestamps from backend metadata (slower, required for point-in-time restore)\n- When `useMetadata=false`: Use fast timestamps from file listing (faster, suitable for replication monitoring)\n\n### Critical Patterns\n\n1. **Eventual Consistency Handling**:\n - Storage may not immediately reflect writes\n - Files may be partially available\n - ALWAYS prefer local files during compaction\n\n2. **Atomic Operations**:\n ```go\n // Write to temp, then rename\n tmpPath := path + \".tmp\"\n // Write to tmpPath\n os.Rename(tmpPath, path)\n ```\n\n3. **Error Types**:\n - Return `os.ErrNotExist` for missing files\n - Wrap errors with context: `fmt.Errorf(\"operation: %w\", err)`\n\n4. **ResumableReader Support**:\n - `OpenLTXFile` MUST support the `offset` parameter for range requests\n - `internal/resumable_reader.go` wraps streams with auto-reconnection on idle timeouts\n - During restore, streams may sit idle while compactor processes other files\n - If `offset` is ignored, restore operations fail on connection timeouts\n\n### ReplicaClientV3 Interface (Optional)\n\nBackends supporting v0.3.x backward-compatible restore should implement:\n```go\ntype ReplicaClientV3 interface {\n GenerationsV3(ctx context.Context) ([]string, error)\n SnapshotsV3(ctx context.Context, generation string) ([]SnapshotInfoV3, error)\n WALSegmentsV3(ctx context.Context, generation string) ([]WALSegmentInfoV3, error)\n OpenSnapshotV3(ctx context.Context, generation string, index int) (io.ReadCloser, error)\n OpenWALSegmentV3(ctx context.Context, generation string, index int, offset int64) (io.ReadCloser, error)\n}\n```\n\nSee `v3.go` for type definitions and `s3/replica_client.go` for reference implementation.\n\n## Implementation Checklist\n\n### New Backend Requirements\n\n- [ ] Implement ReplicaClient interface\n- [ ] Handle partial reads (offset/size)\n- [ ] Support seek parameter for pagination\n- [ ] Preserve CreatedAt timestamps when metadata is available\n- [ ] Handle eventual consistency\n- [ ] Implement proper error types\n- [ ] Add integration tests\n- [ ] Document configuration\n\n### Testing Requirements\n\n```bash\n# Integration test\ngo test -v ./replica_client_test.go -integration [backend]\n\n# Race conditions\ngo test -race -v ./[backend]/...\n\n# Large files (>1GB)\n./bin/litestream-test populate -target-size 2GB\n```\n\n## Existing Backends Reference\n\n### Study These Implementations\n\n- `s3/replica_client.go` - AWS S3 (most complete)\n- `gs/replica_client.go` - Google Cloud Storage\n- `abs/replica_client.go` - Azure Blob Storage\n- `file/replica_client.go` - Local filesystem (simplest)\n- `sftp/replica_client.go` - SSH File Transfer\n- `nats/replica_client.go` - NATS JetStream (newest)\n- `oss/replica_client.go` - Alibaba Cloud OSS\n\n## Common Pitfalls\n\n1. Not handling eventual consistency\n2. Missing atomic write operations\n3. Incorrect error types\n4. Not preserving timestamps\n5. Forgetting partial read support\n6. No retry logic for transient failures\n7. Not supporting `offset` parameter in `OpenLTXFile` — breaks `ResumableReader` during restore\n\n## Configuration Pattern\n\n```yaml\nreplica:\n type: [backend]\n option1: value1\n option2: value2\n```\n\n## References\n\n- docs/REPLICA_CLIENT_GUIDE.md - Complete implementation guide\n- replica_client.go - Interface definition\n- v3.go - ReplicaClientV3 interface and v0.3.x types\n- internal/resumable_reader.go - ResumableReader for restore resilience\n- replica_client_test.go - Test suite\n","frontmatter":{"role":"Replica Client Developer","tools":["read","write","edit","grep","bash"],"priority":"high"},"isInternal":false,"tokens":1087,"sizeBytes":4355},{"name":"sqlite-expert.md","path":".claude/agents/sqlite-expert.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/agents/sqlite-expert.md","title":"Subagent: sqlite-expert","category":"subagent-persona","format":"markdown","content":"---\nrole: SQLite WAL and Page Expert\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# SQLite Expert Agent\n\nYou are a SQLite internals expert specializing in WAL (Write-Ahead Log) operations and page management for the Litestream project.\n\n## Core Knowledge\n\n### Critical SQLite Concepts\n1. **1GB Lock Page** (MUST KNOW):\n - Located at exactly 0x40000000 (1,073,741,824 bytes)\n - Page number varies by page size:\n - 4KB pages: 262145\n - 8KB pages: 131073\n - 16KB pages: 65537\n - 32KB pages: 32769\n - MUST be skipped in all iterations\n - Cannot contain data\n\n2. **WAL Structure**:\n - 32-byte header with magic number\n - Frames with 24-byte headers\n - Cumulative checksums\n - Salt values for verification\n\n3. **Page Types**:\n - B-tree interior/leaf pages\n - Overflow pages\n - Freelist pages\n - Lock byte page (at 1GB)\n\n## Primary Responsibilities\n\n### WAL Monitoring\n- Monitor WAL file changes in `db.go`\n- Ensure proper checksum verification\n- Handle WAL frame reading correctly\n- Convert WAL frames to LTX format\n\n### Page Management\n- Always skip lock page during iteration\n- Handle various page sizes correctly\n- Verify page integrity\n- Manage page caching efficiently\n\n### Testing Requirements\n- Create test databases >1GB\n- Test all page sizes (4KB, 8KB, 16KB, 32KB)\n- Verify lock page skipping\n- Test WAL checkpoint modes\n\n## Code Patterns\n\n### Correct Lock Page Handling\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue // Skip lock page\n}\n```\n\n### WAL Reading\n```go\n// Always verify magic number\nmagic := binary.BigEndian.Uint32(header.Magic[:])\nif magic != 0x377f0682 && magic != 0x377f0683 {\n return errors.New(\"invalid WAL magic\")\n}\n```\n\n## Common Mistakes to Avoid\n1. Not skipping lock page at 1GB\n2. Incorrect checksum calculations\n3. Wrong byte order (use BigEndian)\n4. Not handling all page sizes\n5. Direct file manipulation (use SQLite API)\n\n## References\n- docs/SQLITE_INTERNALS.md - Complete SQLite internals guide\n- docs/LTX_FORMAT.md - LTX conversion details\n- db.go - WAL monitoring implementation\n","frontmatter":{"role":"SQLite WAL and Page Expert","tools":["read","write","edit","grep","bash"],"priority":"high"},"isInternal":false,"tokens":561,"sizeBytes":2113},{"name":"test-engineer.md","path":".claude/agents/test-engineer.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/agents/test-engineer.md","title":"Subagent: test-engineer","category":"subagent-persona","format":"markdown","content":"---\nrole: Test Engineer\ntools:\n - read\n - write\n - edit\n - bash\n - grep\npriority: medium\n---\n\n# Test Engineer Agent\n\nYou specialize in creating and maintaining comprehensive test suites for Litestream, with focus on edge cases and race conditions.\n\n## Critical Test Scenarios\n\n### 1GB Lock Page Testing\n\n**MUST TEST**: Databases crossing the 1GB boundary\n\n```bash\n# Create >1GB test database\nsqlite3 large.db < c.failAfter {\n return errors.New(\"simulated failure\")\n }\n return c.ReplicaClient.WriteLTXFile(...)\n}\n```\n\n## Coverage Requirements\n\n### Minimum Coverage\n- Core packages: >80%\n- Storage backends: >70%\n- Critical paths: 100%\n\n### Generate Coverage Report\n```bash\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\n```\n\n## Common Test Mistakes\n\n1. Not testing with databases >1GB\n2. Forgetting race detector\n3. Not testing all page sizes\n4. Missing eventual consistency tests\n5. No error injection tests\n6. Ignoring benchmark regressions\n\n## CI/CD Integration\n\n```yaml\n# .github/workflows/test.yml\n- name: Run tests with race detector\n run: go test -race -v ./...\n\n- name: Test large databases\n run: ./scripts/test-large-db.sh\n\n- name: Integration tests\n run: ./scripts/test-integration.sh\n```\n\n## References\n- docs/TESTING_GUIDE.md - Complete testing guide\n- replica_client_test.go - Integration test patterns\n- db_test.go - Unit test examples\n","frontmatter":{"role":"Test Engineer","tools":["read","write","edit","bash","grep"],"priority":"medium"},"isInternal":false,"tokens":1100,"sizeBytes":4105},{"name":"add-storage-backend.md","path":".claude/commands/add-storage-backend.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/add-storage-backend.md","title":"Command: /add-storage-backend","category":"command-prompt","format":"markdown","content":"---\ndescription: Create a new storage backend implementation\n---\n\n# Add Storage Backend Command\n\nCreate a new storage backend implementation for Litestream with all required components.\n\n## Steps\n\n1. **Create Package Directory**\n ```bash\n mkdir -p {{backend_name}}\n ```\n\n2. **Implement ReplicaClient Interface**\n Create `{{backend_name}}/replica_client.go`:\n ```go\n package {{backend_name}}\n\n type ReplicaClient struct {\n // Configuration fields\n }\n\n func (c *ReplicaClient) Type() string {\n return \"{{backend_name}}\"\n }\n\n func (c *ReplicaClient) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {\n // List files at level\n // When useMetadata=true, fetch accurate timestamps from backend metadata\n }\n\n func (c *ReplicaClient) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {\n // Open file for reading\n }\n\n func (c *ReplicaClient) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {\n // Write file atomically\n }\n\n func (c *ReplicaClient) DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error {\n // Delete files\n }\n\n func (c *ReplicaClient) DeleteAll(ctx context.Context) error {\n // Remove all files for replica\n }\n ```\n\n3. **Add Configuration Parsing**\n Update `cmd/litestream/config.go`:\n ```go\n case \"{{backend_name}}\":\n client = &{{backend_name}}.ReplicaClient{\n // Parse config\n }\n ```\n\n4. **Create Integration Tests**\n Create `{{backend_name}}/replica_client_test.go`:\n ```go\n func TestReplicaClient_{{backend_name}}(t *testing.T) {\n if !*integration || *backend != \"{{backend_name}}\" {\n t.Skip(\"{{backend_name}} integration test skipped\")\n }\n // Test implementation\n }\n ```\n\n5. **Add Documentation**\n Update README.md with configuration example:\n ```yaml\n replica:\n type: {{backend_name}}\n option1: value1\n option2: value2\n ```\n\n## Key Requirements\n\n- Handle eventual consistency\n- Implement atomic writes (temp file + rename)\n- Support partial reads (offset/size)\n- Preserve CreatedAt timestamps in returned FileInfo\n- Return proper error types (os.ErrNotExist)\n\n## Testing\n\n```bash\n# Run integration tests\ngo test -v ./replica_client_test.go -integration {{backend_name}}\n\n# Test with race detector\ngo test -race -v ./{{backend_name}}/...\n```\n","frontmatter":{"description":"Create a new storage backend implementation"},"isInternal":false,"tokens":634,"sizeBytes":2516},{"name":"analyze-ltx.md","path":".claude/commands/analyze-ltx.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/analyze-ltx.md","title":"Command: /analyze-ltx","category":"command-prompt","format":"markdown","content":"Analyze LTX file issues in Litestream. This command helps diagnose problems with LTX files, including corruption, missing files, and consistency issues.\n\nFirst, understand the context:\n- What error messages are being reported?\n- Which storage backend is being used?\n- Are there any eventual consistency issues?\n\nThen perform the analysis:\n\n1. **Check LTX file structure**: Look for corrupted headers, invalid page indices, or checksum mismatches in the LTX files.\n\n2. **Verify file continuity**: Ensure there are no gaps in the TXID sequence that could prevent restoration.\n\n3. **Check compaction issues**: Look for problems during compaction that might corrupt files, especially with eventually consistent storage.\n\n4. **Analyze page sequences**: Verify that page numbers are sequential and the lock page at 1GB is properly skipped.\n\n5. **Review storage backend behavior**: Check if the storage backend has eventual consistency that might cause partial reads during compaction.\n\nKey files to examine:\n- `db.go`: WAL monitoring and LTX generation\n- `replica_client.go`: Storage interface\n- `store.go`: Compaction logic\n- Backend-specific client in `s3/`, `gs/`, etc.\n\nCommon issues to look for:\n- \"nonsequential page numbers\" errors (corrupted compaction)\n- \"EOF\" errors (partial file reads)\n- Missing TXID ranges (failed uploads)\n- Lock page at 0x40000000 not being skipped\n\nUse the testing harness to reproduce:\n```bash\n./bin/litestream-test validate -source-db test.db -replica-url [URL]\n```\n\n## Recovery Options\n\nWhen analysis reveals corrupted or missing LTX files, two recovery mechanisms are available:\n\n**Manual recovery** with `litestream reset`:\n```bash\nlitestream reset /path/to/database.db\n```\nClears local LTX state from the metadata directory. The database file is not modified.\nNext sync creates a fresh snapshot. See `cmd/litestream/reset.go` for implementation.\n\n**Automatic recovery** with `auto-recover` config:\n```yaml\ndbs:\n - path: /path/to/database.db\n replicas:\n - url: s3://bucket/path\n auto-recover: true\n```\nWhen enabled, Litestream automatically resets local state when LTX errors are detected\nduring sync. Disabled by default. See `replica.go` (auto-recover logic) for implementation.\n","isInternal":false,"tokens":501,"sizeBytes":2229},{"name":"debug-ipc.md","path":".claude/commands/debug-ipc.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/debug-ipc.md","title":"Command: /debug-ipc","category":"command-prompt","format":"markdown","content":"---\ndescription: Debug IPC Unix socket issues\n---\n\n# Debug IPC Command\n\nDiagnose issues with the Litestream IPC control socket (`server.go`).\n\n## 1. Socket Configuration Check\n\nVerify the socket is enabled and correctly configured:\n\n```yaml\n# litestream.yml\nsocket:\n enabled: true # Default: false\n path: /var/run/litestream.sock # Default path\n permissions: 0600 # Default permissions\n```\n\nSource: `SocketConfig` struct in `server.go:17-21`, defaults in `DefaultSocketConfig()`.\n\n## 2. Endpoint Reference\n\nAll endpoints are HTTP over Unix socket (`server.go:74-87`):\n\n| Method | Path | Request Body | Response |\n|--------|------|-------------|----------|\n| `GET` | `/info` | — | `{version, pid, uptime_seconds, started_at, database_count}` |\n| `GET` | `/list` | — | `{databases: [{path, status, last_sync_at}]}` |\n| `GET` | `/txid?path=` | — | `{txid}` |\n| `POST` | `/register` | `{path, replica_url}` | `{status, path}` |\n| `POST` | `/unregister` | `{path, timeout?}` | `{status, path}` |\n| `POST` | `/start` | `{path, timeout?}` | `{status, path}` |\n| `POST` | `/stop` | `{path, timeout?}` | `{status, path}` |\n| `GET` | `/debug/pprof/` | — | Standard Go pprof index |\n| `GET` | `/debug/pprof/profile` | — | CPU profile |\n| `GET` | `/debug/pprof/trace` | — | Execution trace |\n\n## 3. Testing with curl\n\n```bash\n# Server info (version, PID, uptime)\ncurl --unix-socket /var/run/litestream.sock http://localhost/info\n\n# List all managed databases\ncurl --unix-socket /var/run/litestream.sock http://localhost/list\n\n# Get transaction ID for a specific database\ncurl --unix-socket /var/run/litestream.sock \"http://localhost/txid?path=/path/to/db\"\n\n# Register a new database at runtime\ncurl --unix-socket /var/run/litestream.sock -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"path\":\"/path/to/db\",\"replica_url\":\"s3://bucket/path\"}' \\\n http://localhost/register\n\n# Unregister (stop replicating) a database\ncurl --unix-socket /var/run/litestream.sock -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"path\":\"/path/to/db\"}' \\\n http://localhost/unregister\n\n# CPU profile (30 seconds by default)\ncurl --unix-socket /var/run/litestream.sock http://localhost/debug/pprof/profile > cpu.prof\ngo tool pprof cpu.prof\n\n# Heap profile\ncurl --unix-socket /var/run/litestream.sock http://localhost/debug/pprof/heap > heap.prof\ngo tool pprof heap.prof\n```\n\n## 4. Common Issues\n\n### Socket not enabled\n**Symptom**: `curl: (7) Couldn't connect to server`\n**Fix**: Set `socket.enabled: true` in config and restart litestream.\n\n### Permission denied\n**Symptom**: `curl: (7) Permission denied`\n**Fix**: Check `socket.permissions` (default `0600`). The connecting user must match the litestream process owner.\n\n### Stale socket file after crash\n**Symptom**: `bind: address already in use` in logs on startup\n**Fix**: Remove the stale socket file: `rm /var/run/litestream.sock`\n\n### Database not found\n**Symptom**: `{\"error\":\"database not found\"}` from `/txid` or `/start`\n**Fix**: Verify the path matches the expanded path in config. Use `/list` to see registered databases. Note that `$PID` and env vars are expanded in config paths.\n\n### Register fails with \"already exists\"\n**Symptom**: `{\"error\":\"database already registered\"}`\n**Fix**: The database path is already being replicated. Use `/unregister` first, then `/register` with new settings.\n\n## 5. pprof Debugging\n\nAvailable pprof endpoints on the IPC socket:\n\n```bash\n# Interactive CPU profile analysis\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/profile?seconds=30 > cpu.prof\ngo tool pprof -http=:8080 cpu.prof\n\n# Goroutine dump (useful for deadlock investigation)\ncurl --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/goroutine?debug=2\n\n# Memory allocation profile\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/heap > heap.prof\ngo tool pprof -http=:8080 heap.prof\n\n# Execution trace (captures scheduler, GC, goroutine events)\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/trace?seconds=5 > trace.out\ngo tool trace trace.out\n```\n","frontmatter":{"description":"Debug IPC Unix socket issues"},"isInternal":false,"tokens":1130,"sizeBytes":4180},{"name":"debug-wal.md","path":".claude/commands/debug-wal.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/debug-wal.md","title":"Command: /debug-wal","category":"command-prompt","format":"markdown","content":"Debug WAL monitoring issues in Litestream. This command helps diagnose problems with WAL change detection, checkpointing, and replication triggers.\n\nFirst, understand the symptoms:\n- Is replication not triggering on changes?\n- Are checkpoints failing or not happening?\n- Is the WAL growing unbounded?\n\nThen debug the monitoring system:\n\n1. **Check monitor goroutine** (db.go:1499):\n```go\n// Verify monitor is running\nfunc (db *DB) monitor() {\n ticker := time.NewTicker(db.MonitorInterval) // Default: 1s\n // Check if ticker is firing\n // Verify checkWAL() is being called\n}\n```\n\n2. **Verify WAL change detection**:\n```go\n// Check if WAL changes are detected\nfunc (db *DB) checkWAL() (bool, error) {\n // Get WAL size and checksum\n // Compare with previous values\n // Should return true if changed\n}\n```\n\n3. **Debug checkpoint triggers**:\n```go\n// Check checkpoint thresholds\nMinCheckpointPageN int // Default: 1000 pages\nTruncatePageN int // Default: 121359 pages\n\n// Verify WAL page count\nwalPageCount := db.WALPageCount()\nif walPageCount > db.MinCheckpointPageN {\n // Should trigger passive checkpoint\n}\nif walPageCount > db.TruncatePageN {\n // Should trigger truncate checkpoint (emergency brake)\n}\n```\n\n4. **Check long-running read transaction**:\n```go\n// Ensure rtx is maintained\nif db.rtx == nil {\n // Read transaction lost - replication may fail\n}\n```\n\n5. **Monitor notification channel**:\n```go\n// Check if replicas are notified\nselect {\ncase <-db.notify:\n // WAL change detected\ndefault:\n // No changes\n}\n```\n\nCommon issues to check:\n- MonitorInterval too long (default 1s)\n- Checkpoint failing due to active transactions\n- Read transaction preventing checkpoint\n- Notify channel not triggering replicas\n- WAL file permissions issues\n\nDebug commands:\n```sql\n-- Check WAL status\nPRAGMA wal_checkpoint;\nPRAGMA journal_mode;\nPRAGMA page_count;\nPRAGMA wal_autocheckpoint;\n\n-- Check for locks\nSELECT * FROM pragma_lock_status();\n```\n\nTesting WAL monitoring:\n```go\nfunc TestDB_WALMonitoring(t *testing.T) {\n db := setupTestDB(t)\n\n // Set fast monitoring for test\n db.MonitorInterval = 10 * time.Millisecond\n\n // Write data\n writeTestData(t, db, 100)\n\n // Wait for notification\n select {\n case <-db.notify:\n // Success\n case <-time.After(1 * time.Second):\n t.Error(\"WAL change not detected\")\n }\n}\n```\n\nMonitor with logging:\n```go\nslog.Debug(\"wal check\",\n \"size\", walInfo.Size,\n \"checksum\", walInfo.Checksum,\n \"pages\", walInfo.PageCount)\n```\n","isInternal":false,"tokens":619,"sizeBytes":2532},{"name":"fix-common-issues.md","path":".claude/commands/fix-common-issues.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/fix-common-issues.md","title":"Command: /fix-common-issues","category":"command-prompt","format":"markdown","content":"---\ndescription: Fix common Litestream issues\n---\n\n# Fix Common Issues Command\n\nDiagnose and fix common issues in Litestream deployments.\n\n## Issue 1: Lock Page Not Being Skipped\n\n**Symptom**: Errors or corruption with databases >1GB\n\n**Check**:\n```bash\n# Find lock page references\ngrep -r \"LockPgno\" --include=\"*.go\"\n```\n\n**Fix**:\n```go\n// Ensure all page iterations skip lock page\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n## Issue 2: Race Condition in Replica Position\n\n**Symptom**: Data races detected, inconsistent position tracking\n\n**Check**:\n```bash\ngo test -race -v -run TestReplica_Sync ./...\n```\n\n**Fix**:\n```go\n// Change from RLock to Lock for writes\nfunc (r *Replica) SetPos(pos ltx.Pos) {\n r.mu.Lock() // NOT RLock!\n defer r.mu.Unlock()\n r.pos = pos\n}\n```\n\n## Issue 3: Eventual Consistency Issues\n\n**Symptom**: Compaction failures, partial file reads\n\n**Check**:\n```bash\n# Look for remote reads during compaction\ngrep -r \"OpenLTXFile\" db.go | grep -v \"os.Open\"\n```\n\n**Fix**:\n```go\n// Always try local first\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil\n}\n// Only fall back to remote if local doesn't exist\nreturn replica.Client.OpenLTXFile(...)\n```\n\n## Issue 4: CreatedAt Timestamp Loss\n\n**Symptom**: Point-in-time recovery lacks accurate timestamps\n\n**Check**:\n```go\ninfo, err := client.WriteLTXFile(ctx, level, minTXID, maxTXID, r)\nif err != nil {\n t.Fatal(err)\n}\nif info.CreatedAt.IsZero() {\n t.Fatal(\"CreatedAt not set\")\n}\n```\n\n**Fix**:\n```go\n// Ensure storage metadata is copied into the returned FileInfo\nmodTime := resp.LastModified\ninfo.CreatedAt = modTime\n```\n\n## Issue 5: Non-Atomic File Writes\n\n**Symptom**: Partial files, corruption on crash\n\n**Check**:\n```bash\n# Find direct writes without temp files\ngrep -r \"os.Create\\|os.WriteFile\" --include=\"*.go\"\n```\n\n**Fix**:\n```go\n// Write to temp, then rename\ntmpPath := path + \".tmp\"\nif err := os.WriteFile(tmpPath, data, 0644); err != nil {\n return err\n}\nreturn os.Rename(tmpPath, path)\n```\n\n## Issue 6: WAL Checkpoint Blocking\n\n**Symptom**: WAL grows indefinitely, database locks\n\n**Check**:\n```sql\n-- Check WAL size\nPRAGMA wal_checkpoint(PASSIVE);\nSELECT page_count * page_size FROM pragma_page_count(), pragma_page_size();\n```\n\n**Fix**:\n```go\n// Release read transaction periodically\ndb.rtx.Rollback()\ndb.rtx = nil\n// Checkpoint\ndb.db.Exec(\"PRAGMA wal_checkpoint(RESTART)\")\n// Restart read transaction\ndb.initReadTx()\n```\n\n## Issue 7: Memory Leaks\n\n**Symptom**: Growing memory usage over time\n\n**Check**:\n```bash\n# Generate heap profile\ngo test -memprofile=mem.prof -run=XXX -bench=.\ngo tool pprof -top mem.prof\n```\n\n**Fix**:\n```go\n// Use sync.Pool for buffers\nvar pagePool = sync.Pool{\n New: func() interface{} {\n b := make([]byte, pageSize)\n return &b\n },\n}\n\n// Close resources properly\ndefer func() {\n if f != nil {\n f.Close()\n }\n}()\n```\n\n## Issue 8: Corrupted or Missing LTX Files\n\n**Symptom**: Sync failures with errors like \"ltx validation failed\", \"nonsequential page numbers\",\n\"non-contiguous transaction files\", or persistent sync retry backoff loops after unclean shutdowns.\n\n**Check**:\n```bash\n# Look for LTXError messages in logs\nrg -i \"ltx.*error|reset.*local|auto.recover\" /var/log/litestream.log\n\n# Check if meta directory has corrupted state (note dot prefix)\nls -la /path/to/.database.db-litestream/ltx/\n```\n\n**Fix - Manual Reset**:\n```bash\n# Clears local LTX state, forces fresh snapshot on next sync\n# Database file is NOT modified\nlitestream reset /path/to/database.db\n\n# With explicit config file\nlitestream reset -config /etc/litestream.yml /path/to/database.db\n```\n\n**Fix - Automatic Recovery**:\n```yaml\n# Add to replica config in litestream.yml\ndbs:\n - path: /path/to/database.db\n replicas:\n - url: s3://bucket/path\n auto-recover: true # Automatically resets on LTX errors\n```\n\n**When to use which**: Use `auto-recover` for unattended deployments where automatic recovery\nis preferred over manual intervention. Use manual `reset` when you want to investigate the\ncorruption first. `auto-recover` is disabled by default because resetting discards local LTX\nhistory, which may reduce point-in-time restore granularity.\n\n**Reference**: `cmd/litestream/reset.go`, `replica.go` (auto-recover logic), `db.go` (`ResetLocalState`)\n\n## Issue 9: IPC Socket Connection Failures\n\n**Symptom**: `connection refused` or `no such file or directory` when using the control socket\n\n**Check**:\n```bash\n# Verify socket exists and has correct permissions\nls -la /var/run/litestream.sock\n\n# Verify socket is enabled in config\nrg -A3 'socket:' /etc/litestream.yml\n\n# Check if litestream process is running\npgrep -a litestream\n```\n\n**Fix**:\n```yaml\n# Enable socket in litestream.yml\nsocket:\n enabled: true\n path: /var/run/litestream.sock\n permissions: 0600\n```\n\n**Stale socket**: If litestream crashed, the socket file may still exist. The process creates a new socket on startup and will fail if the stale file exists. Remove it manually:\n```bash\nrm /var/run/litestream.sock\n```\n\n**Reference**: `server.go` (`SocketConfig`, `Server.Start`)\n\n## Issue 10: Backup Files Accumulating (Retention Disabled)\n\n**Symptom**: Storage usage growing unbounded, old LTX files never deleted\n\n**Check**:\n```bash\n# Look for retention warning in logs\nrg \"retention disabled\" /var/log/litestream.log\n```\n\n**Fix**:\n```yaml\n# Option 1: Re-enable Litestream retention (default)\nretention:\n enabled: true\n\n# Option 2: Keep disabled, but configure cloud lifecycle policies\n# Example: S3 lifecycle rule to expire objects in ltx/ prefix after 30 days\n```\n\n**Reference**: `store.go` (`SetRetentionEnabled`), `compactor.go` (`RetentionEnabled`), `cmd/litestream/replicate.go:295`\n\n## Issue 11: v0.3.x Restore Not Finding Backups\n\n**Symptom**: Restore fails with \"no snapshots available\" but v0.3.x backups exist in storage\n\n**Check**:\n```bash\n# Verify v0.3.x backup structure exists\naws s3 ls s3://bucket/path/generations/ --recursive | head -20\n```\n\n**Fix**: Ensure the backend implements `ReplicaClientV3`. The S3 backend supports this automatically. The restore process checks for v0.3.x backups when no v0.4.x+ backup is found.\n\n**Reference**: `v3.go` (`ReplicaClientV3` interface), `s3/replica_client.go`\n\n## Diagnostic Commands\n\n```bash\n# Check database integrity\nsqlite3 database.db \"PRAGMA integrity_check;\"\n\n# List replicated LTX files\nlitestream ltx /path/to/db.sqlite\n\n# Check replication status\nlitestream databases\n\n# Reset corrupted local state\nlitestream reset /path/to/database.db\n\n# Test restoration\nlitestream restore -o test.db [replica-url]\n\n# IPC socket diagnostics (requires socket.enabled: true)\ncurl --unix-socket /var/run/litestream.sock http://localhost/info\ncurl --unix-socket /var/run/litestream.sock http://localhost/list\ncurl --unix-socket /var/run/litestream.sock \"http://localhost/txid?path=/path/to/db\"\n```\n\n## Prevention Checklist\n\n- [ ] Always test with databases >1GB\n- [ ] Run with race detector in CI\n- [ ] Test all page sizes (4KB, 8KB, 16KB, 32KB)\n- [ ] Verify eventual consistency handling\n- [ ] Check for proper locking (Lock vs RLock)\n- [ ] Ensure atomic file operations\n- [ ] Preserve timestamps in compaction\n","frontmatter":{"description":"Fix common Litestream issues"},"isInternal":false,"tokens":1955,"sizeBytes":7266},{"name":"run-comprehensive-tests.md","path":".claude/commands/run-comprehensive-tests.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/run-comprehensive-tests.md","title":"Command: /run-comprehensive-tests","category":"command-prompt","format":"markdown","content":"---\ndescription: Run comprehensive test suite for Litestream\n---\n\n# Run Comprehensive Tests Command\n\nExecute a full test suite including unit tests, integration tests, race detection, and large database tests.\n\n## Quick Test Suite\n\n```bash\n# Basic tests with race detection\ngo test -race -v ./...\n\n# With coverage\ngo test -race -cover -v ./...\n```\n\n## Full Test Suite\n\n### 1. Unit Tests\n```bash\necho \"=== Running Unit Tests ===\"\ngo test -v ./... -short\n```\n\n### 2. Race Condition Tests\n```bash\necho \"=== Testing for Race Conditions ===\"\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\ngo test -race -v ./...\n```\n\n### 3. Integration Tests\n```bash\necho \"=== Running Integration Tests ===\"\n\n# S3 (requires AWS credentials)\nAWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy \\\n go test -v ./replica_client_test.go -integration s3\n\n# Google Cloud Storage (requires credentials)\nGOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json \\\n go test -v ./replica_client_test.go -integration gcs\n\n# Azure Blob Storage\nAZURE_STORAGE_ACCOUNT=xxx AZURE_STORAGE_KEY=yyy \\\n go test -v ./replica_client_test.go -integration abs\n\n# SFTP (requires SSH server)\ngo test -v ./replica_client_test.go -integration sftp\n\n# File system (always available)\ngo test -v ./replica_client_test.go -integration file\n```\n\n### 4. Large Database Tests (>1GB)\n```bash\necho \"=== Testing Large Databases ===\"\n\n# Create test database for each page size\nfor pagesize in 4096 8192 16384 32768; do\n echo \"Testing page size: $pagesize\"\n\n # Create >1GB database\n sqlite3 test-${pagesize}.db <70% for core packages\n✅ Lock page correctly skipped for all page sizes\n✅ Restoration works for databases >1GB\n✅ No memory leaks in benchmarks\n","frontmatter":{"description":"Run comprehensive test suite for Litestream"},"isInternal":false,"tokens":1430,"sizeBytes":5144},{"name":"test-compaction.md","path":".claude/commands/test-compaction.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/test-compaction.md","title":"Command: /test-compaction","category":"command-prompt","format":"markdown","content":"Test Litestream compaction logic. This command helps test and debug compaction issues, especially with eventually consistent storage backends.\n\nFirst, understand the test scenario:\n- What storage backend needs testing?\n- What size database is involved?\n- Are there eventual consistency concerns?\n\nThen create comprehensive tests:\n\n1. **Test basic compaction**:\n```go\nfunc TestCompaction_Basic(t *testing.T) {\n // Create multiple LTX files at level 0\n // Run compaction to level 1\n // Verify merged file is correct\n}\n```\n\n2. **Test with eventual consistency**:\n```go\nfunc TestStore_CompactDB_RemotePartialRead(t *testing.T) {\n // Use mock client that returns partial data initially\n // Verify compaction prefers local files\n // Ensure no corruption occurs\n}\n```\n\n3. **Test lock page handling during compaction**:\n```go\nfunc TestCompaction_LockPage(t *testing.T) {\n // Create database > 1GB\n // Compact with data around lock page\n // Verify lock page is skipped (page at 0x40000000)\n}\n```\n\n4. **Test timestamp preservation**:\n```go\nfunc TestCompaction_PreserveTimestamps(t *testing.T) {\n // Compact files with different CreatedAt times\n // Verify earliest timestamp is preserved\n}\n```\n\nKey areas to test:\n- Reading from local files first (db.go:1280-1294)\n- Skipping lock page at 1GB boundary\n- Preserving CreatedAt timestamps\n- Handling partial/incomplete remote files\n- Concurrent compaction safety\n\nRun with race detector:\n```bash\ngo test -race -v -run TestStore_CompactDB ./...\n```\n\nUse the test harness for large databases:\n```bash\n./bin/litestream-test populate -db test.db -target-size 1.5GB\n./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica\n```\n","isInternal":false,"tokens":421,"sizeBytes":1717},{"name":"trace-replication.md","path":".claude/commands/trace-replication.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/trace-replication.md","title":"Command: /trace-replication","category":"command-prompt","format":"markdown","content":"Trace the complete replication flow in Litestream. This command helps understand how changes flow from SQLite through to storage backends.\n\nFollow the replication path step by step:\n\n1. **Application writes to SQLite**:\n```sql\n-- Application performs write\nINSERT INTO table VALUES (...);\n-- SQLite appends to WAL file\n```\n\n2. **DB.monitor() syncs the shadow WAL** (db.go:1499):\n```go\nticker := time.NewTicker(db.MonitorInterval) // default 1s\nfor {\n select {\n case <-db.ctx.Done():\n return\n case <-ticker.C:\n }\n\n if err := db.Sync(db.ctx); err != nil && !errors.Is(err, context.Canceled) {\n db.Logger.Error(\"sync error\", \"error\", err)\n }\n}\n```\n\n3. **Replica.monitor() responds** (replica.go):\n```go\nticker := time.NewTicker(r.SyncInterval)\ndefer ticker.Stop()\n\nnotify := r.db.Notify()\nfor {\n select {\n case <-ctx.Done():\n return\n case <-ticker.C:\n // Enforce minimum sync interval\n case <-notify:\n // WAL changed, time to sync\n }\n\n notify = r.db.Notify()\n\n if err := r.Sync(ctx); err != nil && !errors.Is(err, context.Canceled) {\n r.Logger().Error(\"monitor error\", \"error\", err)\n }\n}\n```\n\n4. **Replica.Sync() uploads new L0 files** (replica.go):\n```go\n// Determine local database position\ndpos, err := r.db.Pos()\nif err != nil {\n return err\n}\nif dpos.IsZero() {\n return fmt.Errorf(\"no position, waiting for data\")\n}\n\n// Upload each unreplicated L0 file\nfor txID := r.Pos().TXID + 1; txID <= dpos.TXID; txID = r.Pos().TXID + 1 {\n if err := r.uploadLTXFile(ctx, 0, txID, txID); err != nil {\n return err\n }\n r.SetPos(ltx.Pos{TXID: txID})\n}\n```\n\n5. **ReplicaClient uploads to storage**:\n```go\nfunc (c *S3Client) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {\n // Stream LTX data to storage and return metadata (size, CreatedAt, checksums)\n}\n```\n\n6. **Checkpoint when thresholds are hit**:\n```go\nif walPageCount > db.MinCheckpointPageN {\n db.Checkpoint(ctx, litestream.CheckpointModePassive)\n}\nif walPageCount > db.TruncatePageN {\n db.Checkpoint(ctx, litestream.CheckpointModeTruncate)\n}\n```\n\nKey synchronization points:\n- WAL monitoring (1s intervals)\n- Replica sync (configurable, default 1s)\n- Checkpoint triggers (page thresholds)\n- Compaction (hourly/daily)\n\nTrace with logging:\n```go\n// Enable debug logging\nslog.SetLogLevel(slog.LevelDebug)\n\n// Key log points:\nslog.Debug(\"wal changed\", \"size\", walSize)\nslog.Debug(\"syncing replica\", \"pos\", r.Pos())\nslog.Debug(\"ltx uploaded\", \"txid\", maxTXID)\nslog.Debug(\"checkpoint complete\", \"mode\", mode)\n```\n\nPerformance metrics to monitor:\n- WAL growth rate\n- Sync latency\n- Upload throughput\n- Checkpoint frequency\n- Compaction duration\n\nCommon bottlenecks:\n1. Slow storage uploads\n2. Large transactions causing big LTX files\n3. Long-running read transactions blocking checkpoints\n4. Eventual consistency delays\n5. Network latency to storage\n\nTest replication flow:\n```bash\n# Start replication with verbose logging\nlitestream replicate -v\n\n# In another terminal, write to database\nsqlite3 test.db \"INSERT INTO test VALUES (1, 'data');\"\n\n# Watch logs for flow:\n# - WAL change detected\n# - Replica sync triggered\n# - LTX file uploaded\n# - Position updated\n```\n\nVerify replication:\n```bash\n# List replicated files\naws s3 ls s3://bucket/path/ltx/0000/\n\n# Restore and verify\nlitestream restore -o restored.db s3://bucket/path\nsqlite3 restored.db \"SELECT * FROM test;\"\n```\n","isInternal":false,"tokens":935,"sizeBytes":3488},{"name":"validate-replica.md","path":".claude/commands/validate-replica.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/.claude/commands/validate-replica.md","title":"Command: /validate-replica","category":"command-prompt","format":"markdown","content":"Validate a ReplicaClient implementation in Litestream. This command helps ensure a replica client correctly implements the interface and handles edge cases.\n\nFirst, identify what needs validation:\n- Which replica client implementation?\n- What storage backend specifics?\n- Any known issues or concerns?\n\nThen validate the implementation:\n\n1. **Interface compliance check**:\n\n ```go\n // Ensure all methods are implemented\n var _ litestream.ReplicaClient = (*YourClient)(nil)\n ```\n\n1. **Verify error types**:\n\n ```go\n // OpenLTXFile must return os.ErrNotExist for missing files\n _, err := client.OpenLTXFile(ctx, 0, 999, 999, 0, 0)\n if !errors.Is(err, os.ErrNotExist) {\n t.Errorf(\"Expected os.ErrNotExist, got %v\", err)\n }\n ```\n\n1. **Test partial reads**:\n\n ```go\n // Must support offset and size parameters\n rc, err := client.OpenLTXFile(ctx, 0, 1, 100, 50, 25)\n data, _ := io.ReadAll(rc)\n if len(data) != 25 {\n t.Errorf(\"Expected 25 bytes, got %d\", len(data))\n }\n ```\n\n1. **Verify timestamp preservation**:\n\n ```go\n // CreatedAt should reflect remote object metadata (or upload time)\n start := time.Now()\n info, _ := client.WriteLTXFile(ctx, 0, 1, 100, reader)\n if info.CreatedAt.IsZero() || info.CreatedAt.Before(start.Add(-time.Second)) {\n t.Error(\"unexpected CreatedAt timestamp\")\n }\n ```\n\n1. **Test eventual consistency handling**:\n - Implement retry logic for transient failures\n - Handle partial file availability\n - Verify write-after-write consistency\n\n1. **Validate cleanup**:\n\n ```go\n // DeleteAll must remove everything\n err := client.DeleteAll(ctx)\n files, _ := client.LTXFiles(ctx, 0, 0, false)\n if files.Next() {\n t.Error(\"Files remain after DeleteAll\")\n }\n ```\n\nKey validation points:\n- Proper error types (os.ErrNotExist, os.ErrPermission)\n- Context cancellation handling\n- Concurrent operation safety\n- Iterator doesn't load all files at once\n- Proper path construction for storage backend\n\nRun integration tests:\n```bash\ngo test -v ./[backend]/replica_client_test.go -integration\n```\n","isInternal":false,"tokens":528,"sizeBytes":2093},{"name":"ARCHITECTURE.md","path":"docs/ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/docs/ARCHITECTURE.md","title":"docs - System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# Litestream Architecture - Technical Deep Dive\n\n## Table of Contents\n- [System Layers](#system-layers)\n- [Core Components](#core-components)\n- [IPC Server & Control Socket](#ipc-server--control-socket)\n- [Distributed Leasing](#distributed-leasing)\n- [Library Convenience Methods](#library-convenience-methods)\n- [LTX File Format](#ltx-file-format)\n- [WAL Monitoring Mechanism](#wal-monitoring-mechanism)\n- [Compaction Process](#compaction-process)\n- [Transaction Management](#transaction-management)\n- [Concurrency Model](#concurrency-model)\n- [State Management](#state-management)\n- [Initialization Flow](#initialization-flow)\n- [Error Handling](#error-handling)\n\n## System Layers\n\nLitestream follows a layered architecture with clear separation of concerns:\n\n```mermaid\ngraph TB\n subgraph \"Application Layer\"\n CLI[CLI Commands
cmd/litestream/]\n Config[Configuration
config.go]\n end\n\n subgraph \"Core Layer\"\n Store[Store Manager
store.go]\n DB[Database Manager
db.go]\n Replica[Replica Manager
replica.go]\n end\n\n subgraph \"Storage Abstraction\"\n RC[ReplicaClient Interface
replica_client.go]\n end\n\n subgraph \"Storage Implementations\"\n S3[s3/replica_client.go]\n GCS[gs/replica_client.go]\n ABS[abs/replica_client.go]\n OSS[oss/replica_client.go]\n File[file/replica_client.go]\n SFTP[sftp/replica_client.go]\n NATS[nats/replica_client.go]\n end\n\n subgraph \"External\"\n SQLite[SQLite Database]\n Cloud[Cloud Storage]\n end\n\n CLI --> Store\n Store --> DB\n DB --> Replica\n Replica --> RC\n RC --> S3\n RC --> GCS\n RC --> ABS\n RC --> OSS\n RC --> File\n RC --> SFTP\n RC --> NATS\n DB <--> SQLite\n S3 --> Cloud\n GCS --> Cloud\n ABS --> Cloud\n OSS --> Cloud\n```\n\n### Layer Responsibilities\n\n#### 1. Application Layer\n- **CLI Commands**: User interface for operations (replicate, restore, etc.)\n- **Configuration**: YAML/environment variable parsing and validation\n\n#### 2. Core Layer\n- **Store**: Multi-database coordination, compaction scheduling\n- **DB**: Single database management, WAL monitoring, checkpointing\n- **Replica**: Replication to single destination, position tracking\n\n#### 3. Storage Abstraction\n- **ReplicaClient Interface**: Uniform API for all storage backends\n\n#### 4. Storage Implementations\n- Backend-specific logic (authentication, retries, optimizations)\n\n#### 5. IPC Layer\n- **Server**: Unix socket HTTP server for runtime control (`server.go`)\n- **Leaser**: Distributed lease interface for coordination (`leaser.go`, `s3/leaser.go`)\n\n## Core Components\n\n### DB Component (db.go)\n\nThe DB component is the heart of Litestream, managing a single SQLite database:\n\n```go\ntype DB struct {\n // Core fields\n path string // Database file path\n metaPath string // Metadata directory path\n db *sql.DB // SQLite connection\n f *os.File // Long-running file descriptor\n rtx *sql.Tx // Long-running read transaction\n pageSize int // Database page size\n\n // Synchronization\n mu sync.RWMutex // Protects struct fields\n chkMu sync.RWMutex // Checkpoint lock\n notify chan struct{} // WAL change notifications\n\n // Lifecycle\n ctx context.Context\n cancel func()\n wg sync.WaitGroup\n\n // Configuration\n MinCheckpointPageN int // Min pages for passive checkpoint\n TruncatePageN int // Pages before emergency truncate checkpoint\n CheckpointInterval time.Duration // Time-based passive checkpoint interval\n MonitorInterval time.Duration // WAL monitoring frequency\n // Note: MaxCheckpointPageN removed (RESTART mode disabled due to #724)\n\n // Metrics\n dbSizeGauge prometheus.Gauge\n walSizeGauge prometheus.Gauge\n txIDGauge prometheus.Gauge\n}\n```\n\n#### Key Methods\n\n```go\n// Lifecycle\nfunc (db *DB) Open() error\nfunc (db *DB) Close(ctx context.Context) error\n\n// Monitoring\nfunc (db *DB) monitor() // Background WAL monitoring\nfunc (db *DB) checkWAL() (bool, error) // Check for WAL changes\n\n// Checkpointing\nfunc (db *DB) Checkpoint(mode string) error\nfunc (db *DB) autoCheckpoint() error\n\n// Replication\nfunc (db *DB) WALReader(pgno uint32) (io.ReadCloser, error)\nfunc (db *DB) Sync(ctx context.Context) error\n\n// Compaction\nfunc (db *DB) Compact(ctx context.Context, destLevel int) (*ltx.FileInfo, error)\n```\n\n### Replica Component (replica.go)\n\nManages replication to a single destination:\n\n```go\ntype Replica struct {\n db *DB // Parent database\n Client ReplicaClient // Storage backend client\n\n mu sync.RWMutex\n pos ltx.Pos // Current replication position\n\n // Configuration\n SyncInterval time.Duration\n MonitorEnabled bool\n\n // Lifecycle\n cancel func()\n wg sync.WaitGroup\n}\n```\n\n#### Replication Position\n\n```go\ntype Pos struct {\n TXID TXID // Transaction ID\n PageNo uint32 // Page number within transaction\n Checksum uint64 // Running checksum\n}\n```\n\n### Store Component (store.go)\n\nCoordinates multiple databases and manages system-wide resources:\n\n```go\ntype Store struct {\n mu sync.Mutex\n dbs []*DB\n levels CompactionLevels\n\n // Configuration\n SnapshotInterval time.Duration\n SnapshotRetention time.Duration\n L0Retention time.Duration\n L0RetentionCheckInterval time.Duration\n CompactionMonitorEnabled bool\n\n // Lifecycle\n ctx context.Context\n cancel func()\n wg sync.WaitGroup\n}\n```\n\n## IPC Server & Control Socket\n\nLitestream exposes a Unix socket HTTP server for runtime control (`server.go`).\n\n### Socket Configuration\n\n```go\ntype SocketConfig struct {\n Enabled bool `yaml:\"enabled\"` // Default: false\n Path string `yaml:\"path\"` // Default: \"/var/run/litestream.sock\"\n Permissions uint32 `yaml:\"permissions\"` // Default: 0600\n}\n```\n\n### HTTP Endpoints\n\nAll endpoints are served over the Unix socket via Go's `net/http` mux:\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `POST` | `/register` | Add a database at runtime |\n| `POST` | `/unregister` | Remove a database at runtime |\n| `GET` | `/txid?path=` | Get current transaction ID for a database |\n| `GET` | `/list` | List all managed databases |\n| `GET` | `/info` | Server info (version, PID, uptime, database count) |\n| `POST` | `/start` | Start replication for a database |\n| `POST` | `/stop` | Stop replication for a database |\n| `GET` | `/debug/pprof/*` | Standard Go pprof endpoints |\n\n### Request/Response Types\n\n```go\ntype RegisterDatabaseRequest struct {\n Path string `json:\"path\"`\n ReplicaURL string `json:\"replica_url\"`\n}\n\ntype UnregisterDatabaseRequest struct {\n Path string `json:\"path\"`\n Timeout int `json:\"timeout,omitempty\"` // Seconds\n}\n\ntype TXIDResponse struct {\n TXID uint64 `json:\"txid\"`\n}\n```\n\n## Distributed Leasing\n\nThe `Leaser` interface (`leaser.go`) enables distributed coordination so multiple Litestream instances can safely share a replica destination.\n\n### Interface\n\n```go\ntype Leaser interface {\n Type() string\n AcquireLease(ctx context.Context) (*Lease, error)\n RenewLease(ctx context.Context, lease *Lease) (*Lease, error)\n ReleaseLease(ctx context.Context, lease *Lease) error\n}\n\ntype Lease struct {\n Generation int64 `json:\"generation\"`\n ExpiresAt time.Time `json:\"expires_at\"`\n Owner string `json:\"owner,omitempty\"`\n ETag string `json:\"-\"`\n}\n```\n\n### S3 Implementation (`s3/leaser.go`)\n\n- **Defaults**: `DefaultLeaseTTL = 30s`, `DefaultLeasePath = \"lock.json\"`\n- **Owner format**: `hostname:pid` (falls back to `pid-N` if hostname unavailable)\n- **Conditional writes**: Uses `If-Match`/`If-None-Match` on S3 PutObject to prevent races\n- **Release**: Uses `DeleteObject` with `If-Match` to ensure only the holder can release\n- **Error types**: `ErrLeaseNotHeld`, `ErrLeaseAlreadyReleased`, `LeaseExistsError{Owner, ExpiresAt}`\n\n### Acquisition Flow\n\n1. Read existing lease from S3 (`lock.json`)\n2. If lease exists and is not expired, return `LeaseExistsError`\n3. Write new lease with `If-None-Match: *` (first acquire) or `If-Match: ` (expired takeover)\n4. If `PreconditionFailed` (412), another instance acquired first\n5. Return lease with ETag for subsequent renewal\n\n## Library Convenience Methods\n\nThese methods on `DB` (`db.go`) simplify common operations when Litestream is used as a library:\n\n```go\ntype SyncStatus struct {\n LocalTXID ltx.TXID // Local transaction ID\n RemoteTXID ltx.TXID // Remote transaction ID\n InSync bool // true if LocalTXID > 0 and equal to RemoteTXID\n}\n\nfunc (db *DB) SyncStatus(ctx context.Context) (SyncStatus, error)\nfunc (db *DB) SyncAndWait(ctx context.Context) error\nfunc (db *DB) EnsureExists(ctx context.Context) error\n```\n\n- **SyncStatus**: Compares local TXID against remote replica position (performs I/O to query remote)\n- **SyncAndWait**: Runs `db.Sync()` (WAL to LTX) then `db.Replica.Sync()` (LTX to remote), blocking until both complete\n- **EnsureExists**: Restores database from replica if local file doesn't exist; no-op if file exists or no backup available. Must be called before `Open()`\n\n## LTX File Format\n\nLTX (Log Transaction) files are immutable files containing database changes:\n\n```\n+------------------+\n| Header | Fixed size header with metadata\n+------------------+\n| |\n| Page Frames | Variable number of page frames\n| |\n+------------------+\n| Page Index | Index for efficient page lookup\n+------------------+\n| Trailer | Metadata and checksums\n+------------------+\n```\n\n### Header Structure\n\n```go\ntype Header struct {\n Magic [4]byte // \"LTX\\x00\"\n Version uint32 // Format version\n PageSize uint32 // Database page size\n MinTXID TXID // Starting transaction ID\n MaxTXID TXID // Ending transaction ID\n Timestamp int64 // Creation timestamp\n Checksum uint64 // Header checksum\n}\n```\n\n### Page Frame Structure\n\n```go\ntype PageFrame struct {\n Header PageHeader\n Data []byte // Page data (pageSize bytes)\n}\n\ntype PageHeader struct {\n PageNo uint32 // Page number in database\n Size uint32 // Size of page data\n Checksum uint64 // Page checksum\n}\n```\n\n### Page Index\n\nBinary search tree for efficient page lookup:\n```go\ntype PageIndexElem struct {\n PageNo uint32 // Page number\n Offset int64 // Offset in file\n Size uint32 // Size of page frame\n}\n```\n\n### Trailer\n\n```go\ntype Trailer struct {\n PageIndexOffset int64 // Offset to page index\n PageIndexSize int64 // Size of page index\n PageCount uint32 // Total pages in file\n Checksum uint64 // Full file checksum\n}\n```\n\n## WAL Monitoring Mechanism\n\n### Monitor Loop (db.go:1499)\n\n```go\nfunc (db *DB) monitor() {\n ticker := time.NewTicker(db.MonitorInterval)\n defer ticker.Stop()\n\n for {\n select {\n case <-ticker.C:\n // Check WAL for changes\n changed, err := db.checkWAL()\n if err != nil {\n slog.Error(\"wal check failed\", \"error\", err)\n continue\n }\n\n if changed {\n // Notify replicas of changes\n db.notifyReplicas()\n\n // Check if checkpoint needed\n if db.shouldCheckpoint() {\n db.autoCheckpoint()\n }\n }\n\n case <-db.ctx.Done():\n return\n }\n }\n}\n```\n\n### WAL Change Detection\n\n```go\nfunc (db *DB) checkWAL() (bool, error) {\n // Get current WAL state\n walInfo, err := db.walInfo()\n if err != nil {\n return false, err\n }\n\n // Compare with previous state\n db.mu.Lock()\n changed := walInfo.Size != db.prevWALSize ||\n walInfo.Checksum != db.prevWALChecksum\n db.prevWALSize = walInfo.Size\n db.prevWALChecksum = walInfo.Checksum\n db.mu.Unlock()\n\n return changed, nil\n}\n```\n\n## Compaction Process\n\nCompaction merges multiple LTX files to reduce storage overhead:\n\n### Compaction Algorithm (store.go:189)\n\nHigh-level compaction flow:\n\n1. Determine whether the level is due for compaction (`Store.shouldCompact`).\n2. Enumerate level-`L-1` files using `ReplicaClient.LTXFiles`, preferring local\n copies via `os.Open(db.LTXPath(...))` and falling back to\n `ReplicaClient.OpenLTXFile` only when necessary.\n3. Stream the source readers through `ltx.NewCompactor`, which performs\n page-level deduplication and enforces lock-page skipping automatically.\n4. Pipe the compactor output into `ReplicaClient.WriteLTXFile` to create the\n merged LTX file for level `L`.\n5. Adjust the returned `ltx.FileInfo.CreatedAt` to the earliest timestamp from\n the source files so point-in-time recovery remains accurate.\n6. Update the cached max file info for the level and delete old L0 files when\n promoting to level 1.\n\n### Compaction Levels\n\n```go\ntype CompactionLevel struct {\n Level int // Level number (0 = raw, 1+ = compacted)\n Interval time.Duration // How often to compact from previous level\n}\n\n// Default configuration\nvar DefaultCompactionLevels = CompactionLevels{\n {Level: 0, Interval: 0}, // Raw LTX files\n {Level: 1, Interval: 1 * Hour}, // Hourly compaction\n {Level: 2, Interval: 24 * Hour}, // Daily compaction\n}\n```\n\n## Transaction Management\n\n### Long-Running Read Transaction\n\nLitestream maintains a long-running read transaction to ensure consistency:\n\n```go\nfunc (db *DB) initReadTx() error {\n // Start read transaction\n tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{\n ReadOnly: true,\n })\n if err != nil {\n return err\n }\n\n // Execute dummy query to start transaction\n var dummy string\n err = tx.QueryRow(\"SELECT ''\").Scan(&dummy)\n if err != nil {\n tx.Rollback()\n return err\n }\n\n db.rtx = tx\n return nil\n}\n```\n\n**Purpose:**\n- Prevents database from being modified during replication\n- Ensures consistent view of database\n- Allows reading historical pages from WAL\n\n### Checkpoint Coordination\n\n```go\nfunc (db *DB) Checkpoint(mode string) error {\n // Acquire checkpoint lock\n db.chkMu.Lock()\n defer db.chkMu.Unlock()\n\n // Close read transaction temporarily\n if db.rtx != nil {\n db.rtx.Rollback()\n db.rtx = nil\n }\n\n // Perform checkpoint\n _, _, err := db.db.Exec(fmt.Sprintf(\"PRAGMA wal_checkpoint(%s)\", mode))\n if err != nil {\n return err\n }\n\n // Restart read transaction\n return db.initReadTx()\n}\n```\n\n## Concurrency Model\n\n### Mutex Usage Patterns\n\n```go\n// DB struct mutexes\ntype DB struct {\n mu sync.RWMutex // Protects struct fields\n chkMu sync.RWMutex // Checkpoint coordination\n}\n\n// Replica struct mutexes\ntype Replica struct {\n mu sync.RWMutex // Protects position\n muf sync.Mutex // File descriptor lock\n}\n\n// Store struct mutex\ntype Store struct {\n mu sync.Mutex // Protects database list\n}\n```\n\n### Thundering Herd Prevention\n\n`Store.Open()` (`store.go`) limits concurrent database opens at startup:\n\n```go\ng, ctx := errgroup.WithContext(ctx)\ng.SetLimit(50) // Max 50 concurrent DB opens\nfor _, db := range s.dbs {\n db := db\n g.Go(func() error { return db.Open() })\n}\n```\n\nThis prevents OS resource exhaustion (file descriptors, memory) when hundreds of databases are configured.\n\n### Lock Ordering (Prevent Deadlocks)\n\nAlways acquire locks in this order:\n1. Store.mu\n2. DB.mu\n3. DB.chkMu\n4. Replica.mu\n\n### Goroutine Management\n\n```go\n// Start background task\nfunc (db *DB) Start() {\n db.wg.Add(1)\n go func() {\n defer db.wg.Done()\n db.monitor()\n }()\n}\n\n// Stop with timeout\nfunc (db *DB) Close(ctx context.Context) error {\n // Signal shutdown\n db.cancel()\n\n // Wait for goroutines with timeout\n done := make(chan struct{})\n go func() {\n db.wg.Wait()\n close(done)\n }()\n\n select {\n case <-done:\n return nil\n case <-ctx.Done():\n return ctx.Err()\n }\n}\n```\n\n## State Management\n\n### Database States\n\n```mermaid\nstateDiagram-v2\n [*] --> Closed\n Closed --> Opening: Open()\n Opening --> Open: Success\n Opening --> Closed: Error\n Open --> Monitoring: Start()\n Monitoring --> Syncing: Changes Detected\n Syncing --> Monitoring: Sync Complete\n Monitoring --> Checkpointing: Threshold Reached\n Checkpointing --> Monitoring: Checkpoint Complete\n Monitoring --> Closing: Close()\n Closing --> Closed: Cleanup Complete\n```\n\n### Replica States\n\n```mermaid\nstateDiagram-v2\n [*] --> Idle\n Idle --> Starting: Start()\n Starting --> Monitoring: Success\n Starting --> Idle: Error\n Monitoring --> Syncing: Timer/Changes\n Syncing --> Uploading: Have Changes\n Uploading --> Monitoring: Success\n Uploading --> Error: Failed\n Error --> Monitoring: Retry\n Monitoring --> Stopping: Stop()\n Stopping --> Idle: Cleanup\n```\n\n### Position Tracking\n\n```go\ntype Pos struct {\n TXID TXID // Current transaction ID\n PageNo uint32 // Current page number\n Checksum uint64 // Running checksum for validation\n}\n\n// Update position atomically\nfunc (r *Replica) SetPos(pos ltx.Pos) {\n r.mu.Lock() // MUST use Lock, not RLock!\n defer r.mu.Unlock()\n r.pos = pos\n}\n\n// Read position safely\nfunc (r *Replica) Pos() ltx.Pos {\n r.mu.RLock()\n defer r.mu.RUnlock()\n return r.pos\n}\n```\n\n## Initialization Flow\n\n### System Startup Sequence\n\n```mermaid\nsequenceDiagram\n participant Main\n participant Store\n participant DB\n participant Replica\n participant Monitor\n\n Main->>Store: NewStore(config)\n Store->>Store: Validate config\n\n Main->>Store: Open()\n loop For each database\n Store->>DB: NewDB(path)\n Store->>DB: Open()\n DB->>DB: Open SQLite connection\n DB->>DB: Read page size\n DB->>DB: Init metadata\n DB->>DB: Start read transaction\n\n loop For each replica\n DB->>Replica: NewReplica()\n DB->>Replica: Start()\n Replica->>Monitor: Start monitoring\n end\n end\n\n Store->>Store: Start compaction monitors\n Store-->>Main: Ready\n```\n\n### Critical Initialization Steps\n\n1. **Database Opening**\n ```go\n // Must happen in order:\n 1. Open SQLite connection\n 2. Read page size (PRAGMA page_size)\n 3. Create metadata directory\n 4. Start long-running read transaction\n 5. Initialize replicas\n 6. Start monitor goroutine\n ```\n\n2. **Replica Initialization**\n ```go\n // Must happen in order:\n 1. Create replica with client\n 2. Load previous position from metadata\n 3. Validate position against database\n 4. Start sync goroutine (if monitoring enabled)\n ```\n\n## Error Handling\n\n### Error Categories\n\n1. **Recoverable Errors**\n - Network timeouts\n - Temporary storage unavailability\n - Lock contention\n\n2. **Fatal Errors**\n - Database corruption\n - Invalid configuration\n - Disk full\n\n3. **Operational Errors**\n - Checkpoint failures\n - Compaction conflicts\n - Sync delays\n\n### Error Propagation\n\n```go\n// Bottom-up error propagation\nReplicaClient.WriteLTXFile() error\n ↓\nReplica.Sync() error\n ↓\nDB.Sync() error\n ↓\nStore.monitorDB() // Logs error, continues\n```\n\n### Retry Logic\n\n```go\nfunc (r *Replica) syncWithRetry(ctx context.Context) error {\n backoff := time.Second\n maxBackoff := time.Minute\n\n for attempt := 0; ; attempt++ {\n err := r.Sync(ctx)\n if err == nil {\n return nil\n }\n\n // Check if error is retryable\n if !isRetryable(err) {\n return err\n }\n\n // Check context\n if ctx.Err() != nil {\n return ctx.Err()\n }\n\n // Exponential backoff\n time.Sleep(backoff)\n backoff *= 2\n if backoff > maxBackoff {\n backoff = maxBackoff\n }\n }\n}\n```\n\n## Performance Characteristics\n\n### Time Complexity\n\n| Operation | Complexity | Notes |\n|-----------|------------|-------|\n| WAL Monitor | O(1) | Fixed interval check |\n| Page Write | O(1) | Append to LTX file |\n| Compaction | O(n) | n = total pages |\n| Restoration | O(n*log(m)) | n = pages, m = files |\n| File List | O(k) | k = files in level |\n\n### Space Complexity\n\n| Component | Memory Usage | Disk Usage |\n|-----------|-------------|------------|\n| DB | O(1) + metrics | Original DB + WAL |\n| Replica | O(1) | LTX files + metadata |\n| Compaction | O(n) pages | Temporary during merge |\n| Page Index | O(p) | p = pages in file |\n\n### Optimization Points\n\n1. **Page Index Caching**\n - Cache frequently accessed indices\n - Use estimated size for initial fetch\n\n2. **Batch Operations**\n - Group small changes into larger LTX files\n - Batch delete operations\n\n3. **Concurrent Operations**\n - Multiple replicas can sync in parallel\n - Compaction runs independently per level\n\n## Security Considerations\n\n### Access Control\n\n- File permissions: 0600 for database files\n- Directory permissions: 0700 for metadata\n- No built-in authentication (rely on storage backend)\n\n## Monitoring & Metrics\n\n### Prometheus Metrics\n\n```go\n// Database metrics\ndb_size_bytes // Current database size\nwal_size_bytes // Current WAL size\ntotal_wal_bytes // Total bytes written to WAL\ncheckpoint_count // Number of checkpoints\nsync_count // Number of syncs\nsync_error_count // Number of sync errors\n\n// Replica metrics\nreplica_lag_seconds // Replication lag\nreplica_position // Current replication position\n\n// Validation metrics have db and level labels.\nlitestream_validation_checks_total // Checks by result: success, invalid, or error.\nlitestream_last_validation_success // Last check: 1 passed, 0 failed or not yet complete.\nlitestream_last_validation_success_timestamp_seconds // Unix time of last success, or 0 until a check passes.\n```\n\n### Health Checks\n\n```go\nfunc (db *DB) HealthCheck() error {\n // Check database connection\n if err := db.db.Ping(); err != nil {\n return fmt.Errorf(\"database ping failed: %w\", err)\n }\n\n // Check replication lag\n for _, r := range db.replicas {\n lag := time.Since(r.LastSync())\n if lag > MaxAcceptableLag {\n return fmt.Errorf(\"replica %s lag too high: %v\", r.Name(), lag)\n }\n }\n\n return nil\n}\n```\n","isInternal":false,"tokens":5572,"sizeBytes":22684},{"name":"README.md","path":"packages/npm/litestream-vfs-darwin-amd64/README.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/packages/npm/litestream-vfs-darwin-amd64/README.md","title":"litestream-vfs-darwin-amd64 Documentation","category":"plugin-manifest","format":"markdown","content":"# litestream-vfs-darwin-amd64\n\nPlatform package for `litestream-vfs` — macOS x64 (Intel).\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","isInternal":false,"tokens":52,"sizeBytes":198},{"name":"README.md","path":"packages/npm/litestream-vfs-darwin-arm64/README.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/packages/npm/litestream-vfs-darwin-arm64/README.md","title":"litestream-vfs-darwin-arm64 Documentation","category":"plugin-manifest","format":"markdown","content":"# litestream-vfs-darwin-arm64\n\nPlatform package for `litestream-vfs` — macOS ARM64 (Apple Silicon).\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","isInternal":false,"tokens":52,"sizeBytes":208},{"name":"README.md","path":"packages/npm/litestream-vfs-linux-amd64/README.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/packages/npm/litestream-vfs-linux-amd64/README.md","title":"litestream-vfs-linux-amd64 Documentation","category":"plugin-manifest","format":"markdown","content":"# litestream-vfs-linux-amd64\n\nPlatform package for `litestream-vfs` — Linux x64.\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","isInternal":false,"tokens":48,"sizeBytes":189},{"name":"README.md","path":"packages/npm/litestream-vfs-linux-arm64/README.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/packages/npm/litestream-vfs-linux-arm64/README.md","title":"litestream-vfs-linux-arm64 Documentation","category":"plugin-manifest","format":"markdown","content":"# litestream-vfs-linux-arm64\n\nPlatform package for `litestream-vfs` — Linux ARM64.\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n","isInternal":false,"tokens":47,"sizeBytes":191},{"name":"README.md","path":"packages/npm/litestream-vfs/README.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/packages/npm/litestream-vfs/README.md","title":"litestream-vfs Documentation","category":"plugin-manifest","format":"markdown","content":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed via npm.\n\nThis package bundles the [Litestream](https://litestream.io) VFS shared library.\nThe correct platform-specific binary is automatically installed via `optionalDependencies`.\n\n## Installation\n\n```bash\nnpm install litestream-vfs\n```\n\n## Usage\n\n```javascript\nconst { getLoadablePath } = require(\"litestream-vfs\");\nconst path = getLoadablePath();\n// Use `path` with better-sqlite3 or other SQLite bindings\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","isInternal":false,"tokens":184,"sizeBytes":720},{"name":"README.md","path":"packages/python/README.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/packages/python/README.md","title":"python Documentation","category":"plugin-manifest","format":"markdown","content":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed as a Python wheel.\n\nThis package bundles the [Litestream](https://litestream.io) VFS shared library\nso you can load it directly into a Python `sqlite3` connection.\n\n## Installation\n\n```bash\npip install litestream-vfs\n```\n\n## Usage\n\n```python\nimport sqlite3\nimport litestream_vfs\n\nconn = sqlite3.connect(\":memory:\")\nlitestream_vfs.load(conn)\n```\n\nTo get the path to the shared library (for use with other SQLite bindings):\n\n```python\npath = litestream_vfs.loadable_path()\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","isInternal":false,"tokens":207,"sizeBytes":780},{"name":"README.md","path":"packages/ruby/README.md","rawUrl":"https://raw.githubusercontent.com/benbjohnson/litestream/HEAD/packages/ruby/README.md","title":"ruby Documentation","category":"plugin-manifest","format":"markdown","content":"# litestream-vfs\n\nLitestream VFS extension for SQLite — distributed as a Ruby gem.\n\nThis gem bundles the [Litestream](https://litestream.io) VFS shared library\nso you can load it directly into a SQLite connection.\n\n## Installation\n\n```bash\ngem install litestream-vfs\n```\n\nOr add to your Gemfile:\n\n```ruby\ngem \"litestream-vfs\"\n```\n\n## Usage\n\n```ruby\nrequire \"litestream_vfs\"\n\ndb = SQLite3::Database.new(\":memory:\")\nLitestreamVfs.load(db)\n```\n\nTo get the path to the shared library:\n\n```ruby\npath = LitestreamVfs.loadable_path\n```\n\n## Platform Support\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64, aarch64 |\n| macOS | x86_64, arm64 |\n\n## License\n\nApache-2.0 — see [LICENSE](https://github.com/benbjohnson/litestream/blob/main/LICENSE).\n","isInternal":false,"tokens":215,"sizeBytes":766}],"systemPromptSnippet":"\n\n# AGENTS.md - Litestream AI Agent Guide\n\nLitestream is a disaster recovery tool for SQLite that runs as a background process, monitors the WAL, converts changes to immutable LTX files, and replicates them to cloud storage. It uses `modernc.org/sqlite` (pure Go, no CGO required).\n\n## Before You Start\n\n1. Read [AI_PR_GUIDE.md](AI_PR_GUIDE.md) for contribution requirements\n2. Check [CONTRIBUTING.md](CONTRIBUTING.md) for what we accept (bug fixes welcome, features need discussion)\n3. Review recent PRs for current patterns\n\n## Critical Rules\n\n- **Lock page at 1GB**: SQLite reserves page at 0x40000000. Always skip it. See [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md)\n- **LTX files are immutable**: Never modify after creation. See [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md)\n- **Single replica per database**: Each DB replicates to exactly one destination\n- **Use `litestream ltx`**: Not `litestream wal` (deprecated)\n- **Use `litestream reset`**: Clears corrupted local LTX state for a database. See `cmd/litestream/reset.go`\n- **`auto-recover` config**: Replica option that automatically resets local state on LTX errors. Disabled by default. See `replica.go`\n- **Retention enabled by default**: `Store.RetentionEnabled` is `true` by default. Disable only when cloud lifecycle policies handle cleanup. See `store.go`\n- **IPC socket disabled by default**: Control socket is off by default. Enable with `socket.enabled: true` in config. See `server.go`\n- **`$PID` config expansion**: Config files support `$PID` to expand to the current process ID, plus standard `$ENV_VAR` expansion. See `cmd/litestream/main.go`\n- **`litestream ltx -level`**: Use `-level 0`–`9` or `-level all` to inspect specific compaction levels. See `cmd/litestream/ltx.go`\n- **Return errors, don't log them**: Always return errors to callers. Never `log.Printf(err)` and continue — this silently hides failures in a disaster recovery tool. Only use DEBUG log for best-effort operations where failure doesn't affect correctness and a valid fallback exists (e.g., reading SHM mxFrame optimization hint). See [docs/PATTERNS.md](docs/PATTERNS.md#error-handling)\n\n## Layer Boundaries\n\n| Layer | File | Responsibility |\n|-------|------|----------------|\n| DB | `db.go` | Database state, restoration, WAL monitoring, library API (`SyncStatus`, `SyncAndWait`, `EnsureExists`) |\n| Replica | `replica.go` | Replication mechanics only |\n| Storage | `**/replica_client.go` | Backend implementations (includes `ReplicaClientV3` for v0.3.x restore) |\n| IPC | `server.go` | Unix socket control API (register/unregister, /txid, pprof) |\n| Leasing | `leaser.go`, `s3/leaser.go` | Distributed lease acquisition via conditional writes |\n\nDatabase state logic belongs in DB layer, not Replica layer.\n\n## Quick Reference\n\n**Build:**\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\n```\n\n**Code quality:**\n\n```bash\npre-commit run --all-files\n```\n\n## Documentation\n\n| Document | When to Read |\n|----------|--------------|\n| [docs/PATTERNS.md](docs/PATTERNS.md) | Code patterns and anti-patterns |\n| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Deep component details |\n| [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) | WAL format, lock page |\n| [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md) | Replication format |\n| [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md) | Test strategies |\n| [docs/REPLICA_CLIENT_GUIDE.md](docs/REPLICA_CLIENT_GUIDE.md) | Adding storage backends |\n| [docs/PROVIDER_COMPATIBILITY.md](docs/PROVIDER_COMPATIBILITY.md) | Provider-specific S3/cloud configs |\n\n## Checklist\n\nBefore submitting changes:\n\n- [ ] Read relevant docs above\n- [ ] Follow patterns in [docs/PATTERNS.md](docs/PATTERNS.md)\n- [ ] Test with race detector (`go test -race`)\n- [ ] Run `pre-commit run --all-files`\n- [ ] For page iteration: test with >1GB databases\n- [ ] Show investigation evidence in PR (see [AI_PR_GUIDE.md](AI_PR_GUIDE.md))\n\n\n# Litestream\n\nDisaster recovery tool for SQLite. Replicates WAL changes to S3, GCS, Azure, SFTP, or local filesystem.\n\n## Quick Start for AI Contributors\n\n1. Read [AI_PR_GUIDE.md](AI_PR_GUIDE.md) - PR quality requirements\n2. Read [AGENTS.md](AGENTS.md) - Project overview and checklist\n3. Check [CONTRIBUTING.md](CONTRIBUTING.md) - What we accept\n4. Show investigation evidence in PRs\n\n## PR Checklist\n\n- [ ] Evidence of problem (logs, file patterns)\n- [ ] Clear scope (what PR does/doesn't do)\n- [ ] Runnable test commands\n- [ ] Race detector tested (`go test -race`)\n\n## Documentation\n\n| Document | Purpose |\n|----------|---------|\n| [AGENTS.md](AGENTS.md) | Project overview, critical rules |\n| [AI_PR_GUIDE.md](AI_PR_GUIDE.md) | PR templates, common mistakes |\n| [docs/PATTERNS.md](docs/PATTERNS.md) | Code patterns and anti-patterns |\n| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Component details |\n| [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) | WAL format, 1GB lock page |\n| [docs/LTX_FORMAT.md](docs/LTX_FORMAT.md) | Replication format |\n| [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md) | Test strategies |\n| [docs/REPLICA_CLIENT_GUIDE.md](docs/REPLICA_CLIENT_GUIDE.md) | Storage backends |\n\n## Core Files\n\n| File | Purpose |\n|------|---------|\n| `db.go` | Database monitoring, WAL, checkpoints |\n| `replica.go` | Replication management |\n| `store.go` | Multi-database coordination |\n| `replica_client.go` | Storage backend interface |\n\n## Storage Backends\n\n- `s3/replica_client.go` - AWS S3\n- `gs/replica_client.go` - Google Cloud Storage\n- `abs/replica_client.go` - Azure Blob Storage\n- `sftp/replica_client.go` - SFTP\n- `file/replica_client.go` - Local filesystem\n- `nats/replica_client.go` - NATS JetStream\n\n## Critical Concepts\n\n- **Lock page at 1GB** - Always skip page at 0x40000000\n- **LTX files are immutable** - Never modify after creation\n- **Single replica per DB** - One destination per database\n- **Layer boundaries** - DB handles state, Replica handles replication\n\n## Build\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\npre-commit run --all-files\n```\n\n\n# CLAUDE.md - Claude Code Configuration\n\nClaude-specific optimizations for Litestream. See [AGENTS.md](AGENTS.md) for project documentation.\n\n## Critical Coding Rules\n\n- **Always return errors, never log and continue.** The only exception is DEBUG logging for best-effort operations that don't affect correctness. This is the #1 cause of PR review feedback. See [docs/PATTERNS.md](docs/PATTERNS.md#error-handling) for the full decision framework.\n\n## Context Window\n\nWith Claude's large context window, load documentation as needed:\n\n- Start with [AGENTS.md](AGENTS.md) for overview and checklist\n- Load [docs/PATTERNS.md](docs/PATTERNS.md) when writing code\n- Load [docs/SQLITE_INTERNALS.md](docs/SQLITE_INTERNALS.md) for WAL/page work\n- Load [docs/PROVIDER_COMPATIBILITY.md](docs/PROVIDER_COMPATIBILITY.md) for storage backend configs\n\n## Claude-Specific Resources\n\n### Specialized Agents (.claude/agents/)\n\n- `sqlite-expert.md` - SQLite WAL and page management\n- `replica-client-developer.md` - Storage backend implementation\n- `ltx-compaction-specialist.md` - LTX format and compaction\n- `test-engineer.md` - Testing strategies\n- `performance-optimizer.md` - Performance optimization\n\n### Commands (.claude/commands/)\n\n- `/analyze-ltx` - Analyze LTX file structure\n- `/debug-ipc` - Debug IPC Unix socket issues\n- `/debug-wal` - Debug WAL replication issues\n- `/test-compaction` - Test compaction scenarios\n- `/trace-replication` - Trace replication flow\n- `/validate-replica` - Validate replica client\n- `/add-storage-backend` - Create new storage backend\n- `/fix-common-issues` - Diagnose common problems\n- `/run-comprehensive-tests` - Execute full test suite\n\n## Quick Commands\n\n```bash\ngo build -o bin/litestream ./cmd/litestream\ngo test -race -v ./...\npre-commit run --all-files\n```\n\n\n../AGENTS.md\n\n\n# Litestream Architecture Reference\n\nCondensed architectural reference for agents working on Litestream.\n\n## System Layers\n\n```\nApplication Layer cmd/litestream/ CLI commands, YAML/env config\nCore Layer store.go Multi-DB coordination, compaction scheduling\n db.go Single DB management, WAL monitoring, checkpoints\n replica.go Replication to single destination, position tracking\nStorage Abstraction replica_client.go ReplicaClient interface\nStorage Backends s3/, gs/, abs/, Backend-specific implementations\n oss/, file/, sftp/,\n nats/, webdav/\n```\n\nDatabase state logic belongs in the DB layer. The Replica layer handles\nreplication mechanics only.\n\n## Core Components\n\n### DB (db.go)\n\nThe heart of Litestream. Manages a single SQLite database.\n\nKey fields:\n- `path` / `metaPath`: Database and metadata paths\n- `db *sql.DB`: SQLite connection (via modernc.org/sqlite, pure Go)\n- `f *os.File`: Long-running file descriptor\n- `rtx *sql.Tx`: Long-running read transaction (prevents checkpoint past read point)\n- `pageSize int`: Database page size (critical for lock page calculation)\n- `mu sync.RWMutex`: Protects struct fields\n- `chkMu sync.RWMutex`: Checkpoint coordination lock\n- `notify chan struct{}`: WAL change notifications\n\nKey methods:\n- `Open()` / `Close()`: Lifecycle management\n- `monitor()`: Background WAL monitoring loop\n- `checkWAL()`: Detect WAL changes by comparing size/checksum\n- `Checkpoint(mode)`: Run SQLite checkpoint (PASSIVE, FULL, TRUNCATE)\n- `autoCheckpoint()`: Threshold-based checkpoint decisions\n- `Sync()`: Synchronize pending changes to replicas\n- `Compact()`: Merge LTX files at a given level\n\nConfiguration:\n- `MinCheckpointPageN`: Minimum pages for passive checkpoint\n- `TruncatePageN`: Emergency truncate checkpoint threshold\n- `CheckpointInterval`: Time-based passive checkpoint interval\n- `MonitorInterval`: WAL monitoring frequency\n- Note: RESTART checkpoint mode permanently removed (issue #724)\n\n### Replica (replica.go)\n\nManages replication to a single destination.\n\nKey fields:\n- `db *DB`: Parent database\n- `Client ReplicaClient`: Storage backend\n- `pos ltx.Pos`: Current replication position (TXID + PageNo + Checksum)\n- `mu sync.RWMutex`: Protects position\n\nPosition tracking: `SetPos()` MUST use `Lock()` not `RLock()`.\n\n### Store (store.go)\n\nCoordinates multiple databases and manages compaction.\n\nKey fields:\n- `dbs []*DB`: Managed databases\n- `levels CompactionLevels`: Compaction level configuration\n\nDefault compaction levels: L0 (raw), L1 (30s), L2 (5min), L3 (1h),\nplus daily snapshots.\n\n## WAL Monitoring\n\nThe monitor loop in `db.go` runs on a ticker:\n\n1. `checkWAL()` compares current WAL size/checksum to previous state\n2. If changed, notify replicas via `notifyReplicas()`\n3. Check if checkpoint needed via `shouldCheckpoint()`\n4. Run `autoCheckpoint()` if thresholds are met\n\nCheckpoint strategy:\n- WAL pages > `TruncatePageN` → TRUNCATE checkpoint (emergency)\n- WAL pages > `MinCheckpointPageN` → PASSIVE checkpoint\n- `CheckpointInterval` elapsed → PASSIVE checkpoint\n\n## Compaction Algorithm\n\nHigh-level flow (in `store.go`):\n\n1. Determine if level is due for compaction (`shouldCompact`)\n2. Enumerate level L-1 files via `ReplicaClient.LTXFiles`\n3. Prefer local copies (`os.Open(db.LTXPath(...))`) over remote reads\n4. Stream through `ltx.NewCompactor` (page deduplication, lock page skipping)\n5. Write merged file via `ReplicaClient.WriteLTXFile` for level L\n6. Set `CreatedAt` to earliest source file timestamp\n7. Update cached max file info; delete old L0 files when promoting to L1\n\n## State Management\n\n### Database States\n\nClosed → Opening → Open → Monitoring ↔ Syncing/Checkpointing → Closing → Closed\n\n### Replica States\n\nIdle → Starting → Monitoring ↔ Syncing/Uploading → Stopping → Idle\n\nError states retry with exponential backoff (1s initial, 1min max).\n\n## Initialization Flow\n\n1. `Store.Open()` iterates databases\n2. For each DB: open SQLite connection → read page size → create metadata dir →\n start long-running read transaction → initialize replicas → start monitor\n3. For each Replica: create with client → load previous position from metadata →\n validate against database → start sync goroutine\n4. Store starts compaction monitors\n\n## Error Handling\n\n### Error Categories\n\n- **Recoverable**: Network timeouts, temporary storage unavailability, lock contention\n- **Fatal**: Database corruption, invalid configuration, disk full\n- **Operational**: Checkpoint failures, compaction conflicts, sync delays\n\n### Error Propagation\n\n```\nReplicaClient.WriteLTXFile() error\n → Replica.Sync() error\n → DB.Sync() error\n → Store.monitorDB() // logs error, continues\n```\n\n## Lock Ordering\n\nAlways acquire in this order to prevent deadlocks:\n\n1. `Store.mu`\n2. `DB.mu`\n3. `DB.chkMu`\n4. `Replica.mu`\n\n## Goroutine Management\n\n- Use `sync.WaitGroup` for lifecycle tracking\n- Signal shutdown via `context.Cancel()`\n- Wait with timeout in `Close()` methods\n- Every goroutine must have a `defer wg.Done()`\n\n## Performance Characteristics\n\n| Operation | Time | Space |\n|-------------|------------|----------------|\n| WAL Monitor | O(1) | O(1) + metrics |\n| Page Write | O(1) | Original DB + WAL |\n| Compaction | O(n pages) | Temporary during merge |\n| Restoration | O(n·log m) | n=pages, m=files |\n| File List | O(k files) | Per-level |\n\n\n# LTX Format Reference\n\nCondensed reference for agents working with Litestream's LTX file format.\n\n## Overview\n\nLTX (Log Transaction) is Litestream's custom format for storing database changes.\nLTX files are:\n- **Immutable**: Once written, never modified\n- **Self-contained**: Each file is independent\n- **Indexed**: Contains page index for efficient seeks\n- **Checksummed**: CRC-64 ECMA integrity verification\n\n## File Structure\n\n```\n+---------------------+\n| Header | 100 bytes\n+---------------------+\n| Page Frames | Variable (4-byte pgno + pageSize data per page)\n+---------------------+\n| Page Index | Binary search index\n+---------------------+\n| Trailer | 16 bytes\n+---------------------+\n\nFileSize = HeaderSize + (PageCount * (4 + PageSize)) + PageIndexSize + TrailerSize\n```\n\n## Header (100 bytes)\n\n```\nOffset Size Field\n0 4 Magic (\"LTX1\")\n4 4 Flags (bit 1 = NoChecksum)\n8 4 PageSize\n12 4 Commit (page count after applying)\n16 8 MinTXID\n24 8 MaxTXID\n32 8 Timestamp (ms since Unix epoch)\n40 8 PreApplyChecksum\n48 8 WALOffset (0 for snapshots)\n56 8 WALSize (0 for snapshots)\n64 4 WALSalt1\n68 4 WALSalt2\n72 8 NodeID\n80 20 Reserved (zeros)\n```\n\nThe version is implied by the magic string. `\"LTX1\"` → `ltx.Version == 2`.\n\n## Page Frames\n\nEach frame contains one database page:\n\n```\nOffset Size Field\n0 4 Page Number (1-based)\n4 PageSize Page Data\n```\n\nConstraints:\n- Pages written in sequential order during creation\n- Lock page at 1 GB boundary is never included\n- In compacted files, only the latest version of each page\n\n## Page Index\n\nBinary search index for efficient random access to pages. Use\n`ltx.DecodePageIndex()` to parse into a map of page number to\n`ltx.PageIndexElem`:\n\n```go\ntype PageIndexElem struct {\n Level int\n MinTXID TXID\n MaxTXID TXID\n Offset int64 // Byte offset of encoded payload\n Size int64 // Bytes occupied by encoded payload\n}\n```\n\n## Trailer (16 bytes)\n\n```\nOffset Size Field\n0 8 PostApplyChecksum (database checksum after applying file)\n8 8 FileChecksum (CRC-64 of entire file)\n```\n\nThe trailer is always at the end of the file. Read it by seeking\n`-TrailerSize` from `io.SeekEnd`.\n\n## File Naming Convention\n\n```\nFormat: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\n MinTXID (16 hex) MaxTXID (16 hex)\n\nExample: 0000000000000001-0000000000000064.ltx (TXID 1-100)\n 0000000000000065-00000000000000c8.ltx (TXID 101-200)\n```\n\nParse with `ltx.ParseFilename()`, format with `ltx.FormatFilename()`.\n\n## Compaction Levels\n\nLTX files are organized in levels for efficient storage:\n\n```\nLevel 0: /ltx/0000/ Raw LTX files (no compaction)\nLevel 1: /ltx/0001/ Compacted (default: every 30s)\nLevel 2: /ltx/0002/ Compacted (default: every 5min)\nLevel 3: /ltx/0003/ Compacted (default: every 1h)\nSnapshots: Full database state (daily)\n```\n\n### Compaction Process\n\n1. Enumerate level L-1 files\n2. Build page map (newer pages overwrite older)\n3. Write merged file skipping lock page\n4. Preserve earliest `CreatedAt` from source files\n5. Delete old L0 files when promoting to L1\n\n```go\n// Page deduplication: latest version wins\nfor _, file := range files {\n for _, page := range file.Pages {\n pageMap[page.Number] = page\n }\n}\n```\n\n## Checksums\n\nLTX uses CRC-64 ECMA checksums (`hash/crc64` with `crc64.ECMA` table):\n\n- **PreApplyChecksum**: Database state before applying this file\n- **PostApplyChecksum**: Database state after applying this file\n- **FileChecksum**: Integrity of the entire LTX file\n\n## Reading LTX Files\n\n```go\ndec := ltx.NewDecoder(reader)\nheader, err := dec.Header()\nfor {\n var hdr ltx.PageHeader\n data := make([]byte, header.PageSize)\n if err := dec.DecodePage(&hdr, data); err == io.EOF {\n break\n }\n // Process hdr.Pgno and data\n}\ntrailer := dec.Trailer()\n```\n\n## Writing LTX Files\n\n```go\nenc := ltx.NewEncoder(writer)\nenc.EncodeHeader(header)\nfor _, page := range pages {\n if page.Number == ltx.LockPgno(pageSize) {\n continue // Skip lock page\n }\n enc.EncodePage(pageHeader, pageData)\n}\nenc.EncodePageIndex(index)\nenc.EncodeTrailer()\nenc.Close()\n```\n\n## CLI Inspection\n\n```bash\nlitestream ltx /path/to/db.sqlite\nlitestream ltx s3://bucket/db\n```\n\nFor low-level inspection, use the Go API with `ltx.NewDecoder`.\n\n## WAL to LTX Conversion\n\nSQLite WAL frames are converted to LTX format during replication:\n1. Read WAL frames from the WAL file\n2. Skip the lock page\n3. Add checksums and build page index\n4. Write as immutable LTX file\n5. Upload to storage backend\n\n\n# Litestream Code Patterns Reference\n\nCondensed code patterns and anti-patterns for agents writing Litestream code.\n\n## Architectural Boundaries\n\n```\nDB Layer (db.go) → Database state, restoration, monitoring\nReplica Layer (replica.go) → Replication mechanics only\nStorage Layer → ReplicaClient implementations\n```\n\n### DO: Handle database state in DB layer\n\n```go\nfunc (db *DB) init() error {\n if db.needsRestore() {\n if err := db.restore(); err != nil {\n return err\n }\n }\n return db.replica.Start() // Replica focuses only on replication\n}\n```\n\n### DON'T: Put database state logic in Replica layer\n\n```go\n// WRONG\nfunc (r *Replica) Start() error {\n if needsRestore() { // Wrong layer!\n restoreDatabase()\n }\n}\n```\n\n## Atomic File Operations\n\n### DO: Write to temp file, then rename\n\n```go\nfunc writeFileAtomic(path string, data []byte) error {\n dir := filepath.Dir(path)\n tmpFile, err := os.CreateTemp(dir, \".tmp-*\")\n if err != nil {\n return fmt.Errorf(\"create temp file: %w\", err)\n }\n tmpPath := tmpFile.Name()\n\n defer func() {\n if tmpFile != nil {\n tmpFile.Close()\n os.Remove(tmpPath)\n }\n }()\n\n if _, err := tmpFile.Write(data); err != nil {\n return fmt.Errorf(\"write temp file: %w\", err)\n }\n if err := tmpFile.Sync(); err != nil {\n return fmt.Errorf(\"sync temp file: %w\", err)\n }\n if err := tmpFile.Close(); err != nil {\n return fmt.Errorf(\"close temp file: %w\", err)\n }\n tmpFile = nil\n\n if err := os.Rename(tmpPath, path); err != nil {\n os.Remove(tmpPath)\n return fmt.Errorf(\"rename to final path: %w\", err)\n }\n return nil\n}\n```\n\n### DON'T: Write directly to final location\n\n```go\n// WRONG - Can leave partial files on failure\nos.WriteFile(path, data, 0644)\n```\n\n## Error Handling\n\n### DO: Return errors immediately\n\n```go\nfunc (db *DB) validatePosition() error {\n dpos, err := db.Pos()\n if err != nil {\n return err\n }\n rpos := replica.Pos()\n if dpos.TXID < rpos.TXID {\n return fmt.Errorf(\"database position (%v) behind replica (%v)\", dpos, rpos)\n }\n return nil\n}\n```\n\n### DON'T: Log and continue on critical errors\n\n```go\n// WRONG\nif err := processFile(file); err != nil {\n log.Printf(\"error: %v\", err) // Just logging!\n // Continuing is dangerous\n}\n```\n\n### DO: Return errors from loops\n\n```go\nfunc (db *DB) processFiles() error {\n for _, file := range files {\n if err := processFile(file); err != nil {\n return fmt.Errorf(\"process file %s: %w\", file, err)\n }\n }\n return nil\n}\n```\n\n## Locking Patterns\n\n### DO: Use Lock() for writes\n\n```go\nr.mu.Lock()\ndefer r.mu.Unlock()\nr.pos = pos\n```\n\n### DON'T: Use RLock for write operations\n\n```go\n// WRONG - Race condition\nr.mu.RLock()\ndefer r.mu.RUnlock()\nr.pos = pos // Writing with RLock!\n```\n\n### Lock Ordering\n\nAlways acquire in order: Store.mu → DB.mu → DB.chkMu → Replica.mu\n\n## Compaction Patterns\n\n### DO: Read from local when available\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil // Local copy is complete and consistent\n}\nreturn replica.Client.OpenLTXFile(...) // Fall back to remote\n```\n\n### DON'T: Read from remote during compaction\n\n```go\n// WRONG - Can get partial/corrupt data from eventually consistent storage\nf, err := client.OpenLTXFile(ctx, level, minTXID, maxTXID, 0, 0)\n```\n\n### DO: Preserve earliest timestamp\n\n```go\ninfo.CreatedAt = oldestSourceFile.CreatedAt\n```\n\n### DON'T: Use current time during compaction\n\n```go\n// WRONG - Loses timestamp granularity for point-in-time restores\ninfo := <x.FileInfo{CreatedAt: time.Now()}\n```\n\n## Lock Page Handling\n\nThe lock page at 1 GB (0x40000000) must always be skipped:\n\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n\n## Common Pitfalls\n\n1. **Mixing architectural concerns**: DB state logic in Replica layer\n2. **Recreating existing functionality**: Use `db.verify()` for snapshots\n3. **Ignoring lock page**: Must skip during replication and compaction\n4. **Generic error types**: Return `os.ErrNotExist` for missing files\n5. **Blocking iterators**: Use lazy pagination for `LTXFiles`\n6. **Ignoring context**: Check `ctx.Done()` in long operations\n\n\n# ReplicaClient Implementation Reference\n\nCondensed reference for agents implementing or modifying Litestream storage backends.\n\n## Interface Contract\n\nFrom `replica_client.go`:\n\n```go\ntype ReplicaClient interface {\n Type() string\n Init(ctx context.Context) error\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\n### Method Contracts\n\n**Type()**: Return identifier string (e.g., \"s3\", \"gcs\", \"file\").\n\n**Init()**: Initialize connection. Must be idempotent (no-op if already initialized).\n\n**LTXFiles()**: Return iterator sorted by MinTXID. `seek` starts from given TXID.\n`useMetadata=true` fetches accurate timestamps (required for point-in-time restore).\n`useMetadata=false` uses fast timestamps for normal operations.\n\n**OpenLTXFile()**: Open file for reading. Must support partial reads via `offset`/`size`.\nReturn `os.ErrNotExist` if file is missing.\n\n**WriteLTXFile()**: Write file to storage. Set `CreatedAt` from backend\nmetadata or upload time.\n\n**DeleteLTXFiles()**: Delete one or more files. Batch if possible.\n\n**DeleteAll()**: Delete all files for this database.\n\n## Implementation Checklist\n\n### Required\n\n- [ ] All interface methods implemented\n- [ ] `Init()` is idempotent\n- [ ] Partial reads supported (`offset`/`size` in `OpenLTXFile`)\n- [ ] `os.ErrNotExist` returned for missing files\n- [ ] Context cancellation handled in all methods\n- [ ] `CreatedAt` timestamps preserved in `WriteLTXFile`\n- [ ] Concurrent operations supported\n- [ ] Proper cleanup in `DeleteAll`\n\n### Optional\n\n- [ ] Connection pooling\n- [ ] Retry logic with exponential backoff\n- [ ] Request batching for deletes\n- [ ] Compression / encryption at rest\n- [ ] Bandwidth throttling\n\n## Eventual Consistency\n\nMany cloud backends (S3, R2, etc.) are eventually consistent:\n- Recently written files may not be immediately visible\n- Listed files may be only partially readable\n- Deletes may not take effect immediately\n\n### Pattern: Read Local First\n\nDuring compaction, always prefer local files:\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil\n}\nreturn replica.Client.OpenLTXFile(...)\n```\n\n### Pattern: Retry with Backoff\n\n```go\nbackoff := 100 * time.Millisecond\nfor i := 0; i < 5; i++ {\n reader, err := c.openFile(ctx, path, offset, size)\n if err == nil {\n return reader, nil\n }\n if errors.Is(err, os.ErrNotExist) {\n return nil, err // Don't retry definitive errors\n }\n select {\n case <-ctx.Done():\n return nil, ctx.Err()\n case <-time.After(backoff):\n backoff *= 2\n }\n}\n```\n\n### Pattern: Lazy Iterator\n\n```go\nfunc (c *Client) LTXFiles(...) (ltx.FileIterator, error) {\n return &lazyIterator{\n client: c,\n level: level,\n seek: seek,\n pageSize: 1000, // Paginate, don't load all at once\n }, nil\n}\n```\n\n## Error Handling\n\n### Standard Error Types\n\n```go\n// File not found\nreturn nil, os.ErrNotExist\n\n// Permission denied\nreturn nil, os.ErrPermission\n\n// Context cancelled\nreturn nil, ctx.Err()\n\n// Wrapped errors\nreturn nil, fmt.Errorf(\"s3 download failed: %w\", err)\n```\n\n### Retryable Error Classification\n\n```go\nfunc isRetryable(err error) bool {\n var netErr net.Error\n if errors.As(err, &netErr) && netErr.Temporary() {\n return true\n }\n // HTTP 429, 500, 502, 503, 504\n if errors.Is(err, context.DeadlineExceeded) {\n return true\n }\n return false\n}\n```\n\n## Common Mistakes\n\n### 1. Not Handling Partial Reads\n\n```go\n// WRONG - Ignores offset/size\nreturn c.storage.Download(path)\n\n// CORRECT\nif offset == 0 && size == 0 {\n return c.storage.Download(path)\n}\nreturn c.storage.DownloadRange(path, offset, offset+size-1)\n```\n\n### 2. Not Preserving CreatedAt\n\n```go\n// WRONG\nreturn <x.FileInfo{CreatedAt: time.Now()}\n\n// CORRECT - Use backend metadata\nreturn <x.FileInfo{CreatedAt: modTime}\n```\n\n### 3. Wrong Error Types\n\n```go\n// WRONG\nreturn nil, fmt.Errorf(\"not found\")\n\n// CORRECT\nif resp.StatusCode == 404 {\n return nil, os.ErrNotExist\n}\n```\n\n### 4. Ignoring Context\n\n```go\n// WRONG - Could run forever\nfor i := 0; i < 1000000; i++ {\n doWork()\n}\n\n// CORRECT\nselect {\ncase <-ctx.Done():\n return nil, ctx.Err()\ndefault:\n}\n```\n\n### 5. Loading All Files at Once\n\n```go\n// WRONG - Could be millions of files\nallFiles, _ := c.loadAllFiles(level)\n\n// CORRECT - Lazy pagination\nreturn &lazyIterator{pageSize: 1000}, nil\n```\n\n## Path Construction\n\n```go\nfunc (c *Client) ltxDir(level int) string {\n if level == SnapshotLevel {\n return path.Join(c.Path, \"snapshots\")\n }\n return path.Join(c.Path, \"ltx\", fmt.Sprintf(\"%04d\", level))\n}\n```\n\n## Reference Implementations\n\n- **Simplest**: `file/replica_client.go` (direct file I/O, no network)\n- **Most complete**: `s3/replica_client.go` (multipart uploads, retries, signing)\n\nStart with the file backend for understanding, then study S3 for advanced\npatterns.\n\n## Testing Requirements\n\n- Unit tests with >80% coverage\n- Integration tests with build tag\n- Test partial reads, concurrent operations, missing files\n- Run with `-race` flag\n- Verify `os.ErrNotExist` for missing files\n- Test context cancellation\n\n\n# SQLite Internals Reference\n\nCondensed reference for agents working with SQLite internals in Litestream.\n\n## SQLite File Structure\n\n```\ndatabase.db Main database file (pages)\ndatabase.db-wal Write-ahead log\ndatabase.db-shm Shared memory file (coordination)\n```\n\n## Write-Ahead Log (WAL)\n\nWAL is SQLite's method for atomic commits:\n- Changes written to WAL first, database unchanged until checkpoint\n- Readers merge WAL + database for consistent view\n- Litestream monitors WAL and converts frames to LTX format\n\n### WAL File Structure\n\n```\n+------------------+\n| WAL Header | 32 bytes\n+------------------+\n| Frame 1 Header | 24 bytes\n| Frame 1 Data | PageSize bytes\n+------------------+\n| Frame 2 Header | 24 bytes\n| Frame 2 Data | PageSize bytes\n+------------------+\n| ... |\n```\n\n### WAL Header (32 bytes)\n\n```\nMagic (4B) | FileFormat (4B) | PageSize (4B) | Checkpoint (4B) |\nSalt1 (4B) | Salt2 (4B) | Checksum1 (4B) | Checksum2 (4B)\n```\n\nMagic: `0x377f0682` or `0x377f0683`\n\n### WAL Frame Header (24 bytes)\n\n```\nPageNumber (4B) | DbSize (4B) | Salt1 (4B) | Salt2 (4B) |\nChecksum1 (4B) | Checksum2 (4B)\n```\n\n## The 1 GB Lock Page\n\nSQLite reserves a page at exactly 0x40000000 bytes (1 GB) for locking. This is\nthe single most important SQLite detail for Litestream development.\n\n```go\nconst PENDING_BYTE = 0x40000000\n\nfunc LockPgno(pageSize int) uint32 {\n return uint32(PENDING_BYTE/pageSize) + 1\n}\n```\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n| 64 KB | 16385 |\n\nRules:\n- Cannot contain data; SQLite never writes user data here\n- Must be skipped during replication and compaction\n- Only affects databases > 1 GB\n- Page number changes with page size\n\nImplementation in Litestream:\n\n```go\nfor pgno := uint32(1); pgno <= maxPgno; pgno++ {\n if pgno == ltx.LockPgno(db.pageSize) {\n continue\n }\n processPage(pgno)\n}\n```\n\n## Page Structure\n\nSQLite divides databases into fixed-size pages (typically 4096 bytes):\n- Page numbers are 1-based\n- Types: B-tree interior, B-tree leaf, overflow, freelist, lock byte page\n\n## Transaction Types\n\n1. **Deferred** (default): Lock acquired on first use\n2. **Immediate**: RESERVED lock acquired immediately\n3. **Exclusive**: EXCLUSIVE lock acquired immediately\n\n### Lock Hierarchy\n\nUNLOCKED → SHARED → RESERVED → PENDING → EXCLUSIVE\n\n- SHARED: Multiple readers allowed\n- RESERVED: Signals intent to write\n- PENDING: Blocks new SHARED locks\n- EXCLUSIVE: Single writer, no readers\n\n## Long-Running Read Transaction\n\nLitestream maintains a read transaction for consistency:\n\n```go\nfunc (db *DB) initReadTx() error {\n tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{ReadOnly: true})\n if err != nil {\n return err\n }\n var dummy string\n err = tx.QueryRow(\"SELECT ''\").Scan(&dummy)\n if err != nil {\n tx.Rollback()\n return err\n }\n db.rtx = tx\n return nil\n}\n```\n\nPurpose:\n- Prevents checkpoint past our read point\n- Ensures consistent database view\n- Allows reading historical pages from WAL\n\n## Checkpoint Modes\n\n| Mode | Behavior |\n|----------|-----------------------------------------------|\n| PASSIVE | Non-blocking; fails if readers present |\n| FULL | Waits for readers; blocks new readers |\n| RESTART | Like FULL + resets WAL start (removed in #724) |\n| TRUNCATE | Like RESTART + truncates WAL to zero |\n\n### Litestream Checkpoint Strategy\n\n```\nWAL pages > TruncatePageN → TRUNCATE (emergency)\nWAL pages > MinCheckpointPageN → PASSIVE\nCheckpointInterval elapsed → PASSIVE\n```\n\nNote: RESTART mode permanently removed due to issue #724 (write-blocking).\n\n## Important SQLite Pragmas\n\n```sql\nPRAGMA journal_mode = WAL; -- Required for Litestream\nPRAGMA page_size; -- Get page size\nPRAGMA page_count; -- Total pages in database\nPRAGMA freelist_count; -- Free pages\nPRAGMA wal_checkpoint(PASSIVE); -- Non-blocking checkpoint\nPRAGMA wal_autocheckpoint = 10000; -- High threshold (prevent interference)\nPRAGMA busy_timeout = 5000; -- Wait 5s for locks\nPRAGMA synchronous = NORMAL; -- Safe with WAL\nPRAGMA cache_size = -64000; -- 64 MB cache\nPRAGMA integrity_check; -- Verify database integrity\n```\n\n## Critical SQLite Behaviors\n\n1. **Automatic checkpoint**: Default at 1000 WAL pages; set high threshold to\n prevent interference with Litestream's control\n2. **Busy timeout**: Default is 0 (immediate failure); set a reasonable timeout\n3. **Synchronous mode**: NORMAL is safe with WAL mode\n4. **Page cache**: In-memory cache; configure with negative KB or positive pages\n\n## WAL to LTX Conversion\n\nLitestream converts WAL frames to LTX:\n\n1. Read WAL header and validate magic number\n2. Iterate frames, reading page number and data\n3. Skip lock page (`pgno == LockPgno(pageSize)`)\n4. Calculate checksums\n5. Write as immutable LTX file with page index\n\n\n# Litestream Testing Reference\n\nCondensed reference for agents writing and debugging Litestream tests.\n\n## Testing Philosophy\n\n1. Test at multiple levels: unit, integration, end-to-end\n2. Focus on edge cases: >1 GB databases, eventual consistency\n3. Use real SQLite: avoid mocking SQLite behavior\n4. Always run with `-race` flag\n5. Use fixed seeds and timestamps for deterministic tests\n\n## Essential Commands\n\n```bash\n# Full test suite with race detection\ngo test -race -v ./...\n\n# Specific test areas\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\n\n# Coverage\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\ngo tool cover -func=coverage.out | grep total\n\n# Integration tests (backend-specific)\ngo test -v ./replica_client_test.go -integration s3\ngo test -v ./replica_client_test.go -integration gcs\n```\n\n## Lock Page Testing\n\nThe lock page at 1 GB must be skipped. Test with databases spanning this\nboundary using multiple page sizes:\n\n```bash\n./bin/litestream-test populate -db test.db -page-size 4096 -target-size 2GB\n./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica\n```\n\nTest all page size variants:\n\n| Page Size | Lock Page | Test Target |\n|-----------|-----------|-------------|\n| 4 KB | 262145 | >1.0 GB |\n| 8 KB | 131073 | >1.0 GB |\n| 16 KB | 65537 | >1.0 GB |\n| 32 KB | 32769 | >1.0 GB |\n\nVerify lock page is not present in any LTX file after replication.\n\n## Race Condition Testing\n\nRace-prone areas (always run with `-race`):\n\n1. **Position updates**: Concurrent `SetPos()` + `Pos()` on Replica\n2. **WAL monitoring**: Concurrent writes + monitor + checkpoint\n3. **Compaction**: Concurrent compaction at different levels\n\n### Pattern: Concurrent Test\n\n```go\nfunc TestConcurrent(t *testing.T) {\n ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)\n defer cancel()\n\n var wg sync.WaitGroup\n // Start writer, monitor, checkpoint goroutines\n // Use ctx for shutdown, wg for cleanup\n wg.Wait()\n}\n```\n\n## Common Test Failures\n\n### 1. Database Locked\n\n```go\n// WRONG - WAL mode not enabled\ndb, _ := sql.Open(\"sqlite\", \"test.db\")\n\n// CORRECT\ndb, _ := sql.Open(\"sqlite\", \"test.db?_journal=WAL\")\n```\n\n### 2. Timing Issues\n\n```go\n// WRONG - Data may not be replicated yet\nWriteData(db)\nresult := ReadReplica()\n\n// CORRECT - Explicit sync\nWriteData(db)\ndb.Sync(context.Background())\nresult := ReadReplica()\n```\n\n### 3. Goroutine Leaks\n\n```go\n// WRONG - Goroutine outlives test\ngo func() {\n time.Sleep(10 * time.Second)\n doWork()\n}()\n\n// CORRECT - Use context + WaitGroup\nctx, cancel := context.WithCancel(context.Background())\ndefer cancel()\nvar wg sync.WaitGroup\nwg.Add(1)\ngo func() {\n defer wg.Done()\n select {\n case <-ctx.Done():\n return\n case <-time.After(10 * time.Second):\n doWork()\n }\n}()\ncancel()\nwg.Wait()\n```\n\n### 4. File Handle Leaks\n\n```go\n// Always use defer for cleanup\nf, err := os.Open(\"test.db\")\nrequire.NoError(t, err)\ndefer f.Close()\n```\n\n## Test Helper Patterns\n\n```go\nfunc NewTestDB(t testing.TB) *litestream.DB {\n t.Helper()\n path := filepath.Join(t.TempDir(), \"test.db\")\n conn, err := sql.Open(\"sqlite\", path+\"?_journal=WAL\")\n require.NoError(t, err)\n _, err = conn.Exec(\"CREATE TABLE test (id INTEGER PRIMARY KEY, data BLOB)\")\n require.NoError(t, err)\n conn.Close()\n\n db := litestream.NewDB(path)\n db.MonitorInterval = 10 * time.Millisecond\n db.MinCheckpointPageN = 100\n err = db.Open()\n require.NoError(t, err)\n t.Cleanup(func() { db.Close(context.Background()) })\n return db\n}\n```\n\n## Coverage Requirements\n\n| Package | Target |\n|--------------------|--------|\n| Core (db, replica) | >80% |\n| Replica clients | >70% |\n| Utilities | >60% |\n\n## Table-Driven Tests\n\n```go\nfunc TestCheckpoint(t *testing.T) {\n tests := []struct {\n name string\n mode string\n wantErr bool\n }{\n {\"Passive\", \"PASSIVE\", false},\n {\"Full\", \"FULL\", false},\n {\"Truncate\", \"TRUNCATE\", false},\n {\"Invalid\", \"INVALID\", true},\n }\n for _, tt := range tests {\n t.Run(tt.name, func(t *testing.T) {\n // ...\n })\n }\n}\n```\n\n\n---\nname: litestream\ndescription: >-\n Expert knowledge for contributing to Litestream, a standalone disaster recovery\n tool for SQLite. Provides architectural understanding, code patterns, critical\n rules, and debugging procedures for WAL monitoring, LTX replication format,\n storage backend implementation, multi-level compaction, and SQLite page\n management. Use when working with Litestream source code, writing storage\n backends, debugging replication issues, implementing compaction logic, or\n handling SQLite WAL operations.\nlicense: Apache-2.0\nmetadata:\n author: benbjohnson\n version: \"1.0\"\n repository: https://github.com/benbjohnson/litestream\n---\n\n# Litestream Agent Skill\n\nLitestream is a standalone disaster recovery tool for SQLite. It runs as a\nbackground process, monitors the SQLite WAL (Write-Ahead Log), converts changes\nto immutable LTX files, and replicates them to cloud storage. It uses\n`modernc.org/sqlite` (pure Go, no CGO required).\n\n## Quick Start\n\n```bash\n# Build\ngo build -o bin/litestream ./cmd/litestream\n\n# Test (always use race detector)\ngo test -race -v ./...\n\n# Code quality\npre-commit run --all-files\n```\n\n## Critical Rules\n\nThese invariants must never be violated:\n\n### 1. Lock Page at 1GB\n\nSQLite reserves a page at byte offset 0x40000000 (1 GB). Always skip it during\nreplication and compaction. The page number varies by page size:\n\n| Page Size | Lock Page Number |\n|-----------|------------------|\n| 4 KB | 262145 |\n| 8 KB | 131073 |\n| 16 KB | 65537 |\n| 32 KB | 32769 |\n\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n### 2. LTX Files Are Immutable\n\nOnce an LTX file is written, it must never be modified. New changes create new\nfiles. This guarantees point-in-time recovery integrity.\n\n### 3. Single Replica per Database\n\nEach database replicates to exactly one destination. The Replica component\nmanages replication mechanics; database state belongs in the DB layer.\n\n### 4. Read Local Before Remote During Compaction\n\nCloud storage is eventually consistent. Always read from local disk first:\n\n```go\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil // Use local copy\n}\nreturn replica.Client.OpenLTXFile(...) // Fall back to remote\n```\n\n### 5. Preserve Timestamps During Compaction\n\nSet the compacted file's `CreatedAt` to the earliest source file timestamp to\nmaintain temporal granularity for point-in-time restoration.\n\n```go\ninfo.CreatedAt = oldestSourceFile.CreatedAt\n```\n\n### 6. Use Lock() Not RLock() for Writes\n\n```go\n// CORRECT\nr.mu.Lock()\ndefer r.mu.Unlock()\nr.pos = pos\n\n// WRONG - race condition\nr.mu.RLock()\ndefer r.mu.RUnlock()\nr.pos = pos\n```\n\n### 7. Atomic File Operations\n\nAlways write to a temp file then rename. Never write directly to the final path.\n\n```go\ntmpFile, err := os.CreateTemp(dir, \".tmp-*\")\n// ... write data, sync ...\nos.Rename(tmpFile.Name(), finalPath)\n```\n\n## Architecture\n\n### System Layers\n\n| Layer | File(s) | Responsibility |\n|---------|--------------------------|-------------------------------------------|\n| App | `cmd/litestream/` | CLI commands, YAML/env config |\n| Store | `store.go` | Multi-DB coordination, compaction |\n| DB | `db.go` | Single DB management, WAL monitoring |\n| Replica | `replica.go` | Replication to one destination |\n| Storage | `*/replica_client.go` | Backend implementations (S3, GCS, etc.) |\n\nDatabase state logic belongs in the DB layer, not the Replica layer.\n\n### ReplicaClient Interface\n\nAll storage backends implement this interface from `replica_client.go`:\n\n```go\ntype ReplicaClient interface {\n Type() string\n Init(ctx context.Context) error\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\nKey contract details:\n- `OpenLTXFile` must return `os.ErrNotExist` when file is missing\n- `WriteLTXFile` must set `CreatedAt` from backend metadata or upload time\n- `LTXFiles` with `useMetadata=true` fetches accurate timestamps (for PIT restore)\n- `LTXFiles` with `useMetadata=false` uses fast timestamps (normal operations)\n\n### Lock Ordering\n\nAlways acquire locks in this order to prevent deadlocks:\n\n1. `Store.mu`\n2. `DB.mu`\n3. `DB.chkMu`\n4. `Replica.mu`\n\n### Core Components\n\n**DB** (`db.go`): Manages SQLite connection, WAL monitoring, checkpointing, and\nlong-running read transaction for consistency. Key fields: `path`, `db`, `rtx`\n(read transaction), `pageSize`, `notify` channel.\n\n**Replica** (`replica.go`): Tracks replication position (`ltx.Pos` with TXID,\nPageNo, Checksum). One replica per database.\n\n**Store** (`store.go`): Coordinates multiple databases and schedules compaction\nacross levels.\n\n## LTX File Format\n\nLTX (Log Transaction) files are immutable, checksummed archives of database\nchanges. Structure:\n\n```\n+------------------+\n| Header | 100 bytes (magic \"LTX1\", page size, TXID range, timestamp)\n+------------------+\n| Page Frames | 4-byte pgno + pageSize bytes data, per page\n+------------------+\n| Page Index | Binary search index for page lookup\n+------------------+\n| Trailer | 16 bytes (post-apply checksum, file checksum)\n+------------------+\n```\n\n### Naming Convention\n\n```\nFormat: MMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\nExample: 0000000000000001-0000000000000064.ltx (TXID 1-100)\n```\n\n### Compaction Levels\n\n```\nLevel 0: /ltx/0000/ Raw LTX files (no compaction)\nLevel 1: /ltx/0001/ Compacted periodically\nLevel 2: /ltx/0002/ Compacted less frequently\n```\n\nDefault compaction levels: L0 (raw), L1 (30s), L2 (5min), L3 (1h), plus daily\nsnapshots. Compaction merges files by deduplicating pages (latest version wins)\nand always skips the lock page.\n\n## Code Patterns\n\n### DO\n\n- Return errors immediately; let callers decide handling\n- Use `fmt.Errorf(\"context: %w\", err)` for error wrapping\n- Handle database state in the DB layer, not Replica\n- Use `db.verify()` to trigger snapshots (don't reimplement)\n- Test with race detector: `go test -race`\n- Use lazy iterators for `LTXFiles` (paginate, don't load all at once)\n\n### DON'T\n\n- Write data at the 1 GB lock page boundary\n- Modify LTX files after creation\n- Put database state logic in the Replica layer\n- Use `RLock()` when writing shared state\n- Write directly to final file paths (use temp + rename)\n- Ignore context cancellation in long operations\n- Return generic errors instead of `os.ErrNotExist` for missing files\n\n## Specialized Knowledge Areas\n\nLoad reference files on demand based on the task:\n\n| Task | Reference File |\n|-----------------------------------|-----------------------------------------|\n| Understanding system design | `references/ARCHITECTURE.md` |\n| Writing or reviewing code | `references/PATTERNS.md` |\n| Working with LTX files | `references/LTX_FORMAT.md` |\n| WAL monitoring or page operations | `references/SQLITE_INTERNALS.md` |\n| Implementing storage backends | `references/REPLICA_CLIENT_GUIDE.md` |\n| Writing or debugging tests | `references/TESTING_GUIDE.md` |\n\n## Common Debugging Procedures\n\n### Replication Not Working\n\n1. Verify WAL mode: `PRAGMA journal_mode` must return `wal`\n2. Check monitor interval and that the monitor goroutine is running\n3. Confirm `db.notify` channel is being signaled on WAL changes\n4. Check replica position: `replica.Pos()` should advance with writes\n5. Look for `os.ErrNotExist` from `OpenLTXFile` (file not replicated yet)\n\n### Large Database Issues (>1 GB)\n\n1. Verify lock page is being skipped: check `ltx.LockPgno(pageSize)`\n2. Test with multiple page sizes (4K, 8K, 16K, 32K)\n3. Run with databases both smaller and larger than 1 GB\n4. Ensure page iteration loops include the `continue` guard for lock page\n\n### Compaction Problems\n\n1. Confirm local L0 files exist before compaction reads them\n2. Check that `CreatedAt` timestamps are preserved (earliest source)\n3. Verify compaction level intervals in `Store.levels`\n4. Look for eventual consistency issues if reading from remote storage\n\n### Storage Backend Issues\n\n1. Return `os.ErrNotExist` for missing files (not generic errors)\n2. Support partial reads via `offset`/`size` in `OpenLTXFile`\n3. Handle context cancellation in all methods\n4. Test concurrent operations with `-race` flag\n5. For eventually consistent backends, add retry logic with backoff\n\n### Corrupted or Missing LTX Files\n\n1. Check logs for `LTXError` messages - they include context (Op, Path, Level, TXID) and recovery hints\n2. Common error messages: \"nonsequential page numbers\", \"non-contiguous transaction files\", \"ltx validation failed\"\n3. Manual fix: `litestream reset ` clears local LTX state and forces fresh snapshot on next sync (database file is not modified)\n4. Automatic fix: set `auto-recover: true` on the replica config to auto-reset on LTX errors (disabled by default)\n5. Reference: `cmd/litestream/reset.go`, `replica.go` (auto-recover logic), `db.go` (`ResetLocalState`)\n\n## Contribution Guidelines\n\n### What's Accepted\n\n- Bug fixes and patches (welcome)\n- Documentation improvements\n- Small code improvements and performance optimizations\n- Security vulnerability reports (report privately)\n\n### Discuss First\n\n- Feature requests: open an issue before implementing\n- Large changes: discuss approach in an issue first\n\n### Pre-Submit Checklist\n\n- [ ] Read relevant docs from the reference table above\n- [ ] Follow patterns in `references/PATTERNS.md`\n- [ ] Run `go test -race -v ./...`\n- [ ] Run `pre-commit run --all-files`\n- [ ] For page iteration: test with >1 GB databases\n- [ ] Show investigation evidence in PR (see CONTRIBUTING.md)\n\n## Testing\n\n```bash\n# Full test suite with race detection\ngo test -race -v ./...\n\n# Specific areas\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\n\n# Coverage\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\n```\n\nKey testing areas:\n- Lock page handling with >1 GB databases and multiple page sizes\n- Race conditions in position updates, WAL monitoring, and checkpointing\n- Eventual consistency in storage backend operations\n- Atomic file operations and cleanup on error paths\n\n## Environment Validation\n\nRun `scripts/validate-setup.sh` to verify your development environment is\ncorrectly configured for Litestream development.\n\n\n---\nrole: LTX Format and Compaction Specialist\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# LTX Compaction Specialist Agent\n\nYou are an expert in the LTX (Log Transaction) format and multi-level compaction strategies for Litestream.\n\n## Core Knowledge\n\n### LTX File Format\n```\n┌─────────────────────┐\n│ Header │ 84 bytes\n├─────────────────────┤\n│ Page Frames │ Variable\n├─────────────────────┤\n│ Page Index │ Binary search structure\n├─────────────────────┤\n│ Trailer │ 16 bytes\n└─────────────────────┘\n```\n\n### File Naming Convention\n```\nMMMMMMMMMMMMMMMM-NNNNNNNNNNNNNNNN.ltx\nWhere:\n M = MinTXID (16 hex digits)\n N = MaxTXID (16 hex digits)\nExample: 0000000000000001-0000000000000064.ltx\n```\n\n## Default Compaction Levels\n\n### Level Structure\n```\nLevel 0: Raw (no compaction)\nLevel 1: 30-second windows\nLevel 2: 5-minute windows\nLevel 3: 1-hour windows\nSnapshots: Daily full database\n```\n\n### Critical Compaction Rules\n\n1. **ALWAYS Read Local First**:\n ```go\n // CORRECT - Handles eventual consistency\n f, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\n if err == nil {\n return f, nil // Use local file\n }\n // Only fall back to remote if local doesn't exist\n return replica.Client.OpenLTXFile(...)\n ```\n\n2. **Preserve Timestamps**:\n ```go\n // Keep earliest CreatedAt\n info, err := replica.Client.WriteLTXFile(ctx, level, minTXID, maxTXID, reader)\n if err != nil {\n return nil, fmt.Errorf(\"write ltx file: %w\", err)\n }\n info.CreatedAt = oldestSourceFile.CreatedAt\n ```\n\n3. **Skip Lock Page**:\n ```go\n if pgno == ltx.LockPgno(pageSize) {\n continue\n }\n ```\n\n## Compaction Algorithm\n\n```go\nfunc compactLTXFiles(files []*LTXFile) (*LTXFile, error) {\n // 1. Create page map (newer overwrites older)\n pageMap := make(map[uint32]Page)\n for _, file := range files {\n for _, page := range file.Pages {\n pageMap[page.Number] = page\n }\n }\n\n // 2. Create new LTX with merged pages\n merged := <XFile{\n MinTXID: files[0].MinTXID,\n MaxTXID: files[len(files)-1].MaxTXID,\n }\n\n // 3. Add pages in order (skip lock page!)\n for pgno := uint32(1); pgno <= maxPgno; pgno++ {\n if pgno == LockPageNumber(pageSize) {\n continue\n }\n if page, ok := pageMap[pgno]; ok {\n merged.Pages = append(merged.Pages, page)\n }\n }\n\n return merged, nil\n}\n```\n\n## Key Properties\n\n### Immutability\n- LTX files are NEVER modified after creation\n- New changes create new files\n- Compaction creates new merged files\n\n### Checksums\n- CRC-64 ECMA for integrity\n- `PreApplyChecksum`/`PostApplyChecksum` on the header/trailer bracketing file state\n- `FileChecksum` covering the entire file contents\n\n### Page Index\n- Exposed via `ltx.DecodePageIndex`\n- Tracks page number plus offset/size of the encoded payload\n- Located by seeking from the end of the file using trailer metadata\n\n## Common Issues\n\n1. **Partial Reads**: Remote storage may return incomplete files\n2. **Race Conditions**: Multiple compactions running\n3. **Timestamp Loss**: Not preserving original CreatedAt\n4. **Lock Page**: Including 1GB lock page in compacted files\n5. **Memory Usage**: Loading entire files for compaction\n6. **Corrupted State**: Unclean shutdowns or storage failures can leave corrupted local LTX files, causing \"nonsequential page numbers\" or \"non-contiguous transaction files\" errors. Recovery: `litestream reset ` (manual) or `auto-recover: true` replica config (automatic). See `cmd/litestream/reset.go` and `replica.go`\n\n## Testing\n\n```bash\n# Test compaction\ngo test -v -run TestStore_CompactDB ./...\n\n# Test with eventual consistency\ngo test -v -run TestStore_CompactDB_RemotePartialRead ./...\n\n# Manual inspection\nlitestream ltx /path/to/db.sqlite\n# For deeper inspection use the Go API (ltx.NewDecoder)\n```\n\n## References\n- docs/LTX_FORMAT.md - Complete format specification\n- store.go - Compaction scheduling\n- db.go - Compaction implementation\n- github.com/superfly/ltx - LTX library\n\n\n---\nrole: Performance Optimizer\ntools:\n - read\n - write\n - edit\n - bash\n - grep\npriority: medium\n---\n\n# Performance Optimizer Agent\n\nYou specialize in optimizing Litestream for speed, memory usage, and resource efficiency.\n\n## Key Performance Areas\n\n### O(n) Operations to Optimize\n\n1. **Page Iteration**\n ```go\n // Cache page index\n const DefaultEstimatedPageIndexSize = 32 * 1024 // 32KB\n\n // Fetch end of file first for page index\n offset := info.Size - DefaultEstimatedPageIndexSize\n if offset < 0 {\n offset = 0\n }\n ```\n\n2. **File Listing**\n ```go\n // Cache file listings\n type FileCache struct {\n files []FileInfo\n timestamp time.Time\n ttl time.Duration\n }\n ```\n\n3. **Compaction**\n ```go\n // Limit concurrent compactions\n sem := make(chan struct{}, maxConcurrentCompactions)\n ```\n\n## Memory Optimization\n\n### Page Buffer Pooling\n```go\nvar pagePool = sync.Pool{\n New: func() interface{} {\n b := make([]byte, 4096) // Default page size\n return &b\n },\n}\n\nfunc getPageBuffer() []byte {\n return *pagePool.Get().(*[]byte)\n}\n\nfunc putPageBuffer(b []byte) {\n pagePool.Put(&b)\n}\n```\n\n### Streaming Instead of Loading\n```go\n// BAD - Loads entire file\ndata, err := os.ReadFile(path)\n\n// GOOD - Streams data\nf, err := os.Open(path)\ndefer f.Close()\nio.Copy(dst, f)\n```\n\n## Concurrency Patterns\n\n### Proper Locking\n```go\n// Read-heavy optimization\ntype Store struct {\n mu sync.RWMutex // Use RWMutex for read-heavy\n}\n\nfunc (s *Store) Read() {\n s.mu.RLock()\n defer s.mu.RUnlock()\n // Read operation\n}\n\nfunc (s *Store) Write() {\n s.mu.Lock()\n defer s.mu.Unlock()\n // Write operation\n}\n```\n\n### Channel Patterns\n```go\n// Batch processing\nbatch := make([]Item, 0, batchSize)\nticker := time.NewTicker(batchInterval)\n\nfor {\n select {\n case item := <-input:\n batch = append(batch, item)\n if len(batch) >= batchSize {\n processBatch(batch)\n batch = batch[:0]\n }\n case <-ticker.C:\n if len(batch) > 0 {\n processBatch(batch)\n batch = batch[:0]\n }\n }\n}\n```\n\n## I/O Optimization\n\n### Buffered I/O\n```go\n// Use buffered writers\nbw := bufio.NewWriterSize(w, 64*1024) // 64KB buffer\ndefer bw.Flush()\n\n// Use buffered readers\nbr := bufio.NewReaderSize(r, 64*1024)\n```\n\n### Parallel Downloads\n```go\nfunc downloadParallel(files []string) {\n var wg sync.WaitGroup\n sem := make(chan struct{}, 5) // Limit to 5 concurrent\n\n for _, file := range files {\n wg.Add(1)\n go func(f string) {\n defer wg.Done()\n sem <- struct{}{}\n defer func() { <-sem }()\n\n download(f)\n }(file)\n }\n wg.Wait()\n}\n```\n\n## Caching Strategy\n\n### LRU Cache Implementation\n```go\ntype LRUCache struct {\n capacity int\n items map[string]*list.Element\n list *list.List\n mu sync.RWMutex\n}\n\nfunc (c *LRUCache) Get(key string) (interface{}, bool) {\n c.mu.RLock()\n elem, ok := c.items[key]\n c.mu.RUnlock()\n\n if !ok {\n return nil, false\n }\n\n c.mu.Lock()\n c.list.MoveToFront(elem)\n c.mu.Unlock()\n\n return elem.Value, true\n}\n```\n\n## Profiling Tools\n\n### CPU Profiling\n```bash\n# Generate CPU profile\ngo test -cpuprofile=cpu.prof -bench=.\n\n# Analyze\ngo tool pprof cpu.prof\n(pprof) top10\n(pprof) list functionName\n```\n\n### Memory Profiling\n```bash\n# Generate memory profile\ngo test -memprofile=mem.prof -bench=.\n\n# Analyze allocations\ngo tool pprof -alloc_space mem.prof\n```\n\n### Trace Analysis\n```bash\n# Generate trace\ngo test -trace=trace.out\n\n# View trace\ngo tool trace trace.out\n```\n\n## Configuration Tuning\n\n### SQLite Pragmas\n```sql\nPRAGMA cache_size = -64000; -- 64MB cache\nPRAGMA synchronous = NORMAL; -- Balance safety/speed\nPRAGMA wal_autocheckpoint = 10000; -- Larger WAL before checkpoint\nPRAGMA busy_timeout = 5000; -- 5 second timeout\n```\n\n### Litestream Settings\n```yaml\n# Optimal intervals\nmin-checkpoint-page-n: 1000\ntruncate-page-n: 121359\nmonitor-interval: 1s\ncheckpoint-interval: 1m\n```\n\n## Benchmarks to Run\n\n```bash\n# Core operations\ngo test -bench=BenchmarkWALRead\ngo test -bench=BenchmarkLTXWrite\ngo test -bench=BenchmarkCompaction\ngo test -bench=BenchmarkPageIteration\n\n# With memory stats\ngo test -bench=. -benchmem\n```\n\n## Common Performance Issues\n\n1. **Not pooling buffers** - Creates garbage\n2. **Loading entire files** - Use streaming\n3. **Excessive locking** - Use RWMutex\n4. **No caching** - Repeated expensive operations\n5. **Serial processing** - Could parallelize\n6. **Small buffers** - Increase buffer sizes\n\n## References\n- Go performance tips: https://go.dev/doc/perf\n- SQLite optimization: https://sqlite.org/optoverview.html\n- Profiling guide: https://go.dev/blog/pprof\n\n\n---\nrole: Replica Client Developer\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# Replica Client Developer Agent\n\nYou specialize in implementing and maintaining storage backend clients for Litestream replication.\n\n## Core Knowledge\n\n### ReplicaClient Interface\n\nEvery storage backend MUST implement:\n```go\ntype ReplicaClient interface {\n Type() string\n LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error)\n OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error)\n WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error)\n DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error\n DeleteAll(ctx context.Context) error\n}\n```\n\n**LTXFiles useMetadata parameter**:\n- When `useMetadata=true`: Fetch accurate timestamps from backend metadata (slower, required for point-in-time restore)\n- When `useMetadata=false`: Use fast timestamps from file listing (faster, suitable for replication monitoring)\n\n### Critical Patterns\n\n1. **Eventual Consistency Handling**:\n - Storage may not immediately reflect writes\n - Files may be partially available\n - ALWAYS prefer local files during compaction\n\n2. **Atomic Operations**:\n ```go\n // Write to temp, then rename\n tmpPath := path + \".tmp\"\n // Write to tmpPath\n os.Rename(tmpPath, path)\n ```\n\n3. **Error Types**:\n - Return `os.ErrNotExist` for missing files\n - Wrap errors with context: `fmt.Errorf(\"operation: %w\", err)`\n\n4. **ResumableReader Support**:\n - `OpenLTXFile` MUST support the `offset` parameter for range requests\n - `internal/resumable_reader.go` wraps streams with auto-reconnection on idle timeouts\n - During restore, streams may sit idle while compactor processes other files\n - If `offset` is ignored, restore operations fail on connection timeouts\n\n### ReplicaClientV3 Interface (Optional)\n\nBackends supporting v0.3.x backward-compatible restore should implement:\n```go\ntype ReplicaClientV3 interface {\n GenerationsV3(ctx context.Context) ([]string, error)\n SnapshotsV3(ctx context.Context, generation string) ([]SnapshotInfoV3, error)\n WALSegmentsV3(ctx context.Context, generation string) ([]WALSegmentInfoV3, error)\n OpenSnapshotV3(ctx context.Context, generation string, index int) (io.ReadCloser, error)\n OpenWALSegmentV3(ctx context.Context, generation string, index int, offset int64) (io.ReadCloser, error)\n}\n```\n\nSee `v3.go` for type definitions and `s3/replica_client.go` for reference implementation.\n\n## Implementation Checklist\n\n### New Backend Requirements\n\n- [ ] Implement ReplicaClient interface\n- [ ] Handle partial reads (offset/size)\n- [ ] Support seek parameter for pagination\n- [ ] Preserve CreatedAt timestamps when metadata is available\n- [ ] Handle eventual consistency\n- [ ] Implement proper error types\n- [ ] Add integration tests\n- [ ] Document configuration\n\n### Testing Requirements\n\n```bash\n# Integration test\ngo test -v ./replica_client_test.go -integration [backend]\n\n# Race conditions\ngo test -race -v ./[backend]/...\n\n# Large files (>1GB)\n./bin/litestream-test populate -target-size 2GB\n```\n\n## Existing Backends Reference\n\n### Study These Implementations\n\n- `s3/replica_client.go` - AWS S3 (most complete)\n- `gs/replica_client.go` - Google Cloud Storage\n- `abs/replica_client.go` - Azure Blob Storage\n- `file/replica_client.go` - Local filesystem (simplest)\n- `sftp/replica_client.go` - SSH File Transfer\n- `nats/replica_client.go` - NATS JetStream (newest)\n- `oss/replica_client.go` - Alibaba Cloud OSS\n\n## Common Pitfalls\n\n1. Not handling eventual consistency\n2. Missing atomic write operations\n3. Incorrect error types\n4. Not preserving timestamps\n5. Forgetting partial read support\n6. No retry logic for transient failures\n7. Not supporting `offset` parameter in `OpenLTXFile` — breaks `ResumableReader` during restore\n\n## Configuration Pattern\n\n```yaml\nreplica:\n type: [backend]\n option1: value1\n option2: value2\n```\n\n## References\n\n- docs/REPLICA_CLIENT_GUIDE.md - Complete implementation guide\n- replica_client.go - Interface definition\n- v3.go - ReplicaClientV3 interface and v0.3.x types\n- internal/resumable_reader.go - ResumableReader for restore resilience\n- replica_client_test.go - Test suite\n\n\n---\nrole: SQLite WAL and Page Expert\ntools:\n - read\n - write\n - edit\n - grep\n - bash\npriority: high\n---\n\n# SQLite Expert Agent\n\nYou are a SQLite internals expert specializing in WAL (Write-Ahead Log) operations and page management for the Litestream project.\n\n## Core Knowledge\n\n### Critical SQLite Concepts\n1. **1GB Lock Page** (MUST KNOW):\n - Located at exactly 0x40000000 (1,073,741,824 bytes)\n - Page number varies by page size:\n - 4KB pages: 262145\n - 8KB pages: 131073\n - 16KB pages: 65537\n - 32KB pages: 32769\n - MUST be skipped in all iterations\n - Cannot contain data\n\n2. **WAL Structure**:\n - 32-byte header with magic number\n - Frames with 24-byte headers\n - Cumulative checksums\n - Salt values for verification\n\n3. **Page Types**:\n - B-tree interior/leaf pages\n - Overflow pages\n - Freelist pages\n - Lock byte page (at 1GB)\n\n## Primary Responsibilities\n\n### WAL Monitoring\n- Monitor WAL file changes in `db.go`\n- Ensure proper checksum verification\n- Handle WAL frame reading correctly\n- Convert WAL frames to LTX format\n\n### Page Management\n- Always skip lock page during iteration\n- Handle various page sizes correctly\n- Verify page integrity\n- Manage page caching efficiently\n\n### Testing Requirements\n- Create test databases >1GB\n- Test all page sizes (4KB, 8KB, 16KB, 32KB)\n- Verify lock page skipping\n- Test WAL checkpoint modes\n\n## Code Patterns\n\n### Correct Lock Page Handling\n```go\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue // Skip lock page\n}\n```\n\n### WAL Reading\n```go\n// Always verify magic number\nmagic := binary.BigEndian.Uint32(header.Magic[:])\nif magic != 0x377f0682 && magic != 0x377f0683 {\n return errors.New(\"invalid WAL magic\")\n}\n```\n\n## Common Mistakes to Avoid\n1. Not skipping lock page at 1GB\n2. Incorrect checksum calculations\n3. Wrong byte order (use BigEndian)\n4. Not handling all page sizes\n5. Direct file manipulation (use SQLite API)\n\n## References\n- docs/SQLITE_INTERNALS.md - Complete SQLite internals guide\n- docs/LTX_FORMAT.md - LTX conversion details\n- db.go - WAL monitoring implementation\n\n\n---\nrole: Test Engineer\ntools:\n - read\n - write\n - edit\n - bash\n - grep\npriority: medium\n---\n\n# Test Engineer Agent\n\nYou specialize in creating and maintaining comprehensive test suites for Litestream, with focus on edge cases and race conditions.\n\n## Critical Test Scenarios\n\n### 1GB Lock Page Testing\n\n**MUST TEST**: Databases crossing the 1GB boundary\n\n```bash\n# Create >1GB test database\nsqlite3 large.db < c.failAfter {\n return errors.New(\"simulated failure\")\n }\n return c.ReplicaClient.WriteLTXFile(...)\n}\n```\n\n## Coverage Requirements\n\n### Minimum Coverage\n- Core packages: >80%\n- Storage backends: >70%\n- Critical paths: 100%\n\n### Generate Coverage Report\n```bash\ngo test -coverprofile=coverage.out ./...\ngo tool cover -html=coverage.out\n```\n\n## Common Test Mistakes\n\n1. Not testing with databases >1GB\n2. Forgetting race detector\n3. Not testing all page sizes\n4. Missing eventual consistency tests\n5. No error injection tests\n6. Ignoring benchmark regressions\n\n## CI/CD Integration\n\n```yaml\n# .github/workflows/test.yml\n- name: Run tests with race detector\n run: go test -race -v ./...\n\n- name: Test large databases\n run: ./scripts/test-large-db.sh\n\n- name: Integration tests\n run: ./scripts/test-integration.sh\n```\n\n## References\n- docs/TESTING_GUIDE.md - Complete testing guide\n- replica_client_test.go - Integration test patterns\n- db_test.go - Unit test examples\n\n\n---\ndescription: Create a new storage backend implementation\n---\n\n# Add Storage Backend Command\n\nCreate a new storage backend implementation for Litestream with all required components.\n\n## Steps\n\n1. **Create Package Directory**\n ```bash\n mkdir -p {{backend_name}}\n ```\n\n2. **Implement ReplicaClient Interface**\n Create `{{backend_name}}/replica_client.go`:\n ```go\n package {{backend_name}}\n\n type ReplicaClient struct {\n // Configuration fields\n }\n\n func (c *ReplicaClient) Type() string {\n return \"{{backend_name}}\"\n }\n\n func (c *ReplicaClient) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {\n // List files at level\n // When useMetadata=true, fetch accurate timestamps from backend metadata\n }\n\n func (c *ReplicaClient) OpenLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, offset, size int64) (io.ReadCloser, error) {\n // Open file for reading\n }\n\n func (c *ReplicaClient) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {\n // Write file atomically\n }\n\n func (c *ReplicaClient) DeleteLTXFiles(ctx context.Context, files []*ltx.FileInfo) error {\n // Delete files\n }\n\n func (c *ReplicaClient) DeleteAll(ctx context.Context) error {\n // Remove all files for replica\n }\n ```\n\n3. **Add Configuration Parsing**\n Update `cmd/litestream/config.go`:\n ```go\n case \"{{backend_name}}\":\n client = &{{backend_name}}.ReplicaClient{\n // Parse config\n }\n ```\n\n4. **Create Integration Tests**\n Create `{{backend_name}}/replica_client_test.go`:\n ```go\n func TestReplicaClient_{{backend_name}}(t *testing.T) {\n if !*integration || *backend != \"{{backend_name}}\" {\n t.Skip(\"{{backend_name}} integration test skipped\")\n }\n // Test implementation\n }\n ```\n\n5. **Add Documentation**\n Update README.md with configuration example:\n ```yaml\n replica:\n type: {{backend_name}}\n option1: value1\n option2: value2\n ```\n\n## Key Requirements\n\n- Handle eventual consistency\n- Implement atomic writes (temp file + rename)\n- Support partial reads (offset/size)\n- Preserve CreatedAt timestamps in returned FileInfo\n- Return proper error types (os.ErrNotExist)\n\n## Testing\n\n```bash\n# Run integration tests\ngo test -v ./replica_client_test.go -integration {{backend_name}}\n\n# Test with race detector\ngo test -race -v ./{{backend_name}}/...\n```\n\n\nAnalyze LTX file issues in Litestream. This command helps diagnose problems with LTX files, including corruption, missing files, and consistency issues.\n\nFirst, understand the context:\n- What error messages are being reported?\n- Which storage backend is being used?\n- Are there any eventual consistency issues?\n\nThen perform the analysis:\n\n1. **Check LTX file structure**: Look for corrupted headers, invalid page indices, or checksum mismatches in the LTX files.\n\n2. **Verify file continuity**: Ensure there are no gaps in the TXID sequence that could prevent restoration.\n\n3. **Check compaction issues**: Look for problems during compaction that might corrupt files, especially with eventually consistent storage.\n\n4. **Analyze page sequences**: Verify that page numbers are sequential and the lock page at 1GB is properly skipped.\n\n5. **Review storage backend behavior**: Check if the storage backend has eventual consistency that might cause partial reads during compaction.\n\nKey files to examine:\n- `db.go`: WAL monitoring and LTX generation\n- `replica_client.go`: Storage interface\n- `store.go`: Compaction logic\n- Backend-specific client in `s3/`, `gs/`, etc.\n\nCommon issues to look for:\n- \"nonsequential page numbers\" errors (corrupted compaction)\n- \"EOF\" errors (partial file reads)\n- Missing TXID ranges (failed uploads)\n- Lock page at 0x40000000 not being skipped\n\nUse the testing harness to reproduce:\n```bash\n./bin/litestream-test validate -source-db test.db -replica-url [URL]\n```\n\n## Recovery Options\n\nWhen analysis reveals corrupted or missing LTX files, two recovery mechanisms are available:\n\n**Manual recovery** with `litestream reset`:\n```bash\nlitestream reset /path/to/database.db\n```\nClears local LTX state from the metadata directory. The database file is not modified.\nNext sync creates a fresh snapshot. See `cmd/litestream/reset.go` for implementation.\n\n**Automatic recovery** with `auto-recover` config:\n```yaml\ndbs:\n - path: /path/to/database.db\n replicas:\n - url: s3://bucket/path\n auto-recover: true\n```\nWhen enabled, Litestream automatically resets local state when LTX errors are detected\nduring sync. Disabled by default. See `replica.go` (auto-recover logic) for implementation.\n\n\n---\ndescription: Debug IPC Unix socket issues\n---\n\n# Debug IPC Command\n\nDiagnose issues with the Litestream IPC control socket (`server.go`).\n\n## 1. Socket Configuration Check\n\nVerify the socket is enabled and correctly configured:\n\n```yaml\n# litestream.yml\nsocket:\n enabled: true # Default: false\n path: /var/run/litestream.sock # Default path\n permissions: 0600 # Default permissions\n```\n\nSource: `SocketConfig` struct in `server.go:17-21`, defaults in `DefaultSocketConfig()`.\n\n## 2. Endpoint Reference\n\nAll endpoints are HTTP over Unix socket (`server.go:74-87`):\n\n| Method | Path | Request Body | Response |\n|--------|------|-------------|----------|\n| `GET` | `/info` | — | `{version, pid, uptime_seconds, started_at, database_count}` |\n| `GET` | `/list` | — | `{databases: [{path, status, last_sync_at}]}` |\n| `GET` | `/txid?path=` | — | `{txid}` |\n| `POST` | `/register` | `{path, replica_url}` | `{status, path}` |\n| `POST` | `/unregister` | `{path, timeout?}` | `{status, path}` |\n| `POST` | `/start` | `{path, timeout?}` | `{status, path}` |\n| `POST` | `/stop` | `{path, timeout?}` | `{status, path}` |\n| `GET` | `/debug/pprof/` | — | Standard Go pprof index |\n| `GET` | `/debug/pprof/profile` | — | CPU profile |\n| `GET` | `/debug/pprof/trace` | — | Execution trace |\n\n## 3. Testing with curl\n\n```bash\n# Server info (version, PID, uptime)\ncurl --unix-socket /var/run/litestream.sock http://localhost/info\n\n# List all managed databases\ncurl --unix-socket /var/run/litestream.sock http://localhost/list\n\n# Get transaction ID for a specific database\ncurl --unix-socket /var/run/litestream.sock \"http://localhost/txid?path=/path/to/db\"\n\n# Register a new database at runtime\ncurl --unix-socket /var/run/litestream.sock -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"path\":\"/path/to/db\",\"replica_url\":\"s3://bucket/path\"}' \\\n http://localhost/register\n\n# Unregister (stop replicating) a database\ncurl --unix-socket /var/run/litestream.sock -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"path\":\"/path/to/db\"}' \\\n http://localhost/unregister\n\n# CPU profile (30 seconds by default)\ncurl --unix-socket /var/run/litestream.sock http://localhost/debug/pprof/profile > cpu.prof\ngo tool pprof cpu.prof\n\n# Heap profile\ncurl --unix-socket /var/run/litestream.sock http://localhost/debug/pprof/heap > heap.prof\ngo tool pprof heap.prof\n```\n\n## 4. Common Issues\n\n### Socket not enabled\n**Symptom**: `curl: (7) Couldn't connect to server`\n**Fix**: Set `socket.enabled: true` in config and restart litestream.\n\n### Permission denied\n**Symptom**: `curl: (7) Permission denied`\n**Fix**: Check `socket.permissions` (default `0600`). The connecting user must match the litestream process owner.\n\n### Stale socket file after crash\n**Symptom**: `bind: address already in use` in logs on startup\n**Fix**: Remove the stale socket file: `rm /var/run/litestream.sock`\n\n### Database not found\n**Symptom**: `{\"error\":\"database not found\"}` from `/txid` or `/start`\n**Fix**: Verify the path matches the expanded path in config. Use `/list` to see registered databases. Note that `$PID` and env vars are expanded in config paths.\n\n### Register fails with \"already exists\"\n**Symptom**: `{\"error\":\"database already registered\"}`\n**Fix**: The database path is already being replicated. Use `/unregister` first, then `/register` with new settings.\n\n## 5. pprof Debugging\n\nAvailable pprof endpoints on the IPC socket:\n\n```bash\n# Interactive CPU profile analysis\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/profile?seconds=30 > cpu.prof\ngo tool pprof -http=:8080 cpu.prof\n\n# Goroutine dump (useful for deadlock investigation)\ncurl --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/goroutine?debug=2\n\n# Memory allocation profile\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/heap > heap.prof\ngo tool pprof -http=:8080 heap.prof\n\n# Execution trace (captures scheduler, GC, goroutine events)\ncurl -s --unix-socket /var/run/litestream.sock \\\n http://localhost/debug/pprof/trace?seconds=5 > trace.out\ngo tool trace trace.out\n```\n\n\nDebug WAL monitoring issues in Litestream. This command helps diagnose problems with WAL change detection, checkpointing, and replication triggers.\n\nFirst, understand the symptoms:\n- Is replication not triggering on changes?\n- Are checkpoints failing or not happening?\n- Is the WAL growing unbounded?\n\nThen debug the monitoring system:\n\n1. **Check monitor goroutine** (db.go:1499):\n```go\n// Verify monitor is running\nfunc (db *DB) monitor() {\n ticker := time.NewTicker(db.MonitorInterval) // Default: 1s\n // Check if ticker is firing\n // Verify checkWAL() is being called\n}\n```\n\n2. **Verify WAL change detection**:\n```go\n// Check if WAL changes are detected\nfunc (db *DB) checkWAL() (bool, error) {\n // Get WAL size and checksum\n // Compare with previous values\n // Should return true if changed\n}\n```\n\n3. **Debug checkpoint triggers**:\n```go\n// Check checkpoint thresholds\nMinCheckpointPageN int // Default: 1000 pages\nTruncatePageN int // Default: 121359 pages\n\n// Verify WAL page count\nwalPageCount := db.WALPageCount()\nif walPageCount > db.MinCheckpointPageN {\n // Should trigger passive checkpoint\n}\nif walPageCount > db.TruncatePageN {\n // Should trigger truncate checkpoint (emergency brake)\n}\n```\n\n4. **Check long-running read transaction**:\n```go\n// Ensure rtx is maintained\nif db.rtx == nil {\n // Read transaction lost - replication may fail\n}\n```\n\n5. **Monitor notification channel**:\n```go\n// Check if replicas are notified\nselect {\ncase <-db.notify:\n // WAL change detected\ndefault:\n // No changes\n}\n```\n\nCommon issues to check:\n- MonitorInterval too long (default 1s)\n- Checkpoint failing due to active transactions\n- Read transaction preventing checkpoint\n- Notify channel not triggering replicas\n- WAL file permissions issues\n\nDebug commands:\n```sql\n-- Check WAL status\nPRAGMA wal_checkpoint;\nPRAGMA journal_mode;\nPRAGMA page_count;\nPRAGMA wal_autocheckpoint;\n\n-- Check for locks\nSELECT * FROM pragma_lock_status();\n```\n\nTesting WAL monitoring:\n```go\nfunc TestDB_WALMonitoring(t *testing.T) {\n db := setupTestDB(t)\n\n // Set fast monitoring for test\n db.MonitorInterval = 10 * time.Millisecond\n\n // Write data\n writeTestData(t, db, 100)\n\n // Wait for notification\n select {\n case <-db.notify:\n // Success\n case <-time.After(1 * time.Second):\n t.Error(\"WAL change not detected\")\n }\n}\n```\n\nMonitor with logging:\n```go\nslog.Debug(\"wal check\",\n \"size\", walInfo.Size,\n \"checksum\", walInfo.Checksum,\n \"pages\", walInfo.PageCount)\n```\n\n\n---\ndescription: Fix common Litestream issues\n---\n\n# Fix Common Issues Command\n\nDiagnose and fix common issues in Litestream deployments.\n\n## Issue 1: Lock Page Not Being Skipped\n\n**Symptom**: Errors or corruption with databases >1GB\n\n**Check**:\n```bash\n# Find lock page references\ngrep -r \"LockPgno\" --include=\"*.go\"\n```\n\n**Fix**:\n```go\n// Ensure all page iterations skip lock page\nlockPgno := ltx.LockPgno(pageSize)\nif pgno == lockPgno {\n continue\n}\n```\n\n## Issue 2: Race Condition in Replica Position\n\n**Symptom**: Data races detected, inconsistent position tracking\n\n**Check**:\n```bash\ngo test -race -v -run TestReplica_Sync ./...\n```\n\n**Fix**:\n```go\n// Change from RLock to Lock for writes\nfunc (r *Replica) SetPos(pos ltx.Pos) {\n r.mu.Lock() // NOT RLock!\n defer r.mu.Unlock()\n r.pos = pos\n}\n```\n\n## Issue 3: Eventual Consistency Issues\n\n**Symptom**: Compaction failures, partial file reads\n\n**Check**:\n```bash\n# Look for remote reads during compaction\ngrep -r \"OpenLTXFile\" db.go | grep -v \"os.Open\"\n```\n\n**Fix**:\n```go\n// Always try local first\nf, err := os.Open(db.LTXPath(info.Level, info.MinTXID, info.MaxTXID))\nif err == nil {\n return f, nil\n}\n// Only fall back to remote if local doesn't exist\nreturn replica.Client.OpenLTXFile(...)\n```\n\n## Issue 4: CreatedAt Timestamp Loss\n\n**Symptom**: Point-in-time recovery lacks accurate timestamps\n\n**Check**:\n```go\ninfo, err := client.WriteLTXFile(ctx, level, minTXID, maxTXID, r)\nif err != nil {\n t.Fatal(err)\n}\nif info.CreatedAt.IsZero() {\n t.Fatal(\"CreatedAt not set\")\n}\n```\n\n**Fix**:\n```go\n// Ensure storage metadata is copied into the returned FileInfo\nmodTime := resp.LastModified\ninfo.CreatedAt = modTime\n```\n\n## Issue 5: Non-Atomic File Writes\n\n**Symptom**: Partial files, corruption on crash\n\n**Check**:\n```bash\n# Find direct writes without temp files\ngrep -r \"os.Create\\|os.WriteFile\" --include=\"*.go\"\n```\n\n**Fix**:\n```go\n// Write to temp, then rename\ntmpPath := path + \".tmp\"\nif err := os.WriteFile(tmpPath, data, 0644); err != nil {\n return err\n}\nreturn os.Rename(tmpPath, path)\n```\n\n## Issue 6: WAL Checkpoint Blocking\n\n**Symptom**: WAL grows indefinitely, database locks\n\n**Check**:\n```sql\n-- Check WAL size\nPRAGMA wal_checkpoint(PASSIVE);\nSELECT page_count * page_size FROM pragma_page_count(), pragma_page_size();\n```\n\n**Fix**:\n```go\n// Release read transaction periodically\ndb.rtx.Rollback()\ndb.rtx = nil\n// Checkpoint\ndb.db.Exec(\"PRAGMA wal_checkpoint(RESTART)\")\n// Restart read transaction\ndb.initReadTx()\n```\n\n## Issue 7: Memory Leaks\n\n**Symptom**: Growing memory usage over time\n\n**Check**:\n```bash\n# Generate heap profile\ngo test -memprofile=mem.prof -run=XXX -bench=.\ngo tool pprof -top mem.prof\n```\n\n**Fix**:\n```go\n// Use sync.Pool for buffers\nvar pagePool = sync.Pool{\n New: func() interface{} {\n b := make([]byte, pageSize)\n return &b\n },\n}\n\n// Close resources properly\ndefer func() {\n if f != nil {\n f.Close()\n }\n}()\n```\n\n## Issue 8: Corrupted or Missing LTX Files\n\n**Symptom**: Sync failures with errors like \"ltx validation failed\", \"nonsequential page numbers\",\n\"non-contiguous transaction files\", or persistent sync retry backoff loops after unclean shutdowns.\n\n**Check**:\n```bash\n# Look for LTXError messages in logs\nrg -i \"ltx.*error|reset.*local|auto.recover\" /var/log/litestream.log\n\n# Check if meta directory has corrupted state (note dot prefix)\nls -la /path/to/.database.db-litestream/ltx/\n```\n\n**Fix - Manual Reset**:\n```bash\n# Clears local LTX state, forces fresh snapshot on next sync\n# Database file is NOT modified\nlitestream reset /path/to/database.db\n\n# With explicit config file\nlitestream reset -config /etc/litestream.yml /path/to/database.db\n```\n\n**Fix - Automatic Recovery**:\n```yaml\n# Add to replica config in litestream.yml\ndbs:\n - path: /path/to/database.db\n replicas:\n - url: s3://bucket/path\n auto-recover: true # Automatically resets on LTX errors\n```\n\n**When to use which**: Use `auto-recover` for unattended deployments where automatic recovery\nis preferred over manual intervention. Use manual `reset` when you want to investigate the\ncorruption first. `auto-recover` is disabled by default because resetting discards local LTX\nhistory, which may reduce point-in-time restore granularity.\n\n**Reference**: `cmd/litestream/reset.go`, `replica.go` (auto-recover logic), `db.go` (`ResetLocalState`)\n\n## Issue 9: IPC Socket Connection Failures\n\n**Symptom**: `connection refused` or `no such file or directory` when using the control socket\n\n**Check**:\n```bash\n# Verify socket exists and has correct permissions\nls -la /var/run/litestream.sock\n\n# Verify socket is enabled in config\nrg -A3 'socket:' /etc/litestream.yml\n\n# Check if litestream process is running\npgrep -a litestream\n```\n\n**Fix**:\n```yaml\n# Enable socket in litestream.yml\nsocket:\n enabled: true\n path: /var/run/litestream.sock\n permissions: 0600\n```\n\n**Stale socket**: If litestream crashed, the socket file may still exist. The process creates a new socket on startup and will fail if the stale file exists. Remove it manually:\n```bash\nrm /var/run/litestream.sock\n```\n\n**Reference**: `server.go` (`SocketConfig`, `Server.Start`)\n\n## Issue 10: Backup Files Accumulating (Retention Disabled)\n\n**Symptom**: Storage usage growing unbounded, old LTX files never deleted\n\n**Check**:\n```bash\n# Look for retention warning in logs\nrg \"retention disabled\" /var/log/litestream.log\n```\n\n**Fix**:\n```yaml\n# Option 1: Re-enable Litestream retention (default)\nretention:\n enabled: true\n\n# Option 2: Keep disabled, but configure cloud lifecycle policies\n# Example: S3 lifecycle rule to expire objects in ltx/ prefix after 30 days\n```\n\n**Reference**: `store.go` (`SetRetentionEnabled`), `compactor.go` (`RetentionEnabled`), `cmd/litestream/replicate.go:295`\n\n## Issue 11: v0.3.x Restore Not Finding Backups\n\n**Symptom**: Restore fails with \"no snapshots available\" but v0.3.x backups exist in storage\n\n**Check**:\n```bash\n# Verify v0.3.x backup structure exists\naws s3 ls s3://bucket/path/generations/ --recursive | head -20\n```\n\n**Fix**: Ensure the backend implements `ReplicaClientV3`. The S3 backend supports this automatically. The restore process checks for v0.3.x backups when no v0.4.x+ backup is found.\n\n**Reference**: `v3.go` (`ReplicaClientV3` interface), `s3/replica_client.go`\n\n## Diagnostic Commands\n\n```bash\n# Check database integrity\nsqlite3 database.db \"PRAGMA integrity_check;\"\n\n# List replicated LTX files\nlitestream ltx /path/to/db.sqlite\n\n# Check replication status\nlitestream databases\n\n# Reset corrupted local state\nlitestream reset /path/to/database.db\n\n# Test restoration\nlitestream restore -o test.db [replica-url]\n\n# IPC socket diagnostics (requires socket.enabled: true)\ncurl --unix-socket /var/run/litestream.sock http://localhost/info\ncurl --unix-socket /var/run/litestream.sock http://localhost/list\ncurl --unix-socket /var/run/litestream.sock \"http://localhost/txid?path=/path/to/db\"\n```\n\n## Prevention Checklist\n\n- [ ] Always test with databases >1GB\n- [ ] Run with race detector in CI\n- [ ] Test all page sizes (4KB, 8KB, 16KB, 32KB)\n- [ ] Verify eventual consistency handling\n- [ ] Check for proper locking (Lock vs RLock)\n- [ ] Ensure atomic file operations\n- [ ] Preserve timestamps in compaction\n\n\n---\ndescription: Run comprehensive test suite for Litestream\n---\n\n# Run Comprehensive Tests Command\n\nExecute a full test suite including unit tests, integration tests, race detection, and large database tests.\n\n## Quick Test Suite\n\n```bash\n# Basic tests with race detection\ngo test -race -v ./...\n\n# With coverage\ngo test -race -cover -v ./...\n```\n\n## Full Test Suite\n\n### 1. Unit Tests\n```bash\necho \"=== Running Unit Tests ===\"\ngo test -v ./... -short\n```\n\n### 2. Race Condition Tests\n```bash\necho \"=== Testing for Race Conditions ===\"\ngo test -race -v -run TestReplica_Sync ./...\ngo test -race -v -run TestDB_Sync ./...\ngo test -race -v -run TestStore_CompactDB ./...\ngo test -race -v ./...\n```\n\n### 3. Integration Tests\n```bash\necho \"=== Running Integration Tests ===\"\n\n# S3 (requires AWS credentials)\nAWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy \\\n go test -v ./replica_client_test.go -integration s3\n\n# Google Cloud Storage (requires credentials)\nGOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json \\\n go test -v ./replica_client_test.go -integration gcs\n\n# Azure Blob Storage\nAZURE_STORAGE_ACCOUNT=xxx AZURE_STORAGE_KEY=yyy \\\n go test -v ./replica_client_test.go -integration abs\n\n# SFTP (requires SSH server)\ngo test -v ./replica_client_test.go -integration sftp\n\n# File system (always available)\ngo test -v ./replica_client_test.go -integration file\n```\n\n### 4. Large Database Tests (>1GB)\n```bash\necho \"=== Testing Large Databases ===\"\n\n# Create test database for each page size\nfor pagesize in 4096 8192 16384 32768; do\n echo \"Testing page size: $pagesize\"\n\n # Create >1GB database\n sqlite3 test-${pagesize}.db <70% for core packages\n✅ Lock page correctly skipped for all page sizes\n✅ Restoration works for databases >1GB\n✅ No memory leaks in benchmarks\n\n\nTest Litestream compaction logic. This command helps test and debug compaction issues, especially with eventually consistent storage backends.\n\nFirst, understand the test scenario:\n- What storage backend needs testing?\n- What size database is involved?\n- Are there eventual consistency concerns?\n\nThen create comprehensive tests:\n\n1. **Test basic compaction**:\n```go\nfunc TestCompaction_Basic(t *testing.T) {\n // Create multiple LTX files at level 0\n // Run compaction to level 1\n // Verify merged file is correct\n}\n```\n\n2. **Test with eventual consistency**:\n```go\nfunc TestStore_CompactDB_RemotePartialRead(t *testing.T) {\n // Use mock client that returns partial data initially\n // Verify compaction prefers local files\n // Ensure no corruption occurs\n}\n```\n\n3. **Test lock page handling during compaction**:\n```go\nfunc TestCompaction_LockPage(t *testing.T) {\n // Create database > 1GB\n // Compact with data around lock page\n // Verify lock page is skipped (page at 0x40000000)\n}\n```\n\n4. **Test timestamp preservation**:\n```go\nfunc TestCompaction_PreserveTimestamps(t *testing.T) {\n // Compact files with different CreatedAt times\n // Verify earliest timestamp is preserved\n}\n```\n\nKey areas to test:\n- Reading from local files first (db.go:1280-1294)\n- Skipping lock page at 1GB boundary\n- Preserving CreatedAt timestamps\n- Handling partial/incomplete remote files\n- Concurrent compaction safety\n\nRun with race detector:\n```bash\ngo test -race -v -run TestStore_CompactDB ./...\n```\n\nUse the test harness for large databases:\n```bash\n./bin/litestream-test populate -db test.db -target-size 1.5GB\n./bin/litestream-test validate -source-db test.db -replica-url file:///tmp/replica\n```\n\n\nTrace the complete replication flow in Litestream. This command helps understand how changes flow from SQLite through to storage backends.\n\nFollow the replication path step by step:\n\n1. **Application writes to SQLite**:\n```sql\n-- Application performs write\nINSERT INTO table VALUES (...);\n-- SQLite appends to WAL file\n```\n\n2. **DB.monitor() syncs the shadow WAL** (db.go:1499):\n```go\nticker := time.NewTicker(db.MonitorInterval) // default 1s\nfor {\n select {\n case <-db.ctx.Done():\n return\n case <-ticker.C:\n }\n\n if err := db.Sync(db.ctx); err != nil && !errors.Is(err, context.Canceled) {\n db.Logger.Error(\"sync error\", \"error\", err)\n }\n}\n```\n\n3. **Replica.monitor() responds** (replica.go):\n```go\nticker := time.NewTicker(r.SyncInterval)\ndefer ticker.Stop()\n\nnotify := r.db.Notify()\nfor {\n select {\n case <-ctx.Done():\n return\n case <-ticker.C:\n // Enforce minimum sync interval\n case <-notify:\n // WAL changed, time to sync\n }\n\n notify = r.db.Notify()\n\n if err := r.Sync(ctx); err != nil && !errors.Is(err, context.Canceled) {\n r.Logger().Error(\"monitor error\", \"error\", err)\n }\n}\n```\n\n4. **Replica.Sync() uploads new L0 files** (replica.go):\n```go\n// Determine local database position\ndpos, err := r.db.Pos()\nif err != nil {\n return err\n}\nif dpos.IsZero() {\n return fmt.Errorf(\"no position, waiting for data\")\n}\n\n// Upload each unreplicated L0 file\nfor txID := r.Pos().TXID + 1; txID <= dpos.TXID; txID = r.Pos().TXID + 1 {\n if err := r.uploadLTXFile(ctx, 0, txID, txID); err != nil {\n return err\n }\n r.SetPos(ltx.Pos{TXID: txID})\n}\n```\n\n5. **ReplicaClient uploads to storage**:\n```go\nfunc (c *S3Client) WriteLTXFile(ctx context.Context, level int, minTXID, maxTXID ltx.TXID, r io.Reader) (*ltx.FileInfo, error) {\n // Stream LTX data to storage and return metadata (size, CreatedAt, checksums)\n}\n```\n\n6. **Checkpoint when thresholds are hit**:\n```go\nif walPageCount > db.MinCheckpointPageN {\n db.Checkpoint(ctx, litestream.CheckpointModePassive)\n}\nif walPageCount > db.TruncatePageN {\n db.Checkpoint(ctx, litestream.CheckpointModeTruncate)\n}\n```\n\nKey synchronization points:\n- WAL monitoring (1s intervals)\n- Replica sync (configurable, default 1s)\n- Checkpoint triggers (page thresholds)\n- Compaction (hourly/daily)\n\nTrace with logging:\n```go\n// Enable debug logging\nslog.SetLogLevel(slog.LevelDebug)\n\n// Key log points:\nslog.Debug(\"wal changed\", \"size\", walSize)\nslog.Debug(\"syncing replica\", \"pos\", r.Pos())\nslog.Debug(\"ltx uploaded\", \"txid\", maxTXID)\nslog.Debug(\"checkpoint complete\", \"mode\", mode)\n```\n\nPerformance metrics to monitor:\n- WAL growth rate\n- Sync latency\n- Upload throughput\n- Checkpoint frequency\n- Compaction duration\n\nCommon bottlenecks:\n1. Slow storage uploads\n2. Large transactions causing big LTX files\n3. Long-running read transactions blocking checkpoints\n4. Eventual consistency delays\n5. Network latency to storage\n\nTest replication flow:\n```bash\n# Start replication with verbose logging\nlitestream replicate -v\n\n# In another terminal, write to database\nsqlite3 test.db \"INSERT INTO test VALUES (1, 'data');\"\n\n# Watch logs for flow:\n# - WAL change detected\n# - Replica sync triggered\n# - LTX file uploaded\n# - Position updated\n```\n\nVerify replication:\n```bash\n# List replicated files\naws s3 ls s3://bucket/path/ltx/0000/\n\n# Restore and verify\nlitestream restore -o restored.db s3://bucket/path\nsqlite3 restored.db \"SELECT * FROM test;\"\n```\n\n\nValidate a ReplicaClient implementation in Litestream. This command helps ensure a replica client correctly implements the interface and handles edge cases.\n\nFirst, identify what needs validation:\n- Which replica client implementation?\n- What storage backend specifics?\n- Any known issues or concerns?\n\nThen validate the implementation:\n\n1. **Interface compliance check**:\n\n ```go\n // Ensure all methods are implemented\n var _ litestream.ReplicaClient = (*YourClient)(nil)\n ```\n\n1. **Verify error types**:\n\n ```go\n // OpenLTXFile must return os.ErrNotExist for missing files\n _, err := client.OpenLTXFile(ctx, 0, 999, 999, 0, 0)\n if !errors.Is(err, os.ErrNotExist) {\n t.Errorf(\"Expected os.ErrNotExist, got %v\", err)\n }\n ```\n\n1. **Test partial reads**:\n\n ```go\n // Must support offset and size parameters\n rc, err := client.OpenLTXFile(ctx, 0, 1, 100, 50, 25)\n data, _ := io.ReadAll(rc)\n if len(data) != 25 {\n t.Errorf(\"Expected 25 bytes, got %d\", len(data))\n }\n ```\n\n1. **Verify timestamp preservation**:\n\n ```go\n // CreatedAt should reflect remote object metadata (or upload time)\n start := time.Now()\n info, _ := client.WriteLTXFile(ctx, 0, 1, 100, reader)\n if info.CreatedAt.IsZero() || info.CreatedAt.Before(start.Add(-time.Second)) {\n t.Error(\"unexpected CreatedAt timestamp\")\n }\n ```\n\n1. **Test eventual consistency handling**:\n - Implement retry logic for transient failures\n - Handle partial file availability\n - Verify write-after-write consistency\n\n1. **Validate cleanup**:\n\n ```go\n // DeleteAll must remove everything\n err := client.DeleteAll(ctx)\n files, _ := client.LTXFiles(ctx, 0, 0, false)\n if files.Next() {\n t.Error(\"Files remain after DeleteAll\")\n }\n ```\n\nKey validation points:\n- Proper error types (os.ErrNotExist, os.ErrPermission)\n- Context cancellation handling\n- Concurrent operation safety\n- Iterator doesn't load all files at once\n- Proper path construction for storage backend\n\nRun integration tests:\n```bash\ngo test -v ./[backend]/replica_client_test.go -integration\n```\n\n\n# Litestream Architecture - Technical Deep Dive\n\n## Table of Contents\n- [System Layers](#system-layers)\n- [Core Components](#core-components)\n- [IPC Server & Control Socket](#ipc-server--control-socket)\n- [Distributed Leasing](#distributed-leasing)\n- [Library Convenience Methods](#library-convenience-methods)\n- [LTX File Format](#ltx-file-format)\n- [WAL Monitoring Mechanism](#wal-monitoring-mechanism)\n- [Compaction Process](#compaction-process)\n- [Transaction Management](#transaction-management)\n- [Concurrency Model](#concurrency-model)\n- [State Management](#state-management)\n- [Initialization Flow](#initialization-flow)\n- [Error Handling](#error-handling)\n\n## System Layers\n\nLitestream follows a layered architecture with clear separation of concerns:\n\n```mermaid\ngraph TB\n subgraph \"Application Layer\"\n CLI[CLI Commands
cmd/litestream/]\n Config[Configuration
config.go]\n end\n\n subgraph \"Core Layer\"\n Store[Store Manager
store.go]\n DB[Database Manager
db.go]\n Replica[Replica Manager
replica.go]\n end\n\n subgraph \"Storage Abstraction\"\n RC[ReplicaClient Interface
replica_client.go]\n end\n\n subgraph \"Storage Implementations\"\n S3[s3/replica_client.go]\n GCS[gs/replica_client.go]\n ABS[abs/replica_client.go]\n OSS[oss/replica_client.go]\n File[file/replica_client.go]\n SFTP[sftp/replica_client.go]\n NATS[nats/replica_client.go]\n end\n\n subgraph \"External\"\n SQLite[SQLite Database]\n Cloud[Cloud Storage]\n end\n\n CLI --> Store\n Store --> DB\n DB --> Replica\n Replica --> RC\n RC --> S3\n RC --> GCS\n RC --> ABS\n RC --> OSS\n RC --> File\n RC --> SFTP\n RC --> NATS\n DB <--> SQLite\n S3 --> Cloud\n GCS --> Cloud\n ABS --> Cloud\n OSS --> Cloud\n```\n\n### Layer Responsibilities\n\n#### 1. Application Layer\n- **CLI Commands**: User interface for operations (replicate, restore, etc.)\n- **Configuration**: YAML/environment variable parsing and validation\n\n#### 2. Core Layer\n- **Store**: Multi-database coordination, compaction scheduling\n- **DB**: Single database management, WAL monitoring, checkpointing\n- **Replica**: Replication to single destination, position tracking\n\n#### 3. Storage Abstraction\n- **ReplicaClient Interface**: Uniform API for all storage backends\n\n#### 4. Storage Implementations\n- Backend-specific logic (authentication, retries, optimizations)\n\n#### 5. IPC Layer\n- **Server**: Unix socket HTTP server for runtime control (`server.go`)\n- **Leaser**: Distributed lease interface for coordination (`leaser.go`, `s3/leaser.go`)\n\n## Core Components\n\n### DB Component (db.go)\n\nThe DB component is the heart of Litestream, managing a single SQLite database:\n\n```go\ntype DB struct {\n // Core fields\n path string // Database file path\n metaPath string // Metadata directory path\n db *sql.DB // SQLite connection\n f *os.File // Long-running file descriptor\n rtx *sql.Tx // Long-running read transaction\n pageSize int // Database page size\n\n // Synchronization\n mu sync.RWMutex // Protects struct fields\n chkMu sync.RWMutex // Checkpoint lock\n notify chan struct{} // WAL change notifications\n\n // Lifecycle\n ctx context.Context\n cancel func()\n wg sync.WaitGroup\n\n // Configuration\n MinCheckpointPageN int // Min pages for passive checkpoint\n TruncatePageN int // Pages before emergency truncate checkpoint\n CheckpointInterval time.Duration // Time-based passive checkpoint interval\n MonitorInterval time.Duration // WAL monitoring frequency\n // Note: MaxCheckpointPageN removed (RESTART mode disabled due to #724)\n\n // Metrics\n dbSizeGauge prometheus.Gauge\n walSizeGauge prometheus.Gauge\n txIDGauge prometheus.Gauge\n}\n```\n\n#### Key Methods\n\n```go\n// Lifecycle\nfunc (db *DB) Open() error\nfunc (db *DB) Close(ctx context.Context) error\n\n// Monitoring\nfunc (db *DB) monitor() // Background WAL monitoring\nfunc (db *DB) checkWAL() (bool, error) // Check for WAL changes\n\n// Checkpointing\nfunc (db *DB) Checkpoint(mode string) error\nfunc (db *DB) autoCheckpoint() error\n\n// Replication\nfunc (db *DB) WALReader(pgno uint32) (io.ReadCloser, error)\nfunc (db *DB) Sync(ctx context.Context) error\n\n// Compaction\nfunc (db *DB) Compact(ctx context.Context, destLevel int) (*ltx.FileInfo, error)\n```\n\n### Replica Component (replica.go)\n\nManages replication to a single destination:\n\n```go\ntype Replica struct {\n db *DB // Parent database\n Client ReplicaClient // Storage backend client\n\n mu sync.RWMutex\n pos ltx.Pos // Current replication position\n\n // Configuration\n SyncInterval time.Duration\n MonitorEnabled bool\n\n // Lifecycle\n cancel func()\n wg sync.WaitGroup\n}\n```\n\n#### Replication Position\n\n```go\ntype Pos struct {\n TXID TXID // Transaction ID\n PageNo uint32 // Page number within transaction\n Checksum uint64 // Running checksum\n}\n```\n\n### Store Component (store.go)\n\nCoordinates multiple databases and manages system-wide resources:\n\n```go\ntype Store struct {\n mu sync.Mutex\n dbs []*DB\n levels CompactionLevels\n\n // Configuration\n SnapshotInterval time.Duration\n SnapshotRetention time.Duration\n L0Retention time.Duration\n L0RetentionCheckInterval time.Duration\n CompactionMonitorEnabled bool\n\n // Lifecycle\n ctx context.Context\n cancel func()\n wg sync.WaitGroup\n}\n```\n\n## IPC Server & Control Socket\n\nLitestream exposes a Unix socket HTTP server for runtime control (`server.go`).\n\n### Socket Configuration\n\n```go\ntype SocketConfig struct {\n Enabled bool `yaml:\"enabled\"` // Default: false\n Path string `yaml:\"path\"` // Default: \"/var/run/litestream.sock\"\n Permissions uint32 `yaml:\"permissions\"` // Default: 0600\n}\n```\n\n### HTTP Endpoints\n\nAll endpoints are served over the Unix socket via Go's `net/http` mux:\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `POST` | `/register` | Add a database at runtime |\n| `POST` | `/unregister` | Remove a database at runtime |\n| `GET` | `/txid?path=` | Get current transaction ID for a database |\n| `GET` | `/list` | List all managed databases |\n| `GET` | `/info` | Server info (version, PID, uptime, database count) |\n| `POST` | `/start` | Start replication for a database |\n| `POST` | `/stop` | Stop replication for a database |\n| `GET` | `/debug/pprof/*` | Standard Go pprof endpoints |\n\n### Request/Response Types\n\n```go\ntype RegisterDatabaseRequest struct {\n Path string `json:\"path\"`\n ReplicaURL string `json:\"replica_url\"`\n}\n\ntype UnregisterDatabaseRequest struct {\n Path string `json:\"path\"`\n Timeout int `json:\"timeout,omitempty\"` // Seconds\n}\n\ntype TXIDResponse struct {\n TXID uint64 `json:\"txid\"`\n}\n```\n\n## Distributed Leasing\n\nThe `Leaser` interface (`leaser.go`) enables distributed coordination so multiple Litestream instances can safely share a replica destination.\n\n### Interface\n\n```go\ntype Leaser interface {\n Type() string\n AcquireLease(ctx context.Context) (*Lease, error)\n RenewLease(ctx context.Context, lease *Lease) (*Lease, error)\n ReleaseLease(ctx context.Context, lease *Lease) error\n}\n\ntype Lease struct {\n Generation int64 `json:\"generation\"`\n ExpiresAt time.Time `json:\"expires_at\"`\n Owner string `json:\"owner,omitempty\"`\n ETag string `json:\"-\"`\n}\n```\n\n### S3 Implementation (`s3/leaser.go`)\n\n- **Defaults**: `DefaultLeaseTTL = 30s`, `DefaultLeasePath = \"lock.json\"`\n- **Owner format**: `hostname:pid` (falls back to `pid-N` if hostname unavailable)\n- **Conditional writes**: Uses `If-Match`/`If-None-Match` on S3 PutObject to prevent races\n- **Release**: Uses `DeleteObject` with `If-Match` to ensure only the holder can release\n- **Error types**: `ErrLeaseNotHeld`, `ErrLeaseAlreadyReleased`, `LeaseExistsError{Owner, ExpiresAt}`\n\n### Acquisition Flow\n\n1. Read existing lease from S3 (`lock.json`)\n2. If lease exists and is not expired, return `LeaseExistsError`\n3. Write new lease with `If-None-Match: *` (first acquire) or `If-Match: ` (expired takeover)\n4. If `PreconditionFailed` (412), another instance acquired first\n5. Return lease with ETag for subsequent renewal\n\n## Library Convenience Methods\n\nThese methods on `DB` (`db.go`) simplify common operations when Litestream is used as a library:\n\n```go\ntype SyncStatus struct {\n LocalTXID ltx.TXID // Local transaction ID\n RemoteTXID ltx.TXID // Remote transaction ID\n InSync bool // true if LocalTXID > 0 and equal to RemoteTXID\n}\n\nfunc (db *DB) SyncStatus(ctx context.Context) (SyncStatus, error)\nfunc (db *DB) SyncAndWait(ctx context.Context) error\nfunc (db *DB) EnsureExists(ctx context.Context) error\n```\n\n- **SyncStatus**: Compares local TXID against remote replica position (performs I/O to query remote)\n- **SyncAndWait**: Runs `db.Sync()` (WAL to LTX) then `db.Replica.Sync()` (LTX to remote), blocking until both complete\n- **EnsureExists**: Restores database from replica if local file doesn't exist; no-op if file exists or no backup available. Must be called before `Open()`\n\n## LTX File Format\n\nLTX (Log Transaction) files are immutable files containing database changes:\n\n```\n+------------------+\n| Header | Fixed size header with metadata\n+------------------+\n| |\n| Page Frames | Variable number of page frames\n| |\n+------------------+\n| Page Index | Index for efficient page lookup\n+------------------+\n| Trailer | Metadata and checksums\n+------------------+\n```\n\n### Header Structure\n\n```go\ntype Header struct {\n Magic [4]byte // \"LTX\\x00\"\n Version uint32 // Format version\n PageSize uint32 // Database page size\n MinTXID TXID // Starting transaction ID\n MaxTXID TXID // Ending transaction ID\n Timestamp int64 // Creation timestamp\n Checksum uint64 // Header checksum\n}\n```\n\n### Page Frame Structure\n\n```go\ntype PageFrame struct {\n Header PageHeader\n Data []byte // Page data (pageSize bytes)\n}\n\ntype PageHeader struct {\n PageNo uint32 // Page number in database\n Size uint32 // Size of page data\n Checksum uint64 // Page checksum\n}\n```\n\n### Page Index\n\nBinary search tree for efficient page lookup:\n```go\ntype PageIndexElem struct {\n PageNo uint32 // Page number\n Offset int64 // Offset in file\n Size uint32 // Size of page frame\n}\n```\n\n### Trailer\n\n```go\ntype Trailer struct {\n PageIndexOffset int64 // Offset to page index\n PageIndexSize int64 // Size of page index\n PageCount uint32 // Total pages in file\n Checksum uint64 // Full file checksum\n}\n```\n\n## WAL Monitoring Mechanism\n\n### Monitor Loop (db.go:1499)\n\n```go\nfunc (db *DB) monitor() {\n ticker := time.NewTicker(db.MonitorInterval)\n defer ticker.Stop()\n\n for {\n select {\n case <-ticker.C:\n // Check WAL for changes\n changed, err := db.checkWAL()\n if err != nil {\n slog.Error(\"wal check failed\", \"error\", err)\n continue\n }\n\n if changed {\n // Notify replicas of changes\n db.notifyReplicas()\n\n // Check if checkpoint needed\n if db.shouldCheckpoint() {\n db.autoCheckpoint()\n }\n }\n\n case <-db.ctx.Done():\n return\n }\n }\n}\n```\n\n### WAL Change Detection\n\n```go\nfunc (db *DB) checkWAL() (bool, error) {\n // Get current WAL state\n walInfo, err := db.walInfo()\n if err != nil {\n return false, err\n }\n\n // Compare with previous state\n db.mu.Lock()\n changed := walInfo.Size != db.prevWALSize ||\n walInfo.Checksum != db.prevWALChecksum\n db.prevWALSize = walInfo.Size\n db.prevWALChecksum = walInfo.Checksum\n db.mu.Unlock()\n\n return changed, nil\n}\n```\n\n## Compaction Process\n\nCompaction merges multiple LTX files to reduce storage overhead:\n\n### Compaction Algorithm (store.go:189)\n\nHigh-level compaction flow:\n\n1. Determine whether the level is due for compaction (`Store.shouldCompact`).\n2. Enumerate level-`L-1` files using `ReplicaClient.LTXFiles`, preferring local\n copies via `os.Open(db.LTXPath(...))` and falling back to\n `ReplicaClient.OpenLTXFile` only when necessary.\n3. Stream the source readers through `ltx.NewCompactor`, which performs\n page-level deduplication and enforces lock-page skipping automatically.\n4. Pipe the compactor output into `ReplicaClient.WriteLTXFile` to create the\n merged LTX file for level `L`.\n5. Adjust the returned `ltx.FileInfo.CreatedAt` to the earliest timestamp from\n the source files so point-in-time recovery remains accurate.\n6. Update the cached max file info for the level and delete old L0 files when\n promoting to level 1.\n\n### Compaction Levels\n\n```go\ntype CompactionLevel struct {\n Level int // Level number (0 = raw, 1+ = compacted)\n Interval time.Duration // How often to compact from previous level\n}\n\n// Default configuration\nvar DefaultCompactionLevels = CompactionLevels{\n {Level: 0, Interval: 0}, // Raw LTX files\n {Level: 1, Interval: 1 * Hour}, // Hourly compaction\n {Level: 2, Interval: 24 * Hour}, // Daily compaction\n}\n```\n\n## Transaction Management\n\n### Long-Running Read Transaction\n\nLitestream maintains a long-running read transaction to ensure consistency:\n\n```go\nfunc (db *DB) initReadTx() error {\n // Start read transaction\n tx, err := db.db.BeginTx(context.Background(), &sql.TxOptions{\n ReadOnly: true,\n })\n if err != nil {\n return err\n }\n\n // Execute dummy query to start transaction\n var dummy string\n err = tx.QueryRow(\"SELECT ''\").Scan(&dummy)\n if err != nil {\n tx.Rollback()\n return err\n }\n\n db.rtx = tx\n return nil\n}\n```\n\n**Purpose:**\n- Prevents database from being modified during replication\n- Ensures consistent view of database\n- Allows reading historical pages from WAL\n\n### Checkpoint Coordination\n\n```go\nfunc (db *DB) Checkpoint(mode string) error {\n // Acquire checkpoint lock\n db.chkMu.Lock()\n defer db.chkMu.Unlock()\n\n // Close read transaction temporarily\n if db.rtx != nil {\n db.rtx.Rollback()\n db.rtx = nil\n }\n\n // Perform checkpoint\n _, _, err := db.db.Exec(fmt.Sprintf(\"PRAGMA wal_checkpoint(%s)\", mode))\n if err != nil {\n return err\n }\n\n // Restart read transaction\n return db.initReadTx()\n}\n```\n\n## Concurrency Model\n\n### Mutex Usage Patterns\n\n```go\n// DB struct mutexes\ntype DB struct {\n mu sync.RWMutex // Protects struct fields\n chkMu sync.RWMutex // Checkpoint coordination\n}\n\n// Replica struct mutexes\ntype Replica struct {\n mu sync.RWMutex // Protects position\n muf sync.Mutex // File descriptor lock\n}\n\n// Store struct mutex\ntype Store struct {\n mu sync.Mutex // Protects database list\n}\n```\n\n### Thundering Herd Prevention\n\n`Store.Open()` (`store.go`) limits concurrent database opens at startup:\n\n```go\ng, ctx := errgroup.WithContext(ctx)\ng.SetLimit(50) // Max 50 concurrent DB opens\nfor _, db := range s.dbs {\n db := db\n g.Go(func() error { return db.Open() })\n}\n```\n\nThis prevents OS resource exhaustion (file descriptors, memory) when hundreds of databases are configured.\n\n### Lock Ordering (Prevent Deadlocks)\n\nAlways acquire locks in this order:\n1. Store.mu\n2. DB.mu\n3. DB.chkMu\n4. Replica.mu\n\n### Goroutine Management\n\n```go\n// Start background task\nfunc (db *DB) Start() {\n db.wg.Add(1)\n go func() {\n defer db.wg.Done()\n db.monitor()\n }()\n}\n\n// Stop with timeout\nfunc (db *DB) Close(ctx context.Context) error {\n // Signal shutdown\n db.cancel()\n\n // Wait for goroutines with timeout\n done := make(chan struct{})\n go func() {\n db.wg.Wait()\n close(done)\n }()\n\n select {\n case <-done:\n return nil\n case <-ctx.Done():\n return ctx.Err()\n }\n}\n```\n\n## State Management\n\n### Database States\n\n```mermaid\nstateDiagram-v2\n [*] --> Closed\n Closed --> Opening: Open()\n Opening --> Open: Success\n Opening --> Closed: Error\n Open --> Monitoring: Start()\n Monitoring --> Syncing: Changes Detected\n Syncing --> Monitoring: Sync Complete\n Monitoring --> Checkpointing: Threshold Reached\n Checkpointing --> Monitoring: Checkpoint Complete\n Monitoring --> Closing: Close()\n Closing --> Closed: Cleanup Complete\n```\n\n### Replica States\n\n```mermaid\nstateDiagram-v2\n [*] --> Idle\n Idle --> Starting: Start()\n Starting --> Monitoring: Success\n Starting --> Idle: Error\n Monitoring --> Syncing: Timer/Changes\n Syncing --> Uploading: Have Changes\n Uploading --> Monitoring: Success\n Uploading --> Error: Failed\n Error --> Monitoring: Retry\n Monitoring --> Stopping: Stop()\n Stopping --> Idle: Cleanup\n```\n\n### Position Tracking\n\n```go\ntype Pos struct {\n TXID TXID // Current transaction ID\n PageNo uint32 // Current page number\n Checksum uint64 // Running checksum for validation\n}\n\n// Update position atomically\nfunc (r *Replica) SetPos(pos ltx.Pos) {\n r.mu.Lock() // MUST use Lock, not RLock!\n defer r.mu.Unlock()\n r.pos = pos\n}\n\n// Read position safely\nfunc (r *Replica) Pos() ltx.Pos {\n r.mu.RLock()\n defer r.mu.RUnlock()\n return r.pos\n}\n```\n\n## Initialization Flow\n\n### System Startup Sequence\n\n```mermaid\nsequenceDiagram\n participant Main\n participant Store\n participant DB\n participant Replica\n participant Monitor\n\n Main->>Store: NewStore(config)\n Store->>Store: Validate config\n\n Main->>Store: Open()\n loop For each database\n Store->>DB: NewDB(path)\n Store->>DB: Open()\n DB->>DB: Open SQLite connection\n DB->>DB: Read page size\n DB->>DB: Init metadata\n DB->>DB: Start read transaction\n\n loop For each replica\n DB->>Replica: NewReplica()\n DB->>Replica: Start()\n Replica->>Monitor: Start monitoring\n end\n end\n\n Store->>Store: Start compaction monitors\n Store-->>Main: Ready\n```\n\n### Critical Initialization Steps\n\n1. **Database Opening**\n ```go\n // Must happen in order:\n 1. Open SQLite connection\n 2. Read page size (PRAGMA page_size)\n 3. Create metadata directory\n 4. Start long-running read transaction\n 5. Initialize replicas\n 6. Start monitor goroutine\n ```\n\n2. **Replica Initialization**\n ```go\n // Must happen in order:\n 1. Create replica with client\n 2. Load previous position from metadata\n 3. Validate position against database\n 4. Start sync goroutine (if monitoring enabled)\n ```\n\n## Error Handling\n\n### Error Categories\n\n1. **Recoverable Errors**\n - Network timeouts\n - Temporary storage unavailability\n - Lock contention\n\n2. **Fatal Errors**\n - Database corruption\n - Invalid configuration\n - Disk full\n\n3. **Operational Errors**\n - Checkpoint failures\n - Compaction conflicts\n - Sync delays\n\n### Error Propagation\n\n```go\n// Bottom-up error propagation\nReplicaClient.WriteLTXFile() error\n ↓\nReplica.Sync() error\n ↓\nDB.Sync() error\n ↓\nStore.monitorDB() // Logs error, continues\n```\n\n### Retry Logic\n\n```go\nfunc (r *Replica) syncWithRetry(ctx context.Context) error {\n backoff := time.Second\n maxBackoff := time.Minute\n\n for attempt := 0; ; attempt++ {\n err := r.Sync(ctx)\n if err == nil {\n return nil\n }\n\n // Check if error is retryable\n if !isRetryable(err) {\n return err\n }\n\n // Check context\n if ctx.Err() != nil {\n return ctx.Err()\n }\n\n // Exponential backoff\n time.Sleep(backoff)\n backoff *= 2\n if backoff > maxBackoff {\n backoff = maxBackoff\n }\n }\n}\n```\n\n## Performance Characteristics\n\n### Time Complexity\n\n| Operation | Complexity | Notes |\n|-----------|------------|-------|\n| WAL Monitor | O(1) | Fixed interval check |\n| Page Write | O(1) | Append to LTX file |\n| Compaction | O(n) | n = total pages |\n| Restoration | O(n*log(m)) | n = pages, m = files |\n| File List | O(k) | k = files in level |\n\n### Space Complexity\n\n| Component | Memory Usage | Disk Usage |\n|-----------|-------------|------------|\n| DB | O(1) + metrics | Original DB + WAL |\n| Replica | O(1) | LTX files + metadata |\n| Compaction | O(n) pages | Temporary during merge |\n| Page Index | O(p) | p = pages in file |\n\n### Optimization Points\n\n1. **Page Index Caching**\n - Cache frequently accessed indices\n - Use estimated size for initial fetch\n\n2. **Batch Operations**\n - Group small changes into larger LTX files\n - Batch delete operations\n\n3. **Concurrent Operations**\n - Multiple replicas can sync in parallel\n - Compaction runs independently per level\n\n## Security Considerations\n\n### Access Control\n\n- File permissions: 0600 for database files\n- Directory permissions: 0700 for metadata\n- No built-in authentication (rely on storage backend)\n\n## Monitoring & Metrics\n\n### Prometheus Metrics\n\n```go\n// Database metrics\ndb_size_bytes // Current database size\nwal_size_bytes // Current WAL size\ntotal_wal_bytes // Total bytes written to WAL\ncheckpoint_count // Number of checkpoints\nsync_count // Number of syncs\nsync_error_count // Number of sync errors\n\n// Replica metrics\nreplica_lag_seconds // Replication lag\nreplica_position // Current replication position\n\n// Validation metrics have db and level labels.\nlitestream_validation_checks_total // Checks by result: success, invalid, or error.\nlitestream_last_validation_success // Last check: 1 passed, 0 failed or not yet complete.\nlitestream_last_validation_success_timestamp_seconds // Unix time of last success, or 0 until a check passes.\n```\n\n### Health Checks\n\n```go\nfunc (db *DB) HealthCheck() error {\n // Check database connection\n if err := db.db.Ping(); err != nil {\n return fmt.Errorf(\"database ping failed: %w\", err)\n }\n\n // Check replication lag\n for _, r := range db.replicas {\n lag := time.Since(r.LastSync())\n if lag > MaxAcceptableLag {\n return fmt.Errorf(\"replica %s lag too high: %v\", r.Name(), lag)\n }\n }\n\n return nil\n}\n```\n\n\n# litestream-vfs-darwin-amd64\n\nPlatform package for `litestream-vfs` — macOS x64 (Intel).\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n\n\n# litestream-vfs-darwin-arm64\n\nPlatform package for `litestream-vfs` — macOS ARM64 (Apple Silicon).\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n\n\n# litestream-vfs-linux-amd64\n\nPlatform package for `litestream-vfs` — Linux x64.\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n\n\n# litestream-vfs-linux-arm64\n\nPlatform package for `litestream-vfs` — Linux ARM64.\n\nThis package is installed automatically by `litestream-vfs`. You should not need to install it directly.\n\n\n"}